Introduce new shared makefile fragments for BASIC components

The `BasicApp` fragment is analogous to the `CApp` fragment in that it
supports both single-file executables and application directories. Rules are
included for building any such application directories into ROMs, with
files destined for `Resources:$.Resources` being exported to the Messages
module during the `resources` phase, and those destinated for
`Resources:$.Apps` being built into a position-independent module, so that
when the module is killed or unplugged, those files disappear from ResourceFS.

Because the module is position-independent, the makefile fragment implements
the `install_rom` rule rather than the `rom_link` rule used by `CApp`. This
is consistent with how `srcbuild` already installs ROM components listed in
the ModuleDB as being of type `BAS`.

The other main difference from `CApp` is that there is no compilation or
linking step. Tokenisation, crunching and squishing of the (untokenised)
source files are roughly analogous to the compilation steps, except that
multiple source files are concatenated (if applicable) as the first step.
There is also no debug build variant, and so no `debug` rule.

Crunching is performed (on native builds) using the `BasCrunch` tool, so if
you are converting an existing component which includes its own Command
script to perform tokenisation and crunching, this will no longer be needed.

Cross-"compilation" is fully suported. This uses the new `toffb` tool for
tokenisation (apologies for the wheel-reinvention, but this was developed in
ignorance about Steve Fryatt's `Tokenize` tool; `toffb` also does do a more
faithful job of replicating BASIC's `CRUNCH` options, so is more useful for
ensuring cross-compilation builds remain identical to the native builds).
It also uses Matrix Brandy (V1.22.9 or later) to execute Squish (at least
RiscOS/Tools/Sources/Squish!1).

We pass the new `-nosavepath` option to Squish to ensure that the results are
identical for cross-compile builds compared to native ones. However, Squish
ignores unrecognised command-line options, so this does not introduce a
dependency on a newer Squish tool for native builds.

One subtle enhancement over previous makefiles is that we create dynamic
dependencies between the crunched-and-squished program and its untokenised
source files.

GNUmakefiles/BasicApp

+# Makefile fragment for Basic applications
+
+INCLUDED_BASICAPP = YES
+
+#
+# This makefile provides the following phony targets:
+#
+#    all  install  resources  rom  install_rom
+#
+#
+# This fragment uses the following macros set by the master makefile.
+#
+#
+# COMPONENT                 (the name of the component)
+# TARGET              (opt) (the leafname of the application - otherwise ${COMPONENT})
+# ROM_MODULE          (opt) (module filename - otherwise ${COMPONENT})
+# INSTAPP             (opt) (the application target directory - otherwise ${INSTDIR}/!${COMPONENT})
+# INSTDIR             (opt) (the target directory - otherwise ${INSTALLDIR}/${TARGET})
+# RESFSDIR            (opt) (actual directory to export non-application-dir resources to - otherwise ${RESDIR}/${COMPONENT})
+# RESFSAPPDIR         (opt) (where to register application-dir resources to within ResourceFS, using RISC OS style directory separators - otherwise Apps.!${COMPONENT})
+# DIRS                (opt) (stamp object for directory creation - otherwise _dirs)
+# SRCS                (opt) (source files to be concatenated, no .bas suffixes - otherwise ${TARGET})
+# INSTTYPE            (opt) (use "tool" or "app" to install executable vs application - defaults to "tool")
+# INSTAPP_FILES       (opt) (list of files to be installed in application directory - use InstRes specification rules)
+# INSTAPP_DEPENDS     (opt) (list of dependencies to be satisfied before doing application install - ${TARGET} assumed if in INSTAPP_FILES)
+# INSTAPP_VERSION     (opt) (list of Messages/!Run/Desc files to insert app version from VersionNum - include in INSTAPP_FILES as well)
+# RES_FILES           (opt) (list of files to be installed in ${RESFSDIR} when building a ROM - use InstRes specification rules. Where these conflict with the files in INSTAPP_FILES, prefer a subdirectory named 'ROM')
+# RES_DEPENDS         (opt) (list of dependencies to be satisfied before doing resources export - ${TARGET} assumed if in RES_FILES)
+# RESAPP_FILES        (opt) (list of files to be installed in ${RESFSAPPDIR} when building a ROM - use InstRes specification rules. Where these conflict with the files in INSTAPP_FILES, prefer a subdirectory named 'ROM')
+# CUSTOMINSTALLAPP    (opt) (set to "custom" to override the install rule for resource files)
+# CUSTOMINSTALLTOOL   (opt) (set to "custom" to override the install rule for target binary)
+# CUSTOMROM           (opt) (set to "custom" to override the rom rules)
+# SOURCES_TO_SYMLINK  (opt) (files which need be linked to by the link farm, in addition to contents of bas directory)
+#
+#
+
+TARGET       ?= ${COMPONENT}
+ROM_MODULE   ?= ${COMPONENT}${SUFFIX_MODULE}
+INSTDIR      ?= ${INSTALLDIR}/${TARGET}
+INSTAPP      ?= ${INSTDIR}/!${COMPONENT}
+RESFSDIR     ?= ${RESDIR}/${COMPONENT}
+RESFSAPPDIR  ?= Apps.!${COMPONENT}
+MERGEDRDIR   ?= _ResData_
+RES_AREA     ?= Resources
+DIRS         ?= _dirs
+SRCS         ?= ${TARGET}
+
+SRCS_         = $(addsuffix .bas,${SRCS})
+
+SOURCES_TO_SYMLINK += $(wildcard bas/*) VersionNum
+
+ifeq ("${INCLUDED_STDTOOLS}","")
+include StdTools
+endif
+
+ifneq (objs,$(notdir ${CURDIR}))
+
+# Makefile invoked from same directory
+# Create link farm, then execute the makefile from within it
+
+ifeq (clean,${MAKECMDGOALS})
+# With a double-colon rule which can have additional actions assigned from the
+# master makefile, we'd normally need the master makefile to include the
+# ${CURDIR} check to ensure that it's performed on the same invocation as us.
+# However, there's no real benefit to performing clean from within the objs
+# directory, and it adds an ordering problem between the different double-colon
+# rules (the one that deletes the objs directory has to be last otherwise the
+# cwd is invalid for the others) so to simplify things, we only ever do cleans
+# from the same directory as the Makefile.
+clean::
+	@echo Cleaning...
+	@rm -rf objs
+	@echo ${COMPONENT}: cleaned
+else
+all install resources rom install_rom links: ${SYMLINK_DEPEND}
+	$(foreach linksource,${SOURCES_TO_SYMLINK}, \
+		$(shell \
+			linkdest=`echo ${linksource} | sed -e 's,\([^/]*\)/\([^/]*\)$$,\2.\1,' -e 's,^,objs/,'`; \
+			linkdestdir=`echo $$linkdest | sed -e 's,/[^/]*$$,,'`; \
+			linkbackpath=`echo $$linkdestdir | sed -e 's,[^/]*,..,g'`; \
+			[ -d ${linksource} ] || [ -L $$linkdest ] || mkdir -p $$linkdestdir; \
+			[ -d ${linksource} ] || [ -L $$linkdest ] || ln -s $$linkbackpath/${linksource} $$linkdest; \
+		 ) \
+	)
+	@[ -L objs/Resources ] || ln -s ../Resources objs/Resources
+	@mkdir -p objs
+ifneq (links,${MAKECMDGOALS})
+	@${MAKE} -C objs -f ../$(firstword ${MAKEFILE_LIST}) ${MAKECMDGOALS}
+endif
+endif
+
+else
+
+# Makefile invoked from objs subdirectory
+
+all: ${TARGET}${SUFFIX_BASIC}
+	@${ECHO} ${COMPONENT}: application built
+
+# GNU make seems to treat any double-colon rule with no dependencies as
+# always out-of-date, therefore always rebuilds it and anything which in turn
+# depends on the target of the double-colon rule. So use a single-colon rule
+# instead. If any cross builds need to create extra directories on a
+# per-component basis, we'll cross that bridge when we get to it.
+${DIRS}:
+	${TOUCH} $@
+
+install: install_${INSTTYPE}
+
+install_: install_tool
+
+INSTAPP_DEPENDS += $(addsuffix ${SUFFIX_BASIC},$(filter ${TARGET},${INSTAPP_FILES}))
+RES_DEPENDS     += $(addsuffix ${SUFFIX_BASIC},$(filter ${TARGET},${RES_FILES}))
+
+install_app${CUSTOMINSTALLAPP}: ${INSTAPP_DEPENDS}
+	${MKDIR} ${INSTAPP}
+	${INSTRES} -I Resources.${USERIF}.${LOCALE},Resources.${USERIF}.UK,Resources.${LOCALE},Resources.UK,Resources ${INSTAPP} ${INSTAPP_FILES}
+ifneq (,$(filter Messages,${INSTAPP_VERSION}))
+	TMP=`mktemp`; ${INSERTVERSION} ${INSTAPP}/Messages > $$TMP; mv $$TMP ${INSTAPP}/Messages
+endif
+ifneq (,$(filter Desc,${INSTAPP_VERSION}))
+	TMP=`mktemp`; ${INSERTVERSION} descmode=1 ${INSTAPP}/Desc ${INSTAPP}/Desc > $$TMP; mv $$TMP ${INSTAPP}/Desc
+endif
+ifneq (,$(filter !Run,${INSTAPP_VERSION}))
+	TMP=`mktemp`; ${INSERTVERSION} obeymode=1 ${INSTAPP}/!Run${SUFFIX_OBEY} > $$TMP; mv $$TMP ${INSTAPP}/!Run${SUFFIX_OBEY}
+endif
+	@${ECHO} ${COMPONENT}: application installation complete
+
+install_tool${CUSTOMINSTALLTOOL}: ${TARGET}${SUFFIX_BASIC}
+	${MKDIR} ${INSTDIR}
+	${CP} ${TARGET}${SUFFIX_BASIC} ${INSTDIR}/${TARGET}${SUFFIX_BASIC} ${CPFLAGS}
+	@${ECHO} ${COMPONENT}: tool installation complete
+
+resources: ${RES_DEPENDS}
+	${MKDIR} ${RESFSDIR}
+ifneq (${RES_FILES},)
+	${INSTRES} -I Resources.${LOCALE}.ROM,Resources.UK.ROM,Resources.ROM,Resources.{LOCALE},Resources.UK,Resources ${RESFSDIR} ${RES_FILES}
+endif
+ifneq (,$(filter Messages,${INSTAPP_VERSION}))
+	${INSERTVERSION} ${RESFSDIR}/Messages > ${RESFSDIR}/_Awk_
+	${CP} ${RESFSDIR}/_Awk_ ${RESFSDIR}/Messages ${CPFLAGS}
+	${RM} ${RESFSDIR}/_Awk_
+	for path in ${LOCALE}/ROM UK/ROM ROM ${LOCALE} UK ""; do if [ -f Resources/$$path/Messages ]; then touch -r Resources/$$path/Messages ${RESFSDIR}/Messages; break; fi; done
+endif
+	@${ECHO} ${COMPONENT}: resources copied to Messages module
+
+rom${CUSTOMROM}: ${ROM_MODULE}
+	@${ECHO} ${COMPONENT}: rom module built
+
+install_rom${CUSTOMROM}: ${ROM_MODULE}
+	${CP} ${ROM_MODULE} ${INSTDIR}/${COMPONENT}${SUFFIX_MODULE} ${CPFLAGS}
+	@${ECHO} ${COMPONENT}: rom module installed
+
+${TARGET}${SUFFIX_BASIC}: ${DIRS} ${SRCS_}
+ifneq (${TARGET},${SRCS})
+	${FAPPEND} ${TARGET}.bas ${SRCS_}
+endif
+	${BASCRUNCH} -1 ${TARGET}.bas ${TARGET}.crunched
+	${SQUISH} ${SQUISHFLAGS} -from ${TARGET}.crunched -to $@
+	@${ECHO} ${TARGET}${SUFFIX_BASIC}: ${SRCS_} > ${TARGET}.d
+
+${ROM_MODULE}:
+	${MKDIR} ${MERGEDRDIR}
+	${INSTRES} -I Resources.${LOCALE}.ROM,Resources.UK.ROM,Resources.ROM,Resources.${LOCALE},Resources.UK,Resources ${MERGEDRDIR} ${RESAPP_FILES}
+	${INSTVIARG} ${MERGEDRDIR} ${RESFSAPPDIR} _ModGen_
+	${MODGEN} -date "$$(awk '/Module_Date / { $$1=""; $$2=""; gsub(/"/, ""); print }' VersionNum)" $@ !${COMPONENT} !${COMPONENT} $$(sed -n -e 's/^.*(\(.*\)).*$$/\1/;1p' VersionNum) -via _ModGen_
+
+include $(wildcard *.d)
+
+endif
+
+export${CUSTOMEXP}: export_${PHASE}
+	@${NOP}
+
+export_ export_hdrs export_libs:
+	@${NOP}
+
+# EOF

GNUmakefiles/StdTools

@@ -9,6 +9,7 @@ INCLUDED_STDTOOLS = YES
 ifneq (Host,${APCS})
 GNUTOOLPREFIX = arm-unknown-riscos-
 SUFFIX_DATA     = ,ffd
+SUFFIX_BASIC    = ,ffb
 SUFFIX_MODULE   = ,ffa
 SUFFIX_ABSOLUTE = ,ff8
 SUFFIX_OBEY     = ,feb
@@ -57,6 +58,7 @@ LDBIN           = armlink -bin -o
 STRIP           = @echo > /dev/null
 endif
 
+BASCRUNCH       = toffb --crunch
 BINAOF          = binaof
 CAT             = cat
 #DATA2AOF        = datatoaof
@@ -85,6 +87,7 @@ RESGEN          = resgen
 RM              = rm -f
 #SETTYPE         = settype
 #SQUASH          = squash
+SQUISH          = sbrandy -quit ${TOOLSDIR}/Build/Squish,ffb --
 ifneq (Host,${APCS})
 SQZ             = armsqueeze
 else
@@ -191,6 +194,7 @@ ARFLAGS     := -c -o
 ASFLAGS     += ${OBJASM_TOOLOPTIONS} -Stamp -quit
 CFLAGS      += ${NCC_TOOLOPTIONS}
 CMHGFLAGS   += ${TOOLOPTIONS}
+SQUISHFLAGS += -nolist -nosavepath -sysfrom ${LIBDIR}/CLib/swis.h
 
 C_WARNINGS  := -fah
 C_OPTSIZE   := -Ospace

Makefiles/BasicApp

+# Copyright 2020 RISC OS Open Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Makefile fragment for Basic applications
+
+INCLUDED_BASICAPP = YES
+
+#
+# This makefile provides the following phony targets:
+#
+#    all  install  resources  rom  install_rom
+#
+#
+# This fragment uses the following macros set by the master makefile.
+#
+#
+# COMPONENT                 (the name of the component)
+# TARGET              (opt) (the leafname of the application - otherwise ${COMPONENT})
+# ROM_MODULE          (opt) (module filename - otherwise rm.${COMPONENT})
+# INSTAPP             (opt) (the application target directory - otherwise ${INSTDIR}.!${COMPONENT})
+# INSTDIR             (opt) (the target directory - otherwise <Install$Dir>.${TARGET})
+# RESFSDIR            (opt) (actual directory to export non-application-dir resources to - otherwise ${RESDIR}.${COMPONENT})
+# RESFSAPPDIR         (opt) (where to register application-dir resources to within ResourceFS - otherwise Apps.!${COMPONENT})
+# DIRS                (opt) (stamp object for directory creation - otherwise o._dirs)
+# SRCS                (opt) (source files to be concatenated, no bas. prefixes - otherwise ${TARGET})
+# INSTTYPE            (opt) (use "tool" or "app" to install executable vs application - defaults to "tool")
+# INSTAPP_FILES       (opt) (list of files to be installed in application directory - use InstRes specification rules)
+# INSTAPP_DEPENDS     (opt) (list of dependencies to be satisfied before doing application install - ${TARGET} assumed if in INSTAPP_FILES)
+# INSTAPP_VERSION     (opt) (list of Messages/!Run/Desc files to insert app version from VersionNum - include in INSTAPP_FILES as well)
+# RES_FILES           (opt) (list of files to be installed in ${RESFSDIR} when building a ROM - use InstRes specification rules. Where these conflict with the files in INSTAPP_FILES, prefer a subdirectory named 'ROM')
+# RES_DEPENDS         (opt) (list of dependencies to be satisfied before doing resources export - ${TARGET} assumed if in RES_FILES)
+# RESAPP_FILES        (opt) (list of files to be installed in ${RESFSAPPDIR} when building a ROM - use InstRes specification rules. Where these conflict with the files in INSTAPP_FILES, prefer a subdirectory named 'ROM')
+# CUSTOMINSTALLAPP    (opt) (set to "custom" to override the install rule for resource files)
+# CUSTOMINSTALLTOOL   (opt) (set to "custom" to override the install rule for target binary)
+# CUSTOMROM           (opt) (set to "custom" to override the rom rules)
+#
+#
+
+INSTALLDIR    = <Install$Dir>
+
+TARGET       ?= ${COMPONENT}
+ROM_MODULE   ?= rm.${COMPONENT}
+INSTDIR      ?= ${INSTALLDIR}.${TARGET}
+INSTAPP      ?= ${INSTDIR}.!${COMPONENT}
+RESFSDIR     ?= ${RESDIR}.${COMPONENT}
+RESFSAPPDIR  ?= Apps.!${COMPONENT}
+MERGEDRDIR   ?= o._ResData_
+RES_AREA     ?= Resources
+DIRS         ?= o._dirs
+SRCS         ?= ${TARGET}
+
+SRCS_         = $(addprefix bas.,${SRCS})
+
+ifeq ("${INCLUDED_STDTOOLS}","")
+include StdTools
+endif
+
+all: ${TARGET}
+        @${ECHO} ${COMPONENT}: application built
+
+${DIRS} ::
+        ${MKDIR} crunched
+        ${MKDIR} o
+        ${MKDIR} rm
+        ${TOUCH} $@
+
+clean ::
+        @IfThere crunched Then ${ECHO} ${WIPE} crunched ${WFLAGS}
+        @IfThere crunched Then ${WIPE} crunched ${WFLAGS}
+        @IfThere o        Then ${ECHO} ${WIPE} o ${WFLAGS}
+        @IfThere o        Then ${WIPE} o ${WFLAGS}
+        @IfThere rm       Then ${ECHO} ${WIPE} rm ${WFLAGS}
+        @IfThere rm       Then ${WIPE} rm ${WFLAGS}
+ifneq (${TARGET},${SRCS})
+        ${RM} bas.${TARGET}
+endif
+        ${RM} ${TARGET}
+        @${ECHO} ${COMPONENT}: cleaned
+
+install: install_${INSTTYPE}
+
+install_: install_tool
+        @${NOP}
+
+INSTAPP_DEPENDS += $(filter ${TARGET},${INSTAPP_FILES})
+RES_DEPENDS     += $(filter ${TARGET},${RES_FILES})
+
+install_app${CUSTOMINSTALLAPP}: ${INSTAPP_DEPENDS}
+        ${MKDIR} ${INSTAPP}
+        ${INSTRES} -I Resources.${USERIF}.${LOCALE},Resources.${USERIF}.UK,Resources.${LOCALE},Resources.UK,Resources ${INSTAPP} ${INSTAPP_FILES}
+ifneq (,$(filter Messages,${INSTAPP_VERSION}))
+        ${CHMOD} +w ${INSTAPP}.Messages
+        ${INSERTVERSION} LocalRes:Messages > ${INSTAPP}.Messages
+endif
+ifneq (,$(filter Desc,${INSTAPP_VERSION}))
+        ${CHMOD} +w ${INSTAPP}.Desc
+        ${INSERTVERSION} descmode=1 LocalRes:Desc > ${INSTAPP}.Desc
+endif
+ifneq (,$(filter !Run,${INSTAPP_VERSION}))
+        ${CHMOD} +w ${INSTAPP}.!Run
+        ${INSERTVERSION} obeymode=1 LocalRes:!Run > ${INSTAPP}.!Run
+        ${SETTYPE} ${INSTAPP}.!Run Obey
+endif
+        @${ECHO} ${COMPONENT}: application installation complete
+
+install_tool${CUSTOMINSTALLTOOL}: ${TARGET}
+        ${MKDIR} ${INSTDIR}
+        ${CP} ${TARGET} ${INSTDIR}.${TARGET} ${CPFLAGS}
+        @${ECHO} ${COMPONENT}: tool installation complete
+
+resources: ${RES_DEPENDS}
+        ${MKDIR} ${RESFSDIR}
+ifneq (${RES_FILES},)
+        ${INSTRES} -I Resources.${LOCALE}.ROM,Resources.UK.ROM,Resources.ROM,Resources.{LOCALE},Resources.UK,Resources ${RESFSDIR} ${RES_FILES}
+endif
+ifneq (,$(filter Messages,${INSTAPP_VERSION}))
+        ${INSERTVERSION} ${RESFSDIR}.Messages > ${RESFSDIR}._Awk_
+        ${CP} ${RESFSDIR}._Awk_ ${RESFSDIR}.Messages ${CPFLAGS}
+        ${RM} ${RESFSDIR}._Awk_
+        ${TOUCH} -r LocalRes:Messages ${RESFSDIR}.Messages
+endif
+        @${ECHO} ${COMPONENT}: resources copied to Messages module
+
+rom${CUSTOMROM}: ${ROM_MODULE}
+        @${ECHO} ${COMPONENT}: rom module built
+
+install_rom${CUSTOMROM}: ${ROM_MODULE}
+        ${CP} ${ROM_MODULE} ${INSTDIR}.${COMPONENT} ${CPFLAGS}
+        @${ECHO} ${COMPONENT}: rom module installed
+
+${TARGET}: ${SRCS_} ${DIRS}
+ifneq (${TARGET},${SRCS})
+        ${FAPPEND} bas.${TARGET} ${SRCS_}
+endif
+        ${BASCRUNCH} -1 bas.${TARGET} crunched.${TARGET}
+        ${SQUISH} ${SQUISHFLAGS} -from crunched.${TARGET} -to $@
+        @${AWK} "BEGIN { print \"${TARGET}: ${SRCS_}\" }" > !Depend
+
+${ROM_MODULE}: ${DIRS}
+        ${MKDIR} ${MERGEDRDIR}
+        ${INSTRES} -I Resources.${LOCALE}.ROM,Resources.UK.ROM,Resources.ROM,Resources.${LOCALE},Resources.UK,Resources ${MERGEDRDIR} ${RESAPP_FILES}
+        ${INSTVIARG} ${MERGEDRDIR} ${RESFSAPPDIR} o._ModGen_
+        ${GETVERSION} BasicApp$BuildV BasicApp$FullV BasicApp$Date
+        ${DO} ${MODGEN} -date "<BasicApp$Date>" $@ !${COMPONENT} !${COMPONENT} <BasicApp$BuildV> -via o._ModGen_
+
+# EOF

Makefiles/StdTools

@@ -31,6 +31,7 @@ STDMAKEKFILE=$Id$
 AOFASM          = objasm
 AR              = libfile
 AS              = objasm
+BASCRUNCH       = BasCrunch
 BINAOF          = binaof
 CAT             = print
 CC              = cc
@@ -113,7 +114,7 @@ CFLAGS  += -c ${STDTOOLOPTIONS} ${CDEFINES} ${CINCLUDES} ${C_NO_FNAMES} ${C_WARN
 C++INCLUDES += -ICPP:
 C++FLAGS += -c ${STDTOOLOPTIONS} ${C++DEFINES} ${C++INCLUDES}
 CMHGFLAGS +=  -p ${STDTOOLOPTIONS} ${CMHGDEFINES} ${CMHGINCLUDES}
-SQUISHFLAGS += -nolist
+SQUISHFLAGS += -nolist -nosavepath
 
 C_WARNINGS  := -fah
 C_OPTSIZE   := -Ospace