Commit 47fd7e2c authored by Robert Sprowson's avatar Robert Sprowson

Extend upper permissable file size to 4GB-1

Tested for ~200,000 cycles in various configurations with FSBash, with no integrity problems, nor bad maps. However, background transfers (currently only used by ADFS) is still being endurance tested, hence is currently disabled.
Users should note that they can create big files without needing to reformat their drives, however if reverting to an older copy of FileCore the files must not be opened, loaded, or deleted (it's fine to view the directory, just don't expect old FileCore to know how to parse such long fragment runs).
Current versions of DiscKnight (1.49) do not understand long fragment runs.

FSBash
------
* Reviewed and expunged various signed comparisons of file pointers in the existing tests
* Added new tests for big files which attempt to aggrevate all the usual problems passing over and up to important boundaries
* Makefile recreated from fragments
* Some compiler warning squashed

Docs
----
* Some notes added for big file support

Misc
----
Commands: swapped to using Command macro
HelpText: labels renamed to help Command macro
GenSWIs: text table name for SectorDiscOp ammended to match exported header and corresponding secondary module names (ADFS_SectorDiscOp et al). The usefulness of calling the base instantiation of FileCore_SectorDiscOp is minimal, especially from BASIC.
Messages: message for attempting to check an old map disc made less terse
hdr/FileCore: typo

Version 3.56. Tagged as 'FileCore-3_56'
parent e8206b6d
......@@ -55,7 +55,7 @@ main module workspace
----------------------------------------------------------------------------
^0 ^ FileBufsStart ^MapSpace ^DefectSpace
The size of the directory cach, file buffer cache, and hard disc maps are
The size of the directory cache, file buffer cache, and hard disc maps are
kept in CMOS RAM. The hard disc maps are claimed in separate blocks from the
RMA so that if the value in CMOS RAM is wrong (due to reformatting or
connecting a different drive) the blocks can be returned and ones of the
......
Some notes on BigFiles
======================
Historically, FileCore has limited the maximum object size to &7FFFFFFF (ie. all extents are positive) despite the FileSwitch API being able to express objects up to &FFFFFFFF in size. Hereafter shorthanded to 4G-1.
The 'BigFiles' switch enhances FileCore to make full use of the FileSwitch range.
FileSwitch API
==============
FSEntry_Open
------------
The return value in R4 saturates at 4G-1 to mean "allocated 4G". While the PRM states must be a multiple
of buffer size this clearly can't be expressed in a 32 bit register when rounded up to the nearest LFAU.
FileSwitch doesn't use the value of R4 internally, it is merely stored in case a client of FileSwitch requests
it.
FSEntry_Args
------------
Args 4, R2 (out) can be 4G-1 to mean "allocated 4G". The PRMs are vague on this API, could be byte sizes, in
which case this is not unusual.
Args 7, R2 (out) can be 4G-1 to mean "allocated 4G". The PRMs are vague on this API, could be byte sizes, in
which case this is not unusual.
Allocations
===========
For internal functions relating to allocations, internal maths has been extended to 33 bit (or in a few
cases 64 bit) through careful use of the carry bit. On exit from these functions it is mapped to a special
value "RoundedTo4G" which has been set to a value that cannot otherwise be encountered because it is not
any supported sector size.
This ensures the calling standard doesn't require any new registers, minimising the differences between the
switch being enabled and not.
Read ahead/write behind
=======================
It is questionable what read-ahead achieves with modern media. Drives made post year 2000 will all feature
1Mbyte or more of on board read-ahead cache. This far exceeds the maximum supported read-ahead buffers in FileCore
of 255 (ie. 255kBytes).
Other media is now solid state, hence doesn't benefit from read-ahead either - there is no step, seek, or settle
overhead since the media is not a spinning platter.
Some gain is still possible with floppy discs by speculatively reading to the end of a track, though how
often the logical fragmentation of a file actually makes this worthwhile is unclear.
Write behind is still useful, since it decouples the blocking FileSwitch API from the media which is often on
an I/O interface that is slower than a memory to memory copy.
There be many dragons in FileCore80!
The simplest approach was deemed to update the background transfers to use unsigned arithmetic and compares but
to limit the read-ahead/write-behind to &FFFFFC00, ie. one FileCore buffer of 1k less than 4G. In principle this
threshold could adapt to the sector size of the media (hence get as close as 256B from 4G) but this extra
complexity didn't merit the possible gain of a cache hit on just 768 bytes.
Attempts to read that last 1k are simply done in the foreground. This can be analysed with the following cases
1. Request to load all 4G-1 directly (OS_File) to the application's memory
Not possible, since the ARM only has 4G of address space.
2. Request to load bytes one at a time (OS_BGet) of the last, say, 1MB of the file
FileSwitch breaks this up into one-FileSwitch-buffer-at-a-time sized transfers from the media, depending
of the LFAU that FileCore reported on return from FSEntry_Open. Therefore the read ahead applies for all
but the last 1k of the file - an acceptable trade off for OS_BGet.
3. Request to load bytes in blocks (OS_GBPB) of the last, say, 1MB of the file
As the end offset is within &FFFFFC00-&FFFFFFFF this actually gets split by FileSwitch to load most
of the sectors direct into application memory, then the last LFAU sized piece is loaded into FileSwitch's
buffer in order that it can copy out LFAU-1 bytes from it. This is because the FSEntry_GetBytes for a
buffered file is guaranteed to only request FileSwitch buffer sized multiples, and since &FFFFFFFF is
not modulo 64/128/256/512/1024/2048 it has to buffer it then memcpy() it out again.
4. Request to save all 4G-1 directly (OS_File) from the application's memory
In theory this is possible, but generally RISC OS doesn't actually map all the logical address space in
at once in a contiguous lump. Assuming it was, then write behind would rapidly fill up (255kB max) and
the bottleneck becomes I/O as the transfer has saturated the buffer.
As noted above, this implementation will push it all into the foreground, at the expense of 255kB out
of 4G that could have been deferred.
5. Request to save bytes one at a time (OS_BPut) of the last, say, 1MB of the file
See notes on OS_BGet. Write behind applies for all FileSwitch to FileCore buffer transfers up until
the very last, which is pushed into the foreground.
6. Request to save bytes in blocks (OS_GBPB) of the last, say, 1MB of the file
See notes on OS_GBPB. The last part of the transfer is not a whole FileSwitch buffer size so it will
linger in FileSwitch's buffer until either an explicit flush is requested or the file is closed.
This last write is forced into the foreground, but both flushes and closing already wait for write
behinds to complete, so little net change.
Notes, in general:
On a failed completion of a background disc operation, the low level driver
should place a pointer to the failed pair in the status word.
r2 is corrupt by background completion routines supplied by FileCore
;>HelpText
[ International_Help=0
HelpBackup
Backup_Help
= "*"
= TokenEscapeChar,Token0
= " copies one whole floppy disc, (except free space) to another.",13
SynBackup
Backup_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= " <source drive> <dest. drive> [Q]"
= 0
HelpBye
Bye_Help
= "*"
= TokenEscapeChar,Token0
= " closes all files, unsets all directories, and parks hard discs.",13
SynBye
Bye_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= 0
HelpCheckMap
CheckMap_Help
= "*"
= TokenEscapeChar,Token0
= " checks that the map of a new map disc has the correct checksums, and is consistent with the directory tree. If only one copy is good it allows you to rewrite the other.",13
SynCheckMap
CheckMap_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= " [<disc spec.>]"
= 0
HelpCompact
Compact_Help
= "*"
= TokenEscapeChar,Token0
= " tries to collect free spaces together by moving files.",13
SynCompact
Compact_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= " [<disc spec.>]"
= 0
HelpDefect
Defect_Help
= "*"
= TokenEscapeChar,Token0
= " maps out a defect from a new map disc if it lies in an unallocated part of the disc. Otherwise it searches for the object containing the defect.",13
SynDefect
Defect_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= " <disc spec.> <disc add.>"
= 0
HelpDismount
Dismount_Help
= "*"
= TokenEscapeChar,Token0
= " closes files, unsets directories and parks the given disc.",13
SynDismount
Dismount_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= " [<disc spec.>]"
= 0
HelpDrive
Drive_Help
= "*"
= TokenEscapeChar,Token0
= " sets the default drive to use if the directory is unset.",13
SynDrive
Drive_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= " <drive>"
= 0
HelpFree
Free_Help
= "*"
= TokenEscapeChar,Token0
= " displays the total free space on a disc.",13
SynFree
Free_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= " [<disc spec.>]"
= 0
HelpMap
Map_Help
= "*"
= TokenEscapeChar,Token0
= " displays a disc's free space map.",13
SynMap
Map_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= " [<disc spec.>]"
= 0
HelpMount
Mount_Help
= "*"
= TokenEscapeChar,Token0
= " sets the directory to the root directory of the disc, sets the library if unset to $.Library if it exists, and unsets the URD.",13
= "The default is the default drive.",13
SynMount
Mount_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= " [<disc spec.>]"
= 0
HelpNameDisc
HelpNameDisk
NameDisc_Help
NameDisk_Help
= "*"
= TokenEscapeChar,Token0
= " alters a disc's name.",13
SynNameDisc
SynNameDisk
NameDisc_Syntax
NameDisk_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= " <disc spec.> <disc name>"
= 0
HelpVerify
Verify_Help
= "*"
= TokenEscapeChar,Token0
= " checks the whole disc is readable.",13
= "The default is the current disc.",13
SynVerify
Verify_Syntax
= "Syntax: "
= "*"
= TokenEscapeChar,Token0
= " [<disc spec.>]"
= 0
|
HelpBackup DCB "HFLCBKU", 0
SynBackup DCB "SFLCBKU", 0
Backup_Help DCB "HFLCBKU", 0
Backup_Syntax DCB "SFLCBKU", 0
HelpBye DCB "HFLCBYE", 0
SynBye DCB "SFLCBYE", 0
Bye_Help DCB "HFLCBYE", 0
Bye_Syntax DCB "SFLCBYE", 0
HelpCheckMap DCB "HFLCCKM", 0
SynCheckMap DCB "SFLCCKM", 0
CheckMap_Help DCB "HFLCCKM", 0
CheckMap_Syntax DCB "SFLCCKM", 0
HelpCompact DCB "HFLCCOM", 0
SynCompact DCB "SFLCCOM", 0
Compact_Help DCB "HFLCCOM", 0
Compact_Syntax DCB "SFLCCOM", 0
HelpDefect DCB "HFLCDEF", 0
SynDefect DCB "SFLCDEF", 0
Defect_Help DCB "HFLCDEF", 0
Defect_Syntax DCB "SFLCDEF", 0
HelpDismount DCB "HFLCDIS", 0
SynDismount DCB "SFLCDIS", 0
Dismount_Help DCB "HFLCDIS", 0
Dismount_Syntax DCB "SFLCDIS", 0
HelpDrive DCB "HFLCDRV", 0
SynDrive DCB "SFLCDRV", 0
Drive_Help DCB "HFLCDRV", 0
Drive_Syntax DCB "SFLCDRV", 0
HelpFree DCB "HFLCFRE", 0
SynFree DCB "SFLCFRE", 0
Free_Help DCB "HFLCFRE", 0
Free_Syntax DCB "SFLCFRE", 0
HelpMap DCB "HFLCMAP", 0
SynMap DCB "SFLCMAP", 0
Map_Help DCB "HFLCMAP", 0
Map_Syntax DCB "SFLCMAP", 0
HelpMount DCB "HFLCMNT", 0
SynMount DCB "SFLCMNT", 0
Mount_Help DCB "HFLCMNT", 0
Mount_Syntax DCB "SFLCMNT", 0
HelpNameDisc
HelpNameDisk DCB "HFLCNDS", 0
SynNameDisc
SynNameDisk DCB "SFLCNDS", 0
NameDisc_Help
NameDisk_Help DCB "HFLCNDS", 0
NameDisc_Syntax
NameDisk_Syntax DCB "SFLCNDS", 0
HelpVerify DCB "HFLCVER", 0
SynVerify DCB "SFLCVER", 0
Verify_Help DCB "HFLCVER", 0
Verify_Syntax DCB "SFLCVER", 0
]
ALIGN
......
......@@ -19,7 +19,7 @@ ExtEscape:Escape
BC0:PLEASE WRITE PROTECT SOURCE DISC|M|J
BC1:Insert source disc in drive %0 then press SPACE bar|M|J
BC2:Insert destination disc in drive %0 then press SPACE bar|M|J
CMC0:Old Map|M|J
CMC0:Can't check old map|M|J
CMC1:Map inconsistent with directory tree|M|J
CMC2:One copy of the map is broken. Overwrite it with the good copy (Y/N) ?
CMC3:Map good|M|J
......
| Copyright 2013 Castle Technology Ltd
|
| Licensed under the Apache License, Version 2.0 (the "License");
| you may not use this file except in compliance with the License.
| You may obtain a copy of the License at
|
| http://www.apache.org/licenses/LICENSE-2.0
|
| Unless required by applicable law or agreed to in writing, software
| distributed under the License is distributed on an "AS IS" BASIS,
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
| See the License for the specific language governing permissions and
| limitations under the License.
|
Dir <Obey$Dir>
amu_machine THROWBACK=-throwback
| Copyright 1999 Pace Micro Technology plc
| Copyright 2013 Castle Technology Ltd
|
| Licensed under the Apache License, Version 2.0 (the "License");
| you may not use this file except in compliance with the License.
......@@ -12,5 +12,5 @@
| See the License for the specific language governing permissions and
| limitations under the License.
|
Prefix <Obey$Dir>
Amu
Dir <Obey$Dir>
amu_machine clean
......@@ -12,19 +12,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
CCflags = -c -depend !Depend -IOS:,C: -throwback
Linkflags = -aif -c++ -o $@
ObjAsmflags = -throwback -NoCache -depend !Depend
Libraries = C:o.Stubs
ObjectFiles = @.o.Args @.o.BGet @.o.BPut @.o.File @.o.Find @.o.FSControl \
@.o.GBPB @.o.logger @.o.mymalloc @.o.Random @.o.Tester
@.FSBash: $(ObjectFiles) $(Libraries)
Link $(Linkflags) $(ObjectFiles) $(Libraries)
# Makefile for FSBash
#
.SUFFIXES: .c .o
COMPONENT = FSBash
OBJS = Args BGet BPut File Find FSControl GBPB logger mymalloc Random BigFiles Tester
.c.o:; cc $(CCflags) -o $@ $<
include CApp
# Dynamic dependencies:
# This is the AMU control file to make the NFS module
CLIB=^.clib
INCLUDEPATH=-I$(CLIB)
CFLAGS=-c
CC=cc $(CFLAGS) $(INCLUDEPATH)
FSBash : lnk.Tester
link -o FSBash -via lnk.Tester
lnk.Tester : $(CLIB).o.stubs\
o.Tester\
o.File\
o.Args\
o.BGet\
o.BPut\
o.GBPB\
o.Find\
o.FSControl\
o.Random\
o.logger\
o.mymalloc
stamp lnk.Tester
o.Tester : c.Tester\
$(CLIB).h.stdio\
$(CLIB).h.stdlib\
$(CLIB).h.string\
$(CLIB).h.kernel\
$(CLIB).h.swis\
h.Tester\
h.logger\
h.version\
h.mymalloc
$(CC) Tester
o.File : c.File\
$(CLIB).h.stdio\
$(CLIB).h.string\
$(CLIB).h.kernel\
$(CLIB).h.swis\
h.tester\
h.logger
$(CC) File
o.Args : c.Args\
h.Tester\
$(CLIB).h.stdio\
$(CLIB).h.kernel\
$(CLIB).h.swis\
h.tester\
h.logger
$(CC) Args
o.BGet : c.BGet\
$(CLIB).h.stdio\
$(CLIB).h.kernel\
$(CLIB).h.swis\
h.Tester\
h.logger
$(CC) BGet
o.BPut : c.BPut\
$(CLIB).h.stdio\
$(CLIB).h.kernel\
$(CLIB).h.swis\
h.Tester\
h.logger
$(CC) BPut
o.GBPB : c.GBPB\
$(CLIB).h.stdio\
$(CLIB).h.string\
$(CLIB).h.kernel\
$(CLIB).h.swis\
h.Tester\
h.logger
$(CC) GBPB
o.Find : c.Find\
$(CLIB).h.stdio\
$(CLIB).h.kernel\
$(CLIB).h.swis\
h.Tester\
h.logger
$(CC) Find
o.FSControl : c.FSControl\
$(CLIB).h.stdio\
$(CLIB).h.swis\
h.tester\
h.logger
$(CC) FSControl
o.Random : c.Random\
$(CLIB).h.stdio\
$(CLIB).h.stdlib\
$(CLIB).h.string\
$(CLIB).h.kernel\
$(CLIB).h.swis\
h.Tester\
h.logger\
h.mymalloc
$(CC) Random
o.logger : c.logger\
$(CLIB).h.stdio\
$(CLIB).h.stdarg\
$(CLIB).h.time\
h.logger
$(CC) logger
o.mymalloc : c.mymalloc\
$(CLIB).h.stdlib\
$(CLIB).h.stdio
$(CC) mymalloc
......@@ -29,7 +29,7 @@ void os_args0( int *file )
r.r[0] = 0;
r.r[1] = *file;
err = _kernel_swi( XOS_Bit | OS_Args, &r, &newr );
err = _kernel_swi( OS_Args, &r, &newr );
if ( err )
{
......@@ -41,7 +41,7 @@ void os_args0( int *file )
r.r[0] = 2;
r.r[1] = *file;
err = _kernel_swi( XOS_Bit | OS_Args, &r, &r );
err = _kernel_swi( OS_Args, &r, &r );
if ( err )
{
......@@ -50,36 +50,31 @@ void os_args0( int *file )
}
else
{
if ( r.r[2] < newr.r[2] )
if ( (unsigned int)r.r[2] < (unsigned int)newr.r[2] )
{
problems++;
logprintf( "sequential file pointer beyond file extent " );
}
else if ( newr.r[2] < 0 )
{
problems++;
logprintf( "sequential file pointer negative " );
}
}
}
logprintf( "\n" );
}
void os_args1( int *file, int pointer )
void os_args1( int *file, unsigned int pointer )
{
_kernel_oserror *err;
_kernel_swi_regs r;
_kernel_swi_regs newr;
int old_extent;
int i;
unsigned int old_extent;
unsigned int i;
logprintf( "os_args 1( %d, %d ) ", *file, pointer );
logprintf( "os_args 1( %d, %u ) ", *file, pointer );
r.r[0] = 2;
r.r[1] = *file;
err = _kernel_swi( XOS_Bit | OS_Args, &r, &r );
err = _kernel_swi( OS_Args, &r, &r );
if ( err )
{
......@@ -88,13 +83,13 @@ void os_args1( int *file, int pointer )
return;
}
old_extent = r.r[2];
old_extent = (unsigned int)r.r[2];
r.r[0] = 1;
r.r[1] = *file;
r.r[2] = pointer;
r.r[2] = (int)pointer;
err = _kernel_swi( XOS_Bit | OS_Args, &r, &newr );
err = _kernel_swi( OS_Args, &r, &newr );
if ( err )
{
......@@ -114,14 +109,14 @@ void os_args1( int *file, int pointer )
r.r[0] = 0;
r.r[1] = *file;
err = _kernel_swi( XOS_Bit | OS_Args, &r, &r );
err = _kernel_swi( OS_Args, &r, &r );
if ( err )
{
pout_error( err );
logprintf( "while reading new sequential file pointer " );
}
else if ( pointer != newr.r[2] )
else if ( pointer != (unsigned int)newr.r[2] )
{
problems++;
logprintf( "new sequential file pointer not same as set one " );
......@@ -131,7 +126,7 @@ void os_args1( int *file, int pointer )
r.r[0] = 2;
r.r[1] = *file;
err = _kernel_swi( XOS_Bit | OS_Args, &r, &r );
err = _kernel_swi( OS_Args, &r, &r );
if ( err )
{
......@@ -149,9 +144,9 @@ void os_args1( int *file, int pointer )
r.r[0] = 1;
r.r[1] = *file;
r.r[2] = old_extent;
r.r[2] = (int)old_extent;
err = _kernel_swi( XOS_Bit | OS_Args, &r, &r );
err = _kernel_swi( OS_Args, &r, &r );
if ( err )
{
......@@ -160,12 +155,12 @@ void os_args1( int *file, int pointer )