GitLab

Given not all page table formats can represent user mode RO + privileged R/W, and there's no reason for FileCore to let the user peek its internal structures, tighten this up and remove user mode access entirely.
Tested in a Titanium ROM with SCSIFS/RamFS/SDFS/ADFS.

Version 3.77. Tagged as 'FileCore-3_77'

Detail:
s.FormSWIs
LayoutBootBlock:
Calculate the checkbyte for the second defect list (>512M) in accordance with the PRM:
"The byte yy is a check-byte calculated from the previous words in the second defect list"
Currently the code includes all of the first defect list including its terminator when calculating the check-byte for the second defect list.

As a result an empty defect list goes from
&20000000
&40000020

to

&20000000
&40000000

I do not believe this code would ever have been used before, since FileCore_LayoutStructure
is only used in the OS
1) To format floppies, which are smaller than 512MB and
2) To format the RAM disc, which up till now has been smaller than 512MB.

FileCore never looks at the >512M defect list outside of FileCore_LayoutStructure, the defect command is not able to add defects to the second list.

HForm already calculates the checkbyte in accordance with the PRM, and this change ensures
that large RAM discs will conform.

Admin:
Tested by examining 600MB RamDisc0.

Version 3.76. Tagged as 'FileCore-3_76'

Having disassembled various old copies of FileCore, make a note of the most likely reason it declares with fsinfo_dontuseload to maximise the use of background transfer buffers.
Not tagged.

When 'BigMap' support was added drive sizes were such that (see table and discussion in Doc/BigDisc/BigDiscs) an upper maximum of 19 bits for idlen was appropriate. Now that drives of 2^29 sectors are readily available (the limit of 'BigDisc' support) the idlen limit of 19 has meant the minimum object size has grown too. This change uses the remaining 2 indirect disc address bits to get a factor x4 reduction in minimum object size, at the expense of a 4x larger map.

Defns.s: Parameterise idlen for big and small maps. Small's limit remains 15, big's limit raised from 19 to 21.
Identify.s: When mounting, use the parameterised idlen upper limits to decide whether it's a valid disc or not.
FileCore32.s: Adjust the BIC masks to knock out the extra 2 bits when forming the hash of the object id. Fix long standing bug updating the wrong entry in the hash table (due to LSL #2 giving 5x the entry index, rather than 3x).
FileCore15.s: Take care not to shift b20 of idlen into C as a side effect of a flags changing opcode. An audit of FileCore showed this to be the only site where the subsequent condition code(s) depended on C.
Tools/basic/DispMap: Updated to support BigMaps at all, previously it only worked with NewMap discs.

Tested with an assortment of drives with FSBash for 300k cycles of scrambling for idlen = 17, 19, 20 & 21.

Version 3.75. Tagged as 'FileCore-3_75'

Previous testing with RAMFS was limited by the size of RAM disc, and hence complexity of layout, meaning some cases with BigSectors were missed.
FileCore20.s: During mount, when loading the defect list from the boot block at &C00, take care to round down the source sector address to account for it only being aligned up to 1k sectors (but not for 2k and 4k).
FileCore25.s: When loading the head of a BigDir (to find out its actual size) and finding it to be > 2k, skip loading the bit from 2k-4k, since that was already loaded when the head was read in, and reading it a second time offset by 2k fails the head/tail check.
FileCore31.s: When building a table to decide which zones to compact, use FindBuffer() to get some temporary store if too large to fit into ScratchSpace.
Identify.s: Comment updated.
BigDirCode.s: (Unrelated) Fix for bad error pointer being passed back, due to R0 being restored on encountering an error during directory rename.

FSBashed for 300k cycles on Titanium with an assortment of manufacturer drives, capacities, and sector sizes.

Version 3.74. Tagged as 'FileCore-3_74'

Detail:
  FileCore is defined as a case insensitive FS (ie. MyFile can't coexist in the same directory as mYfILE).
  Prior to a territory being selected the BigDir binary chop sort would (via the Internat_UpperCase macro) abandon a directory search once it mismatched, and so would never find !Territory in order to select the territory in the first place.
  Have Internat_UpperCase use ASCII uppercase until that time.
Admin:
  Based on a submission from Colin Granville, with modifications.

Version 3.73. Tagged as 'FileCore-3_73'

FileCore would allow objects to be added to BigDir extensible directories until eventually failing and leaving an unopenable broken directory behind.
In fact, the on disc directory was sound, but after updating the parent in GrowBigDir it would reload the child into the dir cache with FindDir, FindDir called SanityCheckBigDirHeader which was rejecting BigDir's of exactly 4MB that was created moments earlier.
Now, during the initial attempt grow, a check is made for exceeding BigDirMaxSize before trying to enlarge the dynamic area (while it would be possible to translate the error from maxing out the dynamic area, there could be situations where there genuinely isn't enough memory to enlarge the area) and a more polite "Dir full" error returned.
Tested with an initially empty drive (so the dynamic area must grow) and with one that previously created a maximal directory (dynamic area already large enough).

Also tightened up a couple of places loading byte wide variables using LDR, relying on the modulo 31 properties of LSR/LSL, when we can simply load the value as a byte for the same cost.

Version 3.72. Tagged as 'FileCore-3_72'

For new map discs NextFs returns the address of the map zone in R10, which was being used to lookup entries in the disc record. However, for old map discs R10 is a multiple-of-3 offset into the free space map. This was aborting on the 2nd call due to an unaligned LDR at line 3205.
Even on platforms that permit unaligned loads the printed results were wrong because the disc address for an old map disc is in bytes but the value printed out was shifted up by Log2SectorSize. A DD floppy disc could have free space fragments starting at 7M - nonsense.
Fixed by checking the DiscFlags and only reading from the map zone/sector shifting for new map.
Tested with an ADFS-L floppy disc.

Version 3.71. Tagged as 'FileCore-3_71'

Module_Title->Module_TitleStr from Kernel-5_54.

Version 3.70. Tagged as 'FileCore-3_70'

Detail:
  s/BigDirCode, s/Identify - Avoid unnecessary remainder calculation in DivRem macro
Admin:
  Tested on Cortex-A15


Version 3.69. Tagged as 'FileCore-3_69'

Don't rely on an STM with either of the bottom 2 bits set to store on a word aligned boundary any more.
Fixes abort when an old style error is reported to FileCore on ARMv7.
Spotted by Jon Abbott, ref https://www.riscosopen.org/forum/forums/4/topics/3682

Version 3.68. Tagged as 'FileCore-3_68'

Detail:
  s/FileCore30 - Update FindBuffer so that if it failed to find enough spare memory for the buffer (and the RMA is allowed) it will try and grow the RMA dynamic area (current code just aims to use the spare heap space without expanding the DA).
Admin:
  Tested on Raspberry Pi
  Fixes "No room for buffer" error being generated in some situations, following the removal of Wimp_ClaimFreeMemory/free pool locking


Version 3.67. Tagged as 'FileCore-3_67'

FSBash shouldn't stop after 65536 goes, it should continue until the drive is full or Escape pressed.
Not tagged.

Detail:
  s/BigDirCode - Fix stack imbalance in GrowBigDir when DefaultMoveData returns an error
Admin:
  Tested on Raspberry Pi


Version 3.66. Tagged as 'FileCore-3_66'

Detail:
  A couple of lines of dead code and an unaligned word load that should have
  been a byte load crept into the last commit. This worked fine when
  softloaded with exceptions off but not so well in a ROM...
Admin:
  Now tested in a ROM build.

Retagged as 'FileCore-3_65'

Detail:
  These two boards don't have functional card detect lines on their microSD
  slots, so follow a different code path from most other RISC OS platforms.
  There was an issue when you accessed an object (other than the root
  directory) for which it (if it was a directory) or any of its parent
  directories were not in the directory cache, and it is specified by
  reference to disc name rather than drive number, and where the disc is not
  currently in a drive (or it's in a drive but hasn't been mounted since it
  was inserted). The additional PollChange inserted by DiscOp with a
  specified boot block (as used to identify the disc format when scanning
  drives to see if the disc in each drive has changed) which was intended to
  support some ADFS floppy drives had the side-effect with SDFS-type
  card-detect-less change detection (which has to wait until at least one
  DiscOp has been issued before it can tell if the card has changed) that
  FileCore's disc and drive records became unlinked part-way through the
  FullLookup routine. In later subroutines, this meant we ended up
  misidentifying the controller to which the disc was attached, and because
  SDFS didn't (until recently) have any hard discs, the hard disc controller
  was uninitialised, resulting in a data abort. This is fixed by making
  WhatDisc check for whether the disc record it's about to return has been
  delinked from the drive record, and repeat the process if so. Also removed
  some dead code (an alternate entry to WhatDiscCommon) while I was at it.
Admin:
  Tested on Raspberry Pi B+ and 2.

Version 3.65. Tagged as 'FileCore-3_65'

Switch 'DebugM' now used exclusively for the read cached sectors disc op, and DebugMt for multifs extensions.
Various spelling mistakes corrected in passing.
Same (non debug) binary, retagged as FileCore-3_64.

Detail:
  Test/Tester/c/Random - Fix random_path_root() reading off the end of the array when myrand() returns its max value
Admin:
  Tested on BB-xM with high processor vectors


Version 3.64. Retagged as 'FileCore-3_64'

Detail:
  There's some questionable code in DoSwi[Sector]DiscOp[64] where it calls
  PollChange in an apparent attempt to support the ADFS floppy state machine.
  In the no-card-detect SDFS case, we can detect from a card's state whether
  it has been changed or not; to be able to signal this to software that
  cares about such things, we increment the sequence number late in the
  process. The downside is that when the card *has* changed, two consecutive
  PollChanged calls will differ from the steady state: the first because
  MaybeChanged is signalled, the second because the sequence number
  mismatches. DetermineDiscType didn't handle this gracefully, ending up
  pointing off the internal array of disc records, resulting in a string of
  gibberish when canonicalising a drive-based file spec. Fixed by having
  DetermineDiscType double-check that the drive is not empty.

  Fixed UpCall_MediaNotPresent and UpCall_MediaNotKnown to actually use the
  media type string from MiscOp 4. For example, you'll now see "Please insert
  card 'RISCOSpi'" being generated by the Wimp under SDFS.

  Improved documentation of MiscOps 4 and 5.

Admin:
  Already in use in RC12a.

Version 3.64. Tagged as 'FileCore-3_64'

s/Commands:
Make sure the defect list (via *DEFECT) does a read/modify/write of the boot block on 2k and 4k drives.
The checksum generation code was largely common and although commented to be called as though it was common wasn't actually due to a cut and paste mistake on label 37. Deleted common/uncalled code.
s/DebugOpts; s/FileCore15:
Peripheral changes.
s/FileCore20:
Read in the defect list via an intermediate buffer then copy out into the dynamic area.
s/FileCore70:
Declare the buffer size as 4k to FileSwitch, internally FileCore expresses this in a single byte * 32 so is now at the limit of the range of FcbBufSz.
s/FileCore80:
Perform transfers in the foreground when the sector size exceeds FileCore's 1k buffers. It's not worth bothering.
s/FormSWIs; s/Identify:
Vet and reject attempts to layout invalid zone/sector size combos (see Doc/1Zone).
Doc/MiscOp:
Correct a couple of calculation mistakes.
Doc/1Zone:
New documentation detailing the possible valid combinations of a 1 zone fixed disc, and why for 2k and 4k sectors this is rejected.
Test/BigSectors:
Patch to apply to RAMFS to make it a 4k filing system. Note, the smallest 2 zone RAM disc with 4k sectors is 4052kB, in which case the map is at +3F0000; copy at +3F2000; root directory object at +3F4000.

Version 3.63. Tagged as 'FileCore-3_63'

Conditional now set in stone.
Not tagged.

Service call table made unconditional.
Typo in Hdr:FileCore corrected.
Comments improved, and other misc housekeeping.

Version 3.62. Tagged as 'FileCore-3_62'

FileCore80.s: The check for whether write behind could proceed only succeeded for fixed discs ('winnie'), but failed for floppies. So, on trying to flush to close the file being written you'd get stuck in a loop trying to write the buffers.
Commands.s: Magic number swapped for defect list marker

Tested with fsbash, and a simple BASIC program writing 128x1K to a floppy.

Version 3.61. Tagged as 'FileCore-3_61'

Although the fragment clamping is likely to kick in first, be more careful when rounding up/down to cylinder boundaries (for read ahead/write behind).

Version 3.60. Tagged as 'FileCore-3_60'

FileCore80.s: Revision 4.14 accidentally lost the calculation at line 1324 after FragLeft, so when a large write occurred it didn't get split either side of the fragment, and then an impossibly long scatter list would be emitted.
Tidied some DebugG debug messages.

Version 3.59. Tagged as 'FileCore-3_59'

Fix for mistargetted write with a file size > 2G when a background scatter list straddles a fragment

When large transfers are being buffered to be written in the background they are attached to the Fcb without considering how the underlying disc fragments are lined up.
When the background process grabs a handful of buffers to build into a scatter list to pass to the low level DiscOp it picks the head of the list and goes forward (SkipWriteBehind) to the fragment end and back (BackwardSkipWriteBehind).
However, BackwardsSkipWriteBehind uses DefFileFrag which substitutes a default offset of 0, and the signed compare resulted in this being left as the believed start of fragment.
In turn, far too many buffers got associated with the fragment, and written to the wrong place on the disc.
In general this doesn't occur, but if the group of buffers happens to straddle a fragment, and that group of buffers is > 2G, the signed compare would trip up.
As fsbash uses the same random number seed this conspired that testing the same drive twice (with and without background transfers enabled) would appear to pass the integrity check step. A new harddisc was used to show this up.

FileCore80.s: Recoded the compare to be in sectors, so 4G files aren't top bit set, but shared files not at the fragment start still are.
Docs/AdfsBuffers: Added some notes before I forget what I just wrote meant.


Version 3.58. Tagged as 'FileCore-3_58'

BigDirCode.s: Retire BigDirFix switch, it wasn't actually a fix, it was more that the directory format was changed early in development, but there's no point keeping support for the prototype any more
FileCore45.s/FileCore25.s/FileCore31.s/FileCore35.s/DebugOpts.s: Retire BigDirFix
Defns.s: Shock addition of some comments
FileCore.s: Manual inclusion of CPU/Arch no longer needed
FileCore70.s: Crucially apply the same 1k dead band to the FileSwitch "write zeros" entry point, since it (along with Get/PutBytes) are the only places file offsets get passed
FileCore80.s: Lots of tedious and subtle boundary cases fixed
InitDieSvc.s: Removed the disabling switch
doc/BigDisc/ADFSBuffer: Detail what the BufFlags mean

Tested on ADFS (the only background-transferring filing system about) with LFAUs of 2k, 4k, 8k, 16k and bashing 65536 iterations.

Version 3.57. Tagged as 'FileCore-3_57'

Tested for ~200,000 cycles in various configurations with FSBash, with no integrity problems, nor bad maps. However, background transfers (currently only used by ADFS) is still being endurance tested, hence is currently disabled.
Users should note that they can create big files without needing to reformat their drives, however if reverting to an older copy of FileCore the files must not be opened, loaded, or deleted (it's fine to view the directory, just don't expect old FileCore to know how to parse such long fragment runs).
Current versions of DiscKnight (1.49) do not understand long fragment runs.

FSBash
------
* Reviewed and expunged various signed comparisons of file pointers in the existing tests
* Added new tests for big files which attempt to aggrevate all the usual problems passing over and up to important boundaries
* Makefile recreated from fragments
* Some compiler warning squashed

Docs
----
* Some notes added for big file support

Misc
----
Commands: swapped to using Command macro
HelpText: labels renamed to help Command macro
GenSWIs: text table name for SectorDiscOp ammended to match exported header and corresponding secondary module names (ADFS_SectorDiscOp et al). The usefulness of calling the base instantiation of FileCore_SectorDiscOp is minimal, especially from BASIC.
Messages: message for attempting to check an old map disc made less terse
hdr/FileCore: typo

Version 3.56. Tagged as 'FileCore-3_56'

Lots of missing function entry/exit register comments added/corrected.
Drive number extraction all now (32-3) to help see where changes are needed in future for more drives.
Removed redundant Tools/basic/BASH (see Test/BASH).
InitDieSvc/BigDirCode/DebugOpts/FileCore25: Comments
FileCore05: Use of DREG with no preceding text
FileCore15: Use of 'mess' with too few arguments
FileCore20: Fetching of NZones made consistent, 3 bit drive number made consistent
FileCore30/FileCore32: Simplify debug switches, driver number extraction, comments
FileCore31: Fetching of NZones made consistent
FileCore40/FileCore33/FileCore35: NZones, function arguments commented
FileCore45: Function arguments commented, debug reinstated
FileCore70/FileCore60: Function arguments commented, drive number extraction
FormSWIs: NZones
Identify/GenSWIs: Drive number extraction, debug tidy
MyMacros: Unused Align16 removed. Hacked DebugError macro to not try loading from internal error numbers all the time, which are often not aligned, or not in sensible memory places

Relatively few code changes, no expected issues.


Version 3.55. Tagged as 'FileCore-3_55'

Detail:
  It appears that this entry (Get usage of offset) was never updated to
  support the "BigDir" directory format (E+/F+, long filenames, >77 entries
  per directory) at RISC OS 4. This results in OS_FSControl 52 also failing
  for such discs.
Admin:
  Tested using SDFS, but should apply equally to any other FileCore filing
  system.

Version 3.54. Tagged as 'FileCore-3_54'

* OsBytes
* Application start &8000
* Fixed disc density
* Unused a4 macro deleted
* Redundant 'todo', 'Help, 'Syntax' removed

Version 3.53. Not tagged

The massive speedup in FileCore 3.50 by not loading the map sector wise (twice) on mounting meant that the code path to resize the dynamic area for the map had changed.
So, with DynamicMaps = {TRUE} mounting an old map disc would try to load the map into a zero size area (the area had an allocated address but was 0 bytes long).
Aside: reading a new map disc first, then swapping to an old map disc would mask the problem since the dynamic area is only shrunk on an active dismount.


Version 3.53. Tagged as 'FileCore-3_53'

Detail:
  s/FileCore32 - The "try double exact fit" bit of the FillGap routine wasn't ensuring the search end address was word aligned, sometimes causing the search to run off the end of the FragSize array and into the memory before it.
This itself is mostly harmless (there's a check at line 1161 against the exact bit address at which the search should stop), except that check only gets perfomed once a nonzero bit is encountered.
Sometimes this check doesn't get performed at all, causing the search to fall off the start of ScratchSpace, and - with relocated zero page - into the unmapped memory beyond.
By rounding the search end address down, we ensure it's word aligned, without altering the actual size of the area.
Admin:
  Tested on BB-xM with high processor vectors


Version 3.52. Tagged as 'FileCore-3_52'

Detail:
  Rather than use its internal idea of the disc's boot option, FileCore now
  asks FileSwitch, which ends up asking either DOSFS or FileCore for the
  boot option, as appropriate for the format in use.
Admin:
  Tested on a beagleboard.

Version 3.51. Tagged as 'FileCore-3_51'

Detail:
  * Added the ability for a filing system to specify to FileCore that floppy
    discs should be mounted like hard discs, to complement the existing flag
    that specifies the opposite. SDFS requires this. Ideally SCSIFS should
    start using this as well, to encapsulate boot block handling within
    FileCore.
  * Added header definitions so the FileCore and the filing systems can
    negotiate the use of a new MiscOp entry, needed as part of partitioning
    support, especially important for filing systems that use the hardware-
    specific section of the boot block, like ADFS. At present, FileCore
    doesn't do anything different (it still doesn't support partitions) but
    this enables filing systems to be modified in preparation.
  * Removed the only place where FileCore paid any attention to the
    EmptyWorks flag returned by MiscOp 1 - an obscure and not very useful
    edge case not used by any current filing systems.
  * Added 650 lines of documentation for the MiscOp SWI and low-level entry
    to the Doc directory, giving the information you need in order to
    implement a FileCore filing system.
  * Added missing header file definitions for the low-level background DiscOp
    reason code and the hot-plug drive removed flag to MiscOp 7.
  * Removed some RISC OS 2 legacy code whereby FileCore maintained its idea
    of the internal disc address of the CSD @, PSD \, Library % and URD & for
    each filing system. Since RISC OS 3.00, FileSwitch handles these
    internally and passes canonicalised paths to FileCore, so FileCore's disc
    addresses for these directories remained unset. This allows some
    considerable simplification of FileCore's path parser in particular, but
    also affects other code like that which decides how to expire disc
    records. Also note that FileCore has been unable to generate certain
    errors (like "Can't delete current directory") since RISC OS 3.00 for the
    same reason.
  * Changed loading of the map for FileCore-format discs so that the map is
    loaded into the dynamic area during the Service_IdentifyDisc handler in
    one big DiscOp rather than loading it twice, sector-by-sector, via one
    big sector cache. For one thing, the sector cache is implemented
    inefficiently - O(n^2) - and for another thing, single-sector DiscOps are
    relatively inefficient on many filing systems, especially on SD cards.
    Some example speed-ups measured for mounting discs are:
      SCSIFS,   1GB drive: was 0.62 sec, now 0.40 sec (1.55 x speedup)
      SCSIFS, 256GB drive: was 1.44 sec, now 0.52 sec (2.77 x speedup)
      SDFS,     2GB drive: was 1.25 sec, now 0.14 sec (8.93 x speedup)
Admin:
  Tested on a beagleboard, with SCSIFS and SDFS.

Version 3.50. Tagged as 'FileCore-3_50'

Detail:
  s/BigDirCode, s/FileCore20, s/FileCore33 - Fixed some operators which were missing their end colon
Admin:
  Identical binary under objasm


Version 3.49. Retagged as 'FileCore-3_49'

Detail:
  Previous change broke builds from clean - sorry! The basic problem is that
  there's no way to enforce a dependency, or at least ordering, between the
  exported asm and C versions of Hdr:FileCoreErr in the AAsmModule shared
  makefile. Worked around this by builing the C header using the local copy
  of the asm header, rather than the exported one.
Admin:
  Tested from clean this time. Retagged as 'FileCore-3_49'

Version 3.49. Not tagged