AMB conversion of system documentation

[mateuszviste]

Last night I improved AMB so the format and toolset supports compression through MVCOMP. This makes the help files about 40% smaller.

A possibly interesting next step could be to convert all system documentation (files that live in %DOSDIR%\DOC) to AMB. This would make the documentation much more compact: no cluster waste when a single package has lots of txt files + all documentation would be now nicely compressed.

The one downside is that it would no longer be possible to consult a readme file using TYPE, but I do not think it is a very common practical need. Myself, I rarely use TYPE for this since I need to be able to go back and forward reading the documentation, so I usually load it in SVED. Using AMB instead makes little difference.

Compacting documentation in such a way would decrease the default install size to below 1.5M, allowing to have a functional SvarDOS installation (with pkg management etc) on a single 1.44 floppy. This would solve #15 at the same time.

Converting a set of TXT files to AMB should be rather straigh-forward with some relatively simple script:

• generate an index file that lists all the TXT files available (with links)
• word-wrap the TXT files to 78 characters
• escape characters that need to be escaped (%)
• compress the files with MVCOMP
• pack it all up with AMBPACK

[mateuszviste]

As a quick test I wrote a crude script that converts a directory of text files to a compressed AMB file (dir2amb.sh).
Used it on the current "FORMAT" documentation: its size went from 49K (incl. cluster waste) to 23K.

Quite promising I think.

[mateuszviste]

...and current FDISK documentation goes from 32K to 15K. Most notably the annoyingly large GPL2 thing shrinks from 18K to 8.3K.

It seems safe to assume at least a 50% compression gain , which means that the footprint of a fully installed system could go from slightly over 1.5M to about 1.3M if AMB compression is applied to all core packages. The "SvarDOS live on a floppy" idea becomes very much realistic.

[boeckmann]

I think this is a good idea :) Looking at the FDISK documentation files, I think I will clean this mess up a little bit for the next release.

[mateuszviste]

Converting a set of TXT files to AMB should be rather straight-forward with some relatively simple script:

* generate an index file that lists all the TXT files available (with links)
* word-wrap the TXT files to 78 characters
* escape characters that need to be escaped (%)
* compress the files with MVCOMP
* pack it all up with AMBPACK

Also:

• convert files from CR/LF endings to LF-only. AMB does not need CRs, and LF-only makes the files a little bit more compact and a little bit more compressible.
• convert TABs to 8x spaces
• trim whitespaces at the end of lines

[mateuszviste]

a little downside to share: while documentation is consistently smaller by 40-60% when packed into AMB, the package files where large txt documentation have been replaced by a compact AMB tend to actually grow in size. Sometimes by only a few bytes, sometimes by up to 4-5%. Not a nice surprise. It seems that ZIP struggles to compress MVCOMP data, and the result of DEFLATE(MVCOMP(TXT)) is less optimal than a straight DEFLATE(TXT). I was low-key hoping this would be offset by the fact that AMB keeps all files together, making documentation a "solid archive" where ZIP would find more compression opportunities, but evidently this is not enough to make up for the worse compressability (higher entropy) of MVCOMP data.

nothing to solve here I guess, that's just how it is.