Compressed ISO (*.cso) on DreamShell

[sundance]

> loader contain: SD driver, syscall emulation, compress algoritm

Thanks for information @megavolt85.

Could this be translated into source code to look at ?
So, I've under my radar :
- modules/isoldr
- modules/isofs
- modules/minilzo
- firmware/isoldr/loader

"SD driver" seems part of the last one (in firmware/).
"compress algorithm" if represented by `minilzo`.
What about syscall emulation ?
Is it about vfs_handler ? => https://github.com/DC-SWAT/DreamShell/b ... 60.c#L1336

Any other place in the code to look at ?

[megavolt85]

vfs_handler is part kernel and don't used in loader

look Makefile.sd

Code: Select all

$(TARGETCC) $(TARGETCFLAGS) $(TARGETLDFLAGS) -o $@ $(LOBJECTS) $(LIBS)

object files is LOBJECTS

Code: Select all

LOBJECTS += $(SD) 

now open Makefile.cfg

Code: Select all

LOBJECTS = $(MAIN) $(KOS)
LOBJECTS += $(CDDA)

all object is MAIN, KOS, SD andCDDA

Code: Select all

KOS = 	$(KOS_DIR)/src/memmove.o \
		$(KOS_DIR)/src/memcpy.o \
		$(KOS_DIR)/src/memset.o \
		$(KOS_DIR)/src/memcmp.o \
		$(KOS_DIR)/src/memchr.o \
		$(KOS_DIR)/src/strlen.o \
		$(KOS_DIR)/src/timer.o \
		$(KOS_DIR)/src/cache.o \
		$(KOS_DIR)/src/biosfont.o

MAIN =  startup.os main.o syscalls.o reader.o mmu.o utils.o \
		 descramble.o gdc_syscall.o menu_syscall.o sys_syscall.o \
		 bfont_syscall.o flash_syscall.o

SD  =   $(FATFS) ./dev/sd/spi.o ./dev/sd/sd.o minilzo.o

FATFS = $(FATFS_DIR)/src/ff.o \
		 $(FATFS_DIR)/src/option/ccsbcs.o \
		 $(FATFS_DIR)/src/diskio.o \
		 $(FATFS_DIR)/fs.o

CDDA =  pcm_split.o cdda.o

[sundance]

Thanks for these additional elements @megavolt85

I noticed the `minilzo.c` source file in the sd firmware source code, and I didn't understand what it is doing there.
I was expecting `sd.bin` to be purely about reading data by sectors from sdcard, and nothing more,
with an upper logic expected to deal with the File System (FAT32 mostly ?)
and then a virtual file system on top of the *.cso file.

Hence I'm a bit surprised to see mention of *.cso file directly in the SDcard firmware.

That being said, this implementation is gated, behind a HAVE_LZO build macro,
which I don't know if it is enabled nor how to enable it.

_edit_ : according to this Makefile :
https://github.com/DC-SWAT/DreamShell/b ... le.cfg#L77
HAVE_LZO is enabled for sd.bin, while it's disabled for ide.bin.
This is consistent with the RC4 release note.

However, some part of ciso implementation are completely commented out, with mention "do not work" !
https://github.com/DC-SWAT/DreamShell/b ... der.c#L790

So I'm a bit confused now.

I initially expected the functionality to be provided in modules/isofs.klf, but that's not clear anymore.
It looks as if it's actually entirely contained with sd.bin (?).
In which case, I really don't understand what modules isofs.klf and isoldr.kld are about,
do they even have a role in this story ?

Also, it's pretty clear that *.cso support is currently broken,
and it has been so for a long time, judging by my previous test of RC4, dating back to 2016!
Yet, the code is still there, and the build script still enable it for SDcard, so it seems this consequence was simply not detected, and remained undetected for quite a long time.

Possibly, the fact that IDE doesn't support *.cso anymore led to a majority of users no longer wondering about this support, later leading to breaking this feature during a patching exercise, with no core developer noticing it because it's not tested (there's no automated test on this project btw). That would be a classical story.

Anyway, if all the important code to look at is in `firmware/loader`, then it actually simplifies things, as there is only one pool of files to worry about.
Note that I'm not completely sure it is that simple, since some previous experience show that DreamShell/isoloader crashes pretty quickly, within isofs.klf, just on file selection, before even starting to load it, so there might be a need to fix this part too. That's a lot to do just to reach the point of having a "working" implementation to improve upon.

Debugging is also going to be an issue. I'm not sure how DreamShell developers are doing, but a minimum capability is to generate traces, in order to analyze an outcome. Yet, a Dreamcast is not a PC, so accessing these traces can be daunting. Excluding owning a special Dreamcast developer cable, which I don't have, I see 2 main ways :
- use an emulator => generally a lot more flexible, but also more limited, especially when it comes to hardware signals, such as the serial port interface.
- storing logs into a file, typically on the sdcard => this adds more complexity and risks (write operations, flush, file system, etc,), but might be the only method available. I note that the code contains trace capabilities, with macros such as `LOGF`, `LOGFF`, `DBGF` and `DBGFF`, but I have no idea how they work.

Without any of these capabilities, it's like blind walking in a dense forest, significantly reducing probabilities of success (especially when starting from a broken state).
Any information on this topic would be welcomed.

_edit_ : found in the code : `LOGFF` macros can be enabled by adding `-DLOG` to compilation flags , this will create file `/isoldr.log`. Hopefully, a good enough beginning. There might still be a need to understand how data is properly flushed and committed to storage, especially in the event of a crash, since buffered logs would be lost, thus removing the very traces that were necessary to understand the bug.

[sundance]

Some more tests :
I could get to generate an `sd.bin` firmware _without_ lzo support, resulting in a slightly smaller binary (26756, vs 28944).
I was hoping that I could observe the effect of trying to load the file without this support.

However, the pb is, I don't even get to that point : just _selecting_ the *.cso file instantly crash the application.

I now suspect that the "modules" are actually loaded by the "apps", a distinct world separated from `sd.bin`.
So in this new theory, `isofs.klf`, `isoldr.klf` and `minzlo.klf` are loaded by `applications/iso_loader`.

The usefulness of modules might be limited to the launch screen, but it's nonetheless enough to break the experience, so it also needs to be fixed, in order to even have a chance to reach `sd.bin`.

Hence, it's more work...

[Ian Micheal]

I have been working on using CSO on dreamneo seems only 5 to 20 meg change between them for neogeo cd isos

[sundance]

Gains can be significant for some DreamCast ISO.
For example, GigaWing2 goes down from 1GB to 410MB using LZO compression.
zlib could go even smaller, but it is not supported for Dreamcast games (memory is needed by the game itself).
Anyway, the objective is not only to save storage space (that's secondary), but rather to improve loading times.
But for that, I would need this interface to at least work, so that there is something to improve upon.