How to compile DreamShell ?

[sundance]

Not sure if it's the right place to ask that question. If not, please redirect towards more appropriate.

I'm trying to compile DreamShell, which is now accessible through github, making retrieving the latest source code quite simple.
The goal is to familiarize myself with generation of binaries for Dreamcast, and to experiment with newer versions of DreamShell, in case they introduce functionalities or fixes which could help.

The project also contains a directory `build`, which stores pre-compiled files which seem destined for the `DS` folder.
But they are 3 years old, hence not produced with most recent code.
The version of my isoloader is v0.6.0, and I noticed that the source on github is branded v0.6.7, so it would be interesting to see if the recompilation effectively updates something.

Of course, I'm not successful.

Environment : Ubuntu Linux, recent (Eoan)

What has worked :
- cloning Dreamshell repository
- running sdk/toolchain/ubuntu_build.sh
- cloning (separately) KallistiOS
- compiling and installing KallistiOS
- compiling, running and testing simple Dreamcast programs built with KOS and emulated on lxdream

The KaillistiOS part is not necessary, but since DreamShell seems to be built on top of KOS, it seems a nice experience to have. Moreover, it proves that my setup is able to generate working Dreamcast binaries, which is a significant step.

I note that ubuntu_build.sh seems to set KOS_BASE and KOS_PORTS to a different path than KOS itself, but it doesn't seem to matter : both contents downloaded and compiled seem identical (and up to date).

After that, and after ensuring that environment variables are correctly set, I try to compile DreamShell itself using make.
Get the following error message :

```
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: cannot find -lcfg
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: cannot find -lmxml
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: cannot find -lSDL_1.2.13
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: cannot find -lSDL_image_1.2.12
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: cannot find -lSDL_ttf_2.0.11
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: cannot find -lSDL_rtf_0.1.1
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: cannot find -lSDL_gfx_2.0.25
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: cannot find -lfreetype
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: cannot find -llua_5.1.4-2
```

So, in essence, a bunch of dependencies are missing.
But I've got no idea how to find and deal with them. I was expecting ubuntu_build.sh to do this job, but apparently it doesn't. The project is pretty short on documentation, which makes it pretty hard to guess what could be the next step.

[sundance]

I could make some progresses by going into `lib` and manually building all libraries present there.
I guess this stage could be automated as part of the main Makefile, or documented.

Anyway, progress there is, but finish line, not yet.
Here is the next batch of error messages :

```
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/fs/fs.o: in function `InitIDE':
/home/cyan/dev/dreamcast/DreamShell/src/fs/fs.c:190: undefined reference to `_g1_ata_max_lba'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/drivers/sd.o: in function `write_data':
/home/cyan/dev/dreamcast/DreamShell/src/drivers/sd.c:505: undefined reference to `_dcache_pref_range'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0x51c): undefined reference to `_dcache_alloc_range'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0x524): undefined reference to `_dcache_pref_range'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0x52c): undefined reference to `_dcache_purge_range'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0xb44): undefined reference to `_g1_ata_is_dcio'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0xb4c): undefined reference to `_g1_ata_max_lba'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0xb7c): undefined reference to `_g1_ata_standby'
collect2: error: ld returned 1 exit status
make: *** [Makefile:141: DS.elf] Error 1
```

So this one is more dangerous.
Some parts of the source code expect the presence of symbols which are not present in the rest of the project. Either the recipe is incomplete, or the code has changed, removing things on one side without updating dependents, resulting in a link error.

The second hypothesis reminded me this commit :
https://github.com/DC-SWAT/DreamShell/c ... 31d0d59109

Not sure if it's related, but notice the title : "removed old g1ata driver "
and notice that one of the "missing" symbols (g1_ata_max_lba) is indeed listed in the deleted files.

What's strange is that this commit is pretty old (Jun 6, 2017) and, if the diagnosis is correct, should have resulted in an immediate compilation fail for anyone else using this project, so it can't be detected just today.

What's going on ...

[sundance]

Some new elements :

I noticed the files in firmware/isoldr/loader,
and decided to compile that.

And sure enough, it created the intended *.bin files, to be stored into DS/firmware/isoldr .

So I did just that, and tested :

- bad news : it doesn't work with RetroDream. For some reason, RetroDream only have a single `ide.bin` file, although it's able to load iso/gdi files from the SDcard, which is kind of surprising. But when this `ide.bin` is replaced by the one from DreamShell, loading from sdcard doesn't work anymore
- good news : the new files work with DreamShell itself. Note that I've not been able to compile DreamShell itself, only the isoldr part is changed, everything else is the same.

Anyway, the new 0.6.7 isoldr is indeed compatible, and will load the games that were already working, so no degradation. But also no gain : it still doesn't load *.cso files from the SD card. So I guess it's a dead end for an easy upgrade scenario.

I still wonder what's the relation between DreamShell and lzo. Is it just to compress the binary in memory ?
I note that the compilation line included these build macros :
```
-DMINILZO_CFG_SKIP_LZO_PTR -DMINILZO_CFG_SKIP_LZO_STRING -DMINILZO_CFG_SKIP_LZO_UTIL -DMINILZO_CFG_SKIP_LZO1X_COMPRESS -DMINILZO_CFG_SKIP_LZO1X_DECOMPRESS_SAFE
```
So maybe LZO was just disabled. It doesn't tell me why, nor how to enable it. I note that `minilzo.c` source code is present in the same directory.

This adventure allowed me to notice the related source code in firmware/isoldr/loader, and so that might be the place to look at in the future to consider such a feature.

[megavolt85]

/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/fs/fs.o: in function `InitIDE':
/home/cyan/dev/dreamcast/DreamShell/src/fs/fs.c:190: undefined reference to `_g1_ata_max_lba'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/drivers/sd.o: in function `write_data':
/home/cyan/dev/dreamcast/DreamShell/src/drivers/sd.c:505: undefined reference to `_dcache_pref_range'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0x51c): undefined reference to `_dcache_alloc_range'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0x524): undefined reference to `_dcache_pref_range'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0x52c): undefined reference to `_dcache_purge_range'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0xb44): undefined reference to `_g1_ata_is_dcio'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0xb4c): undefined reference to `_g1_ata_max_lba'
/opt/toolchains/dc/sh-elf/lib/gcc/sh-elf/5.2.0/../../../../sh-elf/bin/ld: ./src/exports.o:(.data.ds_symtab+0xb7c): undefined reference to `_g1_ata_standby'
collect2: error: ld returned 1 exit status
make: *** [Makefile:141: DS.elf] Error 1
```

all this functions can be present in KOS after apply patch ds/sdk/toolchain/patches/kos.diff
if patch alredy instaled then need rebuild KOS

P.S. last loader is 0.7.0 with low level emulation

[sundance]

> all this functions can be present in KOS after apply patch ds/sdk/toolchain/patches/kos.diff

Well, this story was started with invoking ubuntu_build.sh .
This script, in turn, invokes `make patch`.
`make patch` (is supposed to) install all patches, including `make patch-kos`.
So kos.diff patches are supposed to be already installed.

This is confirmed when trying to install them again :
```
make patch-kos
(...)
Reversed (or previously applied) patch detected! Assume -R? [n]
Apply anyway? [n]
Skipping patch.
```

Even after rebuilding kos, and then rebuilding DreamShell, it doesn't change the picture : the exact same list of missing symbols is still requested

[megavolt85]

check ds/sdk/Makefile.cfg

path DS_BASE

[sundance]

> check ds/sdk/Makefile.cfg

```
DreamShell SDK Makefile.cfg
(...)
DS_BASE = $(KOS_BASE)/ds
DS_SDK = $(DS_BASE)/sdk
DS_BUILD = $(DS_BASE)/build
```

with
```
echo $KOS_BASE /usr/local/dc/kos/kos
```

To be noted :
$(KOS_BASE) does exist, and contains KOS.
But there is nothing at $(KOS_BASE)/ds, this directory does not exist.
Therefore, $(DS_SDK) and $(DS_BUILD) don't exist either.

[megavolt85]

DS_BASE = /path/to/directory/with/dreamshell
or create ds folder in kos and put dreamshell src to this directory

[sundance]

I've added a `ds` symlink into `$KOS_BASE`, aka /usr/local/dc/kos/kos ,
which redirects towards the directory where DreamShell source code is stored.
Now, `DS_BASE = $(KOS_BASE)/ds` exists, and corresponds to DreamShell directory.
Consequently, `DS_SDK = $(DS_BASE)/sdk` and `DS_BUILD = $(DS_BASE)/build` exist too, and should correspond to intended directories.

Unfortunately, from a compilation stand point, this adaptation does not help.
And the list of missing symbols is still exactly the same one.

Investigating a bit deeper,
if one selects a missing symbol as an example, `g1_ata_max_lba()`,
it's declared in both `include/drivers/g1_ide.h` and `firmware/isoldr/loader/dev/ide/ide.h`,
and then defined in `firmware/isoldr/loader/dev/ide/ide.c`.

Unfortunately, this file is not present in the list of object files needed to create for the first target, `DS.elf`.
So it's not compiled, and then it seems normal that the linker cannot find the symbols it contains.

It's as if the recipe was incomplete.

[megavolt85]

linker can't find `g1_ata_max_lba()`, why you look in header? linker don't check header
`g1_ata_max_lba()` is function from g1ata.c, linker can find this symbol in libkallisti.a