Homebrew sequenced music on AICA: What would it take to make it possible? Ideas, and a call for help and suggestions!

[baku-chan]

Hello there! My name is baku-chan, and I'm writing here to inquire about the possibility of creating homebrew sequenced music for the Dreamcast's AICA as well as what it would take to make it happen.

Just to let you know all know about my background with all of this: while I do have some basic technical knowledge, I'm no programmer myself. I'm just a lover of video game music and especially of sequenced video game music; the kind from eras long past when everything was done either in chiptune or on synthesizers, many of which I don’t think have really had their potential tapped out yet. Like, for instance, I think that sound chips like the Mega Drive/Genesis’s YM2612 and the SNES’s SPC700 have really gotten a lot of attention when it comes to authenticity in emulation and pushing the limits of the hardware, and that’s great! But I also think that later sample-based sound hardware like the Sega Saturn’s SCSP, the PS1’s SPU, the Dreamcast’s AICA, the PS2’s SPU2, or the GameCube and Wii’s DSP have been almost forgotten in comparison, or at least not taken as seriously when it comes to the idea of preserving the many legendary OSTs made for those chips as they truly were, or even when it comes to the idea of writing music for them, like we do now with YM2612 and SPC700-based homebrew music in an almost casual, taken-for-granted way that was unthinkable not too long ago!

And I know that there are some very knowledgeable people in this community who I’m hopeful might be able to help make my crazy dream possible, haha, and to help share that dream with others who might be interested in such a thing. Whether you’re a musician or a lover of video game music or just someone looking for a challenge — or if you’re some combination of all of the above! — then I’d love to make your acquaintance.

For the uninitiated or anyone just stopping by here wanting to know why I would want to chase such a dream, I would like to take the time to go over some the things that make both SCSP and AICA special, my perspective on all of that compared to other sound chips, and why I think that they deserve to have the ability to write homebrew sequenced music on them.

To start, I'd like to give some background as to how I got here; that is, how I became so interested in AICA. I've been doing some research for a while on getting homebrew sequenced music working on the PS2, and comparing the capabilites of that console's SPU2 sound chip (along with that of its closely related predecessor SPU in the PS1) to that of the Sega Saturn's SCSP and, indeed, Dreamcast's AICA led me to think about various possibilites for the latter especially. Much has been said — rightly or wrongly — about the capabilities of the rather infamous SCSP, as well as its arguably overshadowed younger sibling AICA. Of course, I'm sure that everyone here knows the basics about these chips, included the much-touted features of the SCSP in particular that have often been considered to be significant advantages over the likes of, say, the SPU found in the PS1. Such as, for instance, FM support via using SCSP's 32 voices as modulators (such that you could supposedly have up to eight channels of 4-op FM, six channels of 6-op FM, or even a single channel of 32-op FM should one be inclined), as well as a DSP with several more effects over the basic reverb, echo, and delay capabilities of the PS1's SPU including chorus, low- and high-pass filters, an equalizer, and more. With said features, the SCSP is theoretically very powerful, and indeed arguably more so than the SPU in many ways (or even later sound chips like AICA and the PS2's SPU2, for that matter). With that said, the SCSP is also well-known for its "fatal flaw" in the form of its lack of hardware ADPCM compression support, leaving it with a significant disadvantage over the PS1's SPU does have over the SCSP that can be heard even with the latter's most accomplished soundtracks (think the likes Radiant Silvergun or Panzer Dragoon Saga) and makes some the former's greatest accomplishments (such as Rhapsody: A Musical Adventure and Aitakute... your smiles in my heart, among many others) seem relatively impossible on the Saturn in comparison. To really and finally drive the point home, I came up with a rough summary of how many total samples Saturn's SCSP, PS1's SPU, Dreamcast's AICA, and Sony's SPU2 can each store in the form of seconds at multiple commonly used sample rates, with any bespoke compression schemes converted to a standard unit of uncompressed PCM and after things like sequence data, tone data, sound driver, and DSP effect RAM usage are taken into consideration:

NOTE: Sound driver, sequence data, tone data estimates don't exist for SPU and SPU2 here because they're all stored in main RAM for the former and the IOP's RAM for the latter, rather than in their separate pools of sound RAM.

SCSP = 384 KiB ((512 KiB - 44 KiB driver [based on "Saturn Sound Driver Implementation Manual" (ST-241-042795, pg.26)] - 48 KiB DSP [average DSP usage guesstimate] - 32 KiB [sequence data guesstimate] - 4 KiB [tone data guesstimate]) * (1/1) [already uncompressed])
@ 22.050 KHz => (384 KiB / 44.1 KiB/sec) = 8.70748 seconds
@ 32.000 KHz => (384 KiB / 64.0 KiB/sec) = 6.00000 seconds
@ 44.100 KHz => (384 KiB / 88.2 KiB/sec) = 4.35375 seconds
SPU = 1479 KiB ((512 KiB - 96 KiB reverb unit [max possible reverb unit RAM usage]) * (32/9) [Sony SPU-ADPCM])
@ 22.050 KHz => (1479 KiB / 44.1 KiB/sec) = 33.53741 seconds
@ 32.000 KHz => (1479 KiB / 64.0 KiB/sec) = 23.10938 seconds
@ 44.100 KHz => (1479 KiB / 88.2 KiB/sec) = 16.76871 seconds
AICA = 7392 KiB ((2048 KiB - 64 KiB driver [guesstimate based on 32 KiB x 2 for Puyo Puyo DA!'s MANATEE.DRV driver] - 96 KiB DSP [average DSP usage guesstimate from SCSP x 2] - 32 KiB [sequence data guesstimate] - 4 KiB [tone data guesstimate]) * (4/1) [Yamaha ADPCM])
@ 22.050 KHz => (7392 KiB / 44.1 KiB/sec) = 167.61905 seconds
@ 32.000 KHz => (7392 KiB / 64.0 KiB/sec) = 115.50000 seconds
@ 44.100 KHz => (7392 KiB / 88.2 KiB/sec) = 83.80952 seconds
SPU2 = 6372 KiB ((2048 KiB - 256 KiB reverb units [based on reverb unit RAM usage for both cores]) * (32/9) [same ADPCM as PS1's SPU])
@ 22.050 KHz => (6372 KiB / 44.1 KiB/sec) = 144.48980 seconds
@ 32.000 KHz => (6372 KiB / 64.0 KiB/sec) = 99.56250 seconds
@ 44.100 KHz => (6372 KiB / 88.2 KiB/sec) = 72.24490 seconds

Yeah. With that said, trying to find ways around the SCSP's weaknesses soon led me to appreciate the silent power and potential of its successor AICA in the Dreamcast. Again coming from the PlayStation scene trying to get sequenced music working there, comparing the PS2's SPU2 to AICA revealed to me that the latter is an essentially a supercharged version of SPU2, having 16 more channels while inheriting the superior DSP of its predecessor and adding hardware ADPCM support that said predecessor failed to include. In that sense, it is also arguably everything that the SCSP should've been, with the ADPCM support in particular being, by far, the single greatest improvement that alone quadruples the effective amount of sound RAM. And combined with an increase of the actual physical sound RAM from 512 KiB to 2 MiB, this means that AICA has a staggering sixteen times (!!!) the effective amount of sound RAM compared to the SCSP. Its sole regression compared to its predecessor is that FM is no longer possible, which is unfortunate but also no great loss compared to everything that's gained in exchange.

I've already mentioned some of the PS1 era's best sequenced OSTs like the aforementioned Rhapsody and Aitakute... such OSTs would represent less than a fourth of what sound chips like AICA are able to handle if we're measuring by how much RAM is available to them, let alone considering the additional features of the latter's DSP that those SPU soundtracks were never able to use. Likewise, I've heard someone say in regards to the SCSP with all of its features that, so long as you can manage its memory deficits, then there is pretty much nothing that you can't do on it; if that's true, then imagine what could be done on AICA! In fact, let's imagine for a moment, using some of the greatest sequenced OSTs on the PS2 and its SPU2 sound chip for comparison. There, you have the likes of Final Fantasy X and X-2, Final Fantasy XI and its expansions, Final Fantasy XII, Kingdom Hearts 1 & 2, the Atelier series (Lilie, Judie, Viorate, Iris), the Summon Knight series (3, 4, Granthese), the Infinity series (Never 7, Ever 7, Remember 11), the Super Robot Wars series (MX, Z, OGs), Tokimeki Memorial 3 ~At the place of our promise~, Namco x Capcom, and Berwick Saga representing some of the greatest pieces of sequenced music ever written for a video game console... and not a single track from any of those games exceeds a single megabyte of RAM compressed; they literally don't even represent half of what SPU2 is capable of, let alone AICA.

Given what we know is possible on the Saturn's SCSP let alone the PS1's SPU and the PS2's SPU2, imagine what's possible with Dreamcast's AICA... that's what driving me here now to see if we can make homebrew music possible for these sound chips! But first, there's a lot of questions to ask and problems to be sorted out, and so I'm posting here hoping if some of you here can answer at least some of those questions for me. Thank you in advance for taking the time to read! Because, please be forewarned, there's a lot of stuff here, haha...

AICA's DSP: How does it work and what can you do with it?
I understand that the SCSP's DSP can execute up to 128 steps of instructions written for it, doing so in an exact order with no looping, conditions, or repeats. It's also apparently programmable to at least some extent, with effects running on there as "modules" that are essentially mini-programs that run on the DSP. Such modules, according to SEGA's documents, include effects such as "Reverb", "Echo (Delay)", "Chorus", "Flanger", "Filter", "Parametric EQ", and others. "SCSP/DSP Effect Module Specifications" (ST-069-121693) provides a list of the DSP modules available as standard on the SCSP. The document describes said list as "tentative" with the values outlined within supposedly subject to change, but it also appears to be the latest reference released that I'm aware of.

From what I've heard, AICA supposedly inherits a similar if not identical DSP. I've heard some rumblings that there have been some changes made to AICA's DSP versus that in the SCSP, but I don't know about any documentation that exists about what said changes actually are. In any case, I'm assuming that at least inherits the same effects that its SCSP equivalent does?

If so, then based on everything that I've read in those documents and elsewhere, it appears that you can use any amount and combination of the DSP modules listed above — including more than one of the same module — on any of the SCSP's 32 hardware voices as long as the total number of "steps" spent executing them doesn't exceed 128 and the total number of modules used doesn't exceed 16, is this correct?

Some more questions, if AICA's DSP is indeed close to that of the SCSP's: are these DSP programs above built into AICA itself (as in, any game running any sound driver can access them) or are they only available via certain SEGA-provided sound drivers? Also, would the programmable nature of the DSP mean that it's theoretically possible to write custom DSP programs that can do whatever you want, as long as it can be run within the DSP's 128 steps and without exceeding the available sound RAM? (See: ST-228-R1-030596, a "dAsms User Manual" where dAsms is apparently assembly for the DSP.)

AICA's ARM CPU: what can it do, if anything?
The current authoritative resource on the capabilities of AICA's ARM CPU appears to be this venerable DCEmulation thread. The main takeaway that I gained from there is that AICA's ARM CPU is a very, very slow piece of kit — much slower than you'd expect from its apparently deceptively high clock speed — and that it's not useful for much else other than running driver code.

Or is it...? For all that we do and don't know about this CPU, one thing that do know sure, by sheer inference, is that — at least for running driver code, anyway — that it's at least as powerful in practical terms as the 68K that ran the SCSP in the Saturn. After all, AICA is a direct superset of said SCSP! If AICA's ARM was actually less capable than its direct predecessor's 68K, that would be pretty pathetic and would make you wonder how the hell it's useful for anything at all, wouldn't it?

Also, there are a few benchmarks — of sorts — available for both SCSP's 68K and AICA's ARM, specifically when it comes to performing decompression of CRI Middleware's proprietary ADX ADPCM. For one, there's ponut64's ponèSound, which according to them allows the SCSP's 68K to play roughly 25,000 ADX samples per second. Furthermore, based on an estimate — but not test — made by them, the same code can play around 50,000 IMA ADPCM samples per second (which, if correct, underscores how much more processor-intensive decompressing ADX ADPCM is vs. IMA ADPCM). Meanwhile, the developer of another homebrew music playing project that I found, this time for AICA's ARM (one that I don't remember the source for and can't find right now, unfortunately) claimed that the number of ADX ADPCM samples playable per second with their code on said processor is higher than 44,100 (equivalent to a single 44.1 KHz PCM mono stream) but less than 88,200 (equivalent to a single 44.1 KHz PCM stereo stream), with the true number presumably being somewhere between the two. Such a result points to around a two-fold increase in power for the AICA's ARM vs. Saturn's SCSP, or for decoding ADX ADPCM specifically, at least. Finally, according to the developer of another ADX player for the Dreamcast called LibADX who published actual results from their code, playing a single stereo stream of ADX supposedly uses only one percent of the Dreamcast's main SH-4 CPU, while playing the same stream on the ARM CPU failed to do so in real-time, or in order words: it took up 100 percent of the CPU and technically exceeded that to obviously no avail. No sample rate was given for either test, but assuming that it was 44.1 KHz for both (in other words: 88,200 total ADX samples per second for the aforementioned stereo stream), then that would roughly track with the claims made by the previous developer and, irrespective of sample rate, would make AICA's ARM CPU more than 100 times slower than the main SH-4 CPU! Again for whatever specific type of calculations are needed to decode ADX on a general-purpose processor, anyway (and also keeping in mind that the ARM CPU apparently lacks cache which, according to the aforementioned Rand Linden from the DCEmulation thread, is a major reason why it's supposedly so pathetically slow in the first place).

Make of all of the above what you will, but what I'm getting at is that I'm wondering if there might be any use. For instance... what about FM? The lack of it on AICA has often been called a disappointment compared to its predecessor. Now, it could be argued that the massively increased effective memory available to AICA vs. the SCSP via just the addition of hardware ADPCM compression by itself — let alone the four-fold increase in actual physical RAM — makes FM completely unnecessary in any practical way in AICA in the first place, and that if you really wanted to have FM in AICA, you could easily just sample FM sounds from a synthesizer like the Yamaha DX7, or even from an FM-capable console like the Mega Drive with its famous YM2612 or, indeed, a Sega Saturn with its bespoke FM capabilities. But where's the fun in that, haha? If we instead wanted to push the boundaries of what can be done on AICA and try to make the impossible possible, how much can its ARM CPU help us with that? Would it be possible for it to run a program that takes a waveform, pushes it through various FM algorithms, and then creates a new waveform that can then be manipulated by AICA with ADSR, DSP effects, and such, just like any other sample? And with such a program, would it be possible to approximate the functions of Yamaha FM chips such as the aforementioned YM2612 or the SCSP in FM mode, possibly even to the point where music written for those sound chips could run on them? Or could it even be possible to approximate more sophisticated Yamaha FM hardware, such as the aforementioned DX7 or exotic arcade sound chips like the YM2151? And if it's not something that the ARM CPU can do, then we haven't even talked about the main SH-4 CPU yet, have we...?

AICA panpot resolution (including versus PS1's SPU and PS2's SPU2)
According to the "Saturn SCSP User's Manual", the SCSP is able to pan voices up to 15 steps each for the left and right channels plus a center step for a total of 31 steps (ST-77-R2-052594, pg.12). On the surface, this seems rather crude compared to the seemingly much higher panpot resolution of the PS1's SPU and by extension the PS2's SPU2, both of which can pan voices up to 63 steps each for the left and right channels plus a center step for a total of 127 steps (a more than four-fold difference!). However, a supposedly extended variant of panning that utilizes the DSP is also mentioned in the SCSP user's manual, referred to as "process panning" (complete with a graphic — Figure 4.55 — on pg.82). Meanwhile, the only source that I was able to find for the panpot resolution of AICA is an old overview from 2002 of some of chip's registers that the author admits might be spotty in accuracy; nonetheless; it suggests that AICA inherits the same panning system as SCSP, as in: 15 steps each for the left and right channel plus a center step for a total of 31 steps. Would anyone here happen to have any knowledge of this?

Mono samples vs. the possibility (or usefulness?) of stereo samples
In the world of sample-based synthesizers, there's often a distinction made between mono samples and stereo samples, with the general assumption typically being made that the latter are of higher quality (one example of that being when trying to simulate a realistic grand acoustic piano sound, for instance). However, the total polyphony of synthesizers is generally calculated based on mono samples, which means that for a synthesizer that has, say, 64-voice polyphony, using only stereo samples means that the synthesizer now effectively only has 32-voice polyphony, as two voices are taken to play a single sample. Now, I'm almost certain that the sample-based sound chips in consoles like the Dreamcast play only mono samples, unless I'm mistaken there. With that said, would it be theoretically possible to make stereo samples work there anyway, and even if it is possible, would it even be worth it? Now, as for whether it's possible or not, my hypothesis is that is could be done, maybe, but probably not using the official file formats which almost certainly had no functions for such a thing given that, again, the sound chips themselves almost certainly don't have that feature. That said, one feature that those sound chips most certainly do have is panning, including — of course — hard panning on either the left or right channel even as the actual sample being played is mono. So then, would it not be possible to just use two different mono samples playing simultaneously hard-panned to the left and right channels respectively — both associated with the same instrument — and use that to represent a single stereo sample? That said, in order to make that possible, we would almost certainly have to go the route of creating a new sequence format to accommodate such a relatively arcane function rather than existing formats that, again, almost certainly don't even have any notion of such a concept as stereo samples. Which, actually, leads me to the next section...

Reverse-engineering the sequence, wave, and instrument definition formats (MSB/MPB) for AICA
There are two "official" file formats for the Dreamcast handling sequenced music that I'm aware of. First, there's MSB/MPB, which I presume is roughly equivalent to the Saturn's SEQ/TON PS1's SEQ/VAB in that MSB is a relatively small MIDI-like series of note events and such while MPB is a relatively large collection of raw sample files and instrument definitions that the MSB file references. I've personally found these files in Puyo Puyo DA! and in the "Kuuzokuban" trial version of Skies of Arcadia. Second, there's MLT, an apparently very versatile container format that uses a bank system to potentially hold several different files at once including — among other things — MIDI-like sequence data, and which I've personally found in the final Skies of Arcadia release. Also based on my experience with that game and the fact that its MLT files are waaaaaaay too small to hold a full sample-based sequenced song of any meaningful complexity (as in not even triple-digit kilobytes, whereas the far less sophisticated Puyo Puyo DA!'s music regularly hovers around the 500 KiB range for its MSB/MPB files), it also appears that MLT files reference one or more external files for at least raw sample data if not instrument definitions as well, or at the very least that it can reference external files for such purposes, if not necessarily must.

Naturally, in order to create homebrew music on AICA and make it possible to share it with people, said music has to be stored on something. And so, there are a few ways to accomplish that. The first option would be to reverse-engineer one or more of the above "official" sequence formats and use those as our deliverables for homebrew music. However, Dreamcast's MSB/MPB file formats appear to be completely unexplored and undocumented, while the closest thing that I could find for documentation of the MLT file format is how to convert it to DSF via another kingshriek Python script named "dsfmake". The source code comments for dsfmake make frequent references to MSB and MPB — among other formats like MDB ("MIDI drum bank") and OSB ("one-shot bank") — and seems to suggest that MLT is not actually a sequence format itself but rather truly just a container format to hold the actual standard sequence formats for the Dreamcast which is indeed MSB/MPB, as well as other essential files for playback like DSP program files. Or rather, it can hold the latter files if one so chooses (and incidentally, the comments say that one indeed should move all of said requisite files into an MLT file before running them through dsfmake). In any case, perhaps one way to approach reverse-engineering these formats is to compare them to MIDI, the widely-used and widely-understood standard for sequenced music. Which I suggest as part of my experience with the SEQ and SQ formats of the PS1 and PS2 respectively, which according to people familiar with both are extremely close — if not practically identical save for a few minor changes — to SMF MIDI format 0. While I have no idea if either MSB or the Saturn's SEQ or anywhere near as close to standardized MIDI as Sony's sequenced formats apparently are, it should be noted that an official MIDI converter specification exists for SEQ on Saturn (ST-066-121593). And so with that said, one question that I'd like to ask is this: how close is Dreamcast MSB to both SMF MIDI format 0 and Saturn's SEQ, and how trivial would it be to reverse-engineer them such that arbitrary data can be written to it using a sequence editor or tracker?

Alternatively, a second if rather potentially insane option exists. While I admit to knowing relatively little about how MIDI works, I've read that it was meant to be an expandable and adaptable format given its need to be compatible with many different synthesizers with their own unique capabilities. So addition to your standard MIDI events, I believe — anyone here familiar with MIDI, please correct me if I'm wrong! — that there are ways to define custom ones in order to cover everything that different pieces of hardware compatible with MIDI has to offer. If this is true, then I wonder: could we simply treat the AICA as such a different synthesizer, create custom events for it in MIDI (like, for instance, activating DSP modules?), and then just throw the resultant MIDI along with the requisite sample data to a sound driver for it to play back on AICA directly? I mean, a part of me is certain that if it were that easy, then SEGA would've done the same thing for themselves from the beginning instead of creating a bespoke format like MSB. But another part of me is wondering if it really is that simple... or at least realistically possible. Honestly, the only reason why I'm bringing up this possibility versus just reverse-engineering MSB/MPB is that I'm thinking about how universal MIDI is as a format compared to ancient, custom file formats designed to work with only a single piece of hardware as deliverables converted from more familiar formats like MIDI, rather than being dealt with directly as if they actually were MIDI. That is to say, practically every single piece of music software in the world recognizes MIDI, while absolutely nothing knows that MSB or any other console sound chip format even exists let alone how to handle them. In addition, I've been thinking about the prospects of compatibility across multiple sound chips. For instance, if I composed a piece of sequenced music for the Dreamcast, what if I wanted to create a version for the PS2? If I created said song directly in MSB/MPB, then it's stuck in that bespoke format forever unless I convert it to PS2's equivalent in SQ/BD/HD. Which means that a converter needs to be written and tested and such (including porting all of the instrument data and converting PCM to Sony's custom ADPCM format and re-doing any DSP effects among many other things), and then that's all repeated for every other system that you want your music to play on with their own unique ways of doing things. Whereas if MIDI is used for everything instead, then at least the most basic things like key on and offs, pitch bend values, velocity, etc. are all standard across any sound chip that supports reading MIDI, with only things unique to certain sound chips needing to be changed. With that said, I'd imagine that the success of such an approach depends on how just close AICA's inner systems track with how MIDI was designed, right? Which is most definitely no guarantee, and perhaps explains why bespoke formats exist for console sound chips in the first place, so please forgive my ramblings, haha. (And of course, it goes without saying that even of all of that was possible, it would all require a completely custom driver, as obviously the official SEGA sound drivers don't work anything like the above theoretical way.)

As such, perhaps the simplest solution to that last prospect especially is a third and final option, which would be to just, well, use MIDI, from the very beginning of the composition process and then convert the final result to each console's bespoke file formats from there, just like the composers attached to actual commercial software projects did back in the day. To not even worry about reverse-engineering formats like MSB/MPB at all as something to actually be edited themselves as if they were MIDI, but rather as a means to figure out how to convert a standard format like MIDI to them as accurately and painlessly as possible. Perhaps, anyway. What does everyone here think would be the best option out of the three here?

Choosing a deliverable format for homebrew AICA music: DSF or...?
If you're familiar with how most people listen to Dreamcast music outside of real hardware or OSTs of the in-game music that are almost certainly recorded from real hardware, then you're probably aware of the homebrew-created DSF file format — themselves offshoots of the original PSF format for PS1 music — that are widely used to store and play back said music ripped from the original games. You might be thinking, then: hey, why not just use that format to distribute our homebrew Dreamcast music on? It makes sense to use such common and well-known formats like DSF, right?

To put it simply, I'm not entirely sure if that would be a very good idea. For many reasons, but mainly because the state of both Dreamcast emulation has, up until very recently, been rather dire. And in particular, the state of the programs that people use to actually listen to DSF files has been especially so in a way that, unlike general-use emulators, has yet to really be rectified as of late. Or rather, program, singular, at this point, said program being Highly Theoretical. Other programs that can play DSF files exist, yes, but Highly Theoretical is the only one that's been updated at anything resembling a recent date (2020/09/02 for version 2.0.53, as of this time of writing).

Now, I've had many encounters with Highly Theoretical going back to when I was creating videos for (now-defunct) my video game music YouTube channel, and the first thing that struck me was how off it was when playing sequenced music from NiGHTS Into Dreams... from the Saturn. And as you might expect given how closely related SCSP and AICA are, it didn't do much better with sequenced Dreamcast music. Skies of Arcadia's DSF rip confirmed that for me, in this case in direct comparison with recordings that I myself made from real Dreamcast hardware using the PAL prototype's sound test. Like with Nights Into Dreams..., you're more than able to make comparisons yourself with the current version of Highly Theoretical against even Skies of Arcadia's flawed (incomplete and missing tracks, weird mastering, and such) but still hardware-accurate OST CD release. To start, "Sky Pirate Hideout" has a brighter sound on Highly Theoretical versus real hardware that appears to be slightly filtered in comparison, while the wind sounds that appear in and out in the background on the latter are barely audible on the former. "Kingdom of Ixa'taka" lacks the bass response of real hardware on Highly Theoretical, even when both are adjusted for volume. "Sudden Storm" seems to have instrument volume balance issues on Highly Theoretical that don't exist on real hardware, causing some parts in the background to cut through the mix more than they should. "Uninhabited Island" sounds noticeably flat and thin on Highly Theoretical with the background strings especially, apparently due to it not rendering the reverb properly where it sounds more abundant and enveloping on real hardware. "Theme of Loneliness" is a track that makes heavy use of reverb and what appears to be low-pass filtering to create a kind of lo-fi concert atmosphere for its sole piano accompaniment; said effects are almost completely missing on Highly Theoretical (save for the tiniest bit of reverb remaining). "Theme of Fina" surprisingly doesn't have the same problem with its own reverb-heavy piano solo featured in its opening seconds, but it falters elsewhere with a lack of bass response and weak, quiet-sounding background strings compared to real hardware. And finally, "The Dark Rift" — a track not featured on the OST — is significantly lacking in reverb on Highly Theoretical as can be heard from the very beginning of the track, an omission that makes itself especially known in a synth-heavy section beginning around the two-and-a-half-minute mark that's utterly awash with reverb — as well as what appears to be a filter of some sort — on real hardware to ominous effect.

So, I've obviously made it clear that I don't consider Highly Theoretical a great choice for emulating AICA. But again, it's currently the only real choice for playing DSF files on. I find this problematic, both in general and in the context of composing homebrew AICA music especially; what's the point of doing so when there's no guarantee that it's even going to play correctly on the players that everyone uses? This doesn't necessarily mean that the DSF file format itself is at fault here or that it's completely useless — even if I would argue that those, and the xSF family of file formats themselves, are ripe for improvements, especially in the metadata department, but I digress — but without either a suitable update to Highly Theoretical or a worthy competitor to it, DSF is largely a dead-end format, I think. That leaves how to properly play sequenced Dreamcast music a bit of an open question... unless one can find a way to make Mednafen play DSF files, perhaps.

In any case, of course, you could just choose to play everything on real hardware instead. And which I'd argue you should do anyway, especially if you're a composer! Which leads nicely into...

Actually playing homebrew AICA music... on real hardware
On a conceptual level, this is very simple! Just create a sequence player that runs as homebrew on a Dreamcast, including on real hardware where music can be played there with 100 percent accuracy.

But then, of course, there are several complications that very quickly come up with that, namely actually acquiring Dreamcast hardware in the first place, and then figuring out how to get homebrew running on them. While the Dreamcast infamously never came close to enjoying the level of sheer market saturation as, say, the PlayStation 2, both appear to be fairly easy to find in working order on the used market nonetheless. However, also unlike the PS2, there isn't, to my knowledge, any easy way to run homebrew on the Dreamcast (à la the PS2's FreeMCBoot) without having to shell out a significant amount of money for an ODE (optical disc emulator). That's more of an accessibility problem than a technical problem, of course, but it is a problem nonetheless for what will likely be a large subset of people who simply cannot afford or justify the cost of an ODE. Thus, if there's any way to run relatively small programs on a Dreamcast another way, I'd be very happy to hear your suggestions!

Meanwhile, once you do find a way to get homebrew running — however that may be — a potentially big problem still remains, or at least if you care about sound quality there is. That problem being the noisy analog output of the Dreamcast, which I can personally attest to (it's not good). Compare that to the PS2, where getting perfect sound quality is an utter non-issue thanks to the fact that every single PS2 ever made has a built-in S/PDIF digital out port. Of course, the Dreamcast can be modded for S/PDIF out (or even HDMI out; I've personally used this with a S/PDIF splitter to create digital hardware recordings of Skies of Arcadia and Puyo Puyo DA! with excellent results), but said mods are neither easy to do nor cheap to have someone do for you. For most people, this is not a practical option. Unfortunately, that means that for most people noisy analog audio out is the only option. Which might make some pine for emulation again, but then the caveats of that road have already been outlined in detail.

In any case, I'd imagine that the sequence player itself would require the most actual work on the programming side of things. You could theoretically do lots of cool things with it: allow users to adjust or turn off DSP programs, create custom playlists from multiple sequences, mute individual instruments... the possibilities are endless! But of course, I'd imagine that just the basics would be difficult enough... maybe? Would we play sequences from the MSB/MPB files themselves with related driver files elsewhere or would the program basically be an DSF hardware player? Do either formats even allow for them to be run on real hardware? (I know that PSF does, as it was literally designed from the very beginning to do that, and 2SF does too, but other xSF formats — such as PSF2 especially with its many hacks to get things running sans the main EE CPU — are much more suspect, I think.) And if not, would perhaps a new xSF-like format need to be created, built from its very foundation to run on real hardware and specifically catered for homebrew music?

I think that I'll stop here for now — finally, haha — and see where any potential conversation goes from here. Again, thank you for reading and in advance for any questions that you can answer here for me!

[Cass]

Way above my pay grade but a great read. Good luck with your project. You might want to check out this Dreamcast discord server here - https://discord.com/invite/TRx94EV. Lots of very technically minded people on Dreamcast things.

[Ian Micheal]

First thing Acia is pretty rubbish not going to be doing anything with the dsp no one knows how to use it..

From Heliophobe (RIP) all this been confirmed with asm tests and benchmarks.

Code: Select all

So, we have 64 channels running at 44100khz. That means that 64 channel * 44100 samples = 2822400 channel calculations/second being calculated. But how many cycles does it take the AICA to calculate a channel? Well, we've got to figure they decided to use a 45mhz crystal for the sound system for a reason. Let's say it's 45,000,000 cycles a second (though it's probably not that exactly) so

45,000,000 / 2822400 = 15.94 =~ 16 (say, that's a nice round number.. in binary at least)

So let's say the AICA runs at 16 * 2822400 = 45158400 cycles/second, .

Okay, so now let's assume it's correct that the AICA takes 16 cycles to calculate a channel. For at least some of these cycles, the AICA will need to access memory. At the very least, it will need one of those 16 cycles to fetch the sample data.

However, it seems to take a lot more than that. For each channel there are 32 4 byte registers - but a number of them are unknown or unused (or it's unknown whether they're unused). I'll assume these are held in RAM and fetched by the AICA rather than being directly mapped to AICA registers, since they seem to be read/write (the ARM can write them at work or byte boundaries, implying a read/modify/write sequence) and there are so many of them.

Code: Select all

 suggest 11 of these have known functions, so the AICA is likely to fetch at least these 11 every cycle while doing it's calculations, possibly more. That means 12 or more of the 16 cycles may be taken up by AICA memory reads. Since the ARM7 doesn't have an instruction cache(or, at least the one in the GBA doesn't) it is effectively frozen while waiting for the sound chip to release the memory bus (as it must always have higher priority).

So, worse case scenario is that the ARM7 only gets 1 out of 16 cycles (in case the AICA ties up the bus for other things), so it functions at 2.8mhz, which is awfully close to my estimate. Maybe it gets two or three or more out of sixteen, increasing the available time in 2.8mhz increments.

Code: Select all

 ARM, but it is old stupid ARM7DI which always does 32bit mem reads, so one more /2 = ~2.8M

The great Rand Linden. below.. Whole thing is cluster F in real world use.

Code: Select all

Reminds me of old post from Rand about the ACIA   Rand Linden » Tue Feb 11, 2003 5:16 pm

Using AICA for sound is definitely possible ;-)

... but it's a significant undertaking, and classifying it as "complicated" is a major understatement.

There are some MAJOR restrictions on what can be done with the hardware, and in addition, AICA runs on the G1 bus, so any amount of data transfer beyond minimal is pretty much out.

Also, there's no cache, and as such, all instructions run from memory -- which is SLOW.... as in SLOW SLOW SLOW.

Oh, and it's not an ARM7TDMI -- there's no Thumb and there's no Multiplier logic... there *may* be the Debug logic, but I'm not sure.

There is an embedded DSP to do all sorts of awesome stuff, but it's an order of magnitude more complicated than the rest of the AICA -- basically, it makes the ARM side of stuff look like a cakewalk.

As for the touted "compression" of 8Megs from 2Megs -- If you convert all your samples to 4bit, that means *at most* you can fit 4Meg samples in the 2Megs of RAM.

If your samples started out at 16bit, you're still only storing 4bits of the original 16bits -- sure, you're storing more samples, but at a major cost in quality. Yes, ADPCM filtering smooths out a lot of the noise, but it still isn't close to the same quality.

ADPCM is great for some things, but as a "generic" compression, no -- think "majorly low-resolution JPEG" and you'll have a good idea of what it's like.

All that being said, a sound system is definitely feasible -- it all depends on how much time you want to spend on it... If you're looking for a research project that'll suck huge amount of time and have little progress to show for it, this is the one.

Rand.

Another thing Audio DMA bandwidth drops by around a third when the AICA is playing 16-bit PCM samples on all channels.

Should read or talk to TapamN https://dcemulation.org/phpBB/viewtopic.php?t=105823

I love it if someone did what your talking about sound system in practice is a lot worst then the specs. Also even the official kit the acia driver is full of bugs work around ..

Now I'm sorry i did not read your whole wall of txt as i'm not well but giving you the in practice we dont have any DSP driver and doing homebrew you cant use official docs not that they would help much anyways.

Here one of the best projects homebrew https://github.com/Aurelien34/DreamcastAicaSoundDriver
https://streamable.com/yilvf1 in action all running on the acia .. s3m and wav at the same time zero cpu main..

Hope this helps..

[PkR]

I'm not very familiar with the hardware side of things but I've looked at some of the formats so I'm going to respond to the parts I know about.

There are two "official" file formats for the Dreamcast handling sequenced music that I'm aware of.

They're all related. MLT, MPB, MSB etc. are all used in games developed with the Katana SDK. MLT is a container format that can store data to be loaded into the sound RAM along with a memory map that specifies the offsets and sizes of that data in sound memory. It doesn't always store all data since it can be loaded separately by the game, but it always has a memory map.

MLT can include several types of data: MIDI Program/Drum Bank (MPB/MDB) is like a soundfont, MIDI Sequence Bank (MSB) is like an archive of MIDI-like sequences (MSD), One-Shot Bank (OSB) is an archive of single sound effects (OSD), FX Program Bank (FPB) is an archive of compiled programs (FPD) to be executed on the sound chip for DSP effects, FX Output Bank (FOB) is used as a buffer for the DSP effect. They're all described in PowerPoint presentations in Dreamcast SDK so I suggest you have a look at those.

A few years ago I was trying to understand MPB and MSB formats a little better. I put together a crude tool to look at the data in MPB files, you can find the structs for MPB files here: https://github.com/X-Hax/sa_tools_resea ... or/Formats

Here's what the program looks like. The screenshot shows the data found in a single program/layer/split found in an MPB file. I think I was able to cover almost the whole MPB file, but there were several bytes I didn't understand, I think it was some kind of checksum. There are also at least two versions of MPB/MDB formats, I haven't looked into the differences between them.


Most of this was reversed using the sound tools in the Dreamcast SDK that work on Mac OS 8 and 9. Those tools require the Dreamcast Sound Box to produce actual sound, but conversion works without it. There are tools that let you create MPB banks, convert MID files to MSB banks, create OSB files and put all that together in an MLT file. Sadly the conversion is only one way, so you can't use the tools to open MLT or MPB files.

For Windows, the same SDK has the program "dls2tb" that converts DLS soundbanks to MPB. It has many restrictions, such as "Sega tone bank can have only 128 patches", but it's possible to convert smaller DLS files to MPB for testing purposes.

As for MSB/MSD, it's certainly very similar to SMF 0. SEGA's slides claim it does some optimization to the MIDI file when it's converted to MSB/MSD. Back when I was looking at this I didn't understand what they were doing with the "Note On" and "Note Off" MIDI commands. The Saturn document you linked may shed some light onto this.

In theory it should be possible to convert MPB to DLS or SF2 and MSB/MSD to MID files, and make it possible to play them as regular MIDI sequences with soundfonts on PC or elsewhere, but this requires knowledge on how audio processing works (I personally have no idea what most of those things mean in the screenshot), and how the SMF, DLS and/or SF2 formats are structured. This also wouldn't include DSP. For DSP, there's a Mac tool in the SDK that lets you put together various preset effects, link them to input and output modules, set their parameters and save an FPY file. I'm not sure if you can get an FPB file out of it without the required hardware. I don't know if replicating the DSP accurately is possible without emulation, but the preset effects seem simple enough so it's probably possible to recreate something similar.

On a conceptual level, this is very simple! Just create a sequence player that runs as homebrew on a Dreamcast, including on real hardware where music can be played there with 100 percent accuracy.

Technically you can already do that (if you are okay using the SDK). You can build MPB/MSB/MLT files with the above tools, then write a simple player to load the sound driver and play the sequences. There are code samples in the Katana SDK that you could use for it. Actually I made a primitive player for the Dreamcast to record sequenced sound effects in Sonic Adventure: https://gitlab.com/PiKeyAr/sa1-mlt-player

Hope this helps.