Retro dev kits. Is SGDK unique?

[shmupsrocks]

I'm interested in what makes certain retro platforms popular for developing new games. Genesis/MD has the SGDK dev kit which seems to play a big part in its success there. How unique is that project? Are there similar projects for other consoles?

Dreamcast seems to have a strong indie/homebrew scene although maybe it was stronger in the past?

[darcagn]

Dreamcast's homebrew scene is still very strong. I'm one of the developers/maintainers of KallistiOS, Dreamcast's homebrew kernel/devkit. We've had a tremendous amount of activity within the project in the past year. We're now on a completely up-to-date toolchain with upstream (GCC 13.2.0, Newlib 4.4.0, Binutils 2.41, GDB 14.1, etc.), we have the latest C23/C++23 support, we have a ports of Lua, MicroPython, mRuby, Tcl, etc., and we are making in-roads for supporting Rust, Objective-C, and D. We recently added a new implementation of OpenGL which was used for Driving Strikers, a commercial indie game with netplay that released last year (soccer with cars, similar to Rocket League). We have a very welcoming community and I am currently working on overhauling the project's documentation to make it even easier for beginners to familiarize themselves with KOS and Dreamcast development.

The three major points that I feel have made KallistiOS and Dreamcast development so successful:
- KallistiOS aims to create a familiar platform. You can expect standard POSIX functions and kernel services to exist. Well-written, portable C code tends to need very few adjustments to run on the Dreamcast.
- Dreamcast lives at the perfect intersection of the old school and new school. Developing for it still feels like retro game dev but it's not too restrictive.
- Dreamcast stock consoles can run homebrew with no modifications needed. Just buy a working console and you're good to go. This not only reduces the barrier to entry for users, but for developers makes creating commercially sold titles pressed to CD-ROM much more appealing than on other systems.

[SavagePencil]

There are dozens of them, at least one for each system. I would argue that SGDK is one of the more popular ones because:

1. C is more accessible than ASM
2. The Genesis has enough horsepower that you can overcome the inefficiencies that come with C vs. raw assembly
3. There is enough of an ecosystem of tools (importers, debuggers, feature-rich emulators) that make development easier
4. The Genesis was a pretty popular system

[Sumez]

2. The Genesis has enough horsepower that you can overcome the inefficiencies that come with C vs. raw assembly

That's not really true, just opting out of assembly entirely would always be a bottleneck for your game. Fine for really simple games, but not very good if you want to push the hardware. Especially with things like heavily used routines for repeatedly calling hardware registers, C would just create a ton of unneeded overhead. Of course that's exactly the sort of thing SGDK helps with (I assume).

What is true however is that the 68000 CPU has a structure that lends itself easier to the way C compilers does things, regarding stack usages, indexed addressing for pointers and structs and so on, compared to other CPU families at the time. It has nothing to do with "horsepower" - if that were actually the case, not using assembly would always be a massive disadvantage. That said, 68k assembly is so basic and straightforward, I think anyone owes themselves to take the 30 minutes it demands to learn it if they want to make something for the MegaDrive.

Anyway, to answer the thread question, there's a Game Boy devkit out there which has been extremely popular lately.

[Guspaz]

I've been doing a bunch of work with the .NET nanoFramework recently (the whole RT4K wifi card thing), and it's got a pretty similar limitation: because the .NET IL is interpreted, running managed (C#) code is very slow. As a result, the framework itself is written in C++ and compiles to native machine code for the target platform (like ESP32) and your C# code acts as a thin orchestrating layer on top so that as much execution time is spent in native code as possible. I'd imagine a lot of these retro SDKs are similar, where they're optimizing the provided libraries so that as much of the performance-intensive stuff as possible can be done in code written in assembly.

I know that for the modern Game Boy devkit, one of the major limitations when I was still active in that community was the low quality of the available compiler, which made very suboptimal decisions. Maybe that's improved since then.

[SavagePencil]

2. The Genesis has enough horsepower that you can overcome the inefficiencies that come with C vs. raw assembly

That's not really true, just opting out of assembly entirely would always be a bottleneck for your game. Fine for really simple games, but not very good if you want to push the hardware. Especially with things like heavily used routines for repeatedly calling hardware registers, C would just create a ton of unneeded overhead. Of course that's exactly the sort of thing SGDK helps with (I assume).

What is true however is that the 68000 CPU has a structure that lends itself easier to the way C compilers does things, regarding stack usages, indexed addressing for pointers and structs and so on, compared to other CPU families at the time. It has nothing to do with "horsepower" - if that were actually the case, not using assembly would always be a massive disadvantage. That said, 68k assembly is so basic and straightforward, I think anyone owes themselves to take the 30 minutes it demands to learn it if they want to make something for the MegaDrive.

Anyway, to answer the thread question, there's a Game Boy devkit out there which has been extremely popular lately.

Don't get me wrong, I wasn't trying to whitewash, but I definitely could have been more specific. My argument is that the overhead of C more severely impacts 8-bit development (for example, my experience with the SMS and Lynx games written with C are way less performant because of it). Those systems benefit the most from ASM. Back to the argument, you can get started and get something onscreen quickly and usable using SGDK.

[orange808]

You might just drop an ARM in a custom cart design and handle the logic there with C++. Probably easiest to spam the hardware with nops when it's convenient. You're just polling inputs and feeding the audio/video hardware. That will also make the cart difficult to dump and emulate, because the game's memory space would be mostly invisible to the game console. Of course, there is still a limited market for retro cart releases, so it's a hobby thing.

[drncurry]

Be a pretty clever idea to get "more power" out of it, but I'm sure you'd get a lot of push-back from the community that it isn't a "real" retro game. Never-the-less, its a good idea for security. There are modern ARM processors that have encrypted ROM to prevent dumping. If the ARM's rom was big enough you could probably do everything in the ARM itself and have no additional memory on board. Would make your retro game pretty secure.

[Sumez]

but I'm sure you'd get a lot of push-back from the community that it isn't a "real" retro game.

Well... it isn't. I mean there aren't any rules about what you can or can't do, but why release a cartridge for, say, megadrive if your goal wasn't to actually develop it for the Megadrive, as opposed to some arm cpu you just dropped in there?

[SavagePencil]

I'm pretty sure that's the approach some of the Atari homebrews are taking, such as with this Mappy release: https://www.youtube.com/watch?v=w-8xNE_Knx8

[Lander]

I've seen the Scorpion Engine pop up in Amiga homebrew stuff recently, such as the in-progress fan port of Rastan.
Appeal-wise, I attribute most of it to retro cool. Perhaps nostalgia, perhaps a younger generation discovering neat old stuff and taking an interest now game development is an accessible pastime.

Though by and large, I don't really see it myself. The accessibility angle, yes, but surely the appeal of developing for an old platform is having to learn an interesting and restricted architecture.
Like those Zachtronics coding games that invent an esoteric system and give you a full-length technical manual to print out and ringbind, or trying to figure out a specific chip whose spec and instructions primarily exist in Japanese.
Why bother if it's all simplified and standardized to the point where you might as well be programming for a modern PC?

I suppose it makes more sense for later machines, since architectures gradually become more standard and less interesting as time goes on.

You might just drop an ARM in a custom cart design and handle the logic there with C++.

Ah, like Hellraiser for NES!

[orange808]

Ah, like Hellraiser for NES!

Yes. Even better actually.

With community support, you could build out a "development kit" around C++, Tiled, and other standardized tools. You'd get better results than legacy software with modern workflow and some protection from the community's insatiable desire for piracy. (Because anything that runs on an "old" system is automatically assumed to be free).

Although, once again, you'd need some kind ecosystem that encouraged devs to participate. And, of course, old people like me that know dead assembly languages would howl that it's not real. (Well, not me. I don't care about cart enhancements.) Emu devs would piss and moan about getting locked out; that's already happened. And, rom beggers would lose their minds ranting that they can't play the new games for free, because of preservation or poverty.

And, best of all, I can say it all out loud to my heart's content where I'm writing this right now.

[drncurry]

Ah, like Hellraiser for NES!

Yes. Even better actually.

With community support, you could build out a "development kit" around C++, Tiled, and other standardized tools. You'd get better results than legacy software with modern workflow and some protection from the community's insatiable desire for piracy. (Because anything that runs on an "old" system is automatically assumed to be free).

Although, once again, you'd need some kind ecosystem that encouraged devs to participate. And, of course, old people like me that know dead assembly languages would howl that it's not real. (Well, not me. I don't care about cart enhancements.) Emu devs would piss and moan about getting locked out; that's already happened. And, rom beggers would lose their minds ranting that they can't play the new games for free, because of preservation or poverty.

And, best of all, I can say it all out loud to my heart's content where I'm writing this right now.

Couldn't agree more, but at what point do you just consider making a new system if you're only using the genesis for inputs and rendering? Its also not a particularly good renderer either with only 64 colors.

I think to make it more retro would be to use the modern ARM for security; e.g. encrypted roms with on the fly decryption and dump detection. Then maybe use some of its components it for compression, memory expansion, or floating point calculations. This would preserve the retro game and only complement it.

Disadvantage is cost. The ARM alone would be $5 to $10 and just by my research wouldn't have enough ROM for both the game code and the ARM code. So additional ROM would be required. The PCB, ROM, uC, and case would probably be close to $30. Let alone profit for the programmer and "publisher". Publisher referring to whoever would manufacture these new carts.

The biggest technology gain in the last 10+ years has been compilers have gotten a lot smarter at turning code into faster assembly. I'm sure this is what SGDK is taking advantage of, but I have no technical knowledge of its inner workings. I believe this is where the Super Mario 64 gained a lot of FPS when optimized on modern compilers. It wasn't that the developers forgot to set a flag, its the optimization was unreliable back then.

[orange808]

Couldn't agree more, but at what point do you just consider making a new system if you're only using the genesis for inputs and rendering? Its also not a particularly good renderer either with only 64 colors.

I don't care about semantics and politics. In my opinion, that's something for geeks to do while they read their comic books. I am a gamer and I've always been obsessed with computers, but I never really enjoyed pocket protectors or console wars that much.

The Genesis has a unique look and feel. From the FM sound to the color pallete, it's distinctive. It's also inherently built for fast paced low latency 2d sprite games.

I think to make it more retro

I hate that word so much.

...use the modern ARM for security; e.g. encrypted roms with on the fly decryption and dump detection. Then maybe use some of its components it for compression, memory expansion, or floating point calculations. This would preserve the retro game and only complement it.

Why? You're not enhancing the system or development environment. So, you're honestly expecting homebrew devs to match or beat legacy developers in their spare time for no money? You think discouraging new devs is a good idea, because a young kid (that doesn't need assembler) didn't learn it? I don't use assembler for anything profitable at all. It's just filling space in my little head. You want to make development more difficult, because you like the way that sounds?

Are you even listening to yourselves? That's ridiculous. It reminds of the way girls dress up for each other. If he likes you, he'll like you with your hair pinned up (with a pencil) in sweatpants. All of this "is it the real thing?" stuff feels like geeky guys dressing up for each other.

Furthermore, how does this encryption work? The STM32 already has measures to protect your code--and all the instructions would still be fed to the machine unencrypted. Why bother?

Disadvantage is cost.

Obviously. But, people are already paying quite a bit for carts and it's a small batch hobby release.

The ARM alone would be $5 to $10 and just by my research wouldn't have enough ROM for both the game code and the ARM code. So additional ROM would be required. The PCB, ROM, uC, and case would probably be close to $30. Let alone profit for the programmer and "publisher". Publisher referring to whoever would manufacture these new carts.

You would use something like the Unocart.
https://github.com/bndtr0/stm32-UnoCart

That uses an STM32**** board. That's probably the best option.

The biggest technology gain in the last 10+ years has been compilers have gotten a lot smarter at turning code into faster assembly. I'm sure this is what SGDK is taking advantage of, but I have no technical knowledge of its inner workings. I believe this is where the Super Mario 64 gained a lot of FPS when optimized on modern compilers. It wasn't that the developers forgot to set a flag, its the optimization was unreliable back then.

Super Matio 64 gained FPS because it wasn't originally complied using all the possible optimization flags. Those optimizations came online shortly after that game's release. It was also natively written in a C dialect--not assembler

I absolutely guarantee that the N64 would technically be capable of more if people went through the headache of programming it directly on the metal.

[Lander]

It'd be good to see the devkit side of things move beyond C-like platforms. If the goal is easier and more productive, something that can typecheck ahead of time and ensure code isn't going to do anything weird or unsafe would be nice.
I'm sure the Rust scene is already hard at work rewriting, as is its mien, but where are the ML backends at?

...On some level I can't believe I just typed that, since the weird and unpredictable bugs that arise from programming fast-and-loose near the metal are (well, can be) more interesting than purely algorithmic ones.
But the academic in me says hell no fuck that - just design the bugs in so they're logically sound

I dunno, I feel like you might as well build a fantasy console (software, hardware, or both) if going down the custom route, though inventing a whole platform is a lot bigger in scope than piggybacking someone else's.
Perhaps a middle point similar to a Super Game Boy, where the ARM cart acts as a sub-platform bridging Mega Drive I/O to a modern CPU, and itself takes smaller ROM carts containing the games that use it.
That'd get you some framework for an ecosystem, as well as cutting the overall expense of shipping multiple games.

Still seems like a lot of work when you could just write code for a widely-implementable restricted VM, though

[orange808]

Sure. Enhancement chips piggy back by definition. They always did.

I don't particularly like fantasy consoles, because they create an awful chicken and egg dilemma. How do you get an install base without games? How do you get games without an install base?

It's still most efficient to build out your own implementation in software, target a viable existing platform, and simulate some elements of older consoles. That way, you can target an active install base, sell games directly as downloads in popular stores, and try to make some profit for all your hard work.

[drncurry]

I don't care about semantics and politics. In my opinion, that's something for geeks to do while they read their comic books. I am a gamer and I've always been obsessed with computers, but I never really enjoyed pocket protectors or console wars that much.

The Genesis has a unique look and feel. From the FM sound to the color pallete, it's distinctive. It's also inherently built for fast paced low latency 2d sprite games.

Wasn't intended to be a political question. It was more of a why bother with the complexity of making it genesis compatible when it can just be its own thing. Which would greatly simplify the implementation. I would be on board if it was to makeup for some things the genesis lacks and expand upon SGDK. Like adding sprite rotation abilities that can be done with 1 line of code compared to doing all the scan line tricks with the genesis. But if your planning on bypassing the 68k all-together. Then its just a completely different system at that point. Maybe call it a 32X.

I hate that word so much.

R E T R O

Why? You're not enhancing the system or development environment. So, you're honestly expecting homebrew devs to match or beat legacy developers in their spare time for no money? You think discouraging new devs is a good idea, because a young kid (that doesn't need assembler) didn't learn it? I don't use assembler for anything profitable at all. It's just filling space in my little head. You want to make development more difficult, because you like the way that sounds?

When did I say any of that? The advantage is to provide security. I personally wouldn't use my time to make a Genesis game because of exactly what you mentioned earlier. That anyone with a eprom reader can steal your game and dump it on a rom site. SGDK already allows for good looking games in C.

Furthermore, how does this encryption work? The STM32 already has measures to protect your code--and all the instructions would still be fed to the machine unencrypted. Why bother?

Some uCs, not all, have internal flash protection/encryption. This protect the uC's code from hacking/cloning.

The game rom would be stored in an encrypted form off the uC due to size. Only unencrypted chunks will exist inside the uC itself. To retrieve the unencrypted information would require sending requests to the uC. At which points you can lay traps to prevent dumping of the entire unencrypted data. I can think of settings a bunch of addresses in the ROM to traps that would cause the uC to start feeding random data out. Reading about the stm32 uno cart, I start to wonder if this is technically feasible as it may be too slow to feed the data bus and require the game to be programmed for bank switching since the uCs memory is too small for the entire game.

The only attack I can think of at that point is playing the game and sniffing the data as you play. While a dump it still possible it would take hours to accomplish.

I'm open to ideas.

Possibly use the uC like the NES CIC for another layer of security. I'm not entirely sure how this would work without a matching chip to handshake with in the Genesis

Obviously. But, people are already paying quite a bit for carts and it's a small batch hobby release.

Typical business model is to make 20%+ on every dollars you spend, when sold in quantity. Adding $10 to something that costs $50-60 is HUGE additional cost especially in small quantity.

You would use something like the Unocart.
https://github.com/bndtr0/stm32-UnoCart

The one they use is the STM32F407VGT6. No security. That particular design (flash cart) wouldn't work as is for a Genesis even if security wasn't the goal. The programmable memory size is 1MB. Some of the early Genesis games may fit, but the later ones are well over 1MB. You'd have to design the game with bank switching in mind. Most of the rom carts I see on the Genesis have a dedicated eeprom that gets programmed.

Super Matio 64 gained FPS because it wasn't originally complied using all the possible optimization flags. Those optimizations came online shortly after that game's release. It was also natively written in a C dialect--not assembler

That's pretty much what I said.

I absolutely guarantee that the N64 would technically be capable of more if people went through the headache of programming it directly on the metal.

I doubt that. N64 suffers from architecture problems with no easy solution. ROM access is slow and the vram is too small to store good textures.

[orange808]

The STM32 doesn't have read and write protection from the manufacturer?

LMAO! Okay... Congratulations on embarrassing yourself.

You couldn't even produce those STM32 boards for commercial purposes without providing some protection for the software, because customers would fear their IP is going to be stolen and reverse engineered.

Here's the relevant documentation on the feature, by the way..
https://stm32world.com/wiki/STM32_Reado ... tion_(RDP)

Have a good one..

STM makes multiple variations of their board with more resources, by the way. I have a more powerful version in my hand.

Also, the details of the hardware don't erase the additional layer the compiler adds between a high level language and pure assembler. It doesn't matter if the hardware is complicated or not. That's nonsense. Working on the metal is theoretically faster. Your explanation makes no sense. ¯⁠\⁠_⁠(⁠ツ⁠)⁠_⁠/⁠¯

[drncurry]

The STM32 doesn't have read and write protection from the manufacturer?

LMAO! Okay... Congratulations on embarrassing yourself.

You couldn't even produce those STM32 boards for commercial purposes without providing some protection for the software, because customers would fear their IP is going to be stolen and reverse engineered.

Here's the relevant documentation on the feature, by the way..
https://stm32world.com/wiki/STM32_Reado ... tion_(RDP)

It doesn't have hardware encryption. The article you linked even has links to why encryption is important for protecting code and not just setting a please don't read bit internally.

Also, the details of the hardware don't erase the additional layer the compiler adds between a high level language and pure assembler. It doesn't matter if the hardware is complicated or not. That's nonsense. Working on the metal is theoretically faster. Your explanation makes no sense. ¯⁠\⁠_⁠(⁠ツ⁠)⁠_⁠/⁠¯

Your point is moot. Maybe they'd get more frames per second like what we saw with Super Mario 64, but the N64 being more capable (I assumed you meant graphically?) is impossible because of the technical limitations of the hardware. You can't right assembly instructions to overcome the latency between the ROM and the CPU.

[orange808]

More frames per second *is* better performance. Fuck.

Here's an idea! Let's use Java and a VM. I mean, why not? Right? The hardware doesn't actually work any faster or slower? R-iiiiight? The memory is still the same speed! R-iiiight?

I didn't say encryption. And, your encryption idea is pointless. If someone is determined enough to start directly hacking at the hardware to dump the cart, they're also going to crack the encryption.

You already have a secondary protection from piracy: the cart is not a standard design and emu devs will struggle to simulate the STM hardware.

I'm done. Have a good day.

[Lander]

When it comes down to it, the hardware ceiling is the hardware ceiling. It's non-negotiable whether you're living at the top and special-casing your code to the arch, or subsisting somewhere in the middle by way of a machine translation from a different computational model - one which may or may not happen to hit various optimization paths depending on the shape of your code, the target platform, build config, and so on.

Though strictly speaking, a sufficiently novel special-cased compiler would be able to overcome the latter issue and produce something as good as hand-coded assembly. The tricky part is inventing an intermediate represesntation that doesn't couple too tightly at either end, and translation smart enough to wrangle it into optimal code for a given target.

Unfortunately that's a niche within a niche, since the hundreds of thousands of man hours necessary to turn cutting-edge whitepapers into working machines are all getting spent on the popular languages that do couple (relatively) tightly to popular architectures. Both of which are buried so deep in complexity and technical debt that it's a wonder modern machines even work half the time!

Hence my wish for a greater focus on ML; simplify the input language enough - rip out all of those inconvenient practicalities like memory management, so the human can think about calculus instead of computational logic - and you eventually reach a point where the compiler can shoulder everything we're bad at, rather than part of it with a big list of caveats.

And all that without reaching for horrible half-step solutions like runtimes and garbage collection. If only.
(Or having to recite Og én til javanissen! to please the little gnome that lives inside the JVM and causes segmentations faults if angered )

[Sumez]

Couldn't agree more, but at what point do you just consider making a new system if you're only using the genesis for inputs and rendering? Its also not a particularly good renderer either with only 64 colors.

I don't care about semantics and politics. In my opinion, that's something for geeks to do while they read their comic books. I am a gamer and I've always been obsessed with computers, but I never really enjoyed pocket protectors or console wars that much.

It's not about "semantics" or "politics" though.
Like I said in a previous point - people can do what they want, there's no right or wrong here. I just don't really understand what the point is?
We aren't talking about using an enhanced cartridge to power up a MegaDrive game, to do things it otherwise wouldn't be able to - we're talking about just running the entire game on the cartridge instead of the MegaDrive.

If the only thing you want from a MegaDrive is the color limitation, it's a really roundabout way to do that. Like Lander says, it makes a lot more sense to create your own fantasy console, or hell just put a video output directly in your "cartridge" and ship it as its own plug-n-play thing.

[drncurry]

More frames per second *is* better performance.

Apologies, you said capable not performance. I assumed you meant graphically since there is usually an argument which console from that generation makes better graphics; PS1, SAT, N64. Just like I said protection instead of encryption. Mistakes happen.

I didn't say encryption. And, your encryption idea is pointless. If someone is determined enough to start directly hacking at the hardware to dump the cart, they're also going to crack the encryption.

You already have a secondary protection from piracy: the cart is not a standard design and emu devs will struggle to simulate the STM hardware.

Agreed, thanks for the feedback. That would unfortunately have the same issue, someone determined enough. Its security through obscurity. If the platform is popular enough, it will eventually get emulated similar to the SVP.

When it comes down to it, the hardware ceiling is the hardware ceiling. It's non-negotiable whether you're living at the top and special-casing your code to the arch, or subsisting somewhere in the middle by way of a machine translation from a different computational model - one which may or may not happen to hit various optimization paths depending on the shape of your code, the target platform, build config, and so on.

I think the idea has some potential. Offering something as simple as sprite rotation that the compiler knows how to handle which CPU to send it to and how to update the genesis RAM automatically would offload a huge amount of work and make games look drastically better. It would definitely be an add-on. The STM-32X(tm).

I would argue on complex code, the compiler could offer better performance. I've worked on projects that were 100,000 lines of code and its hard to remember what all everything does. I can't imagine making something like that in ASM and keeping track of everything AND writing it so its 100% optimized.

Hence my wish for a greater focus on ML; simplify the input language enough - rip out all of those inconvenient practicalities like memory management, so the human can think about calculus instead of computational logic - and you eventually reach a point where the compiler can shoulder everything we're bad at, rather than part of it with a big list of caveats.

C# and Java require garbage collectors that run at runtime for this to work. Does RUST allow for unmanaged memory and doesn't use a runtime garbage collector?

I like the idea, but stuff like C# is both a god-send and super annoying. The unmanaged memory is great, but the handicaps on pointers and mem copy is very annoying. I'm guessing the purpose of ML is to allow pointers, but then somehow find every path and confirm that it will never be invalid before runtime? That's a hell of a feat.

[orange808]

here. I just don't really understand what the point is?
We aren't talking about using an enhanced cartridge to power up a MegaDrive game, to do things it otherwise wouldn't be able to - we're talking about just running the entire game on the cartridge instead of the MegaDrive

If you dig in and explore how an enhanced cart would work in concert with the system, you'll find that's not really true. All you really have is some game logic running in a separate memory space, but that would work in concert with the console.

The console and it's processors would be running in parallel. I suppose you could bypass the Genesis/MD audio completely, but you really shouldn't. I think the Genesis/MD provides pins to send audio passthrough. The cart won't see the console memory space and vice versa. Of course, almost everything that actually gets done happens on the console side. You make it sound like I hooked a Raspberry Pi up inside a Genesis shell. That's a nasty smear. It's a hit job.

You make it sound like the cart would plow directly to the audio and video outs. That's not how it would work. And, everything the system does is going to be stored in the console's ram and registers. All of that has to be fetched and updated every frame.

[drncurry]

[Lander]

C# and Java require garbage collectors that run at runtime for this to work. Does RUST allow for unmanaged memory and doesn't use a runtime garbage collector?

Yep! It's done statically, using a combination of move semantics and data lifetime analysis to automate allocation / freeing, prevent pointers from outliving what they reference (or having their pointed data modified unexpectedly), and so on. Violating memory safety is a compile error, and writing abstractions that can't be used wrong doesn't add any overhead - it's great.

The downside is longer compile times, and discovering that many algorithms you already know are memory-unsafe and need to be replaced, but I'll take that over segfaults and footguns any day!

I like the idea, but stuff like C# is both a god-send and super annoying. The unmanaged memory is great, but the handicaps on pointers and mem copy is very annoying. I'm guessing the purpose of ML is to allow pointers, but then somehow find every path and confirm that it will never be invalid before runtime? That's a hell of a feat.

ML is a pretty general acronym; literally stands for Markup Language, so technically covers things like HTML, but in this case refers to computational MLs like Haskell and OCaml that put a hard wall between the program logic, and how said logic is actually processed.

Traditionally, that's achieved by outright refusing to model machine ideas like mutable memory, pointers, and jumps; everything has to be done using recursion and high-minded theory concepts (Func-what? ) instead. But in exchange, the structure of the code is so simple that the compiler is at liberty to case-fit architectual optimizations like allocation / looping / vectorization without first having to unpick a model that has its own ideas of what those things are.

(Personally I quite like it that way, since the theory stuff is the strongest foundation you can get. Boil any language down enough and you'll find Functors, Monoids and Monads in its bones, even if they're not explicitly called out as such. Might as well cut out the middleman and go straight to the centuries-strong academic source, though it's a lot to learn.)

Things like Rust are a middle point, where the logician desire for machines to be a mere implementation detail meets the desire for a practicality and approachability - looks like C*, feels much smarter.

If it's of interest, the underpinning idea there is called effect typing - essentially tagging function outputs with some metadata that says "I only exist between these points in the program", "I perform I/O", "I might diverge into an endless loop", "I might halt with an exception", etc, and propagating it all the way back up to the entrypoint so the compiler can reason about it ahead of time.

Looping back around to a Mega Drive development example, you might model rendering using a specialized effect type that describes in general terms what should be drawn, and is handed off to the compiler for conversion into an set of concrete sprite ops. Provided that the effect data is specific enough, it should be able to generate truly optimal code, and even throw static compiler errors for things like exceeding the sprite limit or missing the VBLANK window.

Clever stuff; the full scope of it will take a while to hit mainstream, but the outlook for programmers is pretty sunny in the long term.

[orange808]

See how I didn't say encryption?

See how I have asterisks after I mentioned the board? Those are commonly accepted to be wildcards. That's before you cracked back with some bullshit about a specific model.

See how I mentioned no ops when convenient? See the wording?

See how none of my posts are edited?


Moving the goalposts indeed.

The ARM side has to be able to send assembler instructions to the CPU to make it work. You get that it's a cart, right? The ARM races the console CPU and feeds assembler instructions to the machine. Probably have to juggle the program counter while it's done. You write libraries to minimize the need for devs to handcode those bits.

The ARM is only attached via cart slot. I made it clear it would run in a separate memory space. Notice the wording? See it? How in the holy heck did you expect to get things done without using the console's processors? The console won't be able to see the ARM's ram. The "shared" space is rom to the console, but we're going to store all that as const anyway, aren't we?

The console gets assembler instructions from the ARM and accesses rom the way you would expect. The difference is, the ARM is faster, so it can run C++ code with time left over to do other things. At some times, you may choose to idle the 68000 for a moment and simply run a block of code on the ARM. The console doesn't need to know about a lot of logic code. That's the theory behind it, it allows C++ logic to run in a less restricted space and you can hide a lot of the assembly from the dev. Depending on the speed of the ARM, you could do a lot more, but you don't have to. You could also use hardware tricks that would normally take too much CPU time (depending on the target console). Because, of course, the ARM is taking over a lot of logic.

...

[drncurry]

Yep! It's done statically, using a combination of move semantics and data lifetime analysis to automate allocation / freeing, prevent pointers from outliving what they reference (or having their pointed data modified unexpectedly), and so on. Violating memory safety is a compile error, and writing abstractions that can't be used wrong doesn't add any overhead - it's great.

The downside is longer compile times, and discovering that many algorithms you already know are memory-unsafe and need to be replaced, but I'll take that over segfaults and footguns any day!

That's very interesting. I've heard a lot about RUST in college years ago, but never bothered looking into it. Crazy thought, if this is all compiler technology, this could be added to C, but instead of handling the memory for the programmer it could be used to catch faults and instruct them on how to fix them. Or... be like Visual Studio and suggest something nonsensical...

I only says that because I was thinking best of both worlds faster compile times, but the programmer still needs to be somewhat responsible, but I think to argue with myself no matter what the compiler would have to examine those paths whether they're correct or not. Could keep faster compile times and run the analysis stuff like a Jenkins pipeline since its assumed the C code will compile, but checking for memory errors is a separate process.

ML is a pretty general acronym;

I thought you were talking about machine learning...

it's a lot to learn.)

Absolutely, thanks for sharing. I've got a lot to digest.

Looping back around to a Mega Drive development example, you might model rendering using a specialized effect type that describes in general terms what should be drawn, and is handed off to the compiler for conversion into an set of concrete sprite ops. Provided that the effect data is specific enough, it should be able to generate truly optimal code, and even throw static compiler errors for things like exceeding the sprite limit or missing the VBLANK window.

Glad you related it back to the genesis, because that's the realization that I came to last night. The issue with the genesis isn't efficient assembly. Its good looking games are good looking because they're manipulating the hardware. I'm struggling to see how this could be brought up to a higher level. They could be turned into a template, like you're suggesting, but I think you may end up with a bunch of games that do the same effect.

[Lander]

That's very interesting. I've heard a lot about RUST in college years ago, but never bothered looking into it. Crazy thought, if this is all compiler technology, this could be added to C, but instead of handling the memory for the programmer it could be used to catch faults and instruct them on how to fix them. Or... be like Visual Studio and suggest something nonsensical...

I only says that because I was thinking best of both worlds faster compile times, but the programmer still needs to be somewhat responsible, but I think to argue with myself no matter what the compiler would have to examine those paths whether they're correct or not. Could keep faster compile times and run the analysis stuff like a Jenkins pipeline since its assumed the C code will compile, but checking for memory errors is a separate process.

Same; it popped up in my sphere a few times and I largely passed it off as new and trendy. Then one day a work project comes along that has one Rust file in the whole codebase - a tiny Discord integration iirc, not even part of my assignment - and one proper peek was all it took to drag me all the way down the rabbit hole.

And yeah, the compile times mainly happen in static analysis, which tends to be a linear process; i.e. you need to know all the extra info for A in order to derive it for f and B in B = f(A).
Tractable with clever multithreading, but more or less a necessity for a lang that needs to be proven correct at compile time.
Though there's some precedent for doing it decoupled as you suggest; TypeScript is more or less that for JavaScript, adding a full Hindley-Milner typechecker that iirc doesn't stop you from generating a runnable .js file if it fails.

I thought you were talking about machine learning...

Ye gods, definitely not! That's the whole other end of the scale; akin to giving up on understanding the fundament, and hoping the weird AI brain can correctly guess its way to a categorically sound solution. I'll have to watch my acronyms!

Anecdotally, I tried asking ChatGPT to show me some of those theory constructs implemented in Haskell and Rust once; it felt like being a teacher trying to wring the correct answer out of a kid who's just guessing because they've been put on the spot and don't actually understand.
Random program terms being added or dropped with each answer, two things rebroken for each I taught it to fix. I pity the poor fools who become the DevOps of that field; compilers and category theory are harsh mistresses, but at least they don't have capacity for stupid

Glad you related it back to the genesis, because that's the realization that I came to last night. The issue with the genesis isn't efficient assembly. Its good looking games are good looking because they're manipulating the hardware. I'm struggling to see how this could be brought up to a higher level. They could be turned into a template, like you're suggesting, but I think you may end up with a bunch of games that do the same effect.

That all comes down to layers of abstraction i.e. how general are the terms in which we describe what to draw?

At the lowest level, it's just ASM blitter opcodes. Naturally the point is to avoid exposing those directly (like writing inline assembly in C), so we lift them into a list of human-readable instructions that correspond with their description in the manufacturer spec - i.e. a name -> int enum that gets stored in an array, likely with some extra data about arguments and relevant registers.
That's a start, and could be exposed directly as an effect type that gets built in-language and returned, but is still far too bare-metal for the lofty goal of 'just draw this'.

So from there, we look at how those instructions are intended to be used; i.e. what sequences of them are legal within the spec, and turn those sequences into coarser commands. This results in another effect type that stores the previous one - an enum that relates names to the vectors of legal instruction sequences, thus continuing to build spec semantics into the structure of the data.
That's another small step toward the goal, so you then you look at the most granular way those legal sequences might be used, and repeat the process, yielding another semantically-strong layer of abstraction that composes the previous one (and probably some supporting datatypes for things like tiles, palettes, etc.).

Keep doing that, and eventually you arrive at the mentioned template-y level of abstraction; really high-level operations that deal in new vagueries like 'scenes' and 'images' which become harder to categorize (is an image a grid of colors? a set of vector primitives? a function over an analytical color field?) and tend to introduce monolithic building blocks like draw_mickey_mania_moose_chase_effect(render_state, tile_map, curvature) that become an ultimately-overused escape hatch.

The key is in composition; high-level toolkits tend to be built top-down, or from somewhere in the middle outward, rather than by exhaustively enumerating the problem space from the bottom up; the goal is a generally usable framework, so implementation is based on an opinionated conjunction of technical need and user ergonomics.

But, if you've taken the fine-grained approach and abstracted each layer that lies between machine code and creator vision as rigorously as possible, you can freely traverse back down those layers, and use the most appropriate one to model novel phenomena like DMA transfer abuse. That in turn gives rise to new tools in layers between it and the topmost "draw the thing" one, and eventually creates a truly comprehensive framework where every piece is as minimal and reusable as possible.
And since each one is a semantically-strong effect type rather than an opaque instruction, the compiler has access to a mountain of context that can drive optimizations at any layer, so can start reasoning in more human terms like "I can abuse XYZ manufacturer-undocumented hack to fit this memory transfer in between these other two semantically-specific ops to fit within per-frame cycle budget".

Naturally, that's all theory that would require an obsessed domain expert and a really nifty language to implement, but it could be done!

[orange808]

I don't know what could be generalized into a strongly typed prison language, but what hardware tricks are you referring to? Using the VDP to copy into memory? That's not really a big trick.

Demo scene stuff doesn't appear in games because it usually eats up too many resources or exploits an edge case curiosity of the hardware.

Good looking games look good because the devs and artists made them look good. In most cases, that's a direct result of hand designed excellent sprite work, clever choices of pallete/color, and hand tuned animations. Full or partial sprite flicker aren't that difficult to do, either.

I can't think of any cases of extreme "racing the beam" raster effects that manipulate individual sprites. Are there any? Can you even manipulate the memory outside of HBLANK--and if you did, would you get consistent results across all the hardware variants? I doubt it.

And, any useful language for programming the Mega Drive/Genesis would need inline assembly options.