I’ve decided this year to let go of my original blog domain (zenithsal.com) and move the blog to my CMG domain: https://zenithsal.cloudmillgames.com.

I’ve been monitoring visits to the original domain and I was surprised to see so many visitors on a daily bases. My guess is it has something to do with the cheatsheets for tic80 and typescript that I’ve published on this blog a while back. It’ll probably take some time for search engines to point visitors to the new URL, but that’ll be fine as both cheatsheets have been “mirrored” in many other places online.

The reason I moved to CMG is because this domain is most likely to remain alive for as long as my life span, unless we are lucky enough to win an asteroid impact, a nuclear war, or a paperclip maximizer.

Little by little the internet is dying. I felt like ranting on a private discord server which led me into a rabbit hole after. This was my rant raw:

man the internet really sucks nowadays, most forums are gone not even in archive, search engines are crippled with ads and target dumb audiences, every piece of software that’s not opensource is spying on us “to enhance your experience”, pretty much every social discussion network wants to be a walled garden like discord, reddit is almost there.. it seems most people have migrated from internet to social media focused on doom scrolling and pointless angry arguing. It feels like LLMs are the only good thing that happened recently.. but wait til they’re enshitified as well and start to cripple themselves (to enhance experience naturally), spy on us and push ads which will happen once their growth slows down

I think we broke the internet, so many computer people started looking for an escape.. for some they found it in pre-internet retro hardware some are trying to revive the old internet by reviving personal sites and focusing on decentralized services only (I think I’m in both camps)

a part of me just wants to boot all major software from all my computers and switch to FreeBSD or something and start over 😂

These sentiments were all brewing in my consciousness for a while now. I’m glad I was able to just dump the whole thing in one paragraph.

I want to break it down here:

Forums are gone

One day not long ago the internet was full of self-hosted forums. At the beginning of my computing and gamedev journey this was one of the main sources of expanding my knowledge and asking when I’m facing problems. Only a few of these still exist like Gamedev.net, and their activity is a pale shadow of what it was. Forums like these had the following characteristics:

• completely open to search engine indexing and every thread and reply had a unique clean URL (no tracking or fingerprinting)
• no dynamic content loading: pages are rendered on the server and sent to clients. This lowers the cost of accessing them for both the browser and the network bandwidth, makes it easier to index the page, and allows saving the page to a local file with all relevant content
• they did not spy on their users, “enhance” anybody’s experience, nor sell private data to highest bidders like social media does today
• they contained moderated focused collective knowledge, due to the tight scope of each forum it was easier for human moderators to enforce rules and whenever a particular section of a forum gets too unwieldily they split it into smaller sections
• I witnessed many conflicts that started and ended on forums, there was none of the bullying and anti-social behavior common in current social media, even when most users had anonymous handles there was a sense of reputation that no longer exists now due to the sheer number of users

Those forums often were self-hosted using a privately purchased domain and direct use of an opensource forum system like phpbb paired with an SQL database. The moment the domain is not renewed or server bill goes unpaid, the entire forum is gone like it never existed. And this is what happened.

So many communities and knowledge vanished over the last decade, a lot of the larger ones migrated to reddit then discord. But reddit is almost a walled garden now due to them essentially crippling API access and using it to kill rival frontends, while discord was always an anomaly as its a walled garden from the get go yet somehow many communities used it as a safe refuge as there are no domains or server fees to pay and therefore content is assumed to remain there for as long as discord will remain active. Until it’s not, and that will happen sooner or later and we loose everything in there again.

On the game development side, lots of specialized forums are gone but the main big ones are somehow still surviving. Gamedev.net as well as TigSource are notable examples.

As for the web archive, it’s an amazing effort that I believe began at the right time. However if you’ve ever tried using it for deeper than main page queries you’d understand how limited its archiving is despite most sites not using dynamic content loading, nowadays with dynamic content and the sea of bloated javascript/css, it’s harder to archive anything useful.

Enshitified Search Engines

This is a topic that’s been talked about a lot. In my view, search engines went from “indexing the world wide web” to “pushing more ads to make more money this quarter”. They are no longer really search engines but ad networks with a side-feature of sometimes providing relevant results.

Fortunately but also equally cursed, LLMs attack the heart of this problem.

LLMs to the temporary rescue

Temporary because I know that ultimately they will all go through enshitification. Soon OpenAI’s goal will deform from building the smartest LLM to “pushing more ads to make more money this quarter”. And LLMs will transition from intelligent data collators to intelligent ad networks with the side-feature of sometimes providing a useful answer.

But for now, this is an effective solution.

Thankfully, there is already opensource solutions with comparable knowledge. If there is one thing I learned from my journey through computing it’s that open-source software outlasts everything and has the property of remaining very consistent. It’s the only form of software that is immune to enshitifcation.

Social conflict stimulators

Then comes social media. It started with its heart in the right place. Connecting people and sharing interests. A long time ago when I signed up for a facebook account when it opened to everyone, some of us used it to show off our work and post screenshots of current projects. In that way it was great! Then when facebook became toxic-boomer-ville and was filled with vile content left and right. I ran off to Google Plus which had this nice feature called “circles” that connected me to entire communities of like-minded people. For a while we used it to share our stuff until one day the plug was pulled and G+ joined the cemetery of Google.

I feel Google Plus was the end of the golden era of social media for me. After that everything moved away from computing and it became about angry arguing, posting personal everyday life photos and filter-ridden selfies.

If there is any resilient property about the wider social internet, it’s that it caters to the needs of the lowest common denominator. It arrives at that via a process of elimination of useful features via what’s often described as..

“Enhancing your experience”

I wince whenever I hear or read that phrase. It basically announces that this service is in active enshitification right now. The exact steps vary but one I was directly exposed to from a development perspective is AB testing, where they vary button colors, positions, sizes, and telemetry the hell out of their users to determine which was clicked the most. That way they arrive at users more likely to click buttons that lead to spending money and less likely to click buttons that don’t.

Nowadays this is referred to as dark patterns, there is a common prediction that dark patterns are created on purpose. That’s definitely true in summary, however part of why these patterns exist is that they do actually work. And there is a general correlation between the number of users on a platform and how concentrated its dark patterns use gets. This is part of why at the beginning web services and platforms are actually good for their target audience, but overtime as they build a larger userbase they start to apply more and more manipulative UI practices as those are the ones that are most illustratively effective on more users. The metric they optimize for is “growth” of userbase, it’s never quality of services or user satisfaction.

This also explains the last phase of enshitification, descending into complete lunacy in UI design where the platform shoots itself in all limbs and users abandon ship faster than “growth” maximizing tactics can adapt to.

I see this whole process to be similar to the paper-clip optimizer problem. Platforms sacrifice their entire userbase and eventually themselves for the sole purpose of optimizing growth.

To the Escape Pods

How do we escape this? I as many others identify deeply with computing. Not just as a profession, hobby, or passion.. but a literal lifestyle. There has to be some sort of an escape from this degradation?

As I mentioned earlier, opensource software is a beacon of eternal light. In the early 2000s they were an escape from corporate control, now they are an escape from enshitification. I consider it the greatest monument of human creation in the digital realm. Opensource code is owned by us, humans, and those rights of ownership are protected and eternal.

Some of us started looking back driven by nostalgia, to before the internet in its modern form began. There is a huge community of retro computing that’s constantly growing in refuge from big tech. But I recently began to feel this puts us in denial of our present reality and disconnects us from the amazing unstoppable progress of computing despite of all the mess. Two notable examples related to this movement are the misterFPGA project, and 8bitworkshop.

Some of us started an alternative web movement. Here are some examples:

• NeoCities is a great successor to what geocities did
• glitch for live creation of web content (and currently encouraging users to build fediverse apps!)
• tildeverse is a very interesting experiment

Another move for us is focusing on technologies that render old internet obsolete. Knowing what we know now, we can design/migrate to platforms that are basically immune to enshitification like opensource software is. One example of this is Mastodon. It took me a while to understand it but once I did, it basically became the only social media platform I use regularly (and trust).

Decentralized Web

Platforms like Mastodon, Diaspora, and many others are now my beacon of light out of enshitification.

I fell in the rabbit hole today trying to understand what’s possible and how do these things work, and here’s a summary of what I found:

• Fediverse is the middle-ground between fully-centralized and decentralized services
• there is an unfortunate overlap between this and blockchain/crypto-shit. From a shallow look only a little bit of what they created appears to be useful for fighting back enshitification
• there are so many applications already built. An alarming number of them are discontinued but that’s not a death sentence as anyone can pick things back up where they were left off
• ActivityPub is the big player as it defines a protocol for decentralized social networking through which mastodon was built, to be more specific ActivityPub is a REST HTTP API covering all the basics of content creation and fediverse interactions
• IPFS appears to be a central service for many solutions (and has the most badass name: Intergalactic File System 🚀🌍🪐), it stores immutable data and indexes them using a calculated hash
• Scuttlebutt is used for many different apps proving that their approach definitely works (based on gossip)

Storing decentralized data

IPFS seems to be the most straight forward way. You install the tool which lets you upload files to an IPFS node. Here’s a tutorial they provide for how to host a static website on IPFS.

What do I want?

A project that materialized as I was researching this is a trello-like decentralized service. Task boards can be private or public, and using a fediverse approach you may host your own node or join an existing node. This stems from a frustration I have with searching for a practical kanban boards solution that isn’t part of a commercial platform, also allows full access from any device, and ideally doesn’t require manual backing up.

After last post I had a few realizations when I compiled raylib and tested the same examples on my aging Macbook Pro that I’ve revived. The module music example crashes the same way! so the RPiZ/3 were innocent. And the colored cubes example is very expensive to run on the Intel Iris 5100 too which is not a slouch. So I had to redo these two tests to make sure I made a correct evaluation.

First, running a different cubes drawing demo on RPiZ resulted in a solid 60 fps:

Also I realized the reason why the textures_bunnymark demo didn’t show any bunnies on the Raspberries is that I had to modify it to emit bunnies with keyboard input since there is no mouse cursor when running native GL like that, but all the bunnies were being spawned at GetMousePosition() which always returns origin of screen in this case and the bunnies stay under the top rectangle!

So now the bunnies are spawned in the center and it works perfectly:

Can render up to 2000 alpha-blended bunnies to the framebuffer while maintaining a solid 60 fps! that’s not bad at all for 2D games. Once we get to 2100 bunnies the performance drops drastically from 60 to around 52 fps. This kind of predictable performance is only possible in native mode, so I consider this experiment a success. 🙌

I’ve always found the RPiZ interesting due to it being so limited and slow (lol) kind of why I like Amiga A500 or Atari ST.

My vision for the RPiZ was to use it like it’s a low level development gaming platform with a respectable GPU compared to older GPUs before DirectX9. The problem is that modern day linux (Raspbian included) is way too much for this little chip that it makes it almost impossible to use for GUI-based development and the super-outdated and bloated X11 tasks the hardware too much that any OpenGL performance is completely unpredictable and choppy.

The RPiZ comes with an ARM11 processor that uses the arm6vl architecture. It has a floating point co-processor (called vfp) and some not-so-useful extensions like running Java byte code in hardware. The RPiZ runs the processor at 1000MHz which sounds like a lot.. but it pales compared to a Pentium III 600MHz for example. I heard it compared to a Pentium II 300MHz with a small cache and a terribly limiting IO bandwidth.

Its GPU is the Video Core IV which on paper supports OpenGL ES 2.0 and uses a tiled-renderer. It’s quite cool that this GPU is entirely detailed in this freely available document. On paper it has a compute capacity of around 24 GFLOPs. The GameCube is said to have 9.4 GFLOPs in comparison. There is no way the VC4 can produce the same rendering level as the GameCube as it doesn’t have dedicated memory or enough bandwidth to do anything like that.

How can we get the true GPU performance with minimal intrusion from the operating system?

1. we could boot from baremetal and use the Circle library to access the hardware including the vc4 GPU with OpenGL
2. we could run natively from command-line by starting our own framebuffer and using that

I like the first solution as it sounds like a lot of fun and is the fastest, but it has an expensive overhead and iteration during development is going to be a challenge.

I opted to do the second solution as it appeared to be the easiest. I know that SDL can do native build with graphics support but I was able to find a great library similar to one I was developing 2 years ago except it’s actually finished lol and it’s raylib.

Build Raylib with native

I used a fresh DietPi image on RPiZ, to build raylib with native graphics I did:

mkdir build && cd build
cmake .. -DPLATFORM="Raspberry Pi" -DBUILD_EXAMPLES=OFF
make PLATFORM=PLATFORM_RPI
sudo make install

To build the examples there are a few things that need to be done:

• add atomic library to examples/CMakeLists.txt in the build loop at the end of target_link_libraries so it should look like this: target_link_libraries(${example_name} raylib atomic)
• exclude two examples from building due to a conflict in GL, add these before if (${PLATFORM} MATCHES "Android")
  • list(REMOVE_ITEM example_sources ${CMAKE_CURRENT_SOURCE_DIR}/others/rlgl_standalone.c)
  • list(REMOVE_ITEM example_sources ${CMAKE_CURRENT_SOURCE_DIR}/others/raylib_opengl_interop.c)

Then rebuild raylib with examples: -DBUILD_EXAMPLES=ON

Keep in mind if you’re doing this on Raspberry Pi 3, you don’t need any of the changes to build it correctly and you should also build the “DRM” mode rather than “Raspberry Pi” mode which is legacy.

So on RPi3 this is how to build:

cmake .. -DPLATFORM=DRM
make PLATFORM=PLATFORM_DRM

you’ll need a few extra dev libraries to build correctly: libgbm-dev and libdrm-dev

Results

Not great. Mostly due to the driver not being great. OpenGL seems to be missing VAO support and blending is a bit glitchy (running textures_bunnymark shows no bunnies!), also R32G32B32 texture format is not supported which is needed for the skybox example. Also the PiZ hard freezes at random after a while, but this could be my RPiZ only as I always had this issue.

Another odd issue is that running audio_module_playing segfaults.

Other than that, when it works it feels great! instant startup and smooth.

Here’s the spotlight example showing how expensive this is on the GPU even if it was running at 800x450 only:

Here’s the maze example working correctly:

And here’s the cubes example to show how terrible the performance could get, however this demo in particular could be running into a feature reported by the driver to be available but it’s actually running in software but I don’t know that for certain:

Yeah, you could probably make simple graphical demos but this is not what I’d expect from a 24GFLOPs GPU and I blame the vc4 driver.

I tried this same thing on RPi3, compilation runs much faster naturally but the issues graphical and beyond are all there too as it uses the same vc4 driver including the missing bunnies in textures_bunnymark :(

RPi4 might be the one? haven’t tried it yet.

I’ve been using Fedora Kinoite on my main laptop for about a year now. Kinoite is a spin of Fedora Silverblue. This is the first and only immutable operating system I’ve used to date.

To explain why immutability is so important in my view, let’s take a look at some examples from other systems.

Mutability by bad decisions

Let’s start with Windows. In my opinion the worst wide-spread operating system of them all. It’s basically a collection of bad decisions upon bad decisions dating back to the 90s dictated by the requirements of back-compatibility on legacy features that enterprises wish to drag along with them forever.

An example of one of those outdated decisions is that Windows was always designed to be a mutable OS.

What does that mean? you can go into your Windows system folder and delete anything you want. Windows may not stop you. In fact you need to do this every now and then to clean your system folder from 10s of GBs of Windows logs and/or temporary install files that their updater (another broken legacy system) just ignores.

Another example of one of those bad decisions is the registry. You can just mess with it to your hearts content, delete whatever you want, and it’ll mess up the whole system!

Mutability by Design

Linux as an operating system was designed from the ground up to be mutable but it’s done right in my opinion. The user has full complete freedom to do anything they want in the system allowing advanced users to do whatever they can think of.

You want to experiment on a different OS scheduler? a different way to install device drivers? have at it at your own peril.

The drawback of course is that this absolute mutability is hostile to its users. I can’t count how many times I killed my installed distro by attempting to update an NVIDIA display driver.

BSD is an interesting variation on the mutability by design philosophy. It actually achieves better stability and is more friendly to its users simply by being a larger OS that comes with many more batteries included and a slower more sensible evolution process.

The Walled Garden

On the other extreme all modern mainstream phones are powered by immutable systems that require a jailbreak to escape from. An immutable system cannot be changed in anyway by its user, only by the system or software makers themselves issuing updates to it.

An operating system with a walled garden design proved to be vital for stability and long term dependable robust operation. This is part of the reason why we depend on our mobile devices so much. We know they are very unlikely to misbehave or break internally like a Windows laptop often does.

This is also reflected in all Apple OSX devices like Macbooks. They are super dependable and clean.

However these systems come with a large drawback: the user does not really control the system, and the system controls what the user can and cannot do.

Immutable Operating Systems

Between the fragility of mutable systems and the authoritarianism of walled gardens lies a sensible middle.

A system that offers the same immutability you find in OSX/iOS/Android so it’s dependable, robust, and maybe impossible to break by the end user. Yet, it maintains the ability to let the user change most things in their system if they really want to, while also providing a method to reverse these changes non-destructively!

So what are the elements that Fedora Silverblue provides that let’s it achieve all that?

• Immutable system: neither you nor any software can change any system specific files/folders directly as they are protected from writing
• Atomic image updates: the system gets updated as an image rather than the usual package manager approach. This eliminates any issues that may rise from wrong dependencies as well as provide a straight forward way to jump back to a previous image of the system or a future one still in beta to test things out non-destructively
• The system image operates as a version control system via rpm-ostree which acts as a package manager of sorts. It allows you to push changes on top of current system image, this way you can apply any system changes you need (such as install NVIDIA drivers or any tools you may need in the OS) and these changes “shadow” any existing data. At anytime you can revert or cherry-pick these changes or altogether discard all commits and restore original system image. When system image is updated, your committed changes get pushed on top of the new image.
• You may install any software you want from flathub (flatpak is supported by default) and you can also install snapd by committing the change on top of the system image using rpm-ostree to enable snapcraft. And finally my favorite method: AppImage
• For developers, one of my favorite features in Kinoite/Silverblue is toolbox. You can create as many toolboxes as you want each acts as a self-contained mutable install of Fedora/Centos where you can use dnf to your hearts content and can even install and run GUI applications seemlessly like VS Code. I naturally setup a toolbox per project and can go crazy in any of these toolboxes without worrying about my actual system getting poisoned in the process

In my opinion this is the perfect developer OS: as solid as OSX or iOS without sacrificing flexibility to change the system, best dev containers system I’ve used so far (much better than manually creating dev containers), and it still works perfectly for gaming by installing steam using rpm-ostree and steam runs in its own sandbox so that’s all you need.

However it’s anything but perfect. Here are some of the challenges I’ve run into so far:

• sometimes when an OS image update is available, some changes in it conflict with packages you installed using rpm-ostree and it fails to update as a result. The error and info you get are vague but the easiest way is to simply uninstall whichever packages are conflicting with the update
• I could not for the life of me get an NVIDIA RTX GPU to work via Thunderbolt, I followed every single guide I found and eventually gave up and installed Windows on an external drive for playing newer games
• Running GUI applications from within toolbox works, however there is a really annoying bug that occurs when you quickly move between app UI elements with the mouse, the whole system pauses for a second (mouse included) and resumes after. This seems to be much worst in electron apps sigh
• Updating your software works from Fedora’s software manager (Discover) but sometimes some updates just fail for no apparent reason, retrying gets those updates to succeed most of the time
• Can’t change the wallpaper in the Login Screen as it’s in a system image path (might be a Kinoite specific issue?)
• Installing Lutris and other game launchers (ubi, epic, etc) is problematic as wine is not part of the default system image and must be applied using rpm-ostree

And that’s that! since Silverblue there has been other immutable Linux distros, this awesome list keeps track of everything immutable.