How do you ref a node?

One of the unique features of BeOS is the way the Operating System holds a reference to a disk file.
To understand this and why it is difficult to emulate this on other operating systems, you will need a primer on how a traditional Unix file system is built. I will be quick, I promise.

Files are stored on disks. There are a lot of kinds of disks, all referred to as devices. Every device has a unique device_id – a counting number. Each device is divided in blocks (of some hundreds of bytes). When you want to write a file to a device, you take some blocks enough to fit your file in, link them in a chain and write file content to the blocks chain. But how do you store who owns the file? Where (which block) the file starts? This is a job of nodes. Node is a piece of data, bookkeeping all the meta information on the file. Owner, group, permissions, ACL, attributes… Name? No, not the name. On a Unix filesystem one file can be known under many names – all sharing the same meta-information and content, uniquely identified by node_id. A rose by any other name would smell as sweet.
So, how do you identify a file by a /file/path we all are familiar with? There are special files on disk, called directories, which keep the "filename -> node_id" mapping. The entries in the directory file are called links. There is no restrictions on how many times and under how many names you can list one node_id. The entries that point to a node being another directory are called… sub-directories.
Now you see that deep path traversal can be time consuming, as you need to read and scan many directory files.

BeOS just skips all of that directory nonsense. If you want to refer to a disk file, you just need two pieces of data: device_id and node_id on that device. Two numbers – which uniquely identify a file on your, particular machine. You just go to the device, find a node on that device and you have all that is needed to open a file.

It’s not that simple on Linux/Unix. In order to be able to access a file, you need at least x (execute) permission on all directories leading to the file and r (read) permission on the file itself. It is not possible to access a random node number on a disk device. Thus, it is not possible to pass reference to a file by a simple two numbers (device_id and node_id) which BeOS calls node_ref and is using everywhere…

Continue reading →

Welcome to Wayland

On October 31, 2022 D/os displayed its first GUI window.

You’ve already seen this picture, when I’ve explained how do I connect to a Wayland socket of server running on host machine, from a client running in chroot environment.

Yes. Wayland. I’ve chosen Wayland as a display server protocol for my implementation of BeAPI. This is a bare-bones protocol, allowing the application to obtain a buffer of pixels to draw on (part of) the screen and request input devices information as a stream of events. I will not be implementing full app_server protocol. Drawing logic is implemented in-app, in BWindow class (the frame buffer canvas) and in BView (view stacking and repainting logic). There are several reasons for that.

libbe is a compatibility API allowing for running existing BeOS applications (after recompiling for D/os) and will not be the only one to use. I will not extend libbe with new APIs (the exception is porting Haiku extensions, for app compatibility). There is too much legacy to evolve it. Be already realized that and started working on version 2 of the API as seen in Dan0 headers. I will be building libbe2 – possibly based on Dan0/BeNext/OpenBinder. Thus there is a reason to not tie display server, to BeAPI intricacies.

Wayland will allow drop-in compatibility with applications built for generic Linux distribution, without specific requirements for desktop environment. Stuff that brings its own toolkit, drawing, or even takes over full screen. Read: OpenOffice, Blender and… games (with distro-compatibility library like Steam).
I do not plan to package any third-party applications as a part of the main system distribution. The way to install such apps will be Flatpak. Just add a proper repository and you’re good to go.

So…
Once you have your buffer in a window on screen, you want to draw something in it, right?
But, that’s a whole, different story. Next time…

I want to give great kudos to Sergey Bugaev for Writing Wayland clients book and Drew DeVault for The Wayland Protocol book.
Without these, I wouldn’t be able to write Wayland handling code. Thanks guys!

chroot is a development tool

Once I started implementing libbe, I needed a tool for writing and automating unit tests for the library. Ideally something as nice to use as Rust tests, that you can keep together with source code and build&run using one command.

After some searching I’ve found doctest which fits the profile perfectly. You write tests together with the source code, so these serve as a code documentation. It is designed to compile really fast, so it does not affect much your build time. You can also ship tests embedded in your prod-build, so it is possible to use them in user deployments to diagnose user issues.

Once I moved to Test-Driven-Development, it quickly turned out, that having QEmu in my modify-compile-run development loop is annoying. Having to launch whole system under QEmu, su to root and manually run tests binary was really slowing me down. Ideally I wanted something as simple as [arrow-up] + [enter] in the shell window. I’ve investigated sharing a local filesystem to a running QEmu machine, remote execution… It didn’t feel right.

Then my attention turned to a chroot wrapper script I’ve written during toolchain development, to test whether my custom-built clang cross-compiler is able to produce C binaries and libraries (like musl) and then C++ binaries able to link against target and run. I used it only to run several versions of “Hello, World!” so far. I tried using it to run the test binary and to my surprise it worked perfectly! 🎉

This is awesome. I’ve spent a lot of effort to have the IDE experience smooth and this flow fits nicely.

Continue reading →

On the road to BApplication

As the base system integration work is done, I can finally start writing some actual code.
This needed a lot of figuring out again. How to structure the source code? How to build it easily? How to test it easily without restarting QEmu every compilation?

Shell BApplication running on D/os in QEmu
  • I decided to use original BeOS 5.0.1 DE headers as a base to write my own from scratch – taking only the parts that are actually implemented. This will help finding out API incompatibility during app compilation.
  • Build system is still ninja, with build files committed to system/os repository – piggybacking on OS build.ninja file providing build tooling.
  • I’ve copied “Kernel Kit” implementing system functions missing from POSIX/Linux Standard C Library on Linux Kernel facilities, from my previous attempt.
  • I’m writing Support Kit using Android’s libutils and libbinder with missing parts coming from OpenBinder.
  • I’m writing Application Kit using OpenBinder/libbinder with missing parts coming from Haiku.

At least this is the plan. Let’s see how it goes.

There was just one more thing I needed to figure out – the way to implement BRoster, which allows user applications to “see” and message each other. The idea is to have a binder service exposing BRoster API, which every user application connects to. This requires writing an android service in C++, as this is the language of the system. It is possible to write such service – as few fundamental Android services are written in C++, but the official documentation is all about Java. Information about C++ services is scarce.

sampservice – Android service written in C++ running on D/os in QEmu

After a lot of Internet searches and digging through Android source, I’ve assembled a basic sample service implementation. It worked from the get-go, but didn’t respond to a shellCommand sent to. Binder debug shows that the message goes into the service, but does not reach my code.

Digging through libbinder code shown, that 7 years ago shellCommand code was “temporarily” stubbed due to some incompatibilities with pre-built Android binaries and stayed stubbed since. Reverting that commit was easy. Funnily, it turned out that a TODO: item in the stubbed code was fixed in the mean time, so I removed a TODO: line too. SampService started talking to me back.

There is a long way to Milestone 2 ahead, but I don’t see any major roadblocks anymore. The plan is laid out and it “just” needs a lot of typing.

Base system is up

It may not look much, but it was a long journey to get here.

D/os is able to boot on x86_64 PC hardware (QEmu), Linux kernel, Android init system and Android ueventd /dev manager,
to musl based userland running Weston Wayland compositor with virgl OpenGL and Android service manager.

Whole system is built in two steps.
1. One command (make) builds custom, LLVM/musl cross-compilation toolchain.
2. One command (ninja) builds the whole operating system and writes ext4 filesystem image for QEmu.

I can finally start porting/writing libbe.

Houston, we have visuals!

It was a wild run to build all the dependencies required to run Wayland Weston on my weird stack. You do not really realize how complicated our daily-use OSes are, until you try to build from scratch.

I finally got it on QEMU with GPU acceleration over virtio. The major obstacle here was that QEMU recently changed its command line arguments, so all the documentation on the web is outdated. I finally found correct one in a comment on QEMU bugtracker. Then after launching weston I saw an emulated black screen. A lot more digging and another comment on on Ubuntu bugtracker revealed that it does not work with GTK display, but works with SDL. Switching to SDL and Boom! I finally have Weston desktop!

With no visible mouse cursor… Well, you need to connect mouse and keyboard, besides screen, to have a working console. virtio mouse and keyboard it is.

Having a good run with virtio, I also added an option to mount /system directory from host over 9p protocol, so I can test freshly built stuff without “rebooting” QEMU.

virtio is awesome.

Second stage… init

It was a long-ish and very interesting journey of making Android Init run with musl root filesystem.

Today D/os reached a point where first_stage_init running from initramfs is able to

  • mount /system partition,
  • switch root to it,
  • pass over to second_stage_init,
  • run ueventd to handle devices,
  • and start console service giving interactive shell to the user.

https://github.com/D-os/dos/wiki

Orphan Rules ¯\_(ツ)_/¯

The silence on this blog and the lack of progress in Soldank repo is because I am on complete detox from Rust. I got so frustrated trying to make my CLI & scripting code into a reusable crate, that I haven’t touch Rust at all since. Hobby is supposed to be fun, not make you frustrated and angry.

The thing is that the command & scripting code uses types defined in external crates (like glam, hecs, Rhai, etc.) and works with them in a generic way using traits implemented on them. So if I move the engine to external crate, its traits are no more local to my project. So my project would be implementing external traits on external types – and you cannot do that in Rust. Period.

P. S. I am aware of Newtype pattern. Applying a deficiency in language to work around a deficiency in language is a horrible idea.