A couple of weeks ago, Debian Bookworm became stable. I have a few devices running stable, including an Orange Pi Zero I use to open and close my garage door with a Telegram bot.
So, I SSH’ed in it, and ran the following commands:
apt update apt upgrade apt dist-upgrade
Without even changing my /etc/apt/sources.list because I already keep stable instead of the codename 😎️.
End of the story, just like any other major Debian update there in the last 6 years… Or is it?
I just wanted a new kernel…
Many boot files were dated June 2017, so the system probably started as Debian Stretch. And actually, it was an Armbian installation (which uses the official Debian repos, contrarily to Raspbian, plus a custom repository for a few additional packages).
The Linux kernel was one of these custom packages, and its version was 4.11.3, released on May 25, 2017, so quite old. I wanted a more recent kernel with security vulnerabilities fixed and all the other improvements.
Debian supports a configuration they call ARMMP (ARM multiplatform). With a single kernel and DeviceTree, it targets many different devices, and the Orange Pi is one of them. So, I tried to install Debian’s official kernel instead of the one I was using… and for the first time in 6 years, I screwed up to the point I needed to remove the thumb drive I use as storage on that system. It was somehow annoying because the board is in an electrical box, and there were several things I needed to move to get to it. I tried to troubleshoot it for an entire evening without success. Eventually, I went back to the old kernel, closed the box, and moved everything again.
Long story short, ARM computers need a bootloader, often U-Boot, and a boot script. You can think of it as the GRUB and its configuration equivalent on x86 computers. However, it is less integrated, and Debian does not update boot files for you out of the box.
Therefore, I was still using Armbian’s script. Even though it uses generic names and symlinks, making them target the new files was not enough.
…and then I wanted a new installation…
While trying to solve this problem, I realized that Armbian comes with a lot of pre-installed software. My old system used almost 2GB out of the 4GB drive. I would not call it bloatware, but I wanted a lighter system closer to a standard Debian. After all, Debian documentation says my hardware is supported out of the box.
I found an unofficial pre-packaged system for the Orange Pi Zero, so I decided to give it a shot. Using an unofficial image is not the best practice, but the script used to create it is available, so users can recreate it and check nothing has been tampered with. I did it, personally, and the only surprise was that the image had many more files than mine because it also included the usrmerge package, which depends on Perl. However, after installing it, almost all the files matched except for some logs and caches.
The advantage of this image is that it includes the U-Boot script as well as a network configuration for systemd-networkd to use DHCP on all Ethernet interfaces (en* and eth*; mine was eth0 in Armbian and end0 on Debian Bookworm). However, it does not include a /etc/fstab. But more importantly, it comes with pre-generated SSH host keys. This is quite a red flag, but they are generated while OpenSSH is installed; I do not believe they were added with malice. However, I replaced them with my old keys and opened an issue on GitHub.
…but without physical access (almost)
So, at this point, I wanted to replace my Armbian installation with a brand new Debian (I also had other reasons, more about that later).
Usually, you would flash the image on the boot device, which requires physical access. But I did not want to open the electrical box again. So, I wondered if I could switch temporarily to a system on a ramdisk to replace the persistent one. I thought, for example, of something based on kexec. But then I found that Linux has a command (based on an homonymous system call) that helps: pivot_root.
It is somehow similar to chroot. But, as a difference, it also allows accessing the old root from within the new system.
Another difference is that it has some additional requirements, but they are very technical, and I do not understand them very well. For starters, both the new root and the old one need to be mount points. It was not a big deal since I already wanted to use a tmpfs for the temporary system. However, initially, I had this error:
pivot_root: failed to change root from `/tmp/newroot' to `/tmp/newroot/oldroot': Invalid argument
It sounds very generic, and the man page of the command is not very helpful, either. But, the man of the syscall says:
The propagation type of the parent mount of new_root and the parent mount of the current root directory must not be MS_SHARED; similarly, if put_old is an existing mount point, its propagation type must not be MS_SHARED. These restrictions ensure that pivot_root() never propagates any changes to another mount namespace.
I am not sure of the proper way to solve this, but since it was a temporary hack, I solved it with this command:
mount --make-private -o remount /
Then, I downloaded and extracted an official Debian image for Docker to create the temporary system. If interested, you can find the latest one on the Docker hub. It is very light, so I had to chroot to it to install a few helpful packages. One was OpenSSH, and I configured it to listen on a different port since 22 was already used by the original system, and I was connected to it.
Then, before pivoting the roots, I stopped the various services of the old system, just like suggested by the guide above. The init process (Debian’s is systemd) cannot be killed. Thus, the old root cannot be unmounted, formatted, etc.
So, I moved all the content of the old system to a new directory, and then I performed a “dirty” copy/extraction of the new image. The one I linked is a compressed ext4 filesystem. After decompressing it, you can mount it and copy the files. However, the decompressed file is almost 4GB, so I created a much smaller (less than 500MB) tarball on my desktop instead.
I have blurry memories of what I had to do to get the new system, so I am sorry if I missed some passages.
I specified almost no physical access because of the elephant in the room: the reboot. Both halting and rebooting rely on the init system (again, systemd), which refused to do anything for me after the procedure. I could not even rely on system requests because they were disabled on Armbian’s kernel. So, I cheated and powered off and on the entire circuit of my basement.
And with that, I finally got the official Debian Bookworm. I confirm it works out of the box with standard Debian packages. I still cannot believe it worked.
However, my task was not finished yet.
The bot
As I wrote in the introduction, I run a Telegram bot in my Orange Pi Zero.
I wrote the original one in Python several years ago.
I used virtual environments to manage the various dependencies. Even though venv copies a lot of stuff, it symlinks the interpreter to the one in /usr/bin by default on Linux. So, dist-upgrading Debian stable always screws all the virtualenvs.
But it is fine, and I take the occasion to update the various dependencies. When I do not intend to redistribute that project, I do not pin versions to favor updating in case of new deployments.
However, this does not work with python-telegram-bot. I have already rewritten the bot at least once because this library broke compatibility between versions, and they did it again. So, I switched to Telegraf and Node.js instead. The radical change was an additional reason for this clean start.
I enjoyed rewriting the bot with the new library. I found it easier to use, and the resulting code is much more compact.
Memory-backed GPIO
The fun was great, but I still had a challenge ahead.
The bot commands the door with a relay board through GPIO.
Back in the day, I found a library to command GPIO with a C program. I grabbed gpio_lib.c, gpio_lib.h from this GitHub repo, and followed these instructions to write a small program that listened on a UNIX socket to turn on and off GPIO pins. But it did not work in the new system.
mmap failed because of permissions problems, which is strange indeed since the program runs as root. The reason was that starting with Linux 4.16 (released just half a year after Stretch), CONFIG_STRICT_DEVMEM is enabled by default. At a certain point, I feared I needed to recompile the kernel to disable it. Luckily, passing iomem=relaxed is enough. If you used the same image as me, you can add it at the end of the line starting with setenv bootargs … on /boot/boot.cmd and on /etc/initramfs/post-update.d/zz-update-uimg (the former is the current boot script, and the latter is the template to create it). Then, you have to run
mkimage -A arm -T script -C none -d /boot/boot.cmd /boot/boot.scr
and reboot.
Sysfs GPIO
That kernel configuration option has existed for years but became the default only in 2018 as a response to Spectre, Meltdown, and similar vulnerabilities.
And actually, there is an alternative (if not preferred) backend for GPIO: the sysfs implementation. It allows interaction with GPIO by reading and writing files in /sys/class/gpio. Since it is file-based, it can be configured not to require superuser permissions.
I tried it before solving the mmap problem, but a couple of hardware issues prevented me from eventually using it.
One was that my relay board is active-low. When a certain GPIO is not enabled, it is in high impedance state. But as soon as you activate it, the state is set to low, and the relay switches from closed to open. Usually, that should happen only when you start the bot. Still, risking any unwanted actions on the door is not great.
Maybe this would have been solvable, but the second problem is not for sure. In addition to open and close the door, I have a Hall effect sensor to probe whether the door is open or closed. Like all switches, it needs either a pull-up or a pull-down resistor. The SOC can be configured to use internal resistors for this purpose, but only when you use it with memory-backed GPIO!
So, I was obligated either to solve mmap’s problems or to change my circuit to add an external pull-up resistor. I chose the former.
To sum up, this was a long and troubled journey, and I spent far too long than I expected and wanted to. But at least I am pretty satisfied with the final result.