Sometimes you are conservative, you stay the course, you do the cautious, reliable, boring thing. And sometimes you install Arch Linux on your lab machines, but only after flirting with OpenSuse. Welcome to Part 6 of Running a University Linux-based Audio Lab!

Lab Shmab

In “Running a University Linux-based Audio Lab” parts 1-5 I describe my audio lab philosophy, choices, deployment strategies, and more. All of those posts relate to the previous lab configuration: a custom software layer built on KDE Neon which is itself based on Ubuntu LTS. That hardware/software configuration is now close to 10 years old. We do update it every year so every two years the LTS version changes and we get the latest Plasma version among others. The configuration is mostly stable, and it is predictable in the way it breaks, but it is time to move on. There are long-running issues with this configuration as well as some new problems. To name a few:

1. The future of KDE Neon is unclear given the emergence of KDE Linux.
2. Ubuntu LTS is ironically the least stable component of our systems and “stable” packages from two years ago often break requirements of our current software builds.
3. I have to source software from GitHub, multiple repositories, and build some of it from scratch. The install script is annoying to update every year. The software is annoying to update because there is so much manual interaction required.
4. Plasma regularly breaks when Neon updates.
5. The apt/pkcon/Discover ways of updating are confusing and potentially conflicting leading to breakage. You can see this for yourself by trying to update three new KDE Neon installations three different ways: use pkcon update for one, sudo apt full-upgrade for another, and use Discover for the last. That you even *can* do this is a problem¹.
6. More! (but life is short)

Better the Arch you Know (than the OpenSuse you don’t…)

Looking to move on, I settled on Arch. I personally run Arch on multiple laptops and desktops and it has unexpectedly seen better uptime and required less intervention than my KDE Neon machines. I briefly flirted with the idea of using OpenSuse, going so far as to enlist my University ITS in updating our post-install script to work with OpenSuse – no small task if you know how bizarre some of OpenSuse’s design decisions are. But in the end, it ended up being *too* weird and unstable as well. When things broke, it took too long to fix them. Standard installation strategies and scripts required rewriting because of OpenSuse’s non-standard library paths. Worst, it didn’t solve my “I have to do a bunch of custom things and build software from source” problem. Arch does, of course.

The Extra repository has all but three packages I need and the AUR has the others. No more compiling from source. Arch elegantly handles realtime privileges and has an rt kernel. Pipewire is stable and recent. I could go on, but you get the idea.

One thing that I *did* like about OpenSuse that saved me when things did break was the automatic snapshots. So, the new systems will use btrfs with subvolumes so we can use snapper to create system snapshots before and after system updates.

Ansible me this!

The new configuration has some major changes, but all of them streamline my system and make installation and updating more robust.

1. Install Arch with a known good configuration using archinstall²
2. Configure the machine post-install with Ansible

Our Ansible playbook does most of the heavy lifting: install the RT kernel, realtime-privileges package, create random users, add users and admin to specific groups, install nvidia drivers, create necessary system configurations, install the music-specific software layer, configure and distribute ssh-keys for rsync backups, and more. Ultimately, it’s all pretty straightforward with very few custom fixes. It’s almost… boring.

CrankyCuckoo, WhaftyWeetabix…

One big change is that we will no longer be using the university Active Directory for authentication. Instead, each machine will have a number of random users created with the required groups whose passwords that must be changed at first login. Students will be assigned one of these (hilariously named) users when they enroll in one of our courses that requires the lab. There will be a generic lab user account with no privileges for general computing.

This change is a significant experiment. I lost my longtime ITS support person (and good personal friend) to retirement. He was responsible for helping me get the system configured every year to keep up with ITS changes. That occasionally broke (from year-to-year) and cost him some hours, but he always got it going again and did it with a smile. He had incredible institutional knowledge as well as solid, old-guy Linux mojo. Going forward, I do not have confidence in the happiness of whomever would be tasked to help me configure my Arch machines for the AD, nor the myriad of other technologies required to make that all work.

One of the potential downsides of local user accounts is forgotten passwords and the emails that come with them. We will see how bad (if bad) this gets. The plus-side is that the lab desktops as well as the floating laptops will be fully accessible even without working networking, even on the first login. Login will also be immediate rather than *whenever* the domain users and groups are all checked in the AD. Pros and cons and pros.

A Plasma Transfusion

One major difference from the Arch systems I use is that the lab machines will run Plasma Desktop on Arch. I run either i3, Sway, or Hyprland depending on what I’m doing and whether I’m using X or Wayland. On the new machines, I will keep using Plasma to both provide an accessible and familiar desktop experience and maintain look-and-feel continuity for our users. As mentioned above, Plasma has been a point of failure in the current configuration so I’m interested to see whether that continues to be the case on Arch. It could be that we see a move to something more pedestrian (but more stable?) like Gnome in the future.

Updates and Backups

To mitigate the risks of a stale Arch system, I am scheduling regular maintenance routines at 3 AM on Monday mornings. That maintenance will take the following form:

1. Use snapper to create a pre-update snapshot
2. Backup important config files to off-site computer
3. Update software
4. Use snapper to create post-update snapshot
5. Check for update errors, email me if problems found

The offsite backup seems a little weird given we are taking snapshots, but its easy and doesn’t cost anything. So, just in case it can help where a rollback is undesirable or impossible we will do it.

That’s All, Folks?

That’s it for this update. I am currently in the testing and configuration stage with one machine ready to go! Next, I will be deploying to one lab machine to confirm everything works as expected. I will then complete the lab desktops and laptops and report back. It will undoubtedly be only good news, right?

As always, if you want emails when I publish a new article, smash the Subscribe link to sign up!

Footnotes

1. It gets worse. It turns out you probably *have* to be able to do this because pkcon itself regularly breaks in ways that require apt or apt-get or other means to resolve. Discover also does this. But Discover is also the one-stop-shop for *all* system upgrades including any weird things like flatpack installs, snaps, and firmware updates. Otherwise, you have to run fwupd on its own. But, you can’t schedule Discover to run in a script, it requires manual intervention, so the easiest and safest is the least prescribed and that’s to run sudo apt update -y && sudo apt upgrade -y. ↩︎
2. The lab machines have to be easily supported by a student assistant, at least in theory. The downside of going Arch All The Way is the initial installation process is daunting, but not so with archinstall. ↩︎