Installing packages to a read-only filesystem

If you think it's at least a little bit insane to install packages (which involves writing files) to a read-only filesystem (to which files can't be written), you're absolutely correct, but I still had to do it. This all came about thanks to two incompatible decisions, neither one made by me.

The first is that the Dell OpenManage packages for Ubuntu use /opt/ as a dumping ground for all sorts of equipment, including binaries, libraries, configuration files, images, and run-time data. The second is that /opt/ happened to be a read-only NFS share mounted from another host.

Preparation

The structure of the problem is straightforward. We have two machines (S and C) running Ubuntu 16.04 LTS (with Linux kernel 4.4.0). The server S shares its /opt/ directory via NFS, and the client C mounts it as its own /opt/. Both the share and the mount are read-only. We wish to install the OpenManage packages (version 8.4.0) on C, but these packages put files in /opt/dell/ and /opt/lsi/.

To get started on the insanity, we need to get some formalities out of the way on C. We need to have

1
deb https://linux.dell.com/repo/community/ubuntu xenial openmanage

in sources.list and we need to run

1
apt-key adv --keyserver pool.sks-keyservers.net --recv-key ...

to fetch the Dell signing key. Now we're ready to

1
apt update

and do some installing!

Path to failure

Some good news to get our hopes up: the packages that we want to install are already present on S, so the /opt/ directory on C already contains all the necessary files for these packages.

If we convince dpkg to not install any files to /opt/dell/ and /opt/lsi/, then the end result should be just what we're after. This is easily done by creating a file called /etc/dpkg/dpkg.cfg.d/dell_opt_exclude with the contents

1
2
path-exclude /opt/dell/*
path-exclude /opt/lsi/*

Now let's try to install a simple package without any other OpenManage dependencies (don't actually do this):

1
apt install srvadmin-hapi

Everything goes well until

1
2
3
Unpacking srvadmin-hapi (8.4.0-1) ...
dpkg: error processing archive /var/cache/apt/archives/srvadmin-hapi_8.4.0-1_amd64.deb (--unpack):
 unable to securely remove '/opt/dell.dpkg-tmp': Read-only file system

Then we hit bug #838877, where dpkg fails to handle EROFS. This is fixed in version 1.18.11, but the fix isn't currently available in Ubuntu LTS.

In hindsight, this was never going to work even without the bug, because packages aren't just collections of files. They also have scripts that run at various times. The OpenManage package scripts try to write to /opt/, so at least a part of /opt/ must be mounted read/write.

Removing the broken package

If you're following along with the commands despite the warning, at this point you have a broken package. If you try to remove it with

1
apt remove srvadmin-hapi

you will see

1
2
3
dpkg: error processing package srvadmin-hapi (--remove):
 package is in a very bad inconsistent state; you should
 reinstall it before attempting a removal

We can't install it, so we try the obvious thing:

1
dpkg --remove --force-remove-reinstreq srvadmin-hapi

This is usually enough to get rid of a broken package, but dpkg tries to clean up after itself:

1
2
dpkg: error processing package srvadmin-hapi (--remove):
 unable to securely remove '/opt/dell/srvadmin/lib64/libdchesm.so.8.dpkg-tmp': Read-only file system

Since it can't (due to the bug), it gives up. I'm not certain that this command would succeed without the bug either, since it's not possible for dpkg to remove the package contents from /opt/.

The easiest way I've found to get out of this situation is to manually remove all the unwanted files that are reported by

1
dpkg -L srvadmin-hapi

and then edit /var/lib/dpkg/status by hand to remove the srvadmin-hapi section.

Overlaid by success

Another idea is to put bind mounts on /opt/dell/ and /opt/lsi/ to mask the existing package files without hiding everything else in /opt/. This would also allow the package scripts to write where they need to. Sadly, since /opt/ would itself still be read-only, we would run into the same dpkg bug as before.

We need a solution that lets us vandalize all of /opt/ while keeping the changes local to the machine and not masking any directories we don't want to modify. That solution is OverlayFS, and it's surprisingly easy to use. There's an excellent introduction in the kernel documentation.

The first step is to create some local directories:

1
2
mkdir /root/opt_overlay
mkdir /root/opt_overlay_work

These don't have to be in /root/, or even in the same directory, but they do have to be on the same filesystem. The work directory is necessary to be able to write to the overlay. Since we already know the directories we want to mask in the overlay, we can prepare whiteouts:

1
2
mknod /root/opt_overlay/dell c 0 0
mknod /root/opt_overlay/lsi c 0 0

Now we are ready to mount:

1
mount -t overlay -o lowerdir=/opt,upperdir=/root/opt_overlay,workdir=/root/opt_overlay_work overlay /opt

Before the mount:

1
2
3
4
5
# findmnt -R /opt
TARGET SOURCE FSTYPE OPTIONS
/opt   S:/opt nfs4   ro,relatime,...
# ls /opt/dell
srvadmin  toolkit

After the mount:

1
2
3
4
5
6
# findmnt -R /opt
TARGET SOURCE  FSTYPE  OPTIONS
/opt   S:/opt  nfs4    ro,relatime,...
└─/opt overlay overlay rw,relatime,lowerdir=/opt,upperdir=/root/opt_overlay,workdir=/root/opt_overlay_work
# ls /opt/dell
ls: cannot access '/opt/dell': No such file or directory

Now we can install the package we wanted all along:

1
2
3
apt install srvadmin-storageservices
systemctl start dataeng
. /etc/profile.d/srvadmin-path.sh

It works!

1
2
3
4
5
6
7
# omreport system
Health

SEVERITY : COMPONENT
Ok       : Main System Chassis

For further help, type the command followed by -?

Stuck in a bind

There are at least two reasons I can't use OverlayFS as a permanent solution. One is that OverlayFS is incompatible with the root_squash NFS option, making it impossible to open directories in the overlay that are not world-readable. Without the overlay:

1
2
3
4
5
6
$ stat -c %A /opt/good ; ls /opt/good
drwxr-xr--
file1  file2  file3  file4
$ stat -c %A /opt/bad ; ls /opt/bad
drwxr-x---
file1  file2  file3  file4

With the overlay:

1
2
3
4
5
6
$ stat -c %A /opt/good ; ls /opt/good
drwxr-xr--
file1  file2  file3  file4
$ stat -c %A /opt/bad ; ls /opt/bad
drwxr-x---
ls: reading directory '/opt/bad': Permission denied

The other is more fundamental. The documentation says that the underlying filesystem must not be modified when the overlay is mounted, but I need to be able to alter the contents of the NFS share. Although this is guaranteed to not result in a catastrophic event (and I suspect OverlayFS will do the Right Thing anyway), it's best to avoid undefined behaviour.

It may not be obvious why we even need to keep up the facade now that the packages are installed. The dataeng service puts PID files in /opt/dell/srvadmin/var/run/openmanage/ and IPC sockets in /opt/dell/srvadmin/var/lib/openmanage/. I'm not sure why it doesn't use the real /var/, but I figure it's best to let it write where it needs to.

Since we've already gone through the arduous journey of installing the packages, there's nothing stopping us from using bind mounts. First, a couple of preparatory steps to undo some of the things we've done:

1
2
systemctl stop dataeng
umount /opt

We're also free to remove /root/opt_overlay_work/. Using the existing directories from the overlay, making the bind mounts is as simple as

1
2
mount -o bind /root/opt_overlay/dell /opt/dell
mount -o bind /root/opt_overlay/lsi /opt/lsi

Now we have

1
2
3
4
5
# findmnt -R /opt
TARGET      SOURCE                                  FSTYPE OPTIONS
/opt        S:/opt                                  nfs4   ro,relatime,...
├─/opt/dell /dev/mapper/...[/root/opt_overlay/dell] ext4   rw,relatime,...
└─/opt/lsi  /dev/mapper/...[/root/opt_overlay/lsi]  ext4   rw,relatime,...

We can start up dataeng and use omreport once again.

Making it permanent

The above setup won't survive a reboot. To make it permanent, we're going to create two systemd mount units. The first is /etc/systemd/system/opt-dell.mount, with the following contents:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
[Unit]
Requires=local-fs.target opt.mount
After=local-fs.target opt.mount

[Mount]
What=/root/opt_overlay/dell
Where=/opt/dell
Options=_netdev,bind

[Install]
WantedBy=remote-fs.target

There's not much happening here:

The other file is the same, but with lsi instead of dell where appropriate.

All that remains is to install both units with

1
systemctl enable opt-dell.mount opt-lsi.mount

which puts symlinks in /etc/systemd/system/remote-fs.target.wants/. Then /opt/dell/ and /opt/lsi/ are bound on top of the NFS share at boot.

Tweaking the services

We would like to make sure that the OpenManage daemons don't start until the bind mounts are ready. To this end, we put

1
2
3
[Unit]
Requires=remote-fs.target
After=remote-fs.target

in the following files:

This may be done conveniently with

1
systemctl edit dataeng dsm_sa_ipmi instsvcdrv

After running

1
systemctl enable dataeng

we see that dataeng is started at boot after all the mounts.

It's likely that other OpenManage packages will include other daemons. They may be found with

1
2
3
for p in $(dpkg -l | grep -f <(grep '^Package: ' /var/lib/apt/lists/linux.dell.com_repo_community_ubuntu_dists_xenial_openmanage_binary-amd64_Packages | cut -d' ' -f2-) | awk '{print $2}'); do
    dpkg -L $p | grep /etc/init.d/
done

Dropping udev rules

During boot, we now see complaints from systemd-udevd:

1
2
Process '/bin/sh -c '[ -x /opt/dell/srvadmin/sbin/dataeng.hotplug ] && /opt/dell/srvadmin/sbin/dataeng.hotplug pci'' failed with exit code 1.
Process '/bin/sh -c '[ -x /opt/dell/srvadmin/sbin/dataeng.hotplug ] && /opt/dell/srvadmin/sbin/dataeng.hotplug usb'' failed with exit code 1.

These arise from the configuration in /etc/udev/rules.d/dataeng-udev.rules. The rules can't function until /opt/ is mounted, which doesn't happen until later in the boot process. I don't believe it's possible to delay these rules until after all the mounts happen.

However, everything I need seems to function despite these errors. My guess is that dataeng takes a while to start because it's scanning all the hardware anyway. I don't need hotplug functionality, so the clutter in the logs is only cosmetic as far as I'm concerned.

Unfortunately, the package drops this file in /etc/ rather than /lib/, so there is no way to neatly override or disable it. My solution is to just replace it with an empty file:

1
true > /etc/udev/rules.d/dataeng-udev.rules