Installing packages to a read-only filesystem
2017-02-12If you think it's at least a little bit insane to install packages (which involves writing files) to a read-only filesystem (to which files can't be written), you're absolutely correct, but I still had to do it. This all came about thanks to two incompatible decisions, neither one made by me.
The first is that the Dell OpenManage packages for Ubuntu use /opt/
as a dumping ground for all sorts of equipment, including binaries, libraries, configuration files, images, and run-time data.
The second is that /opt/
happened to be a read-only NFS share mounted from another host.
Preparation
The structure of the problem is straightforward.
We have two machines (S
and C
) running Ubuntu 16.04 LTS (with Linux kernel 4.4.0).
The server S
shares its /opt/
directory via NFS, and the client C
mounts it as its own /opt/
.
Both the share and the mount are read-only.
We wish to install the OpenManage packages (version 8.4.0) on C
, but these packages put files in /opt/dell/
and /opt/lsi/
.
To get started on the insanity, we need to get some formalities out of the way on C
.
We need to have
1 | deb https://linux.dell.com/repo/community/ubuntu xenial openmanage
|
in sources.list
and we need to run
1 | apt-key adv --keyserver pool.sks-keyservers.net --recv-key ...
|
to fetch the Dell signing key. Now we're ready to
1 | apt update
|
and do some installing!
Path to failure
Some good news to get our hopes up: the packages that we want to install are already present on S
, so the /opt/
directory on C
already contains all the necessary files for these packages.
If we convince dpkg
to not install any files to /opt/dell/
and /opt/lsi/
, then the end result should be just what we're after.
This is easily done by creating a file called /etc/dpkg/dpkg.cfg.d/dell_opt_exclude
with the contents
1 2 | path-exclude /opt/dell/*
path-exclude /opt/lsi/*
|
Now let's try to install a simple package without any other OpenManage dependencies (don't actually do this):
1 | apt install srvadmin-hapi
|
Everything goes well until
1 2 3 | Unpacking srvadmin-hapi (8.4.0-1) ...
dpkg: error processing archive /var/cache/apt/archives/srvadmin-hapi_8.4.0-1_amd64.deb (--unpack):
unable to securely remove '/opt/dell.dpkg-tmp': Read-only file system
|
Then we hit bug #838877, where dpkg
fails to handle EROFS
.
This is fixed in version 1.18.11, but the fix isn't currently available in Ubuntu LTS.
In hindsight, this was never going to work even without the bug, because packages aren't just collections of files.
They also have scripts that run at various times.
The OpenManage package scripts try to write to /opt/
, so at least a part of /opt/
must be mounted read/write.
Removing the broken package
If you're following along with the commands despite the warning, at this point you have a broken package. If you try to remove it with
1 | apt remove srvadmin-hapi
|
you will see
1 2 3 | dpkg: error processing package srvadmin-hapi (--remove):
package is in a very bad inconsistent state; you should
reinstall it before attempting a removal
|
We can't install it, so we try the obvious thing:
1 | dpkg --remove --force-remove-reinstreq srvadmin-hapi
|
This is usually enough to get rid of a broken package, but dpkg
tries to clean up after itself:
1 2 | dpkg: error processing package srvadmin-hapi (--remove):
unable to securely remove '/opt/dell/srvadmin/lib64/libdchesm.so.8.dpkg-tmp': Read-only file system
|
Since it can't (due to the bug), it gives up.
I'm not certain that this command would succeed without the bug either, since it's not possible for dpkg
to remove the package contents from /opt/
.
The easiest way I've found to get out of this situation is to manually remove all the unwanted files that are reported by
1 | dpkg -L srvadmin-hapi
|
and then edit /var/lib/dpkg/status
by hand to remove the srvadmin-hapi
section.
Overlaid by success
Another idea is to put bind mounts on /opt/dell/
and /opt/lsi/
to mask the existing package files without hiding everything else in /opt/
.
This would also allow the package scripts to write where they need to.
Sadly, since /opt/
would itself still be read-only, we would run into the same dpkg
bug as before.
We need a solution that lets us vandalize all of /opt/
while keeping the changes local to the machine and not masking any directories we don't want to modify.
That solution is OverlayFS, and it's surprisingly easy to use.
There's an excellent introduction in the kernel documentation.
The first step is to create some local directories:
1 2 | mkdir /root/opt_overlay
mkdir /root/opt_overlay_work
|
These don't have to be in /root/
, or even in the same directory, but they do have to be on the same filesystem.
The work directory is necessary to be able to write to the overlay.
Since we already know the directories we want to mask in the overlay, we can prepare whiteouts:
1 2 | mknod /root/opt_overlay/dell c 0 0
mknod /root/opt_overlay/lsi c 0 0
|
Now we are ready to mount:
1 | mount -t overlay -o lowerdir=/opt,upperdir=/root/opt_overlay,workdir=/root/opt_overlay_work overlay /opt
|
Before the mount:
1 2 3 4 5 | # findmnt -R /opt
TARGET SOURCE FSTYPE OPTIONS
/opt S:/opt nfs4 ro,relatime,...
# ls /opt/dell
srvadmin toolkit
|
After the mount:
1 2 3 4 5 6 | # findmnt -R /opt
TARGET SOURCE FSTYPE OPTIONS
/opt S:/opt nfs4 ro,relatime,...
└─/opt overlay overlay rw,relatime,lowerdir=/opt,upperdir=/root/opt_overlay,workdir=/root/opt_overlay_work
# ls /opt/dell
ls: cannot access '/opt/dell': No such file or directory
|
Now we can install the package we wanted all along:
1 2 3 | apt install srvadmin-storageservices
systemctl start dataeng
. /etc/profile.d/srvadmin-path.sh
|
It works!
1 2 3 4 5 6 7 | # omreport system
Health
SEVERITY : COMPONENT
Ok : Main System Chassis
For further help, type the command followed by -?
|
Stuck in a bind
There are at least two reasons I can't use OverlayFS as a permanent solution.
One is that OverlayFS is incompatible with the root_squash
NFS option, making it impossible to open directories in the overlay that are not world-readable.
Without the overlay:
1 2 3 4 5 6 | $ stat -c %A /opt/good ; ls /opt/good
drwxr-xr--
file1 file2 file3 file4
$ stat -c %A /opt/bad ; ls /opt/bad
drwxr-x---
file1 file2 file3 file4
|
With the overlay:
1 2 3 4 5 6 | $ stat -c %A /opt/good ; ls /opt/good
drwxr-xr--
file1 file2 file3 file4
$ stat -c %A /opt/bad ; ls /opt/bad
drwxr-x---
ls: reading directory '/opt/bad': Permission denied
|
The other is more fundamental. The documentation says that the underlying filesystem must not be modified when the overlay is mounted, but I need to be able to alter the contents of the NFS share. Although this is guaranteed to not result in a catastrophic event (and I suspect OverlayFS will do the Right Thing anyway), it's best to avoid undefined behaviour.
It may not be obvious why we even need to keep up the facade now that the packages are installed.
The dataeng
service puts PID files in /opt/dell/srvadmin/var/run/openmanage/
and IPC sockets in /opt/dell/srvadmin/var/lib/openmanage/
.
I'm not sure why it doesn't use the real /var/
, but I figure it's best to let it write where it needs to.
Since we've already gone through the arduous journey of installing the packages, there's nothing stopping us from using bind mounts. First, a couple of preparatory steps to undo some of the things we've done:
1 2 | systemctl stop dataeng
umount /opt
|
We're also free to remove /root/opt_overlay_work/
.
Using the existing directories from the overlay, making the bind mounts is as simple as
1 2 | mount -o bind /root/opt_overlay/dell /opt/dell
mount -o bind /root/opt_overlay/lsi /opt/lsi
|
Now we have
1 2 3 4 5 | # findmnt -R /opt
TARGET SOURCE FSTYPE OPTIONS
/opt S:/opt nfs4 ro,relatime,...
├─/opt/dell /dev/mapper/...[/root/opt_overlay/dell] ext4 rw,relatime,...
└─/opt/lsi /dev/mapper/...[/root/opt_overlay/lsi] ext4 rw,relatime,...
|
We can start up dataeng
and use omreport
once again.
Making it permanent
The above setup won't survive a reboot.
To make it permanent, we're going to create two systemd mount units.
The first is /etc/systemd/system/opt-dell.mount
, with the following contents:
1 2 3 4 5 6 7 8 9 10 11 | [Unit]
Requires=local-fs.target opt.mount
After=local-fs.target opt.mount
[Mount]
What=/root/opt_overlay/dell
Where=/opt/dell
Options=_netdev,bind
[Install]
WantedBy=remote-fs.target
|
There's not much happening here:
[Unit]
: We make sure that all the components of the overlay are ready.[Mount]
: Exactly what you'd expect, but we've added the_netdev
option, which tells systemd that the unit should be part ofremote-fs.target
and notlocal-fs.target
.[Install]
: We inform systemd that we want this mounted with all the other remote filesystems.
The other file is the same, but with lsi
instead of dell
where appropriate.
All that remains is to install both units with
1 | systemctl enable opt-dell.mount opt-lsi.mount
|
which puts symlinks in /etc/systemd/system/remote-fs.target.wants/
.
Then /opt/dell/
and /opt/lsi/
are bound on top of the NFS share at boot.
Tweaking the services
We would like to make sure that the OpenManage daemons don't start until the bind mounts are ready. To this end, we put
1 2 3 | [Unit]
Requires=remote-fs.target
After=remote-fs.target
|
in the following files:
/etc/systemd/system/dataeng.service.d/override.conf
/etc/systemd/system/dsm_sa_ipmi.service.d/override.conf
/etc/systemd/system/instsvcdrv.service.d/override.conf
This may be done conveniently with
1 | systemctl edit dataeng dsm_sa_ipmi instsvcdrv
|
After running
1 | systemctl enable dataeng
|
we see that dataeng
is started at boot after all the mounts.
It's likely that other OpenManage packages will include other daemons. They may be found with
1 2 3 | for p in $(dpkg -l | grep -f <(grep '^Package: ' /var/lib/apt/lists/linux.dell.com_repo_community_ubuntu_dists_xenial_openmanage_binary-amd64_Packages | cut -d' ' -f2-) | awk '{print $2}'); do
dpkg -L $p | grep /etc/init.d/
done
|
Dropping udev rules
During boot, we now see complaints from systemd-udevd
:
1 2 | Process '/bin/sh -c '[ -x /opt/dell/srvadmin/sbin/dataeng.hotplug ] && /opt/dell/srvadmin/sbin/dataeng.hotplug pci'' failed with exit code 1.
Process '/bin/sh -c '[ -x /opt/dell/srvadmin/sbin/dataeng.hotplug ] && /opt/dell/srvadmin/sbin/dataeng.hotplug usb'' failed with exit code 1.
|
These arise from the configuration in /etc/udev/rules.d/dataeng-udev.rules
.
The rules can't function until /opt/
is mounted, which doesn't happen until later in the boot process.
I don't believe it's possible to delay these rules until after all the mounts happen.
However, everything I need seems to function despite these errors.
My guess is that dataeng
takes a while to start because it's scanning all the hardware anyway.
I don't need hotplug functionality, so the clutter in the logs is only cosmetic as far as I'm concerned.
Unfortunately, the package drops this file in /etc/
rather than /lib/
, so there is no way to neatly override or disable it.
My solution is to just replace it with an empty file:
1 | true > /etc/udev/rules.d/dataeng-udev.rules
|