Fedora is a system of many individual pieces, each of which can affect the others. The kernel is usually fairly self-contained, or at least it tries to be. This is a story of two issues triggered by two different kernel changes that show the kernel is not an entity unto itself.
Major kernel version upgrades usually bring out issues and 4.4 was no different. A user filed a bug about a change in behavior: disk partitions on the hard disk were now being mounted automatically. This behavior was unwanted as those partitions previously required root to mount. The kernel is responsible for setting up device nodes but ultimately userspace is responsible for making the system calls to actually mount filesystems past a certain point. The system logs were clearly showing udisksd was making the request to mount those partitions. This really didn't look like a kernel issue so I suggested some other userspace setting had changed the automounting options. The reporter downgraded other userspace packages that had been installed at the same time but this showed no change. Installing just the kernel was enough to change the behavior so something in the kernel must have changed.
Debugging often involves working backwards to figure out how we got somewhere. Working backwards here:
- Partitions are being mounted
- The logs are showing the mount requests coming from udisksd
- udisksd is triggering the mount requests
- udisksd uses ????? to decide what to mount
I bounced this problem off of my teammates and they suggested the partition
options could be coming up differently based on the kernel. So what partition
options was udisks reading to figure out what to mount? The udisks
documentation wasn't very useful. It gave great examples of how to change the
automounting behavior but it didn't answer the question of how it sets the
defaults. I eventually found the
udisksctl dump command which shows what
state udisks has for all the partitions it knows about. This was really
insightful: in the non-working kernel udisks was marking the parititions as
automount and removable which explained the behavior. It didn't explain why
it was making those decisions though. Looking at the code is my preferred
method for understanding some behaviors. The udisks code was less enlightening
than I would have liked. All I could tell was that some call was returning
that the disk was removable. On a whim, I decided to look through the kernel
commit history and found a commit
which matched the exact behavior and a revert confirmed this was the breaking
patch. The ultimate issue was that marking the drive as hotplug capable was
enough to get the drive marked as removable. The original patch author
provided a fix
to not mark hotplug ports as removable.
During the course of discussion on bugzilla, a second person was reporting
the same issue. The fix did not work for this reporter though so we started
a different bug for tracking. In this case, the behavior and bug was
different: the kernel would boot but eventually dracut would spit out a message
that it can't find the lvm partitions. A good starting point for issues where
root partitions won't mount is to drop into the
This lets you run shell commands to collect system information. Ultimately
though someone else reported the same issue in parallel and found the issue
before any debugging took place. In 4.4, the mpt2sas driver was replaced with
the mpt3sas driver. The kernel has alias to take care of loading mpt3sas when
modprobe mpt2sas happens. The mpt3sas module needs to be present for this
to work though. Each time a new kernel is installed, dracut uses the set of
existing loaded modules to figure out what to put in the initramfs.
(Live CDs have a huge number of kernel modules in the initramfs to be able
to run on almost any platform and give a base for installing the kernel).
dracut didn't detect the name change so the mpt3sas module wasn't
present in the initramfs, preventing the file system from being mounted. The
workaround is fairly straight forward: put the mpt3sas module into the
initramfs. Once it is in place, dracut will correctly add it to all future
initramfs that are generated. Ultimately though, dracut needs to properly
follow module aliases for modules which are no longer present.
- The kernel is not always right. Kernel changes need to be aware of userspace assumptions and not break them. The kernel community usually tries very hard not to introduce regressions but they do sometimes happen.
- The kernel is not alway wrong. Sometimes a kernel change can expose a limitation in another program which needs to be fixed.
- Knowing debugging techniques outside the kernel is necessary for kernel developers, or at least Fedora kernel maintainers.
- Sometimes you get really really lucky in finding an answer to your problem
- Early testing of updates in bodhi and good bug reporting helps to find issues fast and get them fixed quicker. Both of these issues were reported, and in the once case fixed, by the 2nd stable release for 4.4.