Maintainerati

| categories: fedora

I spent last Wednesday hanging out in San Francisco for the first annual maintainerati event. The idea was that there are a lot of open source maintainers out there but events are usually separated by technology areas. Javascript framework maintainers may never meet programming language maintainers even if their problems are similar. The idea with this event was to give open source maintainers a chance to vent and problem solve with others.

The event was structured as an 'unconference'. I describe it as a slightly more structured hallway track. We started the morning doing 'dot voting' on topics people wanted to talk about and then broke into groups to discuss the topics that got the highest vote. I chose to go for the discussion about recruiting newcomers and maintainers. We started with some discussion about what is a contribution and pros and cons of structuring the contribution process and eventually getting committer rights. There's no hard and fast rule about when people can/should get commit rights and it mostly comes down to relationships; you need to build relationships with existing maintainers and existing maintainers need to build relationships and mentor new committers. This let to quite a bit of discussion about free vs. paid and company vs. non-company contributors. It's a lot easier to build relationships if you can set up a meeting with a maintainer in your company but that doesn't work for outside contributors. There's also the question of trying to recruit volunteers for your sponsored project. Doing work 'for exposure' is problematic and exploitative yet open source has this idea of doing work for the inherent joy of open source and learning. Promoting unpaid contributions needs to be done very carefully if it is done at all. We ended up running out of time and I think the discussion could have certainly gone longer.

There was a second session in the afternoon about problematic communities. This one is unfortunately near and dear to my heart. We started out defining what makes a community toxic. A big point was that bad behavior prevents the community from making progress. Many of the discussion points were not just open source but other communities that tend to have overlap. Code of conducts are a necessity to make dealing with toxic behavior possible. There was some discussion about how specific these guidelines should be, and interestingly it was pointed out that having slightly less specific guidelines (but not too much) may help to avoid people trying to purposely hang out at the edge of acceptable. If your larger community is problematic, it can be helpful to work on making a smaller subset welcoming and let that influence the larger group. I appreciated everyone who took the time to contribute in the discussion.

Outside structured conversations, I spent time talking about empathy. Several attendees either were or had been in first line customer support positions. To succeed in this type of work, you need to have (or quickly build) empathy skills to keep customers satisfied. Developers are not well known for having large amounts of empathy skills. I'm guilty of this myself; empathy without emotionally draining myself is something I'm constantly working on. Figuring out how to teach empathy skills to others is a challenge. One of the ideas that came up was the need to be outside your comfort bubble. Travel and moving were a common way people cited to force yourself to have new experiences. Traditional developer mind set also tends to be very black and white (hi guilty here too). Most important was the desire to keep improving this skill and not write it off as unnecessary.

There were plenty of other conversations I'm sure I've forgotten about. Notes are available on the github and will be added as people get around to it. I really hope to see this conference happen again. It's filling a space to have important conversations about non-technical topics that tend to get sidelined elsewhere. I met so many cool people and left with a lot to think about. My biggest thanks to the organizers.


DevConf 2017

| categories: fedora

Last week was DevConf in Brno, Czech Republic. I was there!

Thorsten Leemhuis gave a talk about What's up in Kernel Land. This was a general overview about new features and patches that are coming into the Linux kernel targeted at non-kernel developers. I was not the target audience but the talk was fantastic. It was easy to follow and gave a good picture of what the kernel community is doing. I appreciate when non-kernel developers give talks about the kernel since kernel developers can be a bit myopic in our topics (myself included).

Peter Jones talked about Firmware updates on Linux. So long as there has been firmware, there have been firmware bugs. Fixing those bugs involves giving updates. Recent updates to the UEFI firmware specification make it plausible to give firmware updates directly from Linux. This talk described the work to make these Firmware updates actually happen and even included a live demo that did not brick a machine. Having a reliable way to get firwmare updates without needing to boot into Windows is very helpful for improving the overall Linux experience.

There was a talk about performance tuning of RHEL. Most of this was about various userspace knobs that can be set depending on your workload to control aspects like NUMA affinity and page writeback. I've worked enough in the memory management area to know how must of the underlying technology works. This talk was useful in showing how to practically make use of these features to improve your system. Having more knowledge available to help with performance issues is always beneficial as a developer.

In a similar theme, there was a talk about solving memory performance problems. This primarily focused on using the perf tool to find and fix cache performance problems. The primary example was variables that are sharing a single cache line. Intel has added new hardware features to make it possible to aggregate cache data and figure out where cache conflicts are occurring. Cache issues can be a pain to solve so it's great to see perf making this a little easier. Hopefully other architectures will get similar hardware support and integration.

Paul More gave a talk on Lessons from the CII Badge Program. The Core Infrastructure Initiative (CII) came out of the Heartbleed security issue to make the software we rely on more secure. As part of this, the CII has a 'badge' program of best practices for open source software. These practices are less "don't write buffer overflows" and more "are you using source control". The example in the talk was how well libseccomp complied with the best practices. Overall, libseccomp had good compliance with a few areas to improve. None of the requirements are overly burdensome but they are important for project sustainability.

Peter Robinson gave a talk about architecting a secure IoT platform using Fedora. This was mostly an update on his work driving IoT for Fedora. The goal is to pick several devices and use them as a reference platform for IoT gateways (machine to hook up to tiny IoT devices). Part of this work is to make sure that Fedora doesn't become the punchline of any IoT jokes so there was a big emphasis on security and what work needs to happen there. I look forward to seeing Fedora used in this buzzword filled space.

There's a new effort underway to make Fedora run better on laptops within Red Hat and this talk described that effort. The focus is going to be on enabling and testing specific laptops to make sure they work well with Fedora. A big reason bugs go unfixed is because nobody has the time to work on them. Having dedicated people to work on this problem is a huge step in the right direction. Part of this work is also things like making the nVidia driver more usable. Philosophically, I have misgivings about the nv driver but from a usability perspective this is a great effort. This was a nice tie in to a talk about Fedora Workstation on the last day (unfortunately the talk didn't get recorded).

Rawhide recently brought in gcc7 and the kernel hit a bit of a snag so I was eager to see the talk about cool new gcc7 features. The talk started out explaining -Wimplicit-fallthrough which is designed to catch fall through in switch cases. Of course, sometimes you actually want this behavior so the 'solution' is to look for comments like /* fallthrough */. I am not sold on this technique but I can take comfort in the fact that there is a very long thread somewhere debating this. I don't know where it is, but I am sufficiently convinced that it exists and I don't want to read it. Most of the rest of the talk was describing other warnings and features, none of which were quite as controversial (although the talk did digress into C++ specification discussion for a bit). It's always good to see the compiler improving to make programming better.

There does exist a video of my talk about fresh kernels with Fedora. This was an overview of how the upstream kernel release process works and how that drivers a lot of the rebase decisions we make in Fedora. Overall, I was pretty satisfied with how the talk went. I got some good questions and people seemed engaged. I might see about tweaking this talk to give elsewhere.

This was my first time at DevConf and in Brno and I had a fantastic time. There was some hallway track and lots of good times talking with people. Hopefully I'll get a chance to go back again.


That's not what I wanted Linux

| categories: complaining, fedora

Once upon a time, I was an intern at a company doing embedded Linux. This was a pretty good internship for a student. A lot of my work involved making builds of open source packages and fixing them when they failed in unusual embedded environments. One time, I was working in a new environment and halfway through a build of some package I got what was a cryptic message to me:

no: command not found

As a beginning developer, I was really confused by this message. It's saying "no the command isn't found". But what command? I don't remember much of how I debugged this but I eventually went through the build logs and came across

checking for perl ...no

The autoconf script was set up incorrectly and set PERL=no instead of turning off perl or erroring out in the config stage. This was fixable by adding perl to my build environment. Alas, I don't think I fixed the autoconf.

Fast forward to the present day. Someone was reporting a build failure when rebuilding the rawhide kernel locally. I was seeing the same issue on my system:

install: cannot create directory
'/home/labbott/rpmbuild/BUILDROOT/kernel-4.10.0-0.rc2.git3.2.local.fc26.x86_64/usr/lib64':
Not a directory

Checking the build tree, /usr/lib64/ was indeed not a directory. It was a binary file. Disassembling the binary file showed it was part of perf and seemed to be related to java. The build logs had this line.

install libperf-jvmti.so '/home/labbott/rpmbuild/BUILDROOT/kernel-4.10.0-0.rc2.git3.1.fc26.x86_64/usr/lib64'

install here behaves in a very *NIX manner. Without any other options, if lib64 exists as a directory, libperf-jvmti.so gets copied to the directory. This is what we expect to happen. If lib64 does not exist, the .so gets copied as a file named lib64. This is what was happening here. The fix is simple, check and create the directory exists before running the command. You could even add a trailing slash to ensure it's actually a directory.

So what is the moral of these stories? Laura enjoys complaining about Linux Your failure modes can produce really non-obvious behaviors if they don't actually fail. Error checking can be hard and Linux is cold and unfeeling when you screw up. Bugs will always happen, so review your code carefully.


Chatty kernel logs

| categories: fedora

Most people don't care about the kernel until it breaks or they think it is broken. When this happens, usually the first place people look is the kernel logs by using dmesg or journalctl -k. This dumps the output of the in- kernel ringbuffer. The messages from the kernel ring buffer mostly come from the kernel itself calling printk. The ring buffer can't hold an infinite amount of data, and even if it could more information isn't necessarily better. In general, the kernel community tries to limit kernel prints to error messages or limited probe information. Under normal operation the kernel should be neither seen nor heard. The kernel doesn't always match those guidelines though and not every kernel message is an urgent problem to be fixed.

The kernel starts dumping information almost immediately after the bootloader passes it control. Very early information is designed to give an idea what kind of kernel is running and what kind of system it is running on. This may include dumping out CPU features and what areas of RAM were found by the kernel. As the kernel continues booting and initializing, the printed messages get more selective. Drivers may print out only hardware information or nothing at all. The latter is preferred in most cases.

I've had many arguments on both sides about whether a driver should be printing something. There's usually a lot of "but my driver reaaaallly needs this". The preferred solution to this problem is usually to adjust the log level the messages are printed at. The kernel provides several log levels to filter out appropriate messages. Most drivers will make use of KERN_ERR KERN_WARN and KERN_INFO. These have the meaning you would expect: true errors, just gentle warning and some useful information. KERN_DEBUG should be used to provide more verbose debugging/tracing on an as needed basis. The kernel option CONFIG_DYNAMIC_DEBUG can be used to enable and disable individual pr_debug messages at runtime. This option is enabled on Fedora kernels.

Even with the different levels of kernel messages available, it may not always be clear how important a message actually is. It's very common to get Fedora kernel bug reports of "dmesg had message X". If there is nothing else going wrong on the system, the bug may either get closed as NOTABUG or placed at low priority. A common ongoing complaint is with firmware. Drivers may look for multiple firmware versions, starting with the newest. If a particular firmware isn't available, the firmware layer itself will spit out an error even if a matching version may eventually be found. Sometimes the kernel driver may not match the hardware exactly and the driver will choose to be 'helpful' and indicate there may be problems. While it is true that hardware which is not 100% compliant may cause issues, messages like this are unhelpful without a message that this isn't likely to be fixed in the kernel. Even with statements like "the kernel is fine", it can be confusing and difficult to explain this to users.

The kernel logs are a vital piece of information for reporting problems. Not all messages in the logs are an indication of a problem that needs a kernel fix. It's still important to report bugs so we know what bothers people. It may be possible to report issues to upstream driver owners to report better error messages and make the kernel logs more useful for everyone.


virt to phys and back again

| categories: fedora

I've been working on CONFIG_DEBUG_VIRTUAL support for arm64. This is designed to catch bad uses of virt_to_phys1 on non-linear addresses. Translating between virtual and physical addresses is very architecture specific. The kernel expects each architecture to define these appropriately. At the simplest level, supporting a debug check is a matter of adding the appropriate #ifdef and calling a new function. Architectures are rarely that simple though.

I've talked before about kmalloc and vmalloc. kmalloc memory is linearly mapped (can call virt_to_phys) whereas vmalloc memory is not. The kernel image (everything built into a vmlinux for the purposes of this discussion) is yet another2 range of memory. This is mapped linearly but may not be the same as kmalloc memory. On arm64, the kernel image can be placed anywhere in physical memory. Virtually, the kernel image is placed at the end of module space/start of vmalloc space. The virt_to_phys translation currently covers both of these ranges:

#define __virt_to_phys(x) ({                                           \
    phys_addr_t __x = (phys_addr_t)(x);                            \
    __x & BIT(VA_BITS - 1) ? (__x & ~PAGE_OFFSET) + PHYS_OFFSET :  \
                         (__x - kimage_voffset); })

The choice of layout means that bit VA_BITS - 1 in the virtual address space can be used to distinguish between 'regular' linear addresses and kernel image virtual addresses.

Enforcing the debugging ranges gets much easier if this can be broken up into two different parts, one for the standard linear range and one for the kernel image. Fortunately, the kernel already has a macro __pa_symbol for symbols so it's easy to define and use that. Part of the work for implementing CONFIG_DEBUG_VIRTUAL now becomes converting calls of __pa(some_kernel_symbol) to __pa_symbol(some_kernel_symbol). This gets more difficult when the __pa call is embedded in another function. A good example of this is the p*d_populate functions. These setup page tables and call __pa but need to work with both kernel symbols and regular linear memory. So what to do?

The phys_to_virt function is defined like so

#define __phys_to_virt(x)       ((unsigned long)((x) - PHYS_OFFSET) | PAGE_OFFSET)

Notice how this is the reverse of the kernel linear map in __virt_to_phys. The kernel image is actually an alias of part of the regular linear address. Alias here means that two different virtual addresses map to the same physical address. The easiest way to work with functions that need to call __pa is to translate kernel symbol addresses to their linear aliases. Since __va will always return an address in the linear range, we can pass the physical address of a kernel image symbol and translate it back:

#define lm_alias(x)     __va(__pa_symbol(x))

This essentially does a virt -> phys -> virt translation but the starting and ending virtual addresses are not the same. The linear alias of the kernel image can be passed to functions that use __pa without a problem. There's still some limitations on the alias. __va uses PHYS_OFFSET

/* PHYS_OFFSET - the physical address of the start of memory. */
#define PHYS_OFFSET             ({ VM_BUG_ON(memstart_addr & 1); memstart_addr; })

There's a nice VM_BUG_ON to bring down the system if PHYS_OFFSET is used too early. This means that the alias trick can't be used until after a certain point in bootup. Before that, __pa_symbol is the only option to get physical addresses of kernel symbols.

The CONFIG_DEBUG_VIRTUAL patches for arm64 are still under review but most of the major issues have been worked out. Hopefully they will be merged soon.


  1. For purposes of this discussion, virt_to_phys includes several functions that while not identical are similar enough to assume they do the same thing. __pa and __virt_to_phys are included in this category. 

  2. If your distro supports it, try running cat /sys/kernel/debug/kernel_page_tables to see the entire virtual address space. 


Next Page ยป