The opinion nobody asked for

| categories: hottakes

Tired and cranky is probably not the best time to write a blog post but here it goes anyway. The business world seems to love to talk about 'authenticity' and 'bringing your whole self' to work. That applies to what you talk about, especially if you are a Woman in Tech(tm). If you are passionate about talking about diversity and those issues because it's part of who you are, go for it. If you'd rather not talk about those issues because it's not who you are, that's okay. Most important is to know why you are making your choices. It's okay to change your mind too.

None of this is an excuse to not care at all though. The real trick is to figure out how you work best. Maybe you're better at talking to other privately. Maybe you boost up other people who do want to share their story. Maybe you spend your time organizing. There's lots of ways to make the tech industry better and most important is learning how to do it in the way that works for you. We can all do better.

There will always be hardware bugs

| categories: hottakes, fedora

By now everyone has seen the latest exploit, meltdown and spectre, complete with logos and full academic paper. The gist of this is that side channel attacks on CPUs are now actually plausible instead of mostly theoretical. LWN (subscribe!) has a good collection of posts about actual technical details and mitigations. Because this involves hardware and not just software, fixes get more complicated.

In my previous job, I worked on kernels for mobile phones. This involved working with new hardware. I love working with hardware but one thing you learn pretty quickly is that hardware will have bugs. Sometimes the hardware team has already found them and will give a workaround. Other times you spend weeks chasing weird crashes and going back and forth with the hardware team. One of the challenging issues when working across teams in any area is communicating your domain expertise and listening to others expertise. There can be a lot of "well how about we just..." and talking across each other. Once upon a time, some hardware was not working the way we expected and we were talking to the hardware team. They were having trouble reproducing the behavior seen on our complex Android stack so we were running a series of experiments on our setups. Much of the actual work was figuring out how to take the requests from the hardware team and translate them into something reasonable for the kernel (e.g. where does "after each TLB flush" apply). Sometimes the experiments weren't actually feasible due to how the kernel was written.

If you are lucky (or unlucky depending on your view), you may find a hardware bug. The question then comes what to do. There may, again, be back and forth about what's actually an acceptable workaround. "Just run this sequence of code sometimes" may sound simple to the hardware team but might be impractical to actually implement in the kernel. The performance penalties can be high if part of the microarchitecture needs to be turned off. Sometimes the answer turns out to be "pretty please don't run this sequence of code which should never get generated by a reasonable compiler". Obviously, if an issue has security implications you may need to just take a performance hit but not implementing a workaround can be a valid decision.

Part of the discussion around all this has been a call for more open source hardware. This is absolutely a worthwhile goal. Most processors support adjusting various microarchitecture features. This is mostly for verification purposes but it's also useful if there's a need to disable a feature such as a prefetcher or branch predictor. The microarchitecture is usually considered proprietary and as such it's next to impossible to figure out how to make changes without consultation from the hardware team. So an open source hardware design would allow for better insight into the microarchitecture. What most people miss about open hardware is that you still have all the problems of hardware. Unless you're running on an FPGA, you can't just drop in a new hardware revision immediately. You're still going to have to implement software workarounds. The value of open hardware comes from freedom of licensing but not freedom from bugs.

Calling all this an "Intelocolypse" is deeply unfair as basically all modern processors from multiple vendors were affected here. It's a fundamental flaw in most implementations. It's certainly possible for each vendor/architecture to give a workaround but because of the severity here, there are proposals to fix this in generic kernel code. As has been mentioned though, many of the fixes are still under review so we'll have to see what happens. A big shout out to all the hardware and software developers who spent time coming up with proposals.

Open Source Infrastructure and the kernel

| categories: hottakes, fedora

(Given I'm talking about the kernel ecosystem and corporations, this is a reminder that these thoughts are my own)

Nadia Eghbal recently published an excellent report about funding and maintaining open source infrastructure. It's long but very easy to read. There's a nice series of selections available if you don't have time to read all of it (but I recommend you do!). The basic premise of the paper is that many key open source projects are under funded and maintained. It's easy to have projects fall through the cracks. Most of the focus was on various userspace projects. The kernel was briefly mentioned as a success story. Comparatively speaking, the kernel is well funded and maintained. I'm going to expand on some of the good things and bad things about infrastructure and the kernel ecosystem.

LWN1 always puts out a report for each development cycle about who contributes to the kernel. The results are pretty stable at this point. Employees of companies are doing most of the contributions to the kernel. This is a good thing. Companies see the value in Linux and want to pay people to make it better. They don't do this out of some altruistic love for software though (at least not most of them). These corporations have a product they are trying to deliver and paying people to deliver code to the upstream community is part of their strategy. The important point is 'strategy'. The terrible secret of space: most kernel developers are going to be advocating for something their employer wants. This is (usually) not some gigantic conspiracy that ruins everything. Most kernel developers who regularly work with the community are smart enough to not advocate for bad ideas just because an employer wants them. Advocating for garbage is a great way to get people to stop paying attention to you, which defeats the point of working with the community.

Where the kernel is unique (and why it succeeds) is the number of employers willing to sponsor maintainers and not just developers. Kernel maintainers are highly valued; LinuxCon this year is having office hours with kernel maintainers. Employers love being able to say "we employ kernel maintainers of X". It helps to build a reputation as a company where other kernel developers want to be. Employers sometimes think that having maintainers on staff will give them an advantage; "Oh boy, they can review my code before it goes out". This may not be true: most maintainers will advocate for review and discussion of all patches in public. Companies often get the advantage of being forced to follow best community practices when they hire maintainers, which may not be the advantage they expected or wanted. It makes them better open source contributors though.

The Linux kernel is an old project compared to most of the projects discussed in the report. The maintainers and developers have spent years figuring out what actually works for successful support. Groups like the Linux Foundation, Linaro and smaller consulting companies are designed to help companies with their contributions and become good open source participants. This is step #0 to getting sustainable open source: companies have to understand how to participate in the community and feel like they are part of the community. They will not see the value in funding if they aren't participating. Worse, they lose perspective on why funding maintainers is important (It's about supporting community, not having leverage to tell someone what to do).

The funding and corporate hand holding of organizations like The Linux Foundations can raise questions about neutrality. Is the kernel becoming 'pay to play' where unless you are a company with money to throw at the Linux Foundation you won't get to have a say? I don't believe so. I think the kernel community will always be a community at heart, independent of any company. There are enough good people involved to keep things going in the right direction. That said, pretending that corporations are always going to act altruistically is naive. Capitalism is harsh and throwing money around is a good way to make things happen. My plan personally is to stay informed and be an active participant in the kernel community.

This leads into one of the big disadvantages of this setup: the growth of companies participating in the kernel makes it much harder for general community members to find a role. Recruiting new people is something that gets discussed in the kernel community as an open problem. Just coming in and saying "I'm new here, I want to do kernel development" doesn't often lead to long term contributors. The kernel community has gotten very good at helping people get their first patch accepted. The staging tree is full of drivers that needs easy fixups so first time contributors can practice their patch skills. Mos people struggle to figure out what to do after that first patch though. There is a huge gap between fixing style errors and making a more significant contribution. Most community members don't have the time to do significant mentoring of projects outside of structured programs. Companies have the advantage of having a nearly endless supply work and more experienced individuals readily available to help newcomers. It isn't always perfect but it's a much better starting point. Other people in the company can also help provide introductions and connections to get involved with projects. Is it impossible to do kernel development without full time employment? No, but it's much easier. Once you've gotten name recognition in the community it's much easier to work outside of a company. This is incredibly unfortunate for diversity efforts. The most under represented groups in open source are also the least likely to be employed by software companies. What you end up with is a community that's difficult to break into and fairly homogeneous. I'd love to see this change but there's no easy solution, partially because shouting "it's not a pipeline problem" hasn't gotten through yet. (It isn't a pipeline problem. It just isn't.)

The kernel provides some great examples to those looking to expand their infrastructure. Hopefully others can learn from the weaknesses as well.

  1. While we're talking about infrastructure, LWN is an excellent place to put your money. The reporting is top notch. Please subscribe and support.