This week Intel publicly disclosed a group of processor vulnerabilities known as Microarchitectural Data Sampling (MDS), also referred to as “ZombieLoad”. MDS affects systems that host virtual machines from varying security domains and/or that the system owner does not fully trust, which includes Linode’s infrastructure and Linodes themselves. This guide has additional detailed information on these vulnerabilities as well as their mitigation.
We’ve started mitigation efforts and anticipate full mitigation of our fleet in the coming weeks. These mitigation efforts may require interruption to your running systems, but we will clearly communicate any scheduled maintenance or coordination required by our customers via Support ticket.
To address these vulnerabilities on your end, we’ve released a new kernel (5.1.2) with mitigations in place, so make sure you select this kernel in your Linode’s configuration profile, then reboot. If you are using a distribution-supplied kernel, you will need to upgrade your kernel accordingly. As always, you should also ensure your Linode is up to date and secured.
We’ll keep you updated here in the coming weeks as we proceed with our mitigation efforts.
Comments (13)
Is the “Latest 64-bit” Kernel going to be sufficient or do all servers need to be using the 5.x kernel?
Hey, Jim. At the moment, the “Latest 64-bit” kernel is not patched for MDS — we’ve delayed changes due to a kernel bug involving inaccurate ‘uptime’ reports. We instead recommend booting into 5.1.2-x86_64-linode124 for 64-bit systems, or 5.1.2-x86-linode144 for 32-bit systems. Once the kernel bug has been completely resolved, you could then switch back to the “Latest”.
Linode offer two 5.1.2 kernels. One is 5.1.2-x86-linode144 and caused kernel panic on Debian 9. 5.1.2-x86_64-linode124 works. Thanks to support for guiding me to this point. It should be added to the post here.
Hi there, John. The 5.1.2-x86-linode144 kernel is designed for 32-bit systems. It will not work properly on 64-bit systems. For 64-bit systems you will want to use the 5.1.2-x86_64-linode124 kernel. For all 64-bit systems you will want to look for the kernels that include “_64” in the title.
Any particular reason for the “latest-kernel” to stuck at 4.18.6 ?
Update?
Great and useful post, Thanks for sharing
Bookmarking the blog for future reference.
What is the expected performance impact of the mitigation?
Viktor: We don’t anticipate performance impacts coming from ZombieLoad mitigation. However, disabling HyperThreading as part of our mitigation strategy for speculative operation vulnerabilities poses a clear challenge regarding performance, so accordingly we’ve been working to minimize its impact on our platform. (You can find more discussion on HyperThreading here.)
If you’re seeing degraded performance on any of your Linodes, please reach out to us so we can help investigate and find a solution.
Can we use the new kernel for Debian 7 machines?
Hi Mike – Yes. No issues with our latest kernel and Debian 7. I’d recommend reading through our Reboot Survival guide, though, if it’s been awhile since you’ve needed to reboot some of your machines:
> https://www.linode.com/docs/guides/reboot-survival-guide/
If you’ve regularly kept your server up to date, issued reboots periodically and have always used the latest kernel, you’re more likely to be okay if you continue to use it. Having a restore plan in place in the event anything goes wrong is always recommended, though, since there are a lot of variables at play.
Other posts mentioned a live-migration capability now. That doesn’t work for host updates?
Are you moving towards the only needed reboots to be for upgrade/downgrades and updating our kernel?
Hi Avi – Technically speaking, for small scale host updates, live migrations would work. Though since CPU vulnerability mitigation is a much larger effort, it’s logistically more efficient to cold migrate servers to patched hosts, or apply the needed patches during the maintenance window. As for your second question:
> Are you moving towards the only needed reboots to be for upgrade/downgrades and updating our kernel?
Yes, though this is a long term effort, and I don’t have anything immediate to share regarding an ETA.