DO Ideas 2

Give option to use the Droplet's own bootloader!

At present it appears (at least for Ubuntu) that the KVM instance for each Droplet is invoked with a kernel & initrd that are external to the Droplet's storage. This means that security updates/upgrades to the kernel from with the Droplet (eg *distribution security updates*) are ignored and the Droplet continues to boot with the kernel & initrd from when the Digital Ocean image was created.

  • Chris Wilson
  • Sep 11 2018
  • Shipped
  • Sep 11, 2018

    Admin Response

    Hi, In my last update, I noted that certain Droplets in fact already used their own bootloaders and "internally managed" kernels. I'm glad to say that these changes have been backported to other distros as well. All new Droplets now use their own bootloader. We've also provided an upgrade path for older Droplets. In order to use the "internally managed" kernels without rebuilding your Droplet, you can now set the "DigitalOcean GrubLoader" from your Droplet's kernel menu. You can read more about how this works in this tutorial: https://www.digitalocean.com/community/tutorials/how-to-update-a-digitalocean-server-s-kernel If you have any questions about the changes, please open a support ticket so the team can clarify any concerns you have. https://cloud.digitalocean.com/support/tickets/new Thanks!
  • Attach files
  • Jonathan Dieter commented
    September 11, 2018 19:59

    Just want to say thanks for finally fixing this. I've switched my droplets over to the DO GrubLoader.

  • Ananth Surya commented
    September 11, 2018 19:59

    Btw, I need this for ubuntu 14.04.3 (shouldn't matter I think)

  • Ananth Surya commented
    September 11, 2018 19:59

    I just need a specific kernel version. Ideal would be give that flexibility to us, the users... looks like grub is not used by the infra at all

  • Andy Sayler commented
    September 11, 2018 19:59

    Today's secuty announcement is yet another reason why relying on DO to provide kernals is bad for DO users (and by proxy, bad for DO): http://arstechnica.com/security/2016/01/linux-bug-imperils-tens-of-millions-of-pcs-servers-and-android-phones/.

    Ubuntu pushed out an updated kernel early this morning (http://www.ubuntu.com/usn/usn-2871-2/), but I'm still wating on DO to update their list of available kernels - meanwhile my DO VMs remain exposed...

  • Anonymous commented
    September 11, 2018 19:59

    New droplets running recent distributions use the simplfied in-server kernel management method. However, old droplets are stuck with the old method of managing kernels from the control panel, even if the user has upgraded the distribution to a recent version.

    It should be possible to switch to the in-server kernel management method.

  • Luka commented
    September 11, 2018 19:59

    Only thing you can do now is https://en.wikipedia.org/wiki/Kexec and that is what i did. running ok...

  • Ben commented
    September 11, 2018 19:59

    I'm surprised that DO hasn't said anything about progress on this issue - some of the latest images (e.g. the Ubuntu 15.04 image) DO implement internally-managed kernels. The bummer is that it's only good for new droplets. Converting already existing droplets means creating a new droplet and then moving content from the old droplet to the new one. AFAIK (unless it's changed since I found out about this), this has not yet been implemented in the Ubuntu 14.04 (LTS) image. I've got one more droplet to move over (my mail server), and I'll have all my droplets running on the internally-managed kernel.

  • Anonymous commented
    September 11, 2018 19:59

    Fedora 22 has now the feature of internally managed kernels. I wish we could extend that to other droplets including a migration plan for older ones like e.g. older Fedora that were upgraded to recent Fedora (newer or equal than 22).

  • Stef commented
    September 11, 2018 19:59

    Really disappointing… :(

  • mc0e commented
    September 11, 2018 19:59

    Seriously? 3 years? I've recently put a few of a client's servers on DO. No more.

  • Sean commented
    September 11, 2018 19:59

    No, on Fedora 21 you can delete the local kernel/initrd and it still boots...

  • md_5 commented
    September 11, 2018 19:59

    CoreOS, FreeBSD and Fedora 21 all use their own bootloaders.... now for this to just become standard. I'll never understand why it was done this way in the first place.

  • Anonymous commented
    September 11, 2018 19:59

    When I joined DO, I didn't realize that this was a limitation. Luckily, I only joined recently and haven't fully committed to DO just yet. If this doesn't get implemented soon, I'm afraid I may be forced to go elsewhere. That would be a shame because I've so far been very happy about everything else at DO.

  • Sean commented
    September 11, 2018 19:59

    I've finally left for another provider because of this issue. Digital Ocean was stuck on kernel 3.2.60-1+deb7u1 for over two months after Debian released newer versions with SECURITY UPDATES. They only added the new version after I noticed this and opened a support ticket. They didn't even know there were missing updates at the time.

    They are acting very irresponsibly by taking kernel patching upon themselves and then not doing their job. It concerns me that this is probably not the only thing they have neglected -- just the easiest one to see.

  • Gerald commented
    September 11, 2018 19:59
  • Gerald commented
    September 11, 2018 19:59

    How many more votes does Digital Ocean need to implement this?

  • Monkey Bonkey commented
    September 11, 2018 19:59

    I have never have an Arch kernel break on me. The situation here is much worse since we can't patch stuff even when we want to...

  • Anonymous commented
    September 11, 2018 19:59

    Because of this issue, and DO's unresponsiveness on it, I switched to RamNode and haven't looked back. Am very happy with their service!

    You can check out their offerings here http://www.ramnode.com, and if you're feeling kind you can use my referral link to visit their site (thanks very much if you do!): https://clientarea.ramnode.com/aff.php?aff=909

  • Luka commented
    September 11, 2018 19:59

    ETA on this is NEVER.... Same as for more space per droplet...

  • Justin Jett commented
    September 11, 2018 19:59

    1000 votes. What is the ETA for this?

  • d3zorg commented
    September 11, 2018 19:59

    much need. do it please :)

  • Max commented
    September 11, 2018 19:59

    How has this not been implemented yet? Once again, there are kernel updates that I can't apply because of the DO bootloader.

  • Sean commented
    September 11, 2018 19:59

    I just figured I'd mention, because of this problem, there is RIGHT NOW a security update that has been released for Debian that is impossible to install on a DO droplet without kexec trickery. When are you going to fix this???

  • Bas commented
    September 11, 2018 19:59

    We need this...

  • Martin Pescatore commented
    September 11, 2018 19:59

    DigitalOcean keep us updated on this please! Transparency is important!

  • Sean commented
    September 11, 2018 19:59

    The amount of time this issue has existed demonstrates that security is far from the top priority at DigitalOcean.

  • Anonymous commented
    September 11, 2018 19:59

    Please you must finish this improvement.

  • Anonymous commented
    September 11, 2018 19:59

    Where is the security here?

  • Juvioneo commented
    September 11, 2018 19:59

    PLEASE FINISH THIS!!!

  • Juvioneo commented
    September 11, 2018 19:59

    Yes, please

  • Avi Kivity commented
    September 11, 2018 19:59

    Seriously guys, you're using an external kernel with KVM?

  • Louis Matthijssen commented
    September 11, 2018 19:59

    Has been started more than 2 years ago now... any progress?

  • David Ford commented
    September 11, 2018 19:59

    @ WrongPriorities

    that's amusing. i use Arch on servers, remote servers. "wfm", or perhaps i'm simply more diligent in testing things before i deploy them ;) i've had more rh and debian things break on remote servers than on arch servers.

  • WrongPriorities commented
    September 11, 2018 19:59

    @ Mikael Lyngvig

    That's why I never recommend that people use Arch Linux on servers, especially remote servers.

  • Mikael Egevig commented
    September 11, 2018 19:59

    Be careful with the Arch script below; Arch Linux occasionally releases a broken kernel so that you risk ending up with a droplet that cannot access the Internet. In the end, I had to give up Arch Linux and reinstall Ubuntu on all my droplets and Linux boxes because of too many issues with broken Arch Linux kernels.

  • Bas commented
    September 11, 2018 19:59

    Common DO... We really need this.

  • Deni Bertović commented
    September 11, 2018 19:59

    Can we please get an update?

  • Glaudston Soares commented
    September 11, 2018 19:59

    Can we please get an update on this DigitalOcean?

  • Peter Waller commented
    September 11, 2018 19:59

    This problem just caused a waste of time here:

    https://github.com/dotcloud/docker/issues/4536#issuecomment-37205601

  • Alexandr commented
    September 11, 2018 19:59

    Can someone tell me the decision to run a virtual machine with a kernel that is on it? Is that possible?

  • Guruprasad commented
    September 11, 2018 19:59

    It is almost 2 years and still this remains a missing must-have feature.

  • Luka commented
    September 11, 2018 19:59

    how to use kexec on centos to boot other kernel (CloudLinux).

  • Eris Castrogivanni commented
    September 11, 2018 19:59

    Still waiting on this. The fact that it's not already done when it was "started" over a year ago is embarrassing. This should be a high-priority feature.

  • jkuan commented
    September 11, 2018 19:59

    Slightly less hacky way to use kexec on arch:
    pacman -S kexec-tools
    vim /etc/systemd/system/kexec-load-linux.service
    
======code======================
    
[Unit]
    Description=load linux kernel into the current kernel
    Documentation=man:kexec(8)
    DefaultDependencies=no
    Before=shutdown.target umount.target final.target

    [Service]
    Type=oneshot
    ExecStart=/usr/bin/kexec -l /boot/vmlinuz-linux --initrd=/boot/initramfs-linux.img --reuse-cmdline

    [Install]
    WantedBy=kexec-reboot.service

    ======end==code========================

    cp /usr/lib/systemd/system/kexec.target /etc/systemd/system/kexec-reboot.service
    replace the last line of that file with "WantedBy=sysinit.target"
    ======code======================
    ...
    [Install]
    WantedBy=sysinit.target
    ======end==code========================

    cd /etc/systemd/system
    systemctl enable kexec-load-linux.service
    systemctl enable kexec-reboot.service

    vim /etc/pacman.conf
    =======================================
    ...
    #IgnorePkg = linux
    ......
    ======================================
    Change line 'IgnorePkg = linux' - comment it with # (and save file!)
    pacman -Syu
    reboot

    Needless to say, you should probably take a snapshot before doing any of this.

  • Anonymous commented
    September 11, 2018 19:59

    OK, well, for now my solution is to switch hosts. Good thing I didn't get too far engrained here. Sorry to have to say good-bye DO, maybe we'll meet again. Wish you put as much effort into these fundamentals as you do into your fantastic design. I'll miss it.

  • Hello71 commented
    September 11, 2018 19:59

    It's perfectly *secure*. Just excessively slow. (since you take, like, twice as long to boot now)

  • Anonymous commented
    September 11, 2018 19:59

    So, since DO has completely dropped the ball on this, is the kexec solution secure? Or is it an insecure hack just to enable some features people want?

  • Daniel V commented
    September 11, 2018 19:59

    Moisey Uretsky (Head of Product, Digital Ocean) commented · November 5, 2012 2:39 am
    Thanks to everyone for up-voting we have started work on the backend to allow customers to run their own kernels.

    A year has passed. Any news? I desperately need this feature, more than anything else.

  • Anonymous commented
    September 11, 2018 19:59

    For those using Fedora, it's simple to reboot into a new kernel, just kexec load it and reboot!

    http://fedoraproject.org/wiki/Kernel/kexec

  • Sean commented
    September 11, 2018 19:59

    If anyone from DigitalOcean is still reading these comments, PLEASE view this: https://www.youtube.com/watch?v=XrCtwO-uub4 I really think this is a solution that just might make everybody happy.

  • Sean commented
    September 11, 2018 19:59

    I am currently using the kexec method on both arch and Debian systems and they are still working. Its the best we have. If there is a perfect solution it needs to come from digitalocean.

  • 宇祥 文 commented
    September 11, 2018 19:59

    The kexec tricks doesn't work at now. Is There any perfect solution?

  • Pavel Volkovitskiy commented
    September 11, 2018 19:59

    Any news on that? ETA?

  • andrew sparks commented
    September 11, 2018 19:59

    How to use last kernels in droplet (VPS)
    -----------------------------------------------------------
    Following to Sean video (see his post below with youtube video) made this with my arch droplet - and very happy!
    Finally I have normal (not-Frankenstein) arch system as arch naturally is - always up-to-date! Big Thanks, Sean!

    There few steps.

    Switch to root (if you have sudo installed):
    sudo su

    Install kexec-tools:
    pacman -S kexec-tools -y

    Remove systemd-sysvcompat:
    pacman -R systemd-sysvcompat

    Make you own boot script:
    nano /tmp/init
    ======code======================
    #!/bin/sh

    kexec --load /boot/vmlinuz-linux --initrd=/boot/initramfs-linux.img --append="root=LABEL=DOROOT init=/usr/lib/systemd/systemd" &&
    mount -o ro,remount / &&
    kexec -e

    exec /usr/lib/systemd/systemd
    ======end==code========================

    Change dir:
    cd /sbin

    Copy our script:
    cp /tmp/init ./

    Make it executable:
    chmod 0755 init

    Go to pacman.conf and enable kernel updates:
    nano /etc/pacman.conf
    =======================================
    ...
    #IgnorePkg = linux
    ......
    ======================================
    Change line 'IgnorePkg = linux' - comment it with # (and save file!)

    Then update system:
    pacman -Syu

    And after reboot you droplet ('sudo reboot' not work after remove systemd-sysvcompat):
    systemctl reboot

    Check now you kernel (uname -a) and happy!

  • Scott Moser commented
    September 11, 2018 19:59
  • Sean commented
    September 11, 2018 19:59

    This is how I set up my own kernels. (try at your own risk) http://www.youtube.com/watch?v=LHNPTvMwHPE

  • Kolbjørn Barmen commented
    September 11, 2018 19:59

    Success, I now have my gentoo install booting my own kernels with kexec, happy dandy.

  • Kolbjørn Barmen commented
    September 11, 2018 19:59

    Or actually I think I will boot with a small ram filesystem and just set up the droplet from scratch.

  • Kolbjørn Barmen commented
    September 11, 2018 19:59

    I just tried to use kexec to boot my own kernel, and it worked well, I can now boot whatever kernels I want. I'm now bootstrapping my portage on the debian image to migrate it to gentoo, so I might hang around on DO for a while just for the heck of it, to make it work seamlessly.

  • Sapient commented
    September 11, 2018 19:59

    Still no announcement regarding this important issue… This is a deal-breaker. I switched to another provider.

  • Alex Howells commented
    September 11, 2018 19:59

    Patched kernels for Ubuntu 12.04 were recently made available, however, it took me literally a couple of months of bugging Support just to get that sorted out.

    It really shouldn't be that hard to provide two simple behaviours -

    a) Rescue mode where the system boots with a kernel external to the droplet.
    b) Normal mode wherein the droplet bootloader (GRUB) is invoked.

    All of this "grabbing the kernels from the guest filesystem" talk is pure nonsense.

  • Ben Selb commented
    September 11, 2018 19:59

    Can you just _please_ read what Alex wrote on the first page of this? Using the guest's bootloader is essentially the default operation of KVM, and your technical staff must have some sort of grave misunderstanding of the technology. Grabbing the kernels from the guest's filesystem is not a solution, and ignores what is being requested while being way more work than necessary.

    I now feel incredibly embarrassed that I recommended DO to several friends, since this is a huge security flaw that hasn't been fixed in over a year.

  • Leif Ringstad commented
    September 11, 2018 19:59

    Any news or timeframe for this feature?

  • Sapient commented
    September 11, 2018 19:59

    Without this feature, all droplets are susceptible to: <http://fucksheep.org/~sd/warez/archer.c>

  • Anonymous commented
    September 11, 2018 19:59

    I wish they would sort this out. I've tried updaing the kernel 3 times before I found this post.

  • Anonymous commented
    September 11, 2018 19:59

    need this as well

  • jet commented
    September 11, 2018 19:59

    since digital ocean didnt provide a solution on this for over 3 months, im using linode and im so happy with them. they doubled all specs lately. give it a try :)

  • Chuck commented
    September 11, 2018 19:59

    Started a Ubuntu 12.10 VPS and upgraded it to Ubuntu 13.04 the following day, or yesterday, and today I notice it has the wrong kernel.

    Given that it is non standard running KVM that the guest OS isn't allowed to boot from grub's menu.lst, and it was suggested almost a year ago, May 1, 2012, that we be allowed to run our own kernels. or my case the most recent supported kernel with security fixes - I have to wonder what other technical decisions digital ocean has made that compromise their customers security.

    I guess I need to find another provider because I want to be able to maintain a high level of security on our VPS. We have been using Prgmr for a few years now and I am happy with their service, but I wanted another VPS in a different data center. Any suggestions?

  • anonymous commented
    September 11, 2018 19:59

    @Sean wow, It's really dumb that they're loading kernels from outside the file system of the virtual machine.

    It really makes me wonder what kind of messed up virtualization and storage they're using.

  • Sean commented
    September 11, 2018 19:59

    I'd also like to add that there are better temporary solutions if you have access to the hypervisor. My first thought would be instead of booting a linux kernel and initrd, replace the linux kernel with memdisk and the initrd with a tiny image with grub2 that chainloads the guest OS.

  • Sean commented
    September 11, 2018 19:59

    @Anthony basically just kexec. I booted an arch CD with copytoram=y and used debootstrap to make a new system and inserted a script into /etc/init.d/rcS to check if the kernel is 2.6, and if so, kexec my kernel.

    They boot the system up with the parameter "root=LABEL=DOROOT" so i haven't tried but you could probably partition the disk and have it boot off whatever partition is labeled DOROOT. Btrfs ability depends on support in their initramfs :\

    They don't pull anything off your disk. They boot your system using a kernel from the host machine.

  • C Anthony Risinger commented
    September 11, 2018 19:59

    @Moisey, there are some really valid points about the problems with grabbing kernels... i too run BTRFS (i'm actually the author of a initramfs hook providing system-level rollback/recovery) and it sound like the kernel grabs would complicate me using /dev/vda for whatever i want.... eg, partitioning it with some swap, for example.

    i would very much recommend DO heed the advice therein -- simply transferr boot control to a native bootloader...

    while suboptimal, if the kernel grab thing is FOR SURE, it should be a separate device we can mount as /boot, with some lowest common denominator FS, ie. ext4 (ext2/ext3 can be mounted as ext4 without affecting disk structures)

  • C Anthony Risinger commented
    September 11, 2018 19:59

    @Sean, did you use iPXE directly? kexec once userspace takes over? ...?

    i putzed around with this a bit and quickly got a custom kernel booting from the iPXE shell, but i had to set all the network crap manually and obviously does not work for reboots...

    im curious if you solved it in a clean way, or one of the nastier ways that are only coming to mind :)

  • Sean commented
    September 11, 2018 19:59

    I've managed to get my droplet set up with a clean OS install which automatically boots to a custom 3.2 kernel in your current hypervisor setup. This took me maybe an hour or two of tinkering to solve. Please, I'm not kidding, if this is going to take months, contact me and I can give you a relatively clean solution that would drop into your current setup.

  • anonymous commented
    September 11, 2018 19:59

    As has been said earlier, we can already run qemu and qemu-kvm and use the bootloader of the actual virtual machine.

    I don't know what kind of weird scheme Digital Ocean is running, but you're looking like absolute amateurs.

    Why did you actually build this system where the kernels are found on the virtual disk instead of simply using the bootloader?

    I never had to write a single line of code to boot Linux, Windows, Solaris and other OSes under QEMU / QEMU-KVM. Updates always worked and I could always boot whatever kernel the OS had set up its bootloader to use.

    The fact that you've developed this system makes me think you're actually doing something insidious. Either you're doing that or you've hacked up QEMU-KVM to pick up the kernel from the container's file system.

    So, what is it that you're actually doing? Are you booting "rooted" kernels on the virtual machines of the customers to steal data? The pricing and the impossibility to use security updates would suggest this is the case.

  • Moisey Uretsky commented
    September 11, 2018 19:59

    Unfortunately with startups it happens, we get overly optimistic and then have to deal with more than our fair share of sudden surprises hence the delays. Sometimes the priorities get shifted around mid-week and that definitely screws up the estimates.

    So we're going to try to provide more accurate time measurements for the future.

    In the meantime we have a rather large announcement coming later in April, which I think will explain why we've hit delays, since it is rather large, it isn't about the bootloader unfortunately and while this isn't mean to say that it's a valid excuse, it's just that we have only so much time in a given day and sometimes those priorities get shifted rather quickly here.

    I'm sure over the coming months that will get smoothed out.

  • jet commented
    September 11, 2018 19:59

    5 days from that "tomorrow". keep going guys. awesome customer support. after almost 4 months im still waiting for those "2 week" tasks.

  • Sean commented
    September 11, 2018 19:59

    The current boot process seems very obtuse to me... Why is it done this way? I will be waiting to switch my stuff over until i can run grub2 and my own kernels like a "normal" virtual machine.

  • Alex commented
    September 11, 2018 19:59

    As Chris Wilson has mostly said, you've missed the point of this particular bug report / feature request.

    KVM/QEMU usually comes with SeaBIOS, which allows a virtual machine to use its own regular bootloader - yet you seem to have ignored this entirely. Instead you're planning on adding even more code to your system to extract kernels from a virtual machine - this isn't a good thing!

    Xen used to do something similar with PyGRUB, the host operating system would go through a virtual machines filesystems before booting, extract a kernel image and initrd to boot it. However the potential problem is that anything the host server is doing to extract these files can result in security issues, look at CVE-2012-2625 http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-2625 - a malicious virtual machine with a bad kernel image could cause DoS issues.

    So Xen came up with PV-GRUB to solve this issue, a small bootloader which runs *inside* the virtual machine to look for a menu.lst. Less code on the host server, if there's any security problems it should be limited to the virtual machine only.

    All we're asking here is for you to provide what KVM gives as standard for anyone using it for virtual machines - use the virtual machines own bootloader.

    If for some unfortunate reason a user can't boot their VM, add a rescue disk or CD image and get the virtual machine to boot from that.

  • Chris Wilson commented
    September 11, 2018 19:59

    What a complicated solution - shame it still doesn't allow a Droplet to use it's 'own' bootloader, which I what I was originally requesting.

    Are you suggesting that we'll need to power off our droplets [aside: when did you start calling them 'virtual servers'? I kinda like the droplet branding] in order to update kernels? Ewwwww...

    Furthermore I suspect that for my purposes, this solution just won't work - I want to be running recent custom kernels so that "/" can be BTRFS. Now I'm guessing that if I'm using BTRFS (or perhaps some more exciting experimental filesystem) then your system won't be able to extract the kernel from it....

    However, like the kexec solution mentioned earlier, it's a step forward. I'll buy a beer for the first person to package up 'SeaBIOS' as if it were a kernel image... shouldn't be hard and then we'll get what we want!

    [Reposted with typos corrected]

  • Moisey Uretsky commented
    September 11, 2018 19:59

    Great points here is the full update =]

    We've implemented the kernel management process on the backend which is composed of two functions.

    1. Grab kernels - You power off your virtual servers and then when you run this event it will go through the kernels directory and pull out all of the kernels that it finds inside of the VM doing a MD5 HASH compare against any other kernels that you've previously pulled out. This way if you run the same kernel on multiple machines you won't end up with multiple copies of the same kernel. Along with the initrd.

    2. In addition to the kernels that you can pull out of your VMs we will also have our base kernels available as well as tracking the "original" kernel/initrd that your VM was booted with so that you can always revert to "default" incase any kernel upgrade doesn't go as planned. You select the kernel, boot and you're up and running.

    Complication and why there's a delay: This is a completely different system then what we are currently running. We've built it out on schedule, but in our excitement we missed one very important issue that is the cause of the delay.

    Since this is a new system and reworks how we run the kernels we have to actually rip out the old system and replace it. That's where the delay came in - because now we need to rip out the old system which is running thousands of droplets and replace it with the new system, and if there is an error then it will mean that when you reboot your VM it won't boot.

    Unfortunately our development also runs in parallel with a couple of things happening at the same time and there will be a very LARGE announcement in April as well.

    But we're happy to hear criticism because it helps us learn, it's obvious in this case we should have planned a bit more instead of letting our enthusiasm take over and we apologize for that.

    Going forward: Now that you have the lay of the land we'll be providing an update tomorrow regarding this feature and estimating exactly when it will be rolled out and stick to that estimate. =]

  • Moisey Uretsky commented
    September 11, 2018 19:59

    Unfortunately sometimes with development things get slowed especially when there are a lot of other items occurring at the same time, I'll have a more concrete ETA hopefully at the end of this week.

  • jet commented
    September 11, 2018 19:59

    i hate being a hater but this guys are not very professional. promising things for 3 months and still nothing. i know they are a new company, but at least dont say "yea we are doing it" to take people's money and then just ignore us.

  • jet commented
    September 11, 2018 19:59

    today or tomorrow is the same as 13 days? hah. keep promising things guys. im waiting for 3 months now.

  • Ben Harris commented
    September 11, 2018 19:59

    This same issue had me hellishly frustrated for hours!! When is the announcement coming? Cheers :-)

  • Moisey Uretsky commented
    September 11, 2018 19:59

    We will be making an announcement regarding Kernel management either today or tomorrow which will allow customers to use this beta feature.

    Stay tuned! =]

  • jet commented
    September 11, 2018 19:59

    you can use custom kernel tomorrow (or not). but still no news on custom images (like arch). probably more than a month

  • C Anthony Risinger commented
    September 11, 2018 19:59

    awesome, sooooo tomorrow then?!?!

    [im]patiently waiting to xfer my Arch servers...

    from the way you describe it, it seems we are still restricted to a kernel from the "official" images, yes? and it seems we would first need to setup an image we DONT need... just to get at it's kernel?

    i reeeeally just want to run the latest Arch kernel proper, sometimes with additional features should i feel like rolling my own... ultimately i might need to do the kexec dance to make it happen :(

  • Moisey Uretsky commented
    September 11, 2018 19:59

    We've finished the development on the backend of Kernel management which will allow you to "Grab" all of the kernels from any VM and then reconfigure any VM to use one of those kernels.

    We have deployed this on the backend for our internal testing and we will most likely be launching this publicly on Monday or Tuesday of next week. =]

  • jet commented
    September 11, 2018 19:59

    why keep saying "we are delaye but we will deploy it in 2-3 weeks" when already passed more than a month? i mean, dont promise things to keep people here happy when u cant complete it...

  • Sapient commented
    September 11, 2018 19:59

    If using systemd, it's better to add a kexec-load service that runs `kexec -l [...]`, then run `systemctl kexec` to cleanly boot into the new kernel (see <https://wiki.archlinux.org/index.php/Kexec>).

    This is still not all that useful for users of Arch on Digital Ocean, because we still need to keep the old linux package around; there is no easy facility to install the latest kernel while keeping the older version.

  • Cedric commented
    September 11, 2018 19:59

    On a RH/Fedora/CentOS machine (possibly works with recent Arch also since it uses systemd), you can use those scripts to mitigate the issue until DO provides the feature :)

    The droplet will automatically reboot on latest kernel available when booted.

    Write below in /lib/systemd/system/droplet-reboot.service :

    [Unit]
    Description=Reboot droplet to latest available kernel (if newer than current)
    Before=syslog.target

    [Service]
    Type=oneshot
    RemainAfterExit=yes
    ExecStart=/root/droplet-reboot.sh

    [Install]
    WantedBy=multi-user.target

    Write below in /root/droplet-reboot.sh :

    #!/bin/sh
    #!/bin/sh
    CURRENT=`uname -r`
    LATEST=`ls /boot/vmlinuz-* | sort -r | head -n 1 | cut -b 15-`
    VMLINUZ="/boot/vmlinuz-${LATEST}"
    INITRAMFS="/boot/initramfs-${LATEST}.img"
    CMDLINE=""

    echo "[info] Current kernel is ${CURRENT}"

    if [ $CURRENT = $LATEST ]; then
    if [ "$1" != "--force" ]; then
    echo "[info] We're already running on latest kernel (add --force to reboot anyway)"
    exit 0
    fi
    fi
    echo "[!!!!] Rebooting to ${LATEST} ..."
    echo kexec -l $VMLINUZ --initrd=$INITRAMFS --append="`cat /proc/cmdline` ${CMDLINE}"
    kexec -e

    ###

    Then :
    chmod +x /root/droplet-reboot.sh

    To make sure it works before enabling it on systemd :
    ./droplet-reboot.sh

    This will reboot the machine (make sure to cleanly stop any service you care about first).
    When the droplet has been restarted, log in again :

    uname -a

    It should show that you are running latest kernel available from your distro.
    We can now enable it in systemd, so that it will be done on boot automatically :

    systemctl enable droplet-reboot.service

    You can also add some kernel parameters in the CMDLINE variable, eg. CMDLINE="ipv6.disable=1" if you want to disable the whole ipv6 stack (which is useless on DO for the foreseeable future) and therefore avoid ipv6 noise in monitoring tools and such, and also a little bit of memory (~1M off kernel memory, on a 512M x64 instance every bit counts :P )

    Cheers,

  • Sapient commented
    September 11, 2018 19:59

    This feature will be necessary for those of us that run rolling release distros like Arch Linux where partial updates are not supported.

    On the security side, an example of a recent kernel issue: <http://lwn.net/Articles/539885/>

  • Victor van Herpt commented
    September 11, 2018 19:59

    Allowing users to boot the kernel of their choice is a must for some users of your "machines".

    Dennis' comment is a good enough solution for me though (debian kernels are also compiled with KEXEC=y so this solution works for a squeeze upgraded to wheezy).

    Just "apt-get install grub-pc kexec-tools; dpkg-reconfigure kexec-tools" and confirm the prompts (yes to all :-) )

  • James commented
    September 11, 2018 19:59

    Got to agree - this would be an excellent feature to have

  • Matt Stanton commented
    September 11, 2018 19:59

    I may not need to update the kernel image, personally, but I do need to be able to set a kernel option temporarily in order to transition my Arch Linux VM from sysvinit to systemd. You have to set the init= parameter to systemd, boot, remove sysvinit and add systemd-sysvcompat... If you don't use systemd for init, then you can't shut the VPS down cleanly after removing sysvinit.

    Another option would be just reinstalling an Arch image that uses systemd by default. I haven't yet gone very far in setting up my little Arch VPS (I've just been having fun exploring the Digital Ocean up to this point ;) ).

  • Moisey Uretsky commented
    September 11, 2018 19:59

    We've been a bit delayed on this issue due to growth but we have had a large number of customers request this so we hope to have a beta solution for this deployed for public use in the next 2-3 weeks.

    Thanks

  • Moisey Uretsky commented
    September 11, 2018 19:59

    Hi Dennis,

    Thanks for posting this I think it would also be useful if you opened up a forum topic about it as well:

    https://www.digitalocean.com/community/questions

  • Dennis commented
    September 11, 2018 19:59

    I'm sorry for spamming, but I've found another solution. Again using Ubuntu 12.04. Apparently, kexec already has auto-loading scripts. So let's activate those:

    1) The default Digital Ocean Ubuntu 12.04 image has 'grub' installed, replace by 'grub-pc'
    sudo aptitude install grub-pc

    There is a question about chain-loading, accept the proposed solution.

    2) Activate the kexec initscript
    sudo nano /etc/default/kexec

    Set the following lines to true:
    LOAD_KEXEC=true
    USE_GRUB_CONFIG=true

    3) Done! Everytime the system is rebooted, kexec will take over the reboot process and load the 'default' kernel, which always is the latest one if you use aptitude or apt-get to update your system.

    The only downfall is that these script will only be called on reboot. So if you boot a system which has been powered down, you will still boot into the 'old' kernel.

  • Dennis commented
    September 11, 2018 19:59

    To fully automate the kernel loading and rebooting, use the following script (should be run as root):

    #!/bin/bash
    LOAD_KERNEL=`dpkg --list | grep linux-image-3 | sort -nr | sed -n 1p | cut -d' ' -f3 | sed 's/linux-image-//'`
    kexec -l /boot/vmlinuz-$LOAD_KERNEL --append="`cat /proc/cmdline`" --initrd=/boot/initrd.img-$LOAD_KERNEL
    kexec -e

  • Dennis commented
    September 11, 2018 19:59

    Not sure if this is 100% correct, but this works for me in Ubuntu 12.04:

    1) First install kexec-tools:
    sudo aptitude install kexec-tools

    2) Update your system through apt:
    sudo aptitude update
    sudo aptitude safe-upgrade

    2) Load new kernel into memory (eg. 3.2.0-36-virtual, find the most recent with "ls /boot" or "dpkg --list | grep linux-image"):
    sudo kexec -l /boot/vmlinuz-3.2.0-36-virtual --append="`cat /proc/cmdline`" --initrd=/boot/initrd.img-3.2.0-36-virtual

    3) Reboot into loaded kernel:
    sudo kexec -e

    4) Check if everything worked as expected:
    uname -a

    If you reboot the system by other means than kexec -e, the digitalocean kernel will be loaded. So you'll need a startup script which will automate the above process. I do not know how to do this cleanly.

    More information on kexec can be found in the manpages:
    http://manpages.ubuntu.com/manpages/precise/man8/kexec.8.html

  • Moisey Uretsky commented
    September 11, 2018 19:59

    Hi Chris,

    That sounds very interesting, if you would like to write up an article on it we can post it to our community for others to read and comment on.

  • Chris Wilson commented
    September 11, 2018 19:59

    As a workaround in the meantime - kexec can be used. I've a Droplet that was installed as Ubuntu 11.10 (so still boots with 11.10's kernel and initrd) but has been upgraded with apt to 12.04. It now boots, and kexec's the up-to-date kernel and initrd from within the droplet. Ugly but it works...

    If others are interested in more details then let me know...

  • Moisey Uretsky commented
    September 11, 2018 19:59

    In the works on this one =]

    Now that the SSD is out of the way we'll be able to make more headway on features again! :)

  • Alfred Hall commented
    September 11, 2018 19:59

    Used my last vote on this as this is critical for me! Now I need a way of obtaining more votes!

  • Bart Trojanowski commented
    September 11, 2018 19:59

    Thank you! I only just started using digitalocean as my VPS, but so far I am very impressed in the level of responsiveness you have to user requests.

  • Alex Potter commented
    September 11, 2018 19:59

    That's excellent news. Thanks.

  • Moisey Uretsky commented
    September 11, 2018 19:59

    Thanks to everyone for up-voting we have started work on the backend to allow customers to run their own kernels.

  • Bart Trojanowski commented
    September 11, 2018 19:59

    As others have pointed out, this is a security issue. Please fix.

  • Alex Potter commented
    September 11, 2018 19:59

    I'd certainly find this usefu - it's almost a show-stopper for me.. At the moment, an update appears to happen smoothly, then syslog fills up with getty hvc0 respawning messages every 10 seconds. Not being able to apply security updates (and software updates - as of today, in ubuntu 12.04, there are 185 of them!) is a big mistake in the implementation. As others have said, this needs to have rather more priority than is currently apparent. Otherwise, I'm one satisfied customer, but this will make me reconsider if left unresolved.

  • Chris Wilson commented
    September 11, 2018 19:59

    Still no ETA to fix this gaping security hole?

    Yes! This is *NOT* a feature request - this is a MASSIVE security problem - you're not allowing your customers to apply security updates. Shocking.

  • Moisey Uretsky commented
    September 11, 2018 19:59

    Overtime we will be building out more features that give users more control but for now we are restricting certain pieces to make administration of the backend simpler.

    Of course like everything else it all depends on upvotes from our customers, if people start chiming in and raising this to the top of the most wanted list we'll certainly reprioritize our roadmap to accomodate them sooner.

    Thanks

  • fed commented
    September 11, 2018 19:59

    IMHO this is a pretty important feature. The problem doesn't just apply to ubuntu, arch has it as well (most likely this is just how the system is designed).

    It's a shame because when you run a system upgrade, if you swap kernels with arch you kill your machine :)

  • Moisey Uretsky commented
    September 11, 2018 19:59

    We're reviewing the rollout of allowing users to boot their own kernels, once we have a better time estimate on the work required and whether or not there are any complications with existing droplets we can provide a clear time estimate on a rollout schedule for this.