Native mono crashes [kernel fix released]

Help & Support
#109

FYI I pulled the latest early testing builds from the kernel ppa.
All tested on a 14.04.2 trusty vm.

3.13.0-53.x was still crashing. (trusty-updates)
3.13.0-54.61 seems to work. (ppa)

lts-utopic:
3.16.0-38.30 was still crashing. (trusty-updates)
3.16.0-39.31 seems to work. (ppa)

Prolly could use someone versed in linux to try it out on a physical machine. But I take no responsibility if it utterly breaks your machine, coz these are early testing builds. Don’t blindly add the ppa, it gotta be pinned with low priority to avoid unwanted updates.

1 Like
#110

@jrhelbert

What does you cron job look like? I tried to do the same but it didn’t work.

#111

Sonarr does self checks on startup to make sure it’s not already running and quits if it detects a process already running, so my cron is extremely simple:

cat /etc/cron.hourly/start-sonarr
#!/bin/sh

start nzbdrone
#112

@Kelsey_Murphy_Drummo

Mine is just:

*/1 * * * * /etc/init.d/nzbdrone start

@Taloth Im going to wait haha. Looking hopeful though.

#113

Thanks @jrhelbert

I get an error unless I use the following to start Sonarr which doesn’t work with cron.

sudo start nzbdrone

It must be something to do with the user that Sonarr is running under or something.

Edit: Never mind, I figured it out!

#114

And the packages just got promoted to trusty-proposed. (and utopic-proposed for those using 14.10)

I need at least 1 person to test this on a physical server. (please reply if you’re going to try this)

Before you start, be sure you can get into the grub boot menu, which allows you to boot back into an older kernel if needed.

Instructions for 14.04 trusty:

Step 1: Add repository source

Edit /etc/apt/sources.list and add:

deb http://archive.ubuntu.com/ubuntu/ trusty-proposed restricted main multiverse universe

Step 2: Pin it to prevent auto-installs

(using sudoedit for example)
Create /etc/apt/preferences.d/proposed-updates with the following content:

Package: *
Pin: release a=trusty-proposed
Pin-Priority: 400

Step 3: Update apt cache

Run sudo apt-get update

Optional: Verify if pinning is correct:
Run apt-cache policy linux-image-generic
you should see the 3.13.0.53.60 as having the highest (500) priority, with the proposed one at 400 priority.

Step 4: Install kernel

If you have the original trusty 3.13.0 kernel:

  • Run sudo apt-get install linux-image-generic -t trusty-proposed
    OR if you have the lts-utopic backported 3.16.0 kernel:
  • Run sudo apt-get install linux-image-generic-lts-utopic -t trusty-proposed

Step 5: Reboot (assuming you have the default grub config)

Instructions for 14.10 utopic:
I didn’t test this one myself.

Step 1: Add repository source

Edit /etc/apt/sources.list and add:

deb http://archive.ubuntu.com/ubuntu/ utopic-proposed restricted main multiverse universe

Step 2: Pin it to prevent auto-installs

(using sudoedit for example)
Create /etc/apt/preferences.d/proposed-updates with the following content:

Package: *
Pin: release a=utopic-proposed
Pin-Priority: 400

Step 3: Update apt cache

Run sudo apt-get update

Optional: Verify if pinning is correct:
Run apt-cache policy linux-image-generic
you should see the 3.16.0.38.30 as having the highest (500) priority, with the proposed one at 400 priority.

Step 4: Install kernel

Run sudo apt-get install linux-image-generic -t utopic-proposed

Step 5: Reboot (assuming you have the default grub config)

#115

Physical Server here, I just installed and have rebooted into it.

uname -a
Linux server 3.13.0-54-generic #91-Ubuntu SMP Tue May 26 19:15:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I’ve gone ahead and removed any of the workarounds (processor affinity, cron, etc) I had put in place and we’ll see how it goes. I haven’t had anything that was a surefire trigger, so I’m just going to let it run and do its thing.

I’ll post tomorrow with status.

#116

In hindsight that’s pretty poor planning on my part… I rebooted into -53 and ran Sonarr several times without all the workarounds in place. Sure enough it’d crash within min either just by turning on or by kicking off a series refresh.

Booted back into -54 and ran the same tests several times without seeing any failures.

It’s looking optimistic, so I’ll just let it run and do its thing now and report back tomorrow.

#117

@jrhelbert Thank you for trying it out.

At least it seems confirmed now that this specific bug affected physical servers as well.
Lets hope this was the only bug causing crashes :smiley:

#118

Still going strong!

I had seen it go a day or two before without crashing, so I’m going to run longer before I say for certain that it’s gone (As opposed to sheer luck). Given how unstable it was after the last Sonarr update though, it’s very promising.

#119

I just updated to the proposed kernel on 14.04. I’ll let you know how it works - since the last update it was crashing at the least every 4 hours, as often as every 3 seconds. I’d sometimes have to restart it 6 times just to get the web interface up. Just restarted, Sonarr loaded on boot, refreshed all series, and is happily awaiting work. So far so good. I like Sonarr so much better than the competition, but was getting close to walking away in frustration. Thanks for the hard work.

#120

I was experiencing the same issues, just tried it with the proposed LTS kernel and it seems to work fine now. I will let it run and see if everything goes smoothly.

Linux dakar 3.16.0-39-generic #53~14.04.1-Ubuntu SMP Wed May 27 10:03:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

#121

24 hours later, no crashes. That’s the longest uptime I’ve had since the latest update, and close to the longest uptime I’ve had ever.

Not using a VM either, so this is a pleasant surprise.

#122

Still no crashes! Seems relatively safe to say the issue is fixed, and the same issue affected both Physical as well as Virtual Machines.

#123

I followed your instructions for 14.04 trusty, updated my kernel to 3.13.0-54 and didn’t crashed for 24+ hours.
Thank you.

#124

Awesome guys, tnx for letting me know. I’ll reported the success to the ubuntu team.

Now it’s wait and see if fixes the crashes ppl have been seeing for Emby as well (@Sek0n)

#125

Thanks for the post man, this solved my problem. I couldn’t get NZBDrone running for more than 60 seconds. Same exact stack trace, everything the same as this.

I downgraded my kernel from .53 down to .46 like you suggested and bewm problem solved. Good stuff, thanks a lot.

#126

Good to see things have been figured out. I’d love to try the instructions for 14.04 but in case anything goes wrong I don’t really have easy access to the physical server at this moment to hook up a keyboard and screen to it. Can I expect the bugfix to be updated automatically over the following days/weeks?

#127

I didn’t read all the posts but I am assuming you need access to the physical server to deal with the grub menu?

I didn’t have access to my physical server, only remote access. It was a bit sketchy and I definitely knocked on wood, but here’s how I did it.

  • Installed kernel .46
    $ apt-get install linux-image-3.13.0-46-generic

  • Figure out which kernels are installed on ur system
    $ dpkg -l | grep linux-image
    $ apt-get remove $all_kernel_versions_from_previous_command_except_for_version_46
    You will see a message saying grub is broken or vmlinuz doesn’t exist or the symbolic link is busted
    $ sudo update-grub

  • “sudo update-grub” will look for the latest kernel you have installed (which at this point is .46) and will load up next time you reboot by default…Good luck

#128

@sigmundfreund Ubuntu kernel releases are on a 3 week cycle. A public release is expected in just under 2 weeks.

@rizz2pro That would work yes. Although at this point it would either just install -proposed or wait for the actually release.