Native mono crashes [kernel fix released]

jrhelbert · May 28, 2015, 5:03pm

exec is part of the upstart script itself. I just modifying the suggested upstart script provided with Sonarr and inserted the taskset: https://github.com/Sonarr/Sonarr/wiki/Autostart-on-Linux

Yonder · May 28, 2015, 5:15pm

Oh i see, sorry i was using this one:

Taloth · May 28, 2015, 5:29pm

Not sure how you managed to wreck your system badly enough that it required a re-install. You can just pick the correct kernel during grub bootup. Changing the default isn’t strictly required.

I’m waiting with overwhelming anticipation for -54…
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1458618

Taloth · May 28, 2015, 5:29pm

@PCJonathan No we don’t need more reports, it’s just waiting for the kernel update.

Yonder · May 28, 2015, 10:36pm

Not sure, i’m headless and remote, so i will just wait, not a linux pro. Is this bug specifically related to mono per say? Or could it affect other applications? Just wondering because plex seems to also be dropping.

Taloth · May 28, 2015, 10:40pm

FYI I pulled the latest early testing builds from the kernel ppa.
All tested on a 14.04.2 trusty vm.

3.13.0-53.x was still crashing. (trusty-updates)
3.13.0-54.61 seems to work. (ppa)

lts-utopic:
3.16.0-38.30 was still crashing. (trusty-updates)
3.16.0-39.31 seems to work. (ppa)

Prolly could use someone versed in linux to try it out on a physical machine. But I take no responsibility if it utterly breaks your machine, coz these are early testing builds. Don’t blindly add the ppa, it gotta be pinned with low priority to avoid unwanted updates.

Kelsey_Murphy_Drummo · May 28, 2015, 11:55pm

@jrhelbert

What does you cron job look like? I tried to do the same but it didn’t work.

jrhelbert · May 29, 2015, 2:05am

Sonarr does self checks on startup to make sure it’s not already running and quits if it detects a process already running, so my cron is extremely simple:

cat /etc/cron.hourly/start-sonarr
#!/bin/sh

start nzbdrone

Yonder · May 29, 2015, 2:05am

@Kelsey_Murphy_Drummo

Mine is just:

*/1 * * * * /etc/init.d/nzbdrone start

@Taloth Im going to wait haha. Looking hopeful though.

Kelsey_Murphy_Drummo · May 29, 2015, 4:18am

Thanks @jrhelbert

I get an error unless I use the following to start Sonarr which doesn’t work with cron.

sudo start nzbdrone

It must be something to do with the user that Sonarr is running under or something.

Edit: Never mind, I figured it out!

Taloth · May 29, 2015, 9:10pm

And the packages just got promoted to trusty-proposed. (and utopic-proposed for those using 14.10)

I need at least 1 person to test this on a physical server. (please reply if you’re going to try this)

Before you start, be sure you can get into the grub boot menu, which allows you to boot back into an older kernel if needed.

Instructions for 14.04 trusty:

Step 1: Add repository source

Edit /etc/apt/sources.list and add:

deb http://archive.ubuntu.com/ubuntu/ trusty-proposed restricted main multiverse universe

Step 2: Pin it to prevent auto-installs

(using sudoedit for example)
Create /etc/apt/preferences.d/proposed-updates with the following content:

Package: *
Pin: release a=trusty-proposed
Pin-Priority: 400

Step 3: Update apt cache

Run sudo apt-get update

Optional: Verify if pinning is correct:
Run apt-cache policy linux-image-generic
you should see the 3.13.0.53.60 as having the highest (500) priority, with the proposed one at 400 priority.

Step 4: Install kernel

If you have the original trusty 3.13.0 kernel:

Run sudo apt-get install linux-image-generic -t trusty-proposed
OR if you have the lts-utopic backported 3.16.0 kernel:

Run sudo apt-get install linux-image-generic-lts-utopic -t trusty-proposed

Step 5: Reboot (assuming you have the default grub config)

Instructions for 14.10 utopic:
I didn’t test this one myself.

Step 1: Add repository source

Edit /etc/apt/sources.list and add:

deb http://archive.ubuntu.com/ubuntu/ utopic-proposed restricted main multiverse universe

Step 2: Pin it to prevent auto-installs

(using sudoedit for example)
Create /etc/apt/preferences.d/proposed-updates with the following content:

Package: *
Pin: release a=utopic-proposed
Pin-Priority: 400

Step 3: Update apt cache

Run sudo apt-get update

Optional: Verify if pinning is correct:
Run apt-cache policy linux-image-generic
you should see the 3.16.0.38.30 as having the highest (500) priority, with the proposed one at 400 priority.

Step 4: Install kernel

Run sudo apt-get install linux-image-generic -t utopic-proposed

Step 5: Reboot (assuming you have the default grub config)

jrhelbert · May 30, 2015, 4:52am

Physical Server here, I just installed and have rebooted into it.

uname -a
Linux server 3.13.0-54-generic #91-Ubuntu SMP Tue May 26 19:15:08 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I’ve gone ahead and removed any of the workarounds (processor affinity, cron, etc) I had put in place and we’ll see how it goes. I haven’t had anything that was a surefire trigger, so I’m just going to let it run and do its thing.

I’ll post tomorrow with status.

jrhelbert · May 30, 2015, 5:24am

In hindsight that’s pretty poor planning on my part… I rebooted into -53 and ran Sonarr several times without all the workarounds in place. Sure enough it’d crash within min either just by turning on or by kicking off a series refresh.

Booted back into -54 and ran the same tests several times without seeing any failures.

It’s looking optimistic, so I’ll just let it run and do its thing now and report back tomorrow.

Taloth · May 30, 2015, 7:56am

@jrhelbert Thank you for trying it out.

At least it seems confirmed now that this specific bug affected physical servers as well.
Lets hope this was the only bug causing crashes

jrhelbert · May 30, 2015, 3:25pm

Still going strong!

I had seen it go a day or two before without crashing, so I’m going to run longer before I say for certain that it’s gone (As opposed to sheer luck). Given how unstable it was after the last Sonarr update though, it’s very promising.

Aphex242 · May 30, 2015, 5:55pm

I just updated to the proposed kernel on 14.04. I’ll let you know how it works - since the last update it was crashing at the least every 4 hours, as often as every 3 seconds. I’d sometimes have to restart it 6 times just to get the web interface up. Just restarted, Sonarr loaded on boot, refreshed all series, and is happily awaiting work. So far so good. I like Sonarr so much better than the competition, but was getting close to walking away in frustration. Thanks for the hard work.

chignole · May 31, 2015, 4:52pm

I was experiencing the same issues, just tried it with the proposed LTS kernel and it seems to work fine now. I will let it run and see if everything goes smoothly.

Linux dakar 3.16.0-39-generic #53~14.04.1-Ubuntu SMP Wed May 27 10:03:17 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Aphex242 · May 31, 2015, 5:49pm

24 hours later, no crashes. That’s the longest uptime I’ve had since the latest update, and close to the longest uptime I’ve had ever.

Not using a VM either, so this is a pleasant surprise.

jrhelbert · May 31, 2015, 5:49pm

Still no crashes! Seems relatively safe to say the issue is fixed, and the same issue affected both Physical as well as Virtual Machines.

sebastiencs · May 31, 2015, 8:24pm

I followed your instructions for 14.04 trusty, updated my kernel to 3.13.0-54 and didn’t crashed for 24+ hours.
Thank you.