Native mono crashes [kernel fix released]

Reported in to MONO on https://bugzilla.xamarin.com/show_bug.cgi?id=29212

Tnx Michael for posting it to the mono bugtracker. I couldn’t get 4.0.0 to build cleanly yet, much less run.

I’m just afraid that the cause of the issue might be hard to trace and thus might take a while till it’s fixed and released. I’m gonna try to come up with some workarounds for our userbase.
Setting processor affinity prolly works, ugly though.

Hey guys I’m getting a TON of crashes in OSX as well, does this look like it might be related to what you’re trying to troubleshoot? There’s no rhyme or reason to why/when this is occuring. Could be two minutes after start, could be two days. During some crashes Sonarr is idle, during others it’s being hammered.

uname: Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64
Sonarr Version: 2.0.0.3004
Mono Version: 3.12.1 ((detached/b7764aa Fri Mar 6 15:32:47 EST 2015)
MediaInfo: mediainfo @0.7.72_0 (active)
SQLite: sqlite3 @3.8.8.3_0 (active)

http://pastebin.com/f2ssT9TX
http://pastebin.com/a914Bkwe

Same here. Getting this every so often:

Apr 19 14:19:54 emerald sonarr[6166]: [Info] EpisodeSearchService: Episode search completed. 1 reports downloaded.
Apr 19 14:19:54 emerald sonarr[6166]: [Info] NzbSearchService: Searching 2 indexers for [That Mitchell and Webb Look : S02E03]
Apr 19 14:19:54 emerald sonarr[6166]: [Info] DownloadDecisionMaker: Processing 1 reports
Apr 19 14:19:56 emerald sonarr[6166]: [Info] Sabnzbd: Adding report [That.Mitchell.And.Webb.Look.S02E03.DVDRip.XviD-HAGGiS] to the queue.
Apr 19 14:19:56 emerald sonarr[6166]: Stacktrace:
Apr 19 14:19:56 emerald sonarr[6166]: Native stacktrace:
Apr 19 14:19:56 emerald sonarr[6166]: /usr/lib/libmonosgen-2.0.so.1(+0xcf10a) [0x7f75d71c310a]
Apr 19 14:19:56 emerald sonarr[6166]: /usr/lib/libmonosgen-2.0.so.1(+0x12ccbb) [0x7f75d7220cbb]
Apr 19 14:19:56 emerald sonarr[6166]: /usr/lib/libmonosgen-2.0.so.1(+0x47ee6) [0x7f75d713bee6]
Apr 19 14:19:56 emerald sonarr[6166]: /usr/lib/libpthread.so.0(+0x10740) [0x7f75d6ee7740]
Apr 19 14:19:56 emerald sonarr[6166]: /usr/lib/libmonosgen-2.0.so.1(+0x1bb1df) [0x7f75d72af1df]
Apr 19 14:19:56 emerald sonarr[6166]: /usr/lib/libmonosgen-2.0.so.1(+0x1b7ce7) [0x7f75d72abce7]
Apr 19 14:19:56 emerald sonarr[6166]: /usr/lib/libmonosgen-2.0.so.1(+0x254175) [0x7f75d7348175]
Apr 19 14:19:56 emerald sonarr[6166]: /usr/lib/libpthread.so.0(+0x7374) [0x7f75d6ede374]
Apr 19 14:19:56 emerald sonarr[6166]: /usr/lib/libc.so.6(clone+0x6d) [0x7f75d6c1c27d]
Apr 19 14:19:56 emerald sonarr[6166]: Debug info from gdb:
Apr 19 14:19:57 emerald sonarr[6166]: ptrace: Operation not permitted.
Apr 19 14:19:57 emerald sonarr[6166]: No threads.
Apr 19 14:19:57 emerald sonarr[6166]: =================================================================
Apr 19 14:19:57 emerald sonarr[6166]: Got a SIGSEGV while executing native code. This usually indicates
Apr 19 14:19:57 emerald sonarr[6166]: a fatal error in the mono runtime or one of the native libraries
Apr 19 14:19:57 emerald sonarr[6166]: used by your application.
Apr 19 14:19:57 emerald sonarr[6166]: =================================================================
Apr 19 14:19:57 emerald systemd[1]: sonarr.service: main process exited, code=killed, status=6/ABRT

Operation not permitted was likely this (from dmesg):

[267263.762261] ptrace of pid 6166 was attempted by: gdb (pid 9119)

Mono version is 3.12.1-1.

@xgz Try mono 3.10, we’ve had reports on 3.12.1 on OSX being unreliable. Even before we started investigating these crashes on Ubuntu.

@Enverex You’re missing a lot of information on that report: system, linux version, kernel version, sonarr version etc etc

Please note at this point I don’t really look at the individual traces anymore. There isn’t any valuable information in the backtrace itself coz it’s basically saying “Ew, sewer leak, bye bye”

I actually rolled back to 3.6.0 based on your responses on this thread and it’s working rock solid for me. I’ll give 3.10 a shot if I come up with any other apps that need to use mono.

Thanks a ton for the reply and good luck with the bug smashing!

Arch Linux, kernel 3.19.3, latest Sonarr from GIT.

It was more a “me too” post than anything else.

@Michael_Thwaite

If I run taskset 1 mono bug-18026.exe the chance of it crashing is drastically reduced, but still happens particularly with Sonarr.
I also installed kernel 3.16.0 on the 14.04 vm, seems to make the issue more severe but that could be coincidence.

Sadly, since taskset 1 doesn’t work on Sonarr, I don’t have a workaround I can push to our users. Booting using a lower kernel version seems to be the only option atm (other than using a single cpu in case of a vm). Hopefully Alexander can reproduce it and give us insight on whether it’s mono or the kernel.

It’s encouraging that Alexander can’t fault it on Ubuntu 14.10.

Did you sign on to the cc list on the Xamarin buglist to add more weight?

Linux Distribution: Ubuntu 14.04.2 LTS
Kernel: 3.13.0-49-generic
Sonarr: 2.0.0.3004
Mono: 3.10.0 (tarball Wed Nov 5 12:50:04 UTC 2014)
Libmediainfo0: 0.7.67-2ubuntu1
libsqlite3-0: 3.8.2-1ubuntu2

[Debug] Parser: Parsing string ‘Hawaii Five-0 - s02e04 - Mea Makamae - SD DVD.tbn’
[Debug] MetadataService: Episode image already exists: Season 05/Hawaii Five-0 - S05E18 - Pono Kaulike - SDTV-thumb.jpg
[Debug] XbmcMetadata: Generating Episode Metadata for: /mnt/media/Video/Series/Hawaii Five-0/Season 05/Hawaii Five-0 - S05E19 - Kahania - SDTV.mp4
Stacktrace:

Native stacktrace:

[Debug] Parser: Episode Parsed. hawaiifive0 - S02E04
/usr/bin/mono() [0x4b3f7c]
/usr/bin/mono() [0x50c30f]
/usr/bin/mono() [0x423637]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f4fa9f05340]
/usr/bin/mono() [0x4c55ad]
/usr/bin/mono() [0x50bdfe]
/usr/bin/mono() [0x4b25a6]
/usr/bin/mono() [0x4b2be7]
/usr/bin/mono() [0x4b31f6]
/usr/bin/mono() [0x4d28aa]
/usr/bin/mono() [0x510d9d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f4fa9f05340]
/usr/bin/mono() [0x4d1f90]
[0x417dfa4c]

How did you roll back? I followed the instructions here (http://www.mono-project.com/docs/getting-started/install/linux/), but
they did not work for version 3.6.0 or 3.06.0.

JDS, that’s for OSX, it won’t fix the native crashes on ubuntu.
As workaround for ubuntu simply boot into the older kernel 3.13.0-46.

Distro: Ubuntu 14.04.1 LTS
Linux version 3.18.6-aufs (root@mediamaster) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #1 SMP Sat Mar 7 09:18:30 EST 2015
Mono: Mono JIT compiler version 3.6.0 (tarball Sat Apr 18 15:47:52 EDT 2015)
Sonar: 2.0.0.3004
MySQL: mysql Ver 14.14 Distrib 5.5.41, for debian-linux-gnu (x86_64) using readline 6.3

Recent convert to Sonarr, and I’m fully invested in it now, no turning back. It was crashing ALOT until I went from Mono 3.10 to 3.6… now, it continues to run, but will eventually stop grabbing any new shows. Before with Mono 3.10 it would just crash and burn totally.

Anyway, I had to build a custom kernel for my SnapRAID configuration due, so I guess I’m looking for any potential bandaids vs babysitting it daily. I really don’t feel like compiling a new kernel.

A ‘me too’ post, although I presented a little differently at first. My scheduled tasks would just stop running. I updated to current dev release as of now and it started giving the stack trace issues.

Distributor ID: Ubuntu
Description: Ubuntu 14.04.2 LTS
Release: 14.04
Codename: trusty

Linux home-server 3.13.0-49-generic #83-Ubuntu SMP Fri Apr 10 20:11:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

libmediainfo0:amd64 0.7.67-2ubuntu1

sqlite3 3.8.2-1ubuntu2

sonarr Version
2.0.0.3077
Mono Version
3.10.0 (tarball Wed Nov 5 12:50:04 UTC 2014)

@tymanthius Just try booting into 3.13.0-46 via grub, unless you already deleted that old kernel version.

I think I already deleted it, but as I’m not at home, I can’t do that safely. And my wife will NOT be happy if I remotely break it and she can’t watch tv.

BTW, just got this too:

EPIC FAIL: System.NullReferenceException: Object reference not set to an instance of an object
at (wrapper unknown) System.Threading.Monitor:FastMonitorEnterV4 (object,bool&)
at System.Threading.Timer+Scheduler.SchedulerThread () [0x00000] in :0
at System.Threading.Thread.StartInternal () [0x00000] in :0
[Fatal] GlobalExceptionHandlers: EPIC FAIL: Object reference not set to an instance of an object

System.NullReferenceException: Object reference not set to an instance of an object
at (wrapper unknown) System.Threading.Monitor:FastMonitorEnterV4 (object,bool&)
at System.Threading.Timer+Scheduler.SchedulerThread () [0x00000] in :0
at System.Threading.Thread.StartInternal () [0x00000] in :0

[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
at (wrapper unknown) System.Threading.Monitor:FastMonitorEnterV4 (object,bool&)
at System.Threading.Timer+Scheduler.SchedulerThread () [0x00000] in :0
at System.Threading.Thread.StartInternal () [0x00000] in :0
Stacktrace:

Native stacktrace:

Has anyone looked at latest mono and tried building it by hand? I was about to browse over and see what they had.

You can reinstall 46 via apt-get. But you won’t be able to boot into it without physical access (grub bootloader) or you’ll have to reconfigure grub to pick 46 as default.

Those error are all very related.

Compiling mono versions won’t help, by now I’ve tested 3.2.8, 3.6, 3.8, 3.10, 3.12 they all crash, one faster than the other, and other ppl tests newer versions which don’t seem to work properly either.

When you get home, reinstall and boot the 3.13.0-46 kernel.

I have just downgraded my Kernal to 3.13.0-46-generic x86_64 and Sonarr has been stable for about 4 days now!

Same boat here. Downgraded to 3.13.0-46-generic x86_64 on friday and haven’t crashed all weekend (as opposed to hourly on 3.13.0-49-generic x86_64) so defo something new in the kernel there.

Do you still need logs or anything @Taloth

Wanted to add a ‘me too’ post that backtracking to 3.13.0-46 kernel has cleared up all my issues with sonarr.