[Bug report] 9 Crash reports over a month, Debian jessie, mono, SIGSEGV, SIGABRT, EPIC FAIL: System.NullReferenceException

Help & Support
#1

Short description

Over the last month or weeks Sonarr has been running in a stable manner except for a few crashes (once every 20 hours maybe). Lately those crashes have become more frequent. The crash types are:

  • mono SIGSEGV
  • mono SIGABRT
  • EPIC FAIL: System.NullReferenceException

Info

Sonarr version and branch
Sonarr Ver. 2.0.0.3004 (updated over the month. proper version always included in crash reports)

Operating system
Debian jessie (testing)

➜  ~  uname -a
Linux HairyDemon 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt7-1 (2015-03-01) x86_64 GNU/Linux

mono version

➜  ~  apt-cache policy mono-runtime
mono-runtime:
  Installed: 3.10.0-0xamarin2
  Candidate: 3.10.0-0xamarin2
  Version table:
 *** 3.10.0-0xamarin2 0
        500 http://apt.sonarr.tv/ master/main amd64 Packages
        100 /var/lib/dpkg/status
     3.10.0-0xamarin1 0
        500 http://apt.sonarr.tv/ master/main amd64 Packages
     3.2.8+dfsg-10 0
        500 http://ftp.de.debian.org/debian/ testing/main amd64 Packages

Detailed info available here including

  • software versions and sources
  • systemd information
  • user limits
  • cpu information
  • free memory
  • running processes

List of crashes

This list links to both Sonarr and Systemd logs from only the relevant times. A full Sonarr log and Systemd log is available below:

  1. Mar 22 07:20:56 … Mar 22 15:30:58
    [mono] Got a SIGABRT

  2. Mar 26 05:57:54 … Mar 26 13:13:18
    EPIC FAIL: System.NullReferenceException: Object reference not set to an instance of an object

  3. Mar 26 19:07:21 … Mar 27 09:15:31
    EPIC FAIL: System.NullReferenceException: Object reference not set to an instance of an object

  4. Mar 31 21:33:37 … Apr 01 12:16:51
    [mono] Got a SIGSEGV

  5. Apr 02 12:21:58 … Apr 02 16:41:58
    sonarr.service: main process exited, code=killed, status=11/SEGV

  6. Apr 02 16:48:06 … Apr 02 18:39:22
    [mono] Got a SIGSEGV

  7. Apr 02 23:23:06 … Apr 03 01:31:31
    EPIC FAIL: System.NullReferenceException: Object reference not set to an instance of an object

  8. Apr 03 01:34:48 … Apr 03 13:41:52
    [mono] Got a SIGSEGV

  9. Apr 03 14:03:41 … Apr 03 14:53:31
    [mono] Got a SIGABRT

Full logs

Debug logs are unfortunately not yet available. I enabled trace level logs inside Sonarr just now.

Error messages

Just so people have an easier time searching for this. 7 different crashes thrown together.

Apr 03 14:03:41 HairyDemon mono[26312]: * Assertion at mini-exceptions.c:532, condition `class' not met
Apr 03 14:03:41 HairyDemon mono[26312]: Stacktrace:
Apr 03 14:03:41 HairyDemon mono[26312]: at <unknown> <0xffffffff>
Apr 03 14:03:41 HairyDemon mono[26312]: at (wrapper managed-to-native) System.Diagnostics.StackTrace.get_trace (System.Exception,int,bool) <0xffffffff>
Apr 03 14:03:41 HairyDemon mono[26312]: at System.Diagnostics.StackTrace..ctor (System.Exception,int,bool,bool) <0x0004b>
Apr 03 14:03:41 HairyDemon mono[26312]: at System.Exception.get_StackTrace () <0x00187>
Apr 03 14:03:41 HairyDemon mono[26312]: at System.Exception.ToString () <0x00127>
Apr 03 14:03:41 HairyDemon mono[26312]: at NzbDrone.Common.Instrumentation.GlobalExceptionHandlers.HandleAppDomainException (object,System.UnhandledExceptionEventArgs) <0x00084>
Apr 03 14:03:41 HairyDemon mono[26312]: at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object_object (object,intptr,intptr,intptr) <0xffffffff>
Apr 03 14:03:41 HairyDemon mono[26312]: Native stacktrace:
Apr 03 14:03:41 HairyDemon mono[26312]: /usr/bin/mono() [0x4b3f7c]
Apr 03 14:03:41 HairyDemon mono[26312]: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0) [0x7f0ab39148d0]
Apr 03 14:03:41 HairyDemon mono[26312]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7f0ab3591107]
Apr 03 14:03:41 HairyDemon mono[26312]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f0ab35924e8]
Apr 03 14:03:41 HairyDemon mono[26312]: /usr/bin/mono() [0x636e59]
Apr 03 14:03:41 HairyDemon mono[26312]: /usr/bin/mono() [0x637067]
Apr 03 14:03:41 HairyDemon mono[26312]: /usr/bin/mono() [0x6371b6]
Apr 03 14:03:41 HairyDemon mono[26312]: /usr/bin/mono() [0x4b11a6]
Apr 03 14:03:41 HairyDemon mono[26312]: /usr/bin/mono() [0x4b1b7b]
Apr 03 14:03:41 HairyDemon mono[26312]: /usr/bin/mono() [0x4b229d]
Apr 03 14:03:41 HairyDemon mono[26312]: [0x42333e9e]
Apr 03 14:03:41 HairyDemon mono[26312]: Debug info from gdb:
Apr 03 14:03:41 HairyDemon mono[26312]: =================================================================
Apr 03 14:03:41 HairyDemon mono[26312]: Got a SIGABRT while executing native code. This usually indicates
Apr 03 14:03:41 HairyDemon mono[26312]: a fatal error in the mono runtime or one of the native libraries
Apr 03 14:03:41 HairyDemon mono[26312]: used by your application.
Apr 03 14:03:41 HairyDemon mono[26312]: =================================================================
Apr 03 14:03:41 HairyDemon systemd[1]: sonarr.service: main process exited, code=killed, status=6/ABRT
Apr 03 14:03:41 HairyDemon systemd[1]: Unit sonarr.service entered failed state.

Apr 03 01:34:48 HairyDemon mono[4713]: =================================================================
Apr 03 01:34:48 HairyDemon mono[4713]: Got a SIGSEGV while executing native code. This usually indicates
Apr 03 01:34:48 HairyDemon mono[4713]: a fatal error in the mono runtime or one of the native libraries
Apr 03 01:34:48 HairyDemon mono[4713]: used by your application.
Apr 03 01:34:48 HairyDemon mono[4713]: =================================================================

Apr 02 23:23:06 HairyDemon mono[3159]: EPIC FAIL: System.NullReferenceException: Object reference not set to an instance of an object
Apr 02 23:23:06 HairyDemon mono[3159]: at System.Threading.Timer+Scheduler.Add (System.Threading.Timer timer) [0x00000] in <filename unknown>:0
Apr 02 23:23:06 HairyDemon mono[3159]: at System.Threading.Timer+Scheduler.SchedulerThread () [0x00000] in <filename unknown>:0
Apr 02 23:23:06 HairyDemon mono[3159]: at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
Apr 02 23:23:06 HairyDemon mono[3159]: [Fatal] GlobalExceptionHandlers: EPIC FAIL: Object reference not set to an instance of an object
Apr 02 23:23:06 HairyDemon mono[3159]: System.NullReferenceException: Object reference not set to an instance of an object
Apr 02 23:23:06 HairyDemon mono[3159]: at System.Threading.Timer+Scheduler.Add (System.Threading.Timer timer) [0x00000] in <filename unknown>:0
Apr 02 23:23:06 HairyDemon mono[3159]: at System.Threading.Timer+Scheduler.SchedulerThread () [0x00000] in <filename unknown>:0
Apr 02 23:23:06 HairyDemon mono[3159]: at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
Apr 02 23:23:06 HairyDemon mono[3159]: [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
Apr 02 23:23:06 HairyDemon mono[3159]: at System.Threading.Timer+Scheduler.Add (System.Threading.Timer timer) [0x00000] in <filename unknown>:0
Apr 02 23:23:06 HairyDemon mono[3159]: at System.Threading.Timer+Scheduler.SchedulerThread () [0x00000] in <filename unknown>:0
Apr 02 23:23:06 HairyDemon mono[3159]: at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0

Apr 02 16:48:05 HairyDemon mono[1785]: [Error] MediaCoverService: Couldn't download media cover for [292251][The Man in the High Castle]
Apr 02 16:48:05 HairyDemon mono[1785]: NzbDrone.Common.Http.HttpException: HTTP request failed: [404:NotFound] [HEAD] at [http://thetvdb.com/banners/fanart/original/292251-2.jpg]
Apr 02 16:48:05 HairyDemon mono[1785]: at NzbDrone.Common.Http.HttpClient.Execute (NzbDrone.Common.Http.HttpRequest request) [0x00000] in <filename unknown>:0
Apr 02 16:48:05 HairyDemon mono[1785]: at NzbDrone.Common.Http.HttpClient.Head (NzbDrone.Common.Http.HttpRequest request) [0x00000] in <filename unknown>:0
Apr 02 16:48:05 HairyDemon mono[1785]: at NzbDrone.Core.MediaCover.CoverAlreadyExistsSpecification.AlreadyExists (System.String url, System.String path) [0x00000] in <filename unknown>:0
Apr 02 16:48:05 HairyDemon mono[1785]: at NzbDrone.Core.MediaCover.MediaCoverService.EnsureCovers (NzbDrone.Core.Tv.Series series) [0x00000] in <filename unknown>:0

Apr 02 12:21:58 HairyDemon mono[23277]: Stacktrace:
Apr 02 12:21:58 HairyDemon mono[23277]: Native stacktrace:
Apr 02 12:21:58 HairyDemon systemd[1]: sonarr.service: main process exited, code=killed, status=11/SEGV
Apr 02 12:21:58 HairyDemon systemd[1]: Unit sonarr.service entered failed state.

Mar 31 21:33:36 HairyDemon mono[23266]: [Error] DownloadDecisionMaker: Couldn't process report.
Mar 31 21:33:36 HairyDemon mono[23266]: System.NullReferenceException: Object reference not set to an instance of an object
Mar 31 21:33:36 HairyDemon mono[23266]: at Marr.Data.DataMapper.SetSqlMode (System.Data.Common.DbCommand command) [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at Marr.Data.DataMapper.get_Command () [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at Marr.Data.DataMapper.CloseConnection () [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at Marr.Data.DataMapper.Query[Series] (System.String sql, ICollection`1 entityList, Boolean useAltName) [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at Marr.Data.QGen.QueryBuilder`1[NzbDrone.Core.Tv.Series].ToList () [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at Marr.Data.QGen.SortBuilder`1[NzbDrone.Core.Tv.Series].ToList () [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at Marr.Data.QGen.SortBuilder`1[NzbDrone.Core.Tv.Series].GetEnumerator () [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at System.Linq.Enumerable.Single[Series] (IEnumerable`1 source, System.Func`2 predicate, Fallback fallback) [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at System.Linq.Enumerable.SingleOrDefault[Series] (IEnumerable`1 source) [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at NzbDrone.Core.Tv.SeriesRepository.FindByTvdbId (Int32 tvdbId) [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at NzbDrone.Core.Tv.SeriesService.FindByTitle (System.String title) [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at NzbDrone.Core.Parser.ParsingService.GetSeries (NzbDrone.Core.Parser.Model.ParsedEpisodeInfo parsedEpisodeInfo, Int32 tvRageId) [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at NzbDrone.Core.Parser.ParsingService.Map (NzbDrone.Core.Parser.Model.ParsedEpisodeInfo parsedEpisodeInfo, Int32 tvRageId, NzbDrone.Core.IndexerSearch.Definitions.SearchCriteriaBase searchCriteria) [0x00000] in <filename unknown>:0
Mar 31 21:33:36 HairyDemon mono[23266]: at NzbDrone.Core.DecisionEngine.DownloadDecisionMaker+<GetDecisions>d__0.MoveNext () [0x00000] in <filename unknown>:0

Mar 22 07:20:56 HairyDemon mono[712]: * Assertion at mini-exceptions.c:532, condition `class' not met
Mar 22 07:20:56 HairyDemon mono[712]: Stacktrace:
Mar 22 07:20:56 HairyDemon mono[712]: at <unknown> <0xffffffff>
Mar 22 07:20:56 HairyDemon mono[712]: at (wrapper managed-to-native) System.Diagnostics.StackTrace.get_trace (System.Exception,int,bool) <0xffffffff>
Mar 22 07:20:56 HairyDemon mono[712]: at System.Diagnostics.StackTrace..ctor (System.Exception,int,bool,bool) <0x0004b>
Mar 22 07:20:56 HairyDemon mono[712]: at System.Exception.get_StackTrace () <0x00187>
Mar 22 07:20:56 HairyDemon mono[712]: at System.Exception.ToString () <0x00127>
Mar 22 07:20:56 HairyDemon mono[712]: at NzbDrone.Common.Instrumentation.GlobalExceptionHandlers.HandleAppDomainException (object,System.UnhandledExceptionEventArgs) <0x00084>
Mar 22 07:20:56 HairyDemon mono[712]: at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object_object (object,intptr,intptr,intptr) <0xffffffff>
EPIC FAIL: System.OutOfMemoryException: Out of memory
Native mono crashes [kernel fix released]
#2

Messed up the installation

I believe I can reproduce SIGSEGV now by searching for all missing episodes (only verified once so far). This time traces are included.

Now Sonarr won’t start anymore

After the mess-up

sudo aptitude upgrade
The following packages will be upgraded:
  bsd-mailx bzip2 console-setup console-setup-linux debconf debconf-i18n
  geoip-database grub-common grub-pc grub-pc-bin grub2-common
  keyboard-configuration krb5-locales libbz2-1.0 libbz2-dev libc-bin
  libgcrypt20 libgssapi-krb5-2 libjavascriptcoregtk-1.0-0 libk5crypto3
  libkrb5-3 libkrb5support0 libmp3lame-dev libmp3lame0 libmpeg2-4
  libmpeg2-4-dev libpython-dev libpython-stdlib libssl-dev libssl-doc
  libssl1.0.0 libtcl8.6 libtiff5 libtiff5-dev libtiffxx5 libwebkitgtk-1.0-0
  libwebkitgtk-1.0-common multiarch-support openssl python python-dev
  python-imaging python-minimal python-pil python-pygments tcl8.6 tzdata
  tzdata-java vim vim-common vim-runtime vim-tiny
52 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 38.2 MB of archives. After unpacking 264 kB will be freed.
Do you want to continue? [Y/n/?]

and reboot. My NAS complained like it saw no tomorrow. / and my download partition both failed with errors. I had to manually root in and fsck. After a few fscks and reboots, Debian was back up and running. Fully upgraded, Sonarr would start (take quite a while) and communicate with NZBGet again.

Now it is running and even a full search for all missing episodes does not crash Sonarr. Let’s see when/if the next crash occurs. I might note that I received one error in Sonarr’s log. (I can’t really add a real log as trace logging just explodes Sonarr’s log directory. It is generating about a new log file every 20 seconds)

EPIC FAIL: Argument cannot be null.
Parameter name: obj: Argument cannot be null.
Parameter name: obj

System.ArgumentNullException: Argument cannot be null.
Parameter name: obj
  at System.Threading.Monitor.TryEnter (System.Object obj, Int32 millisecondsTimeout, System.Boolean& lockTaken) [0x00000] in <filename unknown>:0 
  at System.Threading.Monitor.Enter (System.Object obj, System.Boolean& lockTaken) [0x00000] in <filename unknown>:0 
  at (wrapper unknown) System.Threading.Monitor:FastMonitorEnterV4 (object,bool&)
  at System.Net.EndPointListener.OnAccept (System.Object sender, System.EventArgs e) [0x00000] in <filename unknown>:0 
  at System.Net.Sockets.SocketAsyncEventArgs.OnCompleted (System.Net.Sockets.SocketAsyncEventArgs e) [0x00000] in <filename unknown>:0 
  at System.Net.Sockets.SocketAsyncEventArgs.AcceptCallback (IAsyncResult ares) [0x00000] in <filename unknown>:0 
  at System.Net.Sockets.SocketAsyncEventArgs.DispatcherCB (IAsyncResult ares) [0x00000] in <filename unknown>:0

Crash again after reboot

Here you can find the 3MB journalctl. This time it was a SIGABRT. I disabled following extras:

  • Drone Factory scanning
  • Kodi connection
  • Pushbullet connection
  • Kodi metadata

Disabling those settings did not help. I re-enabled metadata fetching. I can now reproduce a mono SIGSEGV crash by searching for 181 missing episodes. Here is the log corresponding to the most recent crash.

#3

Just for posterity: Errors were related to physical memory issue as memtest indicated.

#4