Regular crashes on Raspberry Pi 2

Help & Support
#1

Hi,

i have major stability issues with Sonarr on my RPI 2 running Raspbian Jessie. I reinstalled everything from scratch yesterday but still experience the same problems. Sonarr and mono is installed according to the Linux instructions from the GitHub wiki (with the mono-project.com repository for mono-devel).

Today i run Sonarr in debug mode and saved the crash dump on pastebin http://pastebin.com/raw/pDQRUKCq

Version in use:
Raspberry Pi 2 Model B with latest firmware
OS Raspbian Jessie “light” version:

Linux 4.1.15-v7+ #830 SMP Tue Dec 15 17:02:45 GMT 2015 armv7l GNU/Linux

Sonarr:

[Info] Bootstrap: Starting Sonarr - /opt/NzbDrone/NzbDrone.exe - Version 2.0.0.3645

Mono:

Mono JIT compiler version 4.2.1 (Stable 4.2.1.102/6dd2d0d Thu Nov 12 10:06:49 UTC 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       normal
	Notifications: epoll
	Architecture:  armel,vfp+hard
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            sgen

I have prior to Sonarr over 700 MB of free memory, with Sonarr running idle about 600 MB of free memory. In addition to Sonarr i have NZBGet (native), Couchpotato (python), Samba (native) and a Wifi controller (java / tomcat) running without any issues.

Any ideas how to find the root cause of this crash?

#2

These two threads would be the first thing to look at.


#3

Any ideas for a workaround? Is there any experience with the 4.4 kernel?

#4

No workaround as far as I know. Not sure if 4.4 fairs any better.

#5

I am updating to kernel 4.4 right now and will give feedback in one or two days.

#6

I’m reluctant to blindly attribute this one to the native crash issue since the error is somewhat different, still has a reasonably high probability.
I’m also no entirely sure about the stability of mono 4.2.x, not sure why, maybe I heard something on IRC once about 4.1 vs 4.2 mono. (pretty confusing to be talking about kernel 4.1/2 and mono 4.1/2 at the same time :slight_smile: )

@pawi How experienced are you with linux? If you’ve ever compiled a linux kernel yourself, now would be the time to let me know.

#7

Already have crashes with kernel 4.4. The database is now corrupted and Sonarr crashes on startup. I will reimport an old database and start again in debug mode to verify if it is still the same error as initial reported. @Taloth my last kernel compile is about 10 years ago but i should get it done. However i have to find out if i get drivers for the raspberry which may be an issue.

#8

After recovering from the crash, catching up on some shows i started again with debug mode. Sadly i have another (random?) crash log:

Linux Kernel upgraded to 4.4

Linux 4.4.0-rc7-v7+ #831 SMP PREEMPT Mon Dec 28 19:14:55 GMT 2015 armv7l GNU/Linux

For Sonarr and Mono the version are the same as initial reported, there is no update available at the time.

#9

Could you do a memtest for 12h+? To be sure there aren’t any memory issues?

I’m a bit skeptic about having both mono and the kernel at the newest versions. Too many unknowns.
Rolling back the kernel is probably a bit more problematic, so I would just roll back mono to something lower, atm I have 4.0.4.

But anyway, first do the memtest, not sure how you can do that on a rasp.

#10

I only found a userspace memtester which i have run. I disabled almost all services to have max. memory available (about 930 from 1000). I found a bit flip on the second loop of memtester.

I reduced the “recommended” overclock for the sdram to the original setting of 450 MHz (overclock 500 MHz) and rerun the memtester right now. As soon i have verified i get reliable results from my raspberry i will try further with your recommended versions. Its not a problem to “restart” with kernel 4.2 and mono 4.0.4 if you prefer.

#11

I have run the memtester again with 450 MHz RAM speed and tested 900 MB RAM (to have access to Samba without too much performance loss). I had no issues with the Raspberry for 24h. The log can be viewed under: http://pastebin.com/raw/FaW9FpjB (i interrupted after 24h)

I will try again to test Sonarr with the latest version of Kernel, Mono and Sonarr. If i get any errors i will downgrade Mono (as you suggested) and afterwards the Kernel and will post the crash dumps.

Thanks for your suggestions so far! I will continue my quest and report my findings.

#12

Awesome, now lets hope my hunch was correct.

#13

After almost 48h without crashes i assume that my random crashes are solved due to the bit flip of my overclocked Raspberry Pi 2. It is a surprise for me as the other software i mentioned worked fine and only Sonarr crashed for me. I assume that mono is somewhat more fragile on this topic or Sonarr pushed the RAM usage to an level where the bit flip was “visible” or had an impact.

Solution: i reverted to the default SDRAM clock setting of 450 MHz (everything else is still overclocked according to the “RPi 2: Pi2 Settings” from the raspi-config tool. The SDRAM can be set in

/boot/config.txt

Changed sdram_freq to 450 (which is the default sdram frequency):

sdram_freq=450

Thanks @Taloth for pointing me to the RAM. Thanks to the Sonarr Team for this awesome software!

#14

Great to hear you found the issue. (And tnx for not shrugging it off as ‘my hardware is fine’ :smiley: )

I can explain why Sonarr is more susceptible to memory issues, feel free to skip it if you don’t care. :wink:

In normal applications, the binary executable is loaded into memory and remains read-only while being executed.
mono is a jit-compiler, which means that the parts of the application are compiled on demand (at runtime) from the so called Intermediate Language to native machine instructions. This means that mono will reserve a piece of memory, compile the code of a particular function, and then change the flag of that memory to make it ‘executable’ before finally calling into that function.
This means that the executable memory portion is much more fluid and thus more susceptible to corruption (more writes = higher chances of bit-flips).
The total memory usage is, of course, also a contributing factor.

I once had another user with inexplicable crashes, I eventually convinced him to run a memory test. Surprisingly it took over 7 hours before the first errors were detected.

#15

If i come for help, i better listen =)

Just to dig deeper and for my understanding into the jit-compiler topic. Shouldn’t the Wifi controller which runs with java and tomcat behave the same way? On the other hand it may be not that resource intensive as Sonarr and it was just pure luck that i didn’t experience any crashes with the controller.

#16

Could be luck, but Sonarr is definitely more dynamic in terms of allocating and freeing memory.

The mono native crashes we mentioned (which is caused by a issue on the edge with the linux kernel, still dunno if it’s a bug in mono or the kernel). That problem was also intermittent. But would blow up on a VM with multiple cpus and once I got a testcase on it, I could reproduce between 1-30 sec.
Stuff like this is usually a combination of factors, which makes it very hard to debug.

#17

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.