OK. This has been happening for a few months now, but I just went and restarted Sonarr and went on my marry way. However, now it is happening so frequently that I’m missing a lot of shows.
Sonarr will work fine for the most part, but then all of a sudden, it will crash and I have to drop into the shell to restart it (the reload button doesn’t work).
I’m not that great at debugging with linux and would appreciate any help you guys could offer.
My setup:
Linux: Korora 21 (based on Fedora 21)
Sonarr Ver. 2.0.0.4731
Mono Ver. 4.0.1
App Directory: /home/sonarr/.config/NzbDrone
Startup directory: /opt/NzBDrone
Owner @ Permissions on
/opt: nighthawk:nighthawk @ 755
/opt/NzbDrone: sonarr:xbmc @ 755
Files/directories within are owned: sonarr:xbmc @ 644 (directories 755)
DOWNLOADER: NzbGet (currently not working, can’t get it to start)
LOG FILES: https://pastebin.com/0ugJd5sg
Console output: https://pastebin.com/q04nb0hP
Kernel: > uname -a
Linux msi-desktop.nighthawk.local 4.1.13-100.fc21.x86_64 #1 SMP Tue Nov 10 13:13:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
You should check the console output for the crash coz those log files contains none.
My system is systemd based (journalctl).
Can you advise how to do this?
I think google provides an excelent answer for that, don’t you think?
Don’t really need a smart-ass remark. If I knew what to search for then I would search it.
That’s like saying… “Oh, you have a problem, why don’t you just fix it”.
Obviously, I’m coming here because I don’t know HOW to do that.
I have updated the above post to include new logs. After I restarted Sonarr last time, I went into the Settings and changed the logs to TRACE. Then I waited for it to crash again. The logs listed above now include the TRACE logs from within journalctl.
I would appreciate any help possible. Thanks!
OK. Through searching more on this website, I was able to see what the heck Taloth was talking about. I’ve now updated my original post to include a console output from Mono.
It seems that Mono is having an EPIC FAIL! However I don’t know what the rest of the jibberish is.
What do I need to do to fix this?
Thank you. I was typing this up based on the systemd logs, before you posted the separate console logs.
I’m actually surprised that the EPIC FAIL in the console log doesn’t appear in the systemd logs, coz on my system (I still use upstart) the console log gets piped into the log system.
Anyway, first based on systemd logs, what I already typed up:
May 03 10:57:01 mono[12260]: [Debug] MetadataService: Episode image already exists: /media/stora... May 03 10:57:11 systemd[1]: sonarr.service: main process exited, code=killed, status=6/ABRT
So we can rule out a mediainfo crash (something we see happen with certain vids & mediainfo version).
Looks like mono/Sonarr is busy with the metadata, then nothing for 10 sec before that systemd. ABRT (SIGABRT) is a signal that gets sent when a process kills itself due to a critical failure. This could be mono, or just a standard linux library that detects a critical issue, such as memory corruption. But it could be a number of things.
If it happens so frequently and over a long period of time, I would first suggest you do memory test overnight. I’ve seen the most weird shit happen due to damaged physical memory, so it’s worth ruling that out. The ‘getting worst’ part is definitely a sign.
Plenty of linux distributions offer a memtest boot option, I don’t know of fedora has one too. Please check that out.
Btw. With ‘overnight’ I really mean overnight: run the memory test for like 6 hours. I had users find memory errors after hours of memtest running that ultimately proved to be the cause of mono crashing randomly.
Next the console log:
The error is insane. The hash algo 1.2.840.113549.1.1.11 is sha256 with RSA. That algorithm has existed in mono since early 2.x versions, we’re at 4.x now. So it’s basically giving an error that can’t happen unless something else goes horribly wrong.
It could be caused by memory corruption, so I’d definitely test the memory first. If the memory proves to be fine, then we should move on to updating mono.
So TLDR: Do a memtest for 6+ hours.
If you have questions, lemme know.
Thank you. That was tremendously helpful!
Quick question… With a mem test. What I can find seems to basically take over the computer for X hours. Meaning, that is the only thing running? So, I won’t be able to get to the shows on my disks while that test is running?
Or, are you referring to something that runs in the background while everything is operating?
Yup, the memtest86 takes over the computer, it does reads and writes over the entire physical memory and stress tests it with a bunch of patterns to check for ‘bitflips’, that’s why you run it from the boot menu/livecd instead of linux.
There are memtests that run from inside linux, but they are not as thorough, having to avoid parts of the memory that are currently in use.
That’s why I’d suggest you start it at the end of the day, then you’ll be able to check for the results in the morning.
OK. It ran from about midnight to 9:15am – so tons of time. 0 Errors and 4 Passes
Here is the memory details it gave me:
16GB 666mhz
DDR3-13333
9-9-9-24 @ 128 bit
BCLK: 100
– Brand: G. Skill
I know that my motherboard is a gaming motherboard and has the ability to overclock and a ton of other settings. However, I don’t know if I have those settings correct? So if I need to change anything above, let me know please.
It also looked like I had 2 versions of Mono installed for some reason 2.10.xx, but when I went to remove it, yum wanted to remove Libre Office, Banshee and all of my other programs that use Mono. So, I can’t seem to get that removed. I was able to remove Mono 4.0.1 and am working on re-installing 4.8.xx (latest). But, I don’t understand why the system won’t let me remove 2.10.x without removing everything.
Yup 9h ought to be enough for now.
Based on your first post, Sonarr was definitely running mono 4.0.1, although I honestly don’t know if it could be affected by another 2.x install present.
Unfortunately I don’t have enough Fedora knowledge to be able to assist with updating 2.10.x, but install 4.8 and lemme know how it goes.
PS: can you also post the output of uname -a, it gives the linux kernel version, amongst other things. It’s not a priority but it’s good to know.
I have uname -a in the original post at the bottom.
I’m going to get 4.8 installed today and will report back.
OK. I have installed Mono 4.8.1.0 and uninstalled 2.10 completely.
Now I can’t start Sonarr at all. Here is my attempt using # mono NzbDrone
https://pastebin.com/2aqQiefD
OK. I’ve got Sonarr starting now. I had to change the systemd service file I created to point to a different location of mono.
HOWEVER… My database is gone! It looks like all my settings and everything.
That is really weird. I ran the command manually and it loaded a completely different database. I ran the exact same command from a systemctl (service) and it started with my db intact.
I’m hoping that it doesn’t crash now with mono 4.8
Fingers crossed! Thanks for pointing me in the right direction.
Sry about the uname, I missed that.
Sonarr stores the database in the user home directory, which is different depending on which user Sonarr is running under, hence the difference you saw.
Fingers crossed.
Hey… Just wanted to say thank you. Sonarr seems to be running okay now. I’ll start a new thread if it stops working.
I’m currently trying to fix sabnzbd now, since I was using nzbget before and decided to switch.
Thanks again though!
1 Like
Tnx for letting us know.
@markus101 Should we keep a list of potentially broken mono versions? It could’ve been interference from 2.10 being present as well, but just as likely it was caused by 4.0.1.
Probably wouldn’t hurt so we don’t need to try and keep track of it in our heads.