Sonarr Constantly Crashing On Ubuntu 15.10

Help & Support
#1

Sonarr Version: 2.0.0.3953
Mono Version: 4.2.2 (Stable 4.2.2.30/996df3c)
OS: Ubuntu 15.10
Kernel: 4.4.4-std-3

Problem:
Sonarr keeps crashing for me after processing a few downloads. I have it running as a service via systemd. Once it crashes I can no longer access the web interface even though the process is technically still running. When I check the status of the process it shows the following from the logs. Any ideas? I’m guessing it’s a simple fix- I’m just a noob when it comes to Linux.

Mar 14 20:40:07 plex mono[3046]: [Debug] Parser: Release Group parsed: NTb
Mar 14 20:40:07 plex mono[3046]: [Debug] EpisodeFileMovingService: Moving episode file: /home/plex/amazon/Temp/complete/sonarr/Masters.of.Sex.S03E11.Party.Of.Four.720p.HULU.WEBRip.AAC2.0.H.264-NTb/0fc2a75d31ff48cbf94f92ef8875aa85deed5aa1.mkv to /home/plex/amazon/Media/Television/Masters of Sex/Season 03/Masters of Sex - S03E11 - Party of Four [WEBDL-720p - NTb].mkv
Mar 14 20:40:07 plex mono[3046]: [Debug] DiskTransferService: Move [/home/plex/amazon/Temp/complete/sonarr/Masters.of.Sex.S03E11.Party.Of.Four.720p.HULU.WEBRip.AAC2.0.H.264-NTb/0fc2a75d31ff48cbf94f92ef8875aa85deed5aa1.mkv] > [/home/plex/amazon/Media/Television/Masters of Sex/Season 03/Masters of Sex - S03E11 - Party of Four [WEBDL-720p - NTb].mkv]
Mar 14 20:40:07 plex mono[3046]: [Debug] DiskProvider: Hardlink ‘/home/plex/amazon/Temp/complete/sonarr/Masters.of.Sex.S03E11.Party.Of.Four.720p.HULU.WEBRip.AAC2.0.H.264-NTb/0fc2a75d31ff48cbf94f92ef8875aa85deed5aa1.mkv’ to ‘/home/plex/amazon/Temp/complete/sonarr/Masters.of.Sex.S03E11.Party.Of.Four.720p.HULU.WEBRip.AAC2.0.H.264-NTb/0fc2a75d31ff48cbf94f92ef8875aa85deed5aa1.mkv.backup~’ failed.
Mar 14 20:40:07 plex mono[3046]: [v2.0.0.3953] System.IO.IOException: Read-only file system —> Mono.Unix.UnixIOException: Read-only file system [EROFS].
Mar 14 20:40:07 plex mono[3046]: — End of inner exception stack trace —
Mar 14 20:40:07 plex mono[3046]: at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () <0x404edfa0 + 0x00013> in :0
Mar 14 20:40:07 plex mono[3046]: at Mono.Unix.UnixMarshal.ThrowExceptionForLastErrorIf (Int32 retval) <0x404edf80 + 0x00013> in :0
Mar 14 20:40:07 plex mono[3046]: at Mono.Unix.UnixFileSystemInfo.CreateLink (System.String path) <0x404edd00 + 0x0002b> in :0
Mar 14 20:40:07 plex mono[3046]: at NzbDrone.Mono.DiskProvider.TryCreateHardLink (System.String source, System.String destination) <0x404ecfa0 + 0x00037> in :0

#2

I noticed Sonarr just updated and in the change log now has some hardlink setting on. I found the option and turned it off. We’ll see if that helps- I’ll report back if it crashes.

Edit: Nope still crashes with the same mono i/o error and mention of a hardlink in the previous line.

Edit 2: Here are the logs this time. The weird thing is it’s seemingly random. It will download/process a few files fine and then randomly crash.

Mar 14 22:23:34 plex mono[3086]: [Debug] DiskProvider: Hardlink ‘/home/plex/amazon/Temp/complete/sonarr /The.Venture.Bros.S06E07.A.Party.for.Tarzan.720p.WEB-DL.DD5.1.H.264-CtrlHD/0a5a48dee8ea7f6997a6e5ab3ab1 016b123c5952.mkv’ to ‘/home/plex/amazon/Temp/complete/sonarr/The.Venture.Bros.S06E07.A.Party.for.Tarzan .720p.WEB-DL.DD5.1.H.264-CtrlHD/0a5a48dee8ea7f6997a6e5ab3ab1016b123c5952.mkv.backup~’ failed.
Mar 14 22:23:34 plex mono[3086]: [v2.0.0.3953] System.IO.IOException: Read-only file system —> Mono.U nix.UnixIOException: Read-only file system [EROFS].
Mar 14 22:23:34 plex mono[3086]: — End of inner exception stack trace —
Mar 14 22:23:34 plex mono[3086]: at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () <0x41bc3aa0 + 0 x00013> in :0
Mar 14 22:23:34 plex mono[3086]: at Mono.Unix.UnixMarshal.ThrowExceptionForLastErrorIf (Int32 retval) < 0x41bc3a80 + 0x00013> in :0
Mar 14 22:23:34 plex mono[3086]: at Mono.Unix.UnixFileSystemInfo.CreateLink (System.String path) <0x41b c3800 + 0x0002b> in :0
Mar 14 22:23:34 plex mono[3086]: at NzbDrone.Mono.DiskProvider.TryCreateHardLink (System.String source, System.String destination) <0x41bc2aa0 + 0x00037> in :0

Edit 3: On the forums I’ve read that sonarr has some issues with certain kernels. On my Ubuntu 15.04 server Sonarr was running perfectly under kernel 3.19.0-22-generic. My new Ubuntu 15.10 server has a kernel of 4.4.4-std-3 and Sonarr is constantly crashing.

Do you think the kernel version is the problem? If so would it be possible to downgrade my kernel without messing up my OS? Which version of Ubuntu server would you recommend for Sonarr?

#3

There have been a few kernel issues, the issue starts manifesting itself when the kernel is updated, but it could be a mono issue.

This one was eventually fixed:

Then this one happened and it may still be an issue:

I haven’t seen that particular error before and it looks different than the previous kernel issues we saw. Running mono with the --debug flag would give us more information to work with.

#4

Thanks for the reply! So I disabled sonarr as a systemd service and ran it manually with the --debug flag. Intermittently I’ll still see the following mono i/o error in my terminal window but sonarr seems unaffected now. I can access the web interface and and all episodes are processed correctly.

This leads me to believe it might be an issue with sonarr running via systemd on my machine. I’ll give it some time and see if it continues to work properly over the course of the day. In the meantime I’ll try to find the mono logs and post those.

Edit: Also the directory /home/plex/amazon/Temp/complete/sonarr/ is my FUSE mounted Amazon Cloud Drive. Maybe that is the cause of the i/o errors?

Edit 2: Even when run manually it still crashed after about 2 hours. I Google’d a bit but I’m not sure how to find the mono debug logs. If you let me know I can throw them in a pastebin.

[Debug] EpisodeFileMovingService: Moving episode file: /home/plex/amazon/Temp/complete/sonarr/The.Middle.S01E24.DVDRip.XviD-REWARD-Obfuscated/wKwFoLzj1EtcW1PuJgRFDbo.avi to /home/plex/amazon/Media/Television/The Middle/Season 01/The Middle - S01E24 - Average Rules [DVD - REWARD].avi
[Debug] DiskTransferService: Move [/home/plex/amazon/Temp/complete/sonarr/The.Middle.S01E24.DVDRip.XviD-REWARD-Obfuscated/wKwFoLzj1EtcW1PuJgRFDbo.avi] > [/home/plex/amazon/Media/Television/The Middle/Season 01/The Middle - S01E24 - Average Rules [DVD - REWARD].avi]
[Debug] DiskProvider: Hardlink ‘/home/plex/amazon/Temp/complete/sonarr/The.Middle.S01E24.DVDRip.XviD-REWARD-Obfuscated/wKwFoLzj1EtcW1PuJgRFDbo.avi’ to ‘/home/plex/amazon/Temp/complete/sonarr/The.Middle.S01E24.DVDRip.XviD-REWARD-Obfuscated/wKwFoLzj1EtcW1PuJgRFDbo.avi.backup~’ failed.

[v2.0.0.3953] System.IO.IOException: Read-only file system —> Mono.Unix.UnixIOException: Read-only file system [EROFS].
— End of inner exception stack trace —
at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () <0x41e866a0 + 0x00013> in :0
at Mono.Unix.UnixMarshal.ThrowExceptionForLastErrorIf (Int32 retval) <0x41e86680 + 0x00013> in :0
at Mono.Unix.UnixFileSystemInfo.CreateLink (System.String path) <0x41e86400 + 0x0002b> in :0
at NzbDrone.Mono.DiskProvider.TryCreateHardLink (System.String source, System.String destination) [0x00000] in m:\BuildAgent\work\6c3239faf2b92630\src\NzbDrone.Mono\DiskProvider.cs:162

[Debug] EpisodeService: Linking [Season 01/The Middle - S01E24 - Average Rules [DVD - REWARD].avi] > [[5298]Average Rules]
[Debug] QualityUpgradableSpecification: Existing item meets cut-off. skipping.
[Debug] QualityUpgradableSpecification: Existing item meets cut-off. skipping.
[Debug] QualityUpgradableSpecification: Existing item meets cut-off. skipping.
[Debug] QualityUpgradableSpecification: Existing item meets cut-off. skipping.
[Debug] QualityUpgradableSpecification: Existing item meets cut-off. skipping.
[Debug] DiskScanService: Scanning ‘/home/plex/amazon/Temp/complete/sonarr/The.Middle.S01E24.DVDRip.XviD-REWARD-Obfuscated’ for video files
[Debug] DiskScanService: 0 video files were found in /home/plex/amazon/Temp/complete/sonarr/The.Middle.S01E24.DVDRip.XviD-REWARD-Obfuscated
[Debug] DownloadedEpisodesImportService: Deleting folder after importing valid files

#5

When Cntrl+C the sonarr process I got:

/build/gdb-HnfxP_/gdb-7.10/gdb/dwarf2-frame.c:688: internal-error: Unknown CFI encountered.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) [answered Y; input not from terminal]

This is a bug, please report it. For instructions, see:
http://www.gnu.org/software/gdb/bugs/.

/build/gdb-HnfxP_/gdb-7.10/gdb/dwarf2-frame.c:688: internal-error: Unknown CFI encountered.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n) [answered Y; input not from terminal]

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.

Aborted (core dumped)

Is this the kernel issue? I’m running on a Scaleway VPS and I know it is more common on virtual servers. I have no problem reinstalling a different OS. I just want to make sure I pick the right one before I go through the hassle.

#6

I don’t know the specifics as to why, but another using that was using ACD recommended that ACD mounted over FUSE be read-only in Sonarr.

@Ice_Nine I believe this was your suggestion.

#7

Hmm to test I set Sonarr and Sabnzbd to local paths and it still crashed (although I didn’t notice the intermittent mono i/o errors). On my old Ubuntu 15.04 install Sonarr worked fine writing to ACD mounted via FUSE- not a single crash in two moths. I also tried uninstalling Sonarr and Mono then reinstalling with no success.

My limited intuition tells me it’s a problem with Ubuntu 15.10, Scaleway VPS, or their specific Ubuntu image.

#8

There was a time where writing to ACD via acd_cli was actually feasible - mostly because it wasn’t adopted by the masses for the purposes of storing entire video libraries.

As time went on though, more and more people started loading up videos on ACD for all their video storage needs. The result is that ACD is now becoming less and less viable as a storage platform for people’s video content.

This might be because of Amazon throttling it so it’s less viable to be used this way, or simply because too many people are using the service concurrently for this purpose. The end result is that ACD is no longer what it used to be performance-wise, and you would be best served by using it only sparingly for content that isn’t accessed frequently (in my case, cancelled TV shows that are rarely watched, etc).

As for these crash issues, they do seem peculiar. I don’t see any specific crashing issues with regards to ACD, but do be aware that ACD is a “beta” product that isn’t terrific at handling concurrency very well. And yes, I do keep acd mounted as RO, as a failed half-copied file will wind up remaining a failed half-copied file if you rely on sonarr to move your stuff around. Sonarr and acd_cli aren’t “aware” of each other, so if something bombs during a copy at acd_cli doesn’t know how to handle, you’re going to have a bad time. Don’t do it.

#9

Thanks for the info! Just for curiosity’s sake- what is likely the worst that could happen on a failed write? The single file would be damaged/corrupted and I’d have to redownload it? Or would it be possible that my entire Amazon Cloud Drive could become become corrupted?

As far as the crashing, I think it was a problem with Ubuntu 15.10, my VPS provider, or my own ineptitude. I did a fresh install of Ubuntu 14.04 LTS on a new Scaleway VPS instance and Sonarr has had no issues running for the past 24 hours but it may still be too early to tell.

For the most part ACD has been very fast and stable via acd-cli in my limited personal experience. Files transferred to it via acd-cli max out my VPS’ 200mbps upload speed. Plex also has no problem scanning and serving media from it without any buffering/skipping. Although I don’t expect it to always remain this way, it’s working great for me currently.

#10

No, you wouldn’t lose your entire ACD - just the file would be partial and you’d have to re-upload it.

A quick way to show you what could go wrong is to do an ‘acdcli upload’ of a directory with, say, a couple of seasons worth of shows. Sometimes, the ‘acdcli upload’ command will work flawlessly. But more often than not, something will happen during the upload command which will force you to re-upload that file. The same problem exists when acdcli mounts the remote FUSE FS within linux - if during the middle of a copy, ACD decides to barf on you, you’ll wind up with a partial file.

Say you ran the ‘md5sum’ command against a file you’re going to transfer with sonarr:

root@lateralus> md5sum Archer\ (2009)\ -\ S06E01\ -\ The\ Holdout\ Bluray-1080p.mkv

The output would look something like this:

dd4ee1e228e3137fa5b032b4966620be Archer (2009) - S06E01 - The Holdout Bluray-1080p.mkv

The first field is the md5 hash for that file. lf you think the file on acd is iffy after sonarr has finished copying it, you can run the command ‘acdcli metadata’ command against it to ensure the “md5” field in the “contentProperties” section matches once the copy is finished. If it doesn’t, guess what, you have a partial file. And since sonarr and acdcli don’t communicate in any way, sonarr is going to consider the file 100% copied even if acdcli is forced to stop copying because something got throttled on amazon’s end (and yes, this happens all the time).

Anyway, I’ve had pretty good luck with acd as well - as long as I don’t have more than 1 person at a time watching content off of it (which is, sadly, out of my control as plex doesn’t have any way to limit content based on pathname that i’m aware). Even with my 300/300mbit symmetric connection (and a beefy server to boot), stuttering is constant as soon as a second person attempts to stream another file from acd within Plex. I think that’s more because of the interpreted python code in acdcli though, and not a bandwidth issue.

#11

Weird. I’ve had as many as six people streaming off my ACD mounted via Plex at the same time with no issue whatsoever. All of my content is Web-dl 720p (~4500kbps bitrate) though so if you have extremely high bitrate content that might explain it…

Also is your server local or remote? I’m in the US. I originally had Plex/acd-cli setup on a VPS in France (1gbps symmetric connection) and it would die as soon as someone tried to start a 2nd stream. I moved my setup to a VPS in Canada (only 100mbps symmetric connection) and have been able to stream to six different devices at once. This is purely blind speculation but my best guess is Plex does not play well over such a large distance.

Thanks for the pro tip! That command is definitely gonna come in handy for me. The way I have my downloads setup right now is that Sabnzbd downloads files locally then extracts them to a temporary folder FUSE mounted via acd-cli. Sonarr then scans that temporary folder and moves my files around purely on ACD itself. The big assumption I make is that the files are simply moved- not redownloaded and then reuploaded. The good thing about this setup is Sabnzbd will tell me when a file fails to transfer over. And the fix is as simple as hitting “retry”.

Thanks for all the insight into how acd-cli works. I’ve never used Linux before so I’m a super noob to all this. People on various forums and reddit have been so helpful that this task has quickly gone from daunting to fun. :slight_smile:

#12

In the interests of keeping this thread on-topic and uncluttered, you can ask me any questions regarding my (rather involved 120TB) setup by emailing me at admin@notflyx.com or visiting the robust community (including the developers of sonarr) on internet relay chat (irc) at irc.freenode.net in channel #sonarr.

https://webchat.freenode.net/

#13

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.