Anything?..
I just deleted the entire Linux Container, started from scratch with a new Container (Ubuntu 16.04), re-installed mono, re-installed sonarr, and still ended up with the exact same trace log and fatal error above.
1 Like
This is what happens after I’ve already run
systemctl stop sonarr followed by trying to stop the linux container that sonarr is in.
omg, i was thinking to do what you’ve already done and i was like 99% sure that it would solve the problem! thanks for sharing it and please come back with updates if you have a solution!
Nope. Nothing yet. I’m pretty much at my wits end. I just bought a new RPi3 to bypass proxmox altogether, and see if I have any better luck with that.
Radarr has the exact same cloned environment that Sonarr had, and Radarr is running great.
Due to my completed downloads folder also existing on my proxmox host, trying to migrate sonarr elsewhere is proving very difficult, since it needs access to that folder.
Hopefully some folks can come up with some other ideas. I see no good reason why this behavior would be occurring.
best to upgrade to the latest mediainfo release from the mediainfo repo.
I just realized I completely missed this part and was using apt-get like a dummy. I got mediainfo 18.03.1-1 installed and restarted sonarr. The disk scan service and rss sync was noticeably faster, but ultimately it still locked up and froze within minutes. Here’s the last few lines of the tracelog:
18-4-4 03:39:12.1|Info|RefreshEpisodeService|Finished episode refresh for series: [76177][Saturday Night Live].
18-4-4 03:39:12.1|Debug|RefreshSeriesService|Finished series refresh for Saturday Night Live
18-4-4 03:39:12.1|Trace|EventAggregator|Publishing SeriesUpdatedEvent
18-4-4 03:39:12.1|Trace|EventAggregator|SeriesUpdatedEvent -> DiskScanService
18-4-4 03:39:12.5|Info|DiskScanService|Scanning disk for Saturday Night Live
18-4-4 03:39:12.5|Trace|EventAggregator|Publishing CommandUpdatedEvent
18-4-4 03:39:12.5|Trace|EventAggregator|CommandUpdatedEvent -> CommandModule
18-4-4 03:39:12.5|Trace|EventAggregator|CommandUpdatedEvent <- CommandModule
18-4-4 03:39:12.5|Debug|DiskScanService|Scanning '/media/TV Series/Saturday Night Live' for video files
18-4-4 03:39:14.6|Trace|Scheduler|Pending Tasks: 1
18-4-4 03:39:14.6|Trace|CommandQueueManager|Publishing RefreshSeries
18-4-4 03:39:14.6|Trace|CommandQueueManager|Checking if command is queued or started: RefreshSeries
18-4-4 03:39:14.6|Trace|CommandQueueManager|Command is already in progress: RefreshSeries
18-4-4 03:39:17.2|Trace|DiskScanService|93 files were found in /media/TV Series/Saturday Night Live
Is anything additional logged to the console (standard output/error)? If mono or Sonarr is crashing it may not be able to write to the log file before it falls down, but the standard output usually catches something additional.
Unfortunately no, the above paste is actually from the console output (which i assumed was the same as the tracelog, but i guess they could be different). But i couldn’t even navigate or ctrl+c out of the console anyways.
Under normal circumstances they are, but the output looks different and doesn’t have time stamps.
That’s definitely not a console log output, looks more like the sonarr.trace.txt file.
You should really get the console log in this particular case.
Also, you could try this: Stop Sonarr and change Radarr’s port to 8989 to check if you get the same port in use error. Won’t fix anything but might be useful to know.
You’re using LXC, which I know little to nothing about. The network configuration and port sharing might be completely different.
Get that port-in-use error out of the way first.
Sonarr version: 2.0.0.5163
Mono version: 5.10.1.20
Mediainfo version: 18.03.1
OS: Ubuntu 16.04 LTS via LXC on Proxmox Host
Debug logs:
- Output of
mono --trace=N:ConsoleDriver /opt/NzbDrone/NzbDrone.exe: here (I don’t feel like that worked correctly, please let me know a better command to run if you need different info) - Last available contents of
sonarr.trace.txtprior to freeze here
Description of issue:
I previously posted about this issue and wasn’t able to get a resolution. Since then (for reasons not exclusively related to sonarr) I have tried the following:
- Replaced the physical drives with new enterprise-grade SSDs
- Re-installed the entire underlying Proxmox OS
- Re-installed mono, sonarr, and got the latest
mediainfopackage from their official repository - Deleted my
nzbdrone.dbfile and let it rebuild, to rule out a database corruption issue - Created a full blown Ubuntu 16.04 VM to rule out LXC issues with proxmox
- Downgraded from mono 5.10 to mono 5.8
- Used a Debian 9 LXC instead of Ubuntu, along with Mono v5.12 instead of 5.10
And yet, here I am, still in the exact same situation, with all of the above efforts still ending in the same lock-up behavior. It typically seems to happen during MediaInfo doing parsing, which is what led me to upgrade to the version directly from their repository to begin with.
Move to this thread, since it’s the same issue and hasn’t been closed due to inactivity.
- Capture the standard output/error to a file via redirection, something like so: https://askubuntu.com/a/625230/621585 with trace logging enabled
Thanks for the reply and pointing me in the right direction.
I’ve grown to learn/assume (correct me if I’m wrong) that “trace logging” can really mean two different things:
- trace logging of sonarr (accomplished in the .xml config)
- trace logging of mono (accomplished in CLI)
In this case, I’m assuming we’re interested in what mono itself is doing, so i ran this command:
mono --trace /opt/NzbDrone/NzbDrone.exe &> /var/log/sonarr_output.log
…but it generated a 1.8GB file in under 2 minutes. Is there a better filter command that would be more useful in whatever we’re hoping to find in those outputs?
Mono trace logging is very heavy handed, and rarely what we need (there are cases, but we’d be very explicit in that case).
the mono --trace flag is not what we’re looking for here, just Sonarr’s trace logs, with the console output redirected to a file.
mono --debug /opt/NzbDrone/NzbDrone.exe &> /var/log/sonarr_output.log (with the --debug flag for extra information).
Fortunately/unfortunately, sonarr ran for several hours this time without crashing, so it was long enough to make the sonarr_output.log file unwieldy, and i eventually had to delete it, thinking that a new one would start over in its place. Unfortunately it did not, so this time I was only able to get the trace log again and not the console output.
18-4-11 21:28:34.1|Debug|DiskTransferService|Move [/opt/downloader/completed/tv/Call.the.Midwife.S05E07.1080p.WEB-DL.AAC2.0.H.264-ESQ/5pSFWJ0QywhDYwa06Xq2HoPp9AtMMmWW1D0icdcoUHDGj.mkv] > [/media/TV Series/Call the Midwife/Season 5/Call the Midwife - S05E07 - Episode 7 WEBDL-1080p.mkv]
18-4-11 21:28:34.2|Trace|SymbolicLinkResolver|Checking path /media/TV Series/Call the Midwife/Season 5/Call the Midwife - S05E07 - Episode 7 WEBDL-1080p.mkv for symlink returned error ENOENT, assuming it's not a symlink.
18-4-11 21:28:34.2|Trace|DiskTransferService|Attempting to move hardlinked backup.
18-4-11 21:29:02.8|Trace|Scheduler|Pending Tasks: 2
18-4-11 21:29:02.8|Trace|CommandQueueManager|Publishing CheckForFinishedDownload
18-4-11 21:29:02.8|Trace|CommandQueueManager|Checking if command is queued or started: CheckForFinishedDownload
18-4-11 21:29:02.8|Trace|CommandQueueManager|Command is already in progress: CheckForFinishedDownload
18-4-11 21:29:02.8|Trace|CommandQueueManager|Publishing DownloadedEpisodesScan
18-4-11 21:29:02.8|Trace|CommandQueueManager|Checking if command is queued or started: DownloadedEpisodesScan
18-4-11 21:29:02.8|Trace|CommandQueueManager|Inserting new command: DownloadedEpisodesScan
18-4-11 21:29:02.9|Trace|CommandExecutor|DownloadedEpisodesScanCommand -> DownloadedEpisodesCommandService
18-4-11 21:29:02.9|Trace|CommandQueueManager|Marking command as started: DownloadedEpisodesScan
18-4-11 21:29:02.9|Trace|ConfigService|Using default config value for 'downloadedepisodesfolder' defaultValue:''
18-4-11 21:29:02.9|Trace|DownloadedEpisodesCommandService|Drone Factory folder is not configured
18-4-11 21:29:02.9|Trace|CommandQueueManager|Updating command status
18-4-11 21:29:02.9|Trace|EventAggregator|Publishing CommandExecutedEvent
18-4-11 21:29:02.9|Trace|EventAggregator|CommandExecutedEvent -> TaskManager
18-4-11 21:29:02.9|Trace|TaskManager|Updating last run time for: NzbDrone.Core.MediaFiles.Commands.DownloadedEpisodesScanCommand
18-4-11 21:29:02.9|Trace|EventAggregator|CommandExecutedEvent <- TaskManager
18-4-11 21:29:02.9|Trace|EventAggregator|CommandExecutedEvent -> TaskModule
18-4-11 21:29:02.9|Trace|EventAggregator|CommandExecutedEvent <- TaskModule
18-4-11 21:29:02.9|Trace|CommandExecutor|DownloadedEpisodesScanCommand <- DownloadedEpisodesCommandService [00:00:00.0306390]
I’ll run it again to see if I can get the console output this time.
Is it okay if I split the output files into stdout and stderr to help cut down on file size? Is there an argument to include in the command that would generate new log files in a rotating fashion?
A single file would be best so it doesn’t need to be stitched together and console error should have minimal lines. Not sure if you can have it rotate to a new file after a period of time.
Okay. Hopefully this gets us somewhere…
This is the last portion of the console via sonarr_output.log before the lockup: https://pastebin.com/zUuNvf4B
This is the last portion of sonarr.trace.txt before the lockup: https://pastebin.com/SsniCTj6
To me those entries looks exactly the same, but hopefully you find useful information in it.
Without any further response to go on, I got desperate and just started making wholesale changes until I got a functioning product. I have no way of knowing exactly what was wrong, but I strongly suspect that mono did not like my cifs mount.
I changed all of my TV directory paths to NFS shares, (which also failed), so I ended up creating a bind-mount to a fuse directory on my proxmox host system, which is tied to my Google Drive cloud storage (via plexdrive).
Out of all the storage path options for sonarr to use, I would’ve never thought the most complicated scenario would have been the most likely to work. But I’ve heard countless other stories of folks having problems with mono not playing well with file shares.
I think you’re correct. I moved some of my library to a SMB/CIFS share, and now Mono/Sonarr bomb on every library scan to the point where the service cannot be stopped and I have to killall mono, but that still doesn’t keep mono from causing a reboot or shutdown to hang forever. Apparently Mono/Sonarr and windows shares are just asking for your box to shit the bed every few minutes.
To prove this was the issue, I changed the shares from Windows Shares to NFS (since the shares are on a Windows Server 2012R2 box, which support NFS Share Server) and now mapping the same shares on my Ubuntu box to the NFS instead of Windows shares and Mono/Sonarr are no longer hanging.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.