Sonarr seems to have crashed - prevention and explenation

[seePyou]

Hello all,

What I’m looking for here is not a “blame you” kind of thing. Just wondering if you can tell me what you might see, and how I can prevent this from happening again. It may just be me doing something wrong.

So I was doing some backlog catchup this morning, going over my shows that Sonarr is handling, and on some there are a few holes. I went into each one that had something missing, and did a manual or automatic search. All was going fine and episodes were being queued up for download and processing.

Then I made a “mistake” in one show asking it to search for all missing but monitored episodes, the big magnifying glass on top. Up until now, I was just using the search by the episode itself.

It started OK, redownloading the DVD-rip versions of season 1 and then started working on s02 and s03. But the web page was frozen.

I hit F5 and the browser just said “unable to connect”.

I remoted to my box and saw the logs, which have been set to debug as I wanted to catch such a situation.

here are the logs from oldest to newest:

http://paste.ubuntu.com/15130904/
http://paste.ubuntu.com/15130907/
http://paste.ubuntu.com/15130909/
http://paste.ubuntu.com/15130911/
http://paste.ubuntu.com/15130912/
http://paste.ubuntu.com/15130914/
http://paste.ubuntu.com/15130915/

I’m sorry to say that this is too much, and I realize that it may even be too much for you guys. I do not know if the debug logs should have an [error] flag if an error was recorded instead of the [debug] flag, but I did search for an [error] flag and found nothing.

Thinking that this may be the entire Sonarr crashed and not just a web listener, I pulled a second log of the nzbdrone.txt to see what it is doing:

http://paste.ubuntu.com/15130954/

as you see this is the same log file as the 15130915 one, meaning that there is no logging taking place, so I believe that the whole Sonarr crashed.

I restarted it with the command:

mono --debug /opt/NzbDrone/NzbDrone.exe

but that did not seem to bring it back. In fact it even reported that

[Fatal] ConsoleApp: Address already in use. This can happen in another instance of Sonarr is already running another application is using the same port (default : 8989) or the user has insufficient permissions

I checked the port and I saw this.

Doing a ps shows only bash and ps running, so no Sonarr or mono.
Running top also shows no mono or Sonarr running

based on the above then, anyone has any ideas on what I could try to see before I restart the computer ?

[markus101]

First thing to look at:

Native mono crashes [kernel fix released]

“For every action there is an equal and opposite reaction.” We’ve had quite a few reports about odd crashes, most of them known to be so called Native crashes, where mono cries out in terror and dies without telling us why. Initially this thread was to collect important information about the systems on which the crashes occurred. Since this thread was posted I’ve literally spent more than a hundred hours investigating this issue. Debugging Sonarr and mono countless times, compiled mono half …

Also mono version, needs to be 3.10 or higher.

We’d also need to know the reason it crashed, if the logs don’t have it (I didn’t look at all the log files), you’ll need to capture the standard error and output to a separate file as its likely mono crashing and Sonarr is unable to log.

[seePyou]

Thank you for this.

My kernel is reported as 3.2.0-98. The article mentioned that the first detected kernel version for this error is 3.13.0-64. So, mine is older? And it should not happen?

My Mono version is 4.2.1

For the standard error logs that you mention, can you assist me on how to do that?

[markus101]

Yeah, that looks like a much older version, likely not the exact same issue, but it could be similar. We’ll know more once we know why its crashing.

Redirect stderr and stdout in Bash

[seePyou]

If I read this correctly, because it is highly likely I do not, I should reboot my computer, and then do:

mono --debug /opt/NzbDrone/NzbDrone.exe &>stdoutstderr.log

Is that right?

If not, I do not understand what the “command &>filename” means, which is the accepted answer.
(I did read up a lot on this article and the other sources, but it’s too advanced for me. If the files are always open and can be referenced as 0, 1 and 2, why this this command not employing any of these descriptors? Since they are files, why can I not just copy them and have historical data that way? I expect I’m talking nonsense though, as I do not understand the subject matter )

[markus101]

We need to know why Sonarr crashed. If Sonarr crashed because mono crashed Sonarr’s logs won’t have anything (likely), because the entire framework crashed.

That command redirects the output of the console (which Sonarr runs as) to a file, so everything that would have been output to the console window will be written to the file instead.

You’ll need to reboot then run that command, depending on how you run it you may need to keep your terminal session open, or use screen to keep it running.

[seePyou]

the command being the one I provided? or something else?

[markus101]

The one you provided.

[seePyou]

Hello,

ok I did as we planned.

Rebooted the machine, Sonarr was not started, so I started it with this:

Then I tried to connect to it, but I could not. Maybe the command blocks it from working? I do not know. I still cannot connect to it and see the web pages.

I let it like this for a while, many hours actually, until I saw that the stdoutstderr.log file was not changing size. Then I pastebined it:

http://paste.ubuntu.com/15162499/

it’s very big, but I do not see how that differs from the nzbdrone logs, but you are the experts, not me.

I tried stopping the command from running with ctrl-C but it did nothing. Tried all the other interrupts as well, (as one can see from the picture I’ve added up above) but still nothing.

Here is the port usage too by the way:

and all jobs

Hope this exercise helped with diagnosing something

it’s still running like that, I have not terminated forcibly anything or done a reboot, just in case you need me to do something while it is like that.

[markus101]

The logs would only be useful if mono was crashing and the output of the crash wasn’t logged by Sonarr, but that doesn’t seem to be the case here.

I’d recommend running a mem test on that machine to see if there are any issues, memory issues can cause mono and subsequently Sonarr to behave erratically.

[seePyou]

the fact that the command

mono --debug /opt/NzbDrone/NzbDrone.exe &>stdoutstderr.log

did not allow me to see the Sonarr web pages does not matter ? How am I to test and push Sonarr if I cannot see the web interface of Sonarr ?

is this normal, expected?

[markus101]

Does it work if you run mono --debug /opt/NzbDrone/NzbDrone.exe?

[Fatal] ConsoleApp: Address already in use. This can happen if another instance of Sonarr is already running another application is using the same port (default: 8989) or the user has insufficient permissions

Sonarr can’t bind to the port again, is it running under another user? Have you tried another port?

[seePyou]

ok… so apparently I’m a damn ass.

Sonarr starts automatically when the computer boots.

however I’ve looked at the init.d and I cannot find anything that has to do with Sonarr or Nzbdrone

can you help the damn sod that is me to figure out how to find where Sonarr starts by itself, so that I can stop it from doing that and then I can get it to log in the special output

[markus101]

Depends on how it was setup on your system (its not something we setup). These are some examples of how people have done it:

Sonarr/Sonarr