Auto Update fails on Arch Linux

I installed Sonarr on Arch Linux a few months ago but auto update hasn’t really worked. I managed to get a recent update that included torrent support by re installing sonarr with packer command prior to running the auto update.

With the latest update I am again stuck without auto update. What happens is that it stops running and never restarts with the last log being NZBDrone will restart shortly. I have to run it manually again but it is stuck at the previous version. What could be the issue here?

Permissions of files in nzbdrone_update are “-rw-r–r-- 1 sonarr sonarr”. Are they correct? Files generally are not removed from the folder though sometimes it’s empty after an update attempt.

In case it is related I get these warning when re installing:

warning: directory permissions differ on /usr/lib/sonarr/UI/Handlebars/Helpers/
filesystem: 775  package: 755
warning: directory ownership differs on /usr/lib/sonarr/UI/Handlebars/Helpers/
filesystem: 995:995  package: 0:0
warning: directory permissions differ on /var/lib/sonarr/
filesystem: 775  package: 755
warning: directory ownership differs on /var/lib/sonarr/
filesystem: 995:995  package: 0:0

What does the update log show (it should be available in Update logs in the UI (in the logs section). Usually the issue is permissions related on the start up directory (as its called in System -> Info), but can also be caused by an upstart script attempting to restart Sonarr when its stopped for the update.

Since it’s Arch Linux it uses systemd instead of upstart. Can systemd cause same issues? Here is the latest update log:

15-1-27 22:27:59.4|Info|UpdateApp|Starting NzbDrone Update Client
15-1-27 22:27:59.6|Info|UpdateApp|Updating NzbDrone to version 2.0.0.2722
15-1-27 22:27:59.7|Info|AppFolderInfo|Data directory is being overridden to [/var/lib/sonarr]
15-1-27 22:27:59.7|Debug|UpdateApp|NzbDrone process ID: 27586
15-1-27 22:27:59.7|Debug|UpdateApp|Arguments:
15-1-27 22:27:59.7|Debug|UpdateApp|  27586
15-1-27 22:27:59.7|Debug|UpdateApp|  /tmp/nzbdrone_update
15-1-27 22:27:59.7|Debug|UpdateApp|  /usr/lib/sonarr/NzbDrone.exe
15-1-27 22:27:59.7|Debug|UpdateApp|  /data=/var/lib/sonarr
15-1-27 22:27:59.7|Debug|UpdateApp|  /nobrowser
15-1-27 22:27:59.7|Debug|UpdateApp|Using executing application: /usr/lib/sonarr/NzbDrone.exe
15-1-27 22:27:59.7|Debug|UpdateApp|Executable location: /usr/lib/sonarr/NzbDrone.exe
15-1-27 22:27:59.7|Info|UpdateApp|Starting update process. Target Path:/usr/lib/sonarr
15-1-27 22:27:59.7|Info|InstallUpdateService|Verifying requirements before update...
15-1-27 22:27:59.7|Debug|ProcessProvider|Finding process with Id:27586
15-1-27 22:28:00.1|Debug|ProcessProvider|Found process 27586:mono-sgen [/usr/bin/mono-sgen]
15-1-27 22:28:00.1|Info|InstallUpdateService|Verifying Update Folder
15-1-27 22:28:00.1|Info|ProcessProvider|[27586]: Killing process
15-1-27 22:28:00.1|Info|ProcessProvider|[27586]: Waiting for exit
15-1-27 22:28:00.1|Info|ProcessProvider|[27586]: Process terminated successfully
15-1-27 22:28:00.1|Info|TerminateNzbDrone|Killing all running processes
15-1-27 22:28:00.2|Debug|ProcessProvider|Found 0 processes with the name: NzbDrone.Console
15-1-27 22:28:00.2|Debug|ProcessProvider|Found 0 processes to kill

And here’s another form earlier today:

15-1-27 20:08:37.9|Info|UpdateApp|Starting NzbDrone Update Client
15-1-27 20:08:38.8|Info|UpdateApp|Updating NzbDrone to version 2.0.0.2722
15-1-27 20:08:38.9|Info|AppFolderInfo|Data directory is being overridden to [/var/lib/sonarr]
15-1-27 20:08:38.9|Debug|UpdateApp|NzbDrone process ID: 459
15-1-27 20:08:38.9|Debug|UpdateApp|Arguments:
15-1-27 20:08:38.9|Debug|UpdateApp|  459
15-1-27 20:08:38.9|Debug|UpdateApp|  /tmp/nzbdrone_update
15-1-27 20:08:38.9|Debug|UpdateApp|  /usr/lib/sonarr/NzbDrone.exe
15-1-27 20:08:38.9|Debug|UpdateApp|  /data=/var/lib/sonarr
15-1-27 20:08:38.9|Debug|UpdateApp|  /nobrowser
15-1-27 20:08:38.9|Debug|UpdateApp|Using executing application: /usr/lib/sonarr/NzbDrone.exe
15-1-27 20:08:38.9|Debug|UpdateApp|Executable location: /usr/lib/sonarr/NzbDrone.exe
15-1-27 20:08:38.9|Info|UpdateApp|Starting update process. Target Path:/usr/lib/sonarr
15-1-27 20:08:38.9|Info|InstallUpdateService|Verifying requirements before update...
15-1-27 20:08:38.9|Debug|ProcessProvider|Finding process with Id:459
15-1-27 20:08:39.8|Debug|ProcessProvider|Found process 459:mono-sgen [/usr/bin/mono-sgen]
15-1-27 20:08:39.8|Info|InstallUpdateService|Verifying Update Folder
15-1-27 20:08:39.8|Info|ProcessProvider|[459]: Killing process
15-1-27 20:08:39.8|Info|ProcessProvider|[459]: Waiting for exit
15-1-27 20:08:39.8|Info|ProcessProvider|[459]: Process terminated successfully
15-1-27 20:08:39.8|Info|TerminateNzbDrone|Killing all running processes

other looks look like the second one regarding the process not found section.

I have also tried changing permissions to 775 of both sonarr directories with no change. I am running sonarr under sonarr:sonarr user:group. I have noticed that Startup directory files have root owner after re-install is that normal?

Re-install via what mechanism? I know apt-get will do that for our Debian package.

I’m not sure, its Upstart’s respawn flag that causes issues, if thats removed everything works fine. OS X’s plist also had a flag that would terminate any spawned processes when the main process was terminated, which broke the update process because it would get killed after it stopped the main Sonarr process.

Not really anything in the logs there, it kills off the existing process then stops shortly after, feels a bit like the issue with OS X, but I’m 100% sure it is.

Re-installed with sudo pacman -S sonarr. Root owner makes sense, but could that cause issues?

If the files are not 777 then Sonarr wouldn’t be able to update them, unless its running as root or in the same group as root, both of which are scary (worse than 777ing that directory).

I still think there is something else going on though, since if permissions prevent the folder from being emptied and updated, then it should still restart Sonarr.

You are right, running these commands did not fix the issue:

sudo chown -R sonarr:sonarr /usr/lib/sonarr
sudo chmod -R 775 /usr/lib/sonarr

The update folder does get re-created if I delete it btw. What else can I try? Quite a serious issue since I can’t update at all or at least until the AUR package gets updated…

Here’s how the Sonarr systemd file looks like:

[Unit]
Description=Sonarr Service
After=network.target

[Service]
User=sonarr
Group=sonarr
ExecStart=/usr/bin/sonarr -nobrowser -data=/var/lib/sonarr
Type=simple
TimeoutStopSec=20

[Install]
WantedBy=multi-user.target

Is /usr/bin/sonarr normal considering the appData is in /var/lib/sonarr?

That makes sense that’s where is updating from.

Its similar to but different from this one, which might work better? Implementing Sonarr with systemd

I don’t see any issue with it (it would affect the update process).

Who develops the systemd files? Is it the main sonarr dev community or is it Arch package maintainer or something else? Since systemd was included in the package I would assume it is correct or if it not it should be fixed like any other bug…

The systemd you linked to is invalid as there is no NZBDrone directory in /opt. Restart option set to always might interfere with updating too while the default is restart=no.

Increasing timeout value to 60 didn’t help either. I am out of ideas :frowning:

Someone else maintains it. Since it came with the package, likely the maintainer of that package.

Correct is pretty broad, in the sense that it works for Sonarr, its correct, but it doesn’t seem to work for automatic updates.

If you run Sonarr without systemd does the update succeed? This will prove whether or not systemd is affecting it, or its something else.

I think I have it working now, though it does kind of require a bit of a kludgy workaround.

I added the line:

ExecStop=-/usr/sbin/update-sonarr

(note: the minus after = tells systemd to ignore any errors during execution)

And here’s the update-sonarr script:

#!/bin/sh
cd /tmp/nzbdrone_update
/usr/bin/mono ./NzbDrone.Update.exe `ps aux | grep NzbDrone | grep -v grep | awk ‘{ print $2 }’` /tmp/nzbdrone_update /opt/NzbDrone/NzbDrone.exe
#you could optionally do clean-up here by removing the /tmp/nzbdrone_update directory.
#I’m choosing not to. That’s just the type of renegade I am.

Don’t forget to systemctl daemon-reload after editing the .service file, and to chmod 755 the above script.

Now I’m just waiting for an update to happen so I can see if it’s successful.

-k

I tried running manually without systemd and the update worked just fine, so it is indeed the issue with systemd.

Was about to try it but just realized that the update worked without systemd so now there’s no update left to try :smile:

It is a bit messy for me though since the paths are very different on my system mainly there not being anything in /opt dir. I hope this is fixed in Sonarr itself since if it can work in other distros it’s unlikely that it can’t be relatively easily fixed in Arch.

Not sure what the concern about the path is (or what we could fix), just use the path on your system. /opt/NzbDrone is the path we chose when setting up the Debian package, but Sonarr is happy to run from any path.

With systemd, I see two potential ways of attacking this problem.

First, the hackish way I did it (which seems to work just fine)… Create a script that manually calls the updater from /tmp and passing the PID, source path and target executable.

The only “con” I can see for this method is that the “power-off” button in nzbdrone will never work (though restart should be ok, as would a systemctl stop sonarr.service)

The second way to approach it is to configure the daemon to be a forked process. There’s a problem here, though, as mono doesn’t fork the sonarr executable in a traditional sense. What you would have to do is configure the service to be forked - but since the process is expected to terminate, you might have to fork it via ‘exec’. Then you’d have to definitely keep the pid file around.

I haven’t figured out which of the two is the better option (yet), but I hope to have all this systemd business done this week.

-k

I just mean’t is wouldn’t be universal but I guess that’s not a real problem. The real issue is that it should work out of the box like it does on Debian based distros.

What exactly causes the issue in systemd? Is it the same issue as with upstart where it tries to restart when Sonar is stopped for update?

Well, there’s a few contributing factors. First, the way the auto-update system works in Sonarr is that it forks out another mono instance to run the updater, which you feed a pid and directory args. The updater kills the parent pid (I think), updates, and then attempts to fork a new copy of mono (and subsequently a new instance of NzbDrone.exe).

When running under systemd, you have to create the .system file and configure it with a service type. If you configure it as a “simple” service type, systemd only creates space for a single process - the parent. So once you kill the parent via the NzbDrone updater, poof - the process space is destroyed by systemd. That’s how it works by design.

My little “hack” above gets around it by forcing the updater to run (or at least try to run) after the service disappears.

I think the better way to approach this is to configure the service as forked - which it should do nicely if you trail the entire command line for the daemon with a &.

The problem here (which I’m trying to solve - and it is solvable) is that you would have to give systemd the actual pid for the current Sonarr instance for tracking purposes. If that pid disappears (which WILL happen when the updater runs), the daemon is considered dead since it was killed Adam Lanza-style by its child.

So that’s the basic gist of the problem. nzbdrone.exe relies on mono, mono runs under the context of a systemd service, and we basically have to profile what the proper service “type” is for sonarr’s process flow.

I’ll figure it out. I just need another day or so.

Thanks for the explanation, not as simple as I imagined :slight_smile:

OK, Here’s the cleanest and best job we can do at present with systemd (for reasons that will be outlined at the end of this post):

#--start of sonarr.service (which I locate in /etc/systemd/system):
[Unit]
Description=Sonarr Daemon

[Service]
User=sonarr
Group=sonarr
Restart=always
RestartSec=5
Type=simple
ExecStart=/usr/bin/mono /opt/NzbDrone/NzbDrone.exe -nobrowser
# Note: The below ExecStop line is supposed to be on ONE LINE. 
# The forum automatically line-breaks it. Also, if you opt to NOT
# use the built-in Sonarr updater and instead plan to use a package
# manager, delete or disable the ExecStop line by commenting it out. 
ExecStop=-/usr/bin/mono /tmp/nzbdrone_update/NzbDrone.Update.exe \`ps aux | grep 
NzbDrone | grep -v grep | awk '{ print $2 }'\` /tmp/nzbdrone_update /opt/NzbDrone/NzbDrone.exe
TimeoutStopSec=20

[Install]
WantedBy=multi-user.target
#---end of sonarr.service

After putting the file in /etc/systemd/system, run the following commands:

systemctl daemon-reload
systemctl enable sonarr
systemctl start sonarr

That will have it load at startup and set the proper runlevels, etc.

Now, here’s a few explanations for why I did things this way.

-All you really need is a service type of ‘simple’. Nothing exotic is required. Notice the ExecStop command line. It has a minus sign before the command. That means to simply ignore any error codes it may run into. It’s probably not necessary, but if the executable to upgrade Sonarr doesn’t currently exist in /tmp, then you MIGHT get some kind of service failure. I figure it’s better safe than sorry.

-The Sonarr updater takes 3 main args, first is the pid of the current sonarr process (which it will kill) and then the source and target directories of the update. Once Sonarr kills the process, the ExecStop command line argument runs - which updates Sonarr. Much thanks to markus for helping me out with the updater on irc :smile:

-The Restart=always will notice that Sonarr’s pid has disappeared and attempt to execute the updater. It will wait for the duration of RestartSec to restart it afterwards, though the updater itself respawns Sonarr automatically anyway (so you’re covered either way if for some reason the update fails). NOTE: If you have a very slow server, you will probably want to increase this time so the update has time to finish. 30 seconds is probably a VERY safe number, even if it means waiting 30 seconds for Sonarr to come back to life.

There is ONE caveat to this whole thing. While implenting Sonarr under systemd this way will preserve all functionality, there is ONE function that won’t work. The “Power Off” icon under the system tab will do the same thing as the restart icon. Why? Because systemd has no exit code to read. The updater and the NzbDrone.exe binary under mono will ALWAYS exit with an error code of 0, so the systemd service has no way to determine exactly how Sonarr has been terminated. Perhaps in the future the Sonarr team can implement exit codes for both the stop and restart functions. That would allow us to further tailor Sonarr’s behavior under systemd.

But for now, this is the best that we can do - which is still pretty good. I can live without the power-off button, since i can just do a systemctl stop sonarr anyway.

One last item of note: If you plan on using the updating method further down in this thread (using sonarr-update.timer and sonarr-update.service), make sure you remove the ExecStop= line, as it is no longer necessary - and COULD potentially downgrade you if you have a stale copy sitting in /tmp.