Auto Update fails on Arch Linux

I tried running manually without systemd and the update worked just fine, so it is indeed the issue with systemd.

Was about to try it but just realized that the update worked without systemd so now there’s no update left to try :smile:

It is a bit messy for me though since the paths are very different on my system mainly there not being anything in /opt dir. I hope this is fixed in Sonarr itself since if it can work in other distros it’s unlikely that it can’t be relatively easily fixed in Arch.

Not sure what the concern about the path is (or what we could fix), just use the path on your system. /opt/NzbDrone is the path we chose when setting up the Debian package, but Sonarr is happy to run from any path.

With systemd, I see two potential ways of attacking this problem.

First, the hackish way I did it (which seems to work just fine)… Create a script that manually calls the updater from /tmp and passing the PID, source path and target executable.

The only “con” I can see for this method is that the “power-off” button in nzbdrone will never work (though restart should be ok, as would a systemctl stop sonarr.service)

The second way to approach it is to configure the daemon to be a forked process. There’s a problem here, though, as mono doesn’t fork the sonarr executable in a traditional sense. What you would have to do is configure the service to be forked - but since the process is expected to terminate, you might have to fork it via ‘exec’. Then you’d have to definitely keep the pid file around.

I haven’t figured out which of the two is the better option (yet), but I hope to have all this systemd business done this week.

-k

I just mean’t is wouldn’t be universal but I guess that’s not a real problem. The real issue is that it should work out of the box like it does on Debian based distros.

What exactly causes the issue in systemd? Is it the same issue as with upstart where it tries to restart when Sonar is stopped for update?

Well, there’s a few contributing factors. First, the way the auto-update system works in Sonarr is that it forks out another mono instance to run the updater, which you feed a pid and directory args. The updater kills the parent pid (I think), updates, and then attempts to fork a new copy of mono (and subsequently a new instance of NzbDrone.exe).

When running under systemd, you have to create the .system file and configure it with a service type. If you configure it as a “simple” service type, systemd only creates space for a single process - the parent. So once you kill the parent via the NzbDrone updater, poof - the process space is destroyed by systemd. That’s how it works by design.

My little “hack” above gets around it by forcing the updater to run (or at least try to run) after the service disappears.

I think the better way to approach this is to configure the service as forked - which it should do nicely if you trail the entire command line for the daemon with a &.

The problem here (which I’m trying to solve - and it is solvable) is that you would have to give systemd the actual pid for the current Sonarr instance for tracking purposes. If that pid disappears (which WILL happen when the updater runs), the daemon is considered dead since it was killed Adam Lanza-style by its child.

So that’s the basic gist of the problem. nzbdrone.exe relies on mono, mono runs under the context of a systemd service, and we basically have to profile what the proper service “type” is for sonarr’s process flow.

I’ll figure it out. I just need another day or so.

Thanks for the explanation, not as simple as I imagined :slight_smile:

OK, Here’s the cleanest and best job we can do at present with systemd (for reasons that will be outlined at the end of this post):

#--start of sonarr.service (which I locate in /etc/systemd/system):
[Unit]
Description=Sonarr Daemon

[Service]
User=sonarr
Group=sonarr
Restart=always
RestartSec=5
Type=simple
ExecStart=/usr/bin/mono /opt/NzbDrone/NzbDrone.exe -nobrowser
# Note: The below ExecStop line is supposed to be on ONE LINE. 
# The forum automatically line-breaks it. Also, if you opt to NOT
# use the built-in Sonarr updater and instead plan to use a package
# manager, delete or disable the ExecStop line by commenting it out. 
ExecStop=-/usr/bin/mono /tmp/nzbdrone_update/NzbDrone.Update.exe \`ps aux | grep 
NzbDrone | grep -v grep | awk '{ print $2 }'\` /tmp/nzbdrone_update /opt/NzbDrone/NzbDrone.exe
TimeoutStopSec=20

[Install]
WantedBy=multi-user.target
#---end of sonarr.service

After putting the file in /etc/systemd/system, run the following commands:

systemctl daemon-reload
systemctl enable sonarr
systemctl start sonarr

That will have it load at startup and set the proper runlevels, etc.

Now, here’s a few explanations for why I did things this way.

-All you really need is a service type of ‘simple’. Nothing exotic is required. Notice the ExecStop command line. It has a minus sign before the command. That means to simply ignore any error codes it may run into. It’s probably not necessary, but if the executable to upgrade Sonarr doesn’t currently exist in /tmp, then you MIGHT get some kind of service failure. I figure it’s better safe than sorry.

-The Sonarr updater takes 3 main args, first is the pid of the current sonarr process (which it will kill) and then the source and target directories of the update. Once Sonarr kills the process, the ExecStop command line argument runs - which updates Sonarr. Much thanks to markus for helping me out with the updater on irc :smile:

-The Restart=always will notice that Sonarr’s pid has disappeared and attempt to execute the updater. It will wait for the duration of RestartSec to restart it afterwards, though the updater itself respawns Sonarr automatically anyway (so you’re covered either way if for some reason the update fails). NOTE: If you have a very slow server, you will probably want to increase this time so the update has time to finish. 30 seconds is probably a VERY safe number, even if it means waiting 30 seconds for Sonarr to come back to life.

There is ONE caveat to this whole thing. While implenting Sonarr under systemd this way will preserve all functionality, there is ONE function that won’t work. The “Power Off” icon under the system tab will do the same thing as the restart icon. Why? Because systemd has no exit code to read. The updater and the NzbDrone.exe binary under mono will ALWAYS exit with an error code of 0, so the systemd service has no way to determine exactly how Sonarr has been terminated. Perhaps in the future the Sonarr team can implement exit codes for both the stop and restart functions. That would allow us to further tailor Sonarr’s behavior under systemd.

But for now, this is the best that we can do - which is still pretty good. I can live without the power-off button, since i can just do a systemctl stop sonarr anyway.

One last item of note: If you plan on using the updating method further down in this thread (using sonarr-update.timer and sonarr-update.service), make sure you remove the ExecStop= line, as it is no longer necessary - and COULD potentially downgrade you if you have a stale copy sitting in /tmp.

I changed your service file slightly since the default location when installing Sonarr from the AUR is “/usr/lib/sonarr” but it comes up with an error.
Service file
Service file error

Your particular error is caused by the fact that ExecStop line is split into 3 lines. Just format it so it is just one line. There are other issues that prevent it form working for me though, trying some things out now.

Well, here’s another alternative for ensuring your Sonarr installation is up to date. Reproduce the same functionality within systemd using an update service and an update timer.

#Step 1 - Disable internal updater and configure pkg sources
First, disable the internal updater within sonarr via the web interface.

Then configure your package manager to get updates from the sonarr repo (directions below for ubuntu, alter as necessary):

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys FDA5DFFC
echo “deb http://apt.sonarr.tv/ develop main” | sudo tee etc/apt/sources.list.d/sonarr.list

#Step 2 - Sonarr updater service
Now go into /etc/systemd/system and create the file ‘sonarr-update.service’

[Unit]
Description=Sonarr Updater 
Wants=sonarr-update.timer

[Service]
Type=oneshot
ExecStart=-/bin/systemctl stop sonarr
ExecStart=-/usr/bin/apt-get update
ExecStart=-/usr/bin/apt-get install --only-upgrade nzbdrone
ExecStart=-/bin/systemctl start sonarr
TimeoutStopSec=20

[Install]
WantedBy=multi-user.target

#Step 3: Create timer for Sonarr Updater
Now go into /etc/systemd/system and create the file ‘sonarr-update.timer’

[Unit]
Description=Sonarr updater timer

[Timer]
# Feel free to change the "OnCalendar" setting to any value you wish.
# Sonarr's internal process does it every 4 hours, but I think once a day is more than enough.
# More info: http://www.freedesktop.org/software/systemd/man/systemd.time.html
OnCalendar=daily
Persistent=true
Unit=sonarr-update.service

[Install]
WantedBy=multi-user.target

#Step 4: Enable timer
Perform the following:

systemctl daemon-reload
systemctl enable sonarr-update.timer
systemctl start sonarr-update.timer

Note: Do NOT “enable” the sonarr-update.service via systemctl enable. The service file is called by the timer.

Ok, I have the issues on Arch resolved.

The link to the executable can be either /usr/bin/mono /usr/lib/sonarr/NzbDrone.exe or just /usr/bin/sonarr

For ExecStart -data=/var/lib/sonarr argument is also required.

I updated [Install Sonarr on Arch Linux][1] post with the systemd file corrected for Arch. @Ice_Nine btw, let me know if you want me to link somewhere to you as an author of the script :smile:
[1]: http://dominicm.com/install-nzbdrone-sonarr-on-arch-linux/

No need, like anything else, it’s going to need editing to satisfy the needs of any particular distribution :slight_smile: A lot of this is simple trial and error while I learn all the nuances of systemd.

I’ll report in on how the timer-based updates work out. I have it configured now, I figure in a couple of days I’ll have some good data regarding how reliable it is.

-k

I had issues with autoupdate working on Ubuntu that were related to Sonarr (Drone at the time) not exiting gracefully at update time. It was running as an Upstart job, and Upstart would try to restart the service, borking the update. I had to turn off “respawn” in the .conf file.

There (if I remember right) were permission issues with Sonarr (which runs as you) modifying its default install directory (not owned by you) --which I solved by installing sonarr into my home folder and running it from there. I suppose I could have chown/chmod’d the default install directory as well, but figured this was easier and more portable should I need to migrate the install.

Well, you should always chown a directory that has to run as a certain user - unless you want to restrict writes (then you could simply chmod or use an acl/stickybit).

This is why I think it’s simply easier to just have systemd handle the whole upgrade process. Since systemd controls the underlying process, it has no problem killing it, allowing root to update it, and restart it again as the proper user without incident.

I know there’s a lot of misguided hate towards systemd. Say what you will about it, it’s very well thought-out. I understand that the linux community prefers decentralization above most things, but systemd just makes perfect sense to me (except for the whole journal thing not being a flat file, that kinda sucks, but it’s not a horrible ordeal since everything gets dumped in cleartext to syslog anyway).

I moved a post to a new topic: Arch Linux Updater Requests Windows Update Package

The service script doesn’t seem to work for me. It seems like the update exe is called with the wrong arguments?

Here’s what my service looks like: http://pastebin.com/V7sTqC9T

After getting the update package and moving everything where it should be, the updater is then called like this:

15-2-10 01:19:44.8|Debug|mono|Starting mono /tmp/nzbdrone_update/NzbDrone.Update.exe 638 /tmp/nzbdrone_update /usr/lib/sonarr/NzbDrone.exe /data=/var/lib/sonarr /nobrowser

Which seems wrong? Shouldn’t the third argument be /usr/lib/sonarr or have I misunderstood how the updater works?

Edit

Got it working. Think there was some funky-ness somewhere about the ‘\`’ in the ExecStop command which should take care of taking over the update job when the updater kills the process. Used " instead and it works fine.

It looks like formatting struck again :slight_smile: Thanks for sharing it. Updated Install Sonarr on Arch Linux post too so now it should work out of the box.