Sonarr cannot finish importing when run via systemd

Joeman · September 16, 2018, 9:38am

Sonarr version (exact version): 2.0.0.5228
Mono version (if Sonarr is not running on Windows): Mono JIT compiler version 4.6.2 (Debian 4.6.2.7+dfsg-1)
OS: Debian 9.5 (inside a proxmox VM)
Debug logs: https://www.dropbox.com/s/jklcvuaqzcb8tp9/sonarr.trace.22.txt.zip?dl=0
Description of issue: Import fails because sonarr cannot make a local backup copy?
I am very happy with sonarr:
shows detection works
interface with sabnzb works

What does not work is automatic import at the end of a download.
Sonarr detects as expected the end of the download. The downloaded files are in a folder where the sonarr user has read/write rights.
I can see the show being copied to the target directory until the copy is complete and then, sonarr decides it wants to rollback and deletes the show from the target directory.
Those steps show this should not be a rights issue.
And this is were my problem must lay: both the download folder and the target “shows” folder are on different NFS mounts.

A look at the logs show

18-9-15 22:49:06.6|Debug|EpisodeFileMovingService|Moving episode file: /data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS/6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv to /data/MediaNFS/series/Simpsons/Saison29/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS.mkv
18-9-15 22:49:06.6|Debug|DiskTransferService|Move [/data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS/6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv] > [/data/MediaNFS/series/Simpsons/Saison29/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS.mkv]
18-9-15 22:49:06.6|Trace|SymbolicLinkResolver|Checking path /data/MediaNFS/series/Simpsons/Saison29/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS.mkv for symlink returned error ENOENT, assuming it’s not a symlink.
18-9-15 22:49:06.6|Trace|DiskTransferService|Attempting to move hardlinked backup.
18-9-15 22:49:06.6|Debug|DiskProvider|Hardlink ‘/data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS/6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv’ to ‘/data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS/6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv.backup~’ failed.

[v2.0.0.5228] System.InvalidOperationException: Opération non permise —> Mono.Unix.UnixIOException: Opération non permise [EPERM].
— End of inner exception stack trace —
at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00005] in :0
at Mono.Unix.UnixMarshal.ThrowExceptionForLastErrorIf (System.Int32 retval) [0x00007] in :0
at Mono.Unix.UnixFileSystemInfo.CreateLink (System.String path) [0x0000d] in :0
at NzbDrone.Mono.Disk.DiskProvider.TryCreateHardLink (System.String source, System.String destination) [0x00013] in C:\BuildAgent\work\5d7581516c0ee5b3\src\NzbDrone.Mono\Disk\DiskProvider.cs:182

18-9-15 22:49:06.6|Trace|DiskTransferService|Hardlink move failed, reverting to move.
18-9-15 22:49:11.6|Debug|DiskTransferService|Rolling back incomplete file move [/data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS/6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv] to [/data/MediaNFS/series/Simpsons/Saison29/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS.mkv].
18-9-15 22:49:14.6|Trace|DiskProviderBase|Deleting file: /data/MediaNFS/series/Simpsons/Saison29/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS.mkv
18-9-15 22:49:14.6|Warn|ImportApprovedEpisodes|Couldn’t import episode /data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS/6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv

[v2.0.0.5228] System.UnauthorizedAccessException: Access to the path is denied.

As I understand, sonarr tries to make a local backup copy of the transfered file using a hardlink? Why ?
There is “Media management” setting “Use Hardlinks instead of Copy” (set to NO in my case) but it seems to only apply to torrents.

Once it has failed, the task is retried every minute, each time copying, then rollbacking.
At this point, if I do a manual import while selecting “copy” instead of “move” it works! (well if I am lucky to not interfere with the scheduled retry, because then, even the manual import fails, and I have to try again).

What I want to know is if there is a parameter or mecanism I am missing to disable the use of hardlinks on the source download directory?
Sonarr runs inside a VM with limited space regarding some of the downloads, thus I use a NFS mount from my NAS. If nothing else works, setting up an iscsi mount on the VM should allow hardlinks to work, but I would very much like to find a solution within sonarr and my current setup

Thanks in advance

markus101 · September 16, 2018, 6:12pm

Because of unreliable SMB shares Sonarr makes a “copy” locally then moves the file across, by utilizing a hard link and a move Sonarr can take advantage of the hardlink being instantaneous (when supported) and avoid a slow copy when a move would be quicker. Unfortunately Sonarr doesn’t know if it’s dealing with an SMB share and does this process whenever it moves to a different volume. NFS does not have this issue in our experience, but it appears you’re running into the same thing.

The hardlink failing isn’t failing the import, the fact that the file is not completely transferred is the issue, which looks to be caused by being unable to delete the partially imported file.

Joeman · September 16, 2018, 8:04pm

Thank you for answer.
You’re right. The import itself does not fail.It is only rollbacked.

As I am not familiar with hardlinks, I just made a few tries:

nzbdrone@hermes:/data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS$ ls -lhrt
total 292M
-rw-rw-r-- 1 sabnzbd users 292M mai 21 12:52 6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv
nzbdrone@hermes:/data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS$ ln /data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS/6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv /data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS/6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv.backup~
nzbdrone@hermes:/data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS$ ls -lhrt
total 583M
-rw-rw-r-- 2 sabnzbd users 292M mai 21 12:52 6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv.backup~
-rw-rw-r-- 2 sabnzbd users 292M mai 21 12:52 6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv
nzbdrone@hermes:/data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS$ rm 6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv
nzbdrone@hermes:/data/download/Backlog/The.Simpsons.S29E08.REPACK.720p.HDTV.x264-KILLERS$ ls -lhrt
total 292M
-rw-rw-r-- 1 sabnzbd users 292M mai 21 12:52 6ecc4bcfc01c4f6b9e2b2596fd9bd131.mkv.backup~

I did this when logged in with the same user sonarr is run by. There does not seem to be any problem when doing it manually.

As for the mechanism you described in your post (local hardlink, then move), is there a way to disable it or change the behavior ?

markus101 · September 17, 2018, 5:15pm

That’s without altering any permissions? And Sonarr is definitely running as the nzbdrone user?

Offhand I’m not sure how the hardlink happens through Sonarr, but given the same file and the same user it should behave the same way.

No, it’s not possible to disable that behaviour.

Joeman · September 17, 2018, 8:46pm

Indeed, I did not alter any permission, the test would have been useless …

I think I excluded NFS mount as a reason.
I changed on sabnzb the finished directory to a local directory. There is no room enough on the server for some of my downloads, but for a test it is fine.
This did not change anything

18-9-17 21:11:01.8|Debug|Parser|Release Group parsed: SiGMA
18-9-17 21:11:01.8|Debug|EpisodeFileMovingService|Moving episode file: /data/download_local/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA/11831485e43847b4b1efad356a6bf9e8.mkv to /data/MediaNFS/series/Simpsons/Saison29/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA.mkv
18-9-17 21:11:01.8|Debug|DiskTransferService|Move [/data/download_local/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA/11831485e43847b4b1efad356a6bf9e8.mkv] > [/data/MediaNFS/series/Simpsons/Saison29/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA.mkv]
18-9-17 21:11:01.8|Trace|SymbolicLinkResolver|Checking path /data/MediaNFS/series/Simpsons/Saison29/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA.mkv for symlink returned error ENOENT, assuming it’s not a symlink.
18-9-17 21:11:01.8|Trace|DiskTransferService|Attempting to move hardlinked backup.
18-9-17 21:11:01.8|Debug|DiskProvider|Hardlink ‘/data/download_local/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA/11831485e43847b4b1efad356a6bf9e8.mkv’ to ‘/data/download_local/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA/11831485e43847b4b1efad356a6bf9e8.mkv.backup~’ failed.

[v2.0.0.5228] System.InvalidOperationException: Opération non permise —> Mono.Unix.UnixIOException: Opération non permise [EPERM].
— End of inner exception stack trace —
at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00005] in :0
at Mono.Unix.UnixMarshal.ThrowExceptionForLastErrorIf (System.Int32 retval) [0x00007] in :0
at Mono.Unix.UnixFileSystemInfo.CreateLink (System.String path) [0x0000d] in :0
at NzbDrone.Mono.Disk.DiskProvider.TryCreateHardLink (System.String source, System.String destination) [0x00013] in C:\BuildAgent\work\5d7581516c0ee5b3\src\NzbDrone.Mono\Disk\DiskProvider.cs:182

18-9-17 21:11:01.8|Trace|DiskTransferService|Hardlink move failed, reverting to move.
18-9-17 21:11:19.1|Debug|DiskTransferService|Rolling back incomplete file move [/data/download_local/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA/11831485e43847b4b1efad356a6bf9e8.mkv] to [/data/MediaNFS/series/Simpsons/Saison29/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA.mkv].
18-9-17 21:11:22.1|Trace|DiskProviderBase|Deleting file: /data/MediaNFS/series/Simpsons/Saison29/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA.mkv
18-9-17 21:11:22.2|Warn|ImportApprovedEpisodes|Couldn’t import episode /data/download_local/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA/11831485e43847b4b1efad356a6bf9e8.mkv

[v2.0.0.5228] System.UnauthorizedAccessException: Access to the path is denied.
at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0
at NzbDrone.Common.Disk.DiskProviderBase.MoveFileInternal (System.String source, System.String destination) [0x00000] in C:\BuildAgent\work\5d7581516c0ee5b3\src\NzbDrone.Common\Disk\DiskProviderBase.cs:232
at NzbDrone.Mono.Disk.DiskProvider.MoveFileInternal (System.String source, System.String destination) [0x00076] in C:\BuildAgent\work\5d7581516c0ee5b3\src\NzbDrone.Mono\Disk\DiskProvider.cs:170
at NzbDrone.Common.Disk.DiskProviderBase.MoveFile (System.String source, System.String destination, System.Boolean overwrite) [0x000e3] in C:\BuildAgent\work\5d7581516c0ee5b3\src\NzbDrone.Common\Disk\DiskProviderBase.cs:227
at NzbDrone.Common.Disk.DiskTransferService.TryMoveFileVerified (System.String sourcePath, System.String targetPath, System.Int64 originalSize) [0x00000] in C:\BuildAgent\work\5d7581516c0ee5b3\src\NzbDrone.Common\Disk\DiskTransferService.cs:566

Again, I could then manually and successfully copy/import the file via the sonarr interface.
The downloaded file staying in its directory, I tried to manually create the hardlink

nzbdrone@hermes:/data/download_local/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA$ ln /data/download_local/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA/11831485e43847b4b1efad356a6bf9e8.mkv /data/download_local/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA/11831485e43847b4b1efad356a6bf9e8.mkv.backup~
nzbdrone@hermes:/data/download_local/The.Simpsons.S29E16.King.Leer.1080p.REPACK.AMZN.WEB-DL.DD.5.1.H.264.1-SiGMA$ ls -lhrt
total 1,3G
-rw-rw-r-- 2 sabnzbd users 641M avril 22 02:20 11831485e43847b4b1efad356a6bf9e8.mkv.backup~
-rw-rw-r-- 2 sabnzbd users 641M avril 22 02:20 11831485e43847b4b1efad356a6bf9e8.mkv

!!!
I have found the problem.
(I leave the above trials for reference)

After trying to restart, without any effect, I tried something else: I launched sonarr directly from a shell instead of through systemd, still using a local download directory for sabnzdb. And it worked!
There is no error and no import rollback.

18-9-17 22:16:35.8|Trace|DiskTransferService|Attempting to move hardlinked backup.
18-9-17 22:16:42.6|Trace|DiskTransferService|Hardlink move succeeded, deleting source.
18-9-17 22:16:43.8|Trace|DiskProviderBase|Deleting file: /data/download_local/The.Simpsons.S29E18.720p.HDTV.x264-AVS/229fc4ebd8f14813b40cd44dcef4b92e.mkv

I then confirmed using my initial setup with a NFS mount worked too.

So, this is where I failed:
In my sonarr.service, the “group” parameter was wrong. Sonarr was then launched with the right user, but the wrong group. Apparently it only caused an issue regarding this hardlink creation and nothing else …

Late edit: it seems it is not over yet. I will see tomorrow …

Joeman · September 18, 2018, 5:33am

There is definitely something with running sonarr via systemd. When run manually, move and import worked today. I may still have an issue about unix permissions, as the download directory was not removed, although correctly emptied.
I will try to complete tonight.

Joeman · September 19, 2018, 8:16pm

I can confirm that when sonarr is run manually via “/usr/bin/mono /opt/NzbDrone/NzbDrone.exe -nobrowser” it works, even when both download directory and target directory are on a NFS mount.

Here is the content of my sonarr.service file that I configured using the systemd section of this page

[Unit]
Description=Sonarr Daemon
After=network.target

[Service]
User=nzbdrone
Group=users

Type=simple
ExecStart=/usr/bin/mono --debug /opt/NzbDrone/NzbDrone.exe -nobrowser
TimeoutStopSec=20
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target
The user “nzbdrone” is the same I use when running manually the application. It only belongs to one group “users”.

I am completely dumbfounded about why it will not work when run from systemd instead of manually.

john · September 23, 2018, 12:30am

I looked at the page where you got your service file and it is different from what I use.

First, I use Fedora but it also runs systemd.

Second I have Sabnzbd, Radarr and Sonarr on one server and Plex and all files on a separate server. I use NFS and download everything on sonarr’s server but then it appears on Plex and the files are stored there. NFS is set up via fstab on Sonarr’s server and /etc/exports on Plex server. I use john:users as user/group on all files.

I have been using this service files (/etc/systemd/system/sonarr.service for a couple of years without a problem. I am putting it in here in hopes that it will be helpful. I know Fedora is different from what you are using but it might be close.

[Unit]
Description=Sonarr Daemon
After=syslog.target network.target

[Service]
User=john
Group=users
Type=simple
ExecStart=/usr/bin/mono /home/bin/Apps/NzbDrone/NzbDrone.exe -nobrowser -data /home/bin/apps/NzbDrone
TimeoutStopSec=20

[Install]
WantedBy=multi-user.target

john

Joeman · September 23, 2018, 3:52pm

Thank you @john for this post.
Apart from the use of “-data” at the ExecStart line, I do not see any significant difference between your sonarr.service and mine. At least nothing that would explain why specifically, the creation of the hardlink at the end of the move step fails.
Do you know what “-data” switch at the end of the line is for ?

john · September 23, 2018, 4:23pm

I used the line about data from what was recommended from the install page for Fedora. I honestly do not know what data refers to, sorry.
The only other difference I see is you have in KillMode and Restart. I don’t have either and if you are running from cli they wouldn’t be there either. I would suggest commenting them out and trying it again. Outside of that I have no idea what could be going wrong. Wish I could be of more help.

john

Joeman · September 29, 2018, 5:35pm

Thank God! Well thank @john I mean
Your last post helped me find a solution.
For the record, I have been using what I consider a dirty workaround: running Sonarr using nohup&. Of course, there is no automatic restart in case of crash, and well, this obviously lack elegance.

So, I used a copy of your sonarr.service file, adapting it to my setup and it worked!
In order to precisely identify what was missing and now makes it work, I tried again using my old file, only adding the "-data " parameter at the end of the command line. And as suspected, it does the trick!

Thanks again @john

I am going to contact an admin, perhaps @markus101 ? to correct the sonarr.service example fil in this page https://github.com/Sonarr/Sonarr/wiki/Autostart-on-Linux by adding the "-data " parameter.

john · September 29, 2018, 7:44pm

Happy to hear that you are up and running. Glad to have been of help. I know I did a lot of reading and asking questions when I started with Sonarr and then Radarr. Unfortunately a lot of the documentation online is either wrong or outdated.

john

markus101 · September 29, 2018, 10:22pm

-data changes where Sonarr stores it’s configuration, it is not mandatory and doesn’t change the behaviour of Sonarr beyond that.

The wiki itself is editable by anyone, but changes such as these aren’t included in the base configuration.

Joeman · September 30, 2018, 9:28am

To be honest, I am no linux expert, no complete noob either, yet I do not see how skipping this “data” parameter might explain the behavior I observed.
Just to be sure, when I have more time, I will try to remove the parameter and check the final move/import fails as I first described. For the moment, my sonarr instance works as intended, this is all asked for.

Joeman · September 30, 2018, 9:31am

I suspected as much. The value I configured is the exact same path that seemed used by the application by default. As I said to @john, I cannot see the link between setting this parameter and my impossibility to hardlink at the end of the import. Yet it seems to work.
The only way to check will be to remove it and try again. I will see later and update my post.

system · November 29, 2018, 9:31am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.