Path mappings so hardlink works with Deluge & Sonarr in Docker

[kars85]

Sonarr version (exact version): 2.0.0.5250
Mono version (if Sonarr is not running on Windows): 5.14.0.177
OS: Ubuntu 18.04 running linuxserver.io docker container
Debug logs: Now it is turned on. I don’t think it is an error with anything within Sonarr, I’m just doing something wrong. I will monitor on the next managed download and update this with what I find (if relevant)
Description of issue: I have a Deluge docker container and Sonarr docker container running with /mnt/nfs/Torrents mapped to /data within the container of both. If I do a ls -al (or find . -type l) on both the location the file is moved to after completion in Deluge, as well as the Plex location /mnt/nfs/TVShows, the hardlink is not present.

Am I missing something by not having any entries in the Download Client -> Remote Path Mappings section?

Docker env variables:

DOCKERCONFDIR=~/config
DOCKERSHAREDDIR=~/config/shared
DOWNLOADSDIR=/mnt/nfs/Torrents
MEDIADIR_BOOKS=/mnt/nfs/Books
MEDIADIR_COMICS=/mnt/nfs/Comics
MEDIADIR_MOVIES=/mnt/nfs/Movies
MEDIADIR_MUSIC=/mnt/nfs/Music
MEDIADIR_TV=/mnt/nfs/TVShows
MEDIADIR_HOMEVIDEOS=/mnt/nfs/HomeVideos
MEDIADIR_PHOTOS=/mnt/nfs/Photos

service for Sonarr defined in my docker-compose.yml

  sonarr:
    container_name: sonarr
    environment:
    - PGID=${PGID}
    - PUID=${PUID}
    - TZ=${TZ}
    image: linuxserver/sonarr:develop
    ports:
    - ${SONARR_PORT_8989}:8989
    restart: always
    volumes:
    - /etc/localtime:/etc/localtime:ro
    - ${DOCKERCONFDIR}/sonarr:/config
    - ${DOCKERSHAREDDIR}:/shared
    - ${DOWNLOADSDIR}:/data
    - ${MEDIADIR_TV}:/tv

service for Deluge defined in my docker-compose.yml

  delugevpn:
    cap_add:
    - NET_ADMIN
    container_name: delugevpn
    environment:
    - LAN_NETWORK=${LAN_NETWORK}
    - NAME_SERVERS=${NS1},${NS2}
    - PGID=${PGID}
    - PUID=${PUID}
    - TZ=${TZ}
    - VPN_ENABLED=${VPN_ENABLE}
    - VPN_OPTIONS=${VPN_OPTIONS}
    - VPN_PASS=${VPN_PASS}
    - VPN_PROV=${VPN_PROV}
    - VPN_USER=${VPN_USER}
    image: binhex/arch-delugevpn
    ports:
    - ${DELUGEVPN_PORT_58846}:58846
    - ${DELUGEVPN_PORT_58946}:58946
    - ${DELUGEVPN_PORT_8112}:8112
    restart: always
    volumes:
    - /etc/localtime:/etc/localtime:ro
    - ${DOCKERCONFDIR}/delugevpn:/config
    - ${DOCKERSHAREDDIR}:/shared
    - ${DOWNLOADSDIR}:/data

Download Client View

Media Management View

[markus101]

Hardlinks require the both files to be on the same volume, including docker (the location on the host doesn’t matter) but you’re using /data for the deluge downloads and /tv for the shows.

Something like:

${DOWNLOADSDIR}:/data/downloads and ${MEDIADIR_TV}:/data/tv would work.

[kars85]

Thank you @markus101 ! It seems like the out of the box docker file volume mappings contradict hardlinks working (possibly by design, I don’t know?) I added comments to my compose file so I remember why I have the stock volume mapping (/tv & /downloads) still present in the service’s config. Otherwise, leaving them out results in a asd2342348… volume attached to the container. I think I could fix this easily by mapping the docker volumes in sonarr and deluge to /data and calling it a day, but then they’d have access to everything below that.

  sonarr:
    container_name: sonarr
    environment:
    - PGID=${PGID}
    - PUID=${PUID}
    - TZ=${TZ}
    image: linuxserver/sonarr:develop
    ports:
    - ${SONARR_PORT_8989}:8989
    restart: always
    volumes:
    - /etc/localtime:/etc/localtime:ro
    - ${DOCKERCONFDIR}/sonarr:/config
    - ${DOCKERSHAREDDIR}:/shared
    - ${DOWNLOADSDIR}:/data
    #/downloads and {MEDIADIR_} needs to be in here because of the docker file calling for it
    #doesn't need to be used, just assigned otherwise you get a 324252398... volume name
    #setting mediadir to same volume path (/data/tv) as download path lets hardlink work
    - ${DOWNLOADSDIR}:/downloads
    - ${MEDIADIR_TV}:/data/tv
    - ${MEDIADIR_TV}:/tv

I had a show ready to download, so I enabled Deluge again as my download client, but it appears that I’m still having problems with my configuration to allow hardlinking to work as expected once the download is complete.

The completed download handling works as expected within Deluge - it moves it to the appropriate location after downloading and Sonarr picks it up fine, just throws an exception and defaults back to copy when it fails to create the hardlink.

Debug Log entry:

18-9-17 09:08:07.3|Debug|DiskProvider|Hardlink '/data/deluge/complete/sonarr/Big.Brother.S20E36.Head.of.Household.#13.Nominations.#13.1080p.AMZN.WEB-DL.DDP2.0.H.264-KiNGS.mkv' to '/data/tv/Big Brother/Season 20/Big Brother - S20E36 - [WEBDL-1080P][EAC3 2.0][H264]-KINGS.mkv' failed. 

[v2.0.0.5250] Mono.Unix.UnixIOException: Invalid cross-device link [EXDEV]. at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00005] in <6ecdf4972be240398b3e321fbad963dd>:0 at Mono.Unix.UnixMarshal.ThrowExceptionForLastErrorIf (System.Int32 retval) [0x00004] in <6ecdf4972be240398b3e321fbad963dd>:0 at Mono.Unix.UnixFileSystemInfo.CreateLink (System.String path) [0x0000c] in <6ecdf4972be240398b3e321fbad963dd>:0 at NzbDrone.Mono.Disk.DiskProvider.TryCreateHardLink (System.String source, System.String destination) [0x00013] in C:\BuildAgent\work\5d7581516c0ee5b3\src\NzbDrone.Mono\Disk\DiskProvider.cs:182`

18-9-17 09:09:19.1|Debug|EpisodeService|Linking [Season 20/Big Brother - S20E36 - [WEBDL-1080P][EAC3 2.0][H264]-KINGS.mkv] > [[948]Head of Household #13; Nominations #13]

Sonarr:

Deluge:

Docker Compose Info:

[markus101]

AFAIK /data and /data/tv are still two separate mounts, even if they are in the same structure.

This is an OS limitation (that is the OS running in the container), AFAIK there is no way around it.

[kars85]

Thanks - I talked to the guys from Dockstarter who I based my compose file off of, and they said that the ${DOCKERSHAREDDIR}:/shared volume in each of those services was for hardlinking use cases.

So, I updated my .env variable to have DOCKERSHAREDDIR=/mnt/nfs

I shuffled my Deluge/Sonarr/Radarr/etc… paths around to reference this, so I should be back in business.

Thank you so much for your help.

[kars85]

I didn’t have any tv shows to grab, but I could upgrade a few of my movies. Does this in the debug log indicate hardlinking was successful (no errors or anything before or after this)

18-9-17 17:03:20.2|Info|MovieService|Linking [Video (2013) - [REMUX-1080P][DTS 5.1]-HIFI.mkv] > [[tt1800241][Video(2013)]]

I ran ls -al and find . -type l and neither indicated it was a hardlinked file, but I also read that it is hard to distinguish hardlinked files. Is there anyway you know of to verify?

[markus101]

You need to look at the inode for the files (they should match). That linking message is linking the file to the movie in the DB, not hardlinking the file.

[kars85]

Thanks again - I checked the inodes value for both the completed download file location and the final file location Sonarr/Radarr post-process to and they don’t match. Despite every file system interaction between the Sonarr and Deluge containers going through the /shared mapped volume.

I can certainly pastebin config files and create an imgur album of screenshots of all my configs and application settings if you have more time to help me isolate this issue. Everything I’ve asked others on Reddit and Discord ended up being what I’ve done - map the directories to a higher level and tree down within the download/Sonarr apps, but that didn’t work

I can only assumed you’re busy and do this on this side for fun, so I appreciate all your help and don’t take it for granted. I’m really out of ideas at this point though unless you have one last hail mary idea.

[markus101]

You checked the inode on the host?

Yeah, this is spare time thing and I’m out of ideas on this one (and it’s outside the an issue with Sonarr), but here are my configs for Sonarr/Transmission that work perfectly for hardlinking with a remote path mapping in Sonarr (though could be done without that mapping if I wanted to fix all the torrent paths).

• I’m running Sonarr v3, though there is no difference in how hardlinking works between v2 and v3
• Sonarr has access to everything with this setup, but I’m okay with that
• Mounting /volume1/downloads/:/volume1/downloads/ and /volume1/tv/:/volume1/tv/ are still two different volumes within the docker container and hardlinking will not work

Sonarr:

version: '3'
services:
  sonarr:
    image: "linuxserver/sonarr"
    ports:
     - "8989:8989"
    environment:
      - TZ=America/Vancouver
      - PGID=65537
      - PUID=1035
    volumes:
     - ./data/:/config
     - /volume1/:/volume1/
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "10"

Transmission:

version: '3'
services:
  transmission:
    image: "linuxserver/transmission:latest"
    ports:
      - "9091:9091"
      - "55555:55555"
      - "55555:55555/udp"
    environment:
      - TZ=America/Vancouver
      - PGID=65537
      - PUID=1030
    volumes:
     - ./data/:/config
     - /volume1/downloads/:/downloads

Remote Path Mapping:

[kars85]

I found the issue - one that I was hoping to avoid (reorganizing 15TB of media), but one that I’ll do for long term seeding.

In my container’s shell console (here’s deluge)

overlay                           245082396    22940612   209622616  10% /
tmpfs                                 65536           0       65536   0% /dev
tmpfs                               8143280           0     8143280   0% /sys/fs/cgroup
/dev/sda2                         245082396    22940612   209622616  10% /config
192.168.1.6:/volume2/HomeVideos 30892594304 17658038912 13234555392  58% /shared/HomeVideos
192.168.1.6:/volume2/Comics     30892594304 17658038912 13234555392  58% /shared/Comics
192.168.1.6:/volume2/Books      30892594304 17658038912 13234555392  58% /shared/Books
192.168.1.6:/volume2/Movies     30892594304 17658038912 13234555392  58% /shared/Movies
192.168.1.6:/volume2/Music      30892594304 17658038912 13234555392  58% /shared/Music
192.168.1.6:/volume2/TVShows    30892594304 17658038912 13234555392  58% /shared/TVShows
192.168.1.6:/volume2/Photos     30892594304 17658038912 13234555392  58% /shared/Photos
192.168.1.6:/volume2/Torrents   30892594304 17658038912 13234555392  58% /data
shm                                   65536           0       65536   0% /dev/shm
tmpfs                               8143280           0     8143280   0% /proc/acpi
tmpfs                               8143280           0     8143280   0% /proc/scsi
tmpfs                               8143280           0     8143280   0% /sys/firmware

And here’s Sonarr:

overlay 245082396 22938536 209624692 10% /
tmpfs 65536 0 65536 0% /dev
tmpfs 8143280 0 8143280 0% /sys/fs/cgroup
192.168.1.6:/volume2/Torrents 30892594304 17658032896 13234561408 58% /data
/dev/sda2 245082396 22938536 209624692 10% /shared
192.168.1.6::/volume2/HomeVideos 30892594304 17658032896 13234561408 58% /shared/HomeVideos
192.168.1.6::/volume2/Comics 30892594304 17658032896 13234561408 58% /shared/Comics
192.168.1.6::/volume2/Books 30892594304 17658032896 13234561408 58% /shared/Books
192.168.1.6::/volume2/Movies 30892594304 17658032896 13234561408 58% /movies
192.168.1.6::/volume2/Music 30892594304 17658032896 13234561408 58% /shared/Music
192.168.1.6::/volume2/TVShows 30892594304 17658032896 13234561408 58% /shared/TVShows
192.168.1.6::/volume2/Photos 30892594304 17658032896 13234561408 58% /shared/Photos
shm 65536 8 65528 1% /dev/shm
tmpfs 8143280 0 8143280 0% /proc/acpi
tmpfs 8143280 0 8143280 0% /proc/scsi
tmpfs 8143280 0 8143280 0% /sys/firmware

I don’t know why the formatting is goofy, but notice the missing mount /Shared (with no path after it). Basically, I need to create a physical parent folder called something like “media” and nest all my Torrents/Movies/TVShows under that, and reference that /mnt/nfs/media to /shared within the docker containers for everything.

I found this out by trying to create the hardlink through the bash console, seeing it error when I replicated what Sonarr would’ve done. Only then did it make sense. I’m a slow learner.