-
Last Edited:01 SEP 2017 (minor edits)
-
EDIT: THIS WILL ONLY PARTIALLY WORK WITH RADARR given current code. The tool that is used (NZBToMedia) will auto-extract the movie, but no method is in place to call Radarr for importing. So, you can use NTM as an auto extractor for Radarr, however, you’ll still need to use the drone factory to import. See this post which I authored for more details. I also got word that there is work to make this fully functional with Radarr.
-
DO NOT LET THE LENGTH OF THIS GUIDE SCARE YOU, the actual configuration using the guide should take no more than 15 minutes.
-
If you want to skip all the intro stuff and go directly to the config - go to Para 7 “Configuration”
1. OBJECTIVE: This NEW and tested method will enable the user to configure their environment to do the following:
a. Process all Torrent downloads (compressed and uncompressed) automatically and unmonitored with no intervention required by the user and minimal additional software.
b. Migrate from using the “Drone Factory” within Sonarr as it will soon no longer exist as a function.
c. Server can complete all tasks using system services (user does NOT need to be logged on)
d. Avoid file locks, media duplication, and the use of hard/sym links (if preferred)
e. Ensures that Torrent client (Deluge) + NZB downloader (if used) do not conflict.
f. Enables Sonarr to Handle completed and failed downloads – HUGE
2. Background:
Sonarr is fucking dope. Huge shout out to the devs @Taloth and @markus101 and @clinton-hall who created the software to make this work. However, If you are using Sonarr and Radarr, there is one issue that has been plaguing Windows users who use torrents: The inability to auto-extract and process compressed torrent downloads (.RAR). NZBs are not an issue because software like SABnzbd extracts releases before post-processing. And if you’re using Linux, please close this page now.There were several methods to complete this with somewhat complicated methods( including this one I authored), that all relied on the Sonarr function “Drone Factory”. However, this function is being depreciated . This left many users no method to address this issue. This guide will resolve that.
- If you have a better suggestion, please comment below, however, ensure that your suggestions address all the objectives listed above. If you say, “but bro, just download a Linux docker and you’ll be golden bro.” I’m going to find you and poop in your beer.
3. Software Needed:
NOTE: This guide assumes you already have the necessary software installed to run Sonarr/Radarr, your DL clients, and their dependancy software. If that software is not working properly, I suggest you debug those issues first before implementing this (especially file permission problems). Although the implementation of this feature is somewhat easy, it can be considered an advanced configuration.
- Sonarr: https://github.com/Sonarr/Sonarr
-
NZBtoMedia: https://github.com/clinton-hall/nzbToMedia
NOTE: Outstanding issue via GitHub: This guide requires you to edit code that is cloned from the REPO, this means that if you update the REPO and the custom code referenced here is overwritten, unexpected things may happen. Please see this issue for reference: https://github.com/clinton-hall/nzbToMedia/issues/1276 -
Torrent Client (i.e. Deluge): http://dev.deluge-torrent.org/wiki/Download
- Torrent Client Plug-Ins (3):
- Label: Default plugin that auto-categorizes torrents i.e., “TV”, “Movies”, ’Music”
- Auto-Add: http://dev.deluge-torrent.org/wiki/Plugins/AutoAdd Monitors folders and moves the torrents around based on DIR monitoring and labels. I know this is baked into most clients, but for deluge, this is called Auto-add (this is different from the static settings you configure in deluge - preferences -> downloads )
- Auto-Execute: http://dev.deluge-torrent.org/wiki/Plugins/Execute Default Plugin that will execute a script upon torrent completion
NOTE: It’s important that this plug-in can execute .BAT OR . PY scripts. For example, the Deluge execute plug-in will NOT execute .py scripts, so I will cover how to convert the .py script to a .bat script.
- Torrent Client Plug-Ins (3):
4. Non-standard vocab:
a. “NTM” - NzbToMedia – Software that extracts releases from torrent client and sends command to Sonarr for post-processing.
b. “Release” – raw .torrent/NZB unprocessed data sent from Sonarr to download client.
c. .RAR/.NZB – Compressed files downloaded via download client
d. “Finished DIR” – Directory where torrent/NZB client sends finished downloads for seeding/processing by NTM (NOT Sonarr).
e. “Done DIR” – Intermediate/temporary directory where NTM sends fully extracted media for post-processing by Sonarr.
g. “Media DIR” - Directory where the media is placed via Sonarr at end of job. This is usually the directory where your front end media player (PLEX/KODI) serves media.
5. Environment: This has been tested on two different rigs, I personally, am using the following:
-
Server: Windows 10E 1703 CU x64 (Sonarr, PLEX, Deluge all run as system services)
-
Seedbox (LAN): VM Windows 10P 1703 (VPN, Deluge, NTM)
-
Directory/File Structure (example, however, don’t screw this up, it’s important):
NOTE: (I have everything below d:\downloads hard-linked to my seedbox, however, a mapped network drive, FTP, etc would all work as long as Sonarr and your DL client can write to them, make sure to test your file structure FIRST before implementing this configuration)
├── D:\ [root non-system directory]
├── Media\ [Media DIR which PLEX/Kodi uses to serve fully processed media]
├── Movies
├── TV
├── Downloads\ [root folder where all downloads reside]
├── Drone\ [temp DIR]
├── TV [created by NTM]
├── Movies [created by NTM]
├── Incomplete\ [currently downloading]
├── TV
├── Movies
├── TV\ [finished/seeding DIR]
├── Torrents
├── NZBs
├── Movies\ [finished/seeding dir]
├── Torrents
├── NZBs
├── Empty [empty DIR which will be used for Sonarr’s “remote path mapping” **important**
6. Process explanation (How this shit works):
- non-Rar release flow:
- Sonarr sends release (non-.RAR) to deluge appends “TV” Label ->
- Deluge begins downloading in Incomplete DIR ->
- Sonarr tracks download because “TV” label is appended to torrent ->
- Deluge completes download, moves torrent data to finished DIR, begins seeding->
- Sonarr doesn’t import because remote path is EMPTY. ->
- Deluge executes NTM script ->
- NTM doesn’t detect .rar, COPIES release to drone folder in sub dir \TV\releasename\release.mkv ->
- NTM sends EPISODEASCAN command to sonarr ->
- Sonarr finds extracted media file in dir \TV\releasename\ ->
- Sonarr imports, post-processes, and MOVES release to final destination.
- COMPLETE: Release is copied to final media DIR, Drone DIR is clean, torrent is still being seeding in its original finished dir.
- Rar-Release flow:
- Sonarr sends release (.RAR) to deluge appends “TV” Label ->
- no change
- no change
- no change
- Sonarr doesn’t import because remote path is EMPTY. ->
- Deluge executes NTM script ->
- NTM detects .RAR, auto pauses torrent, extracts release, sends ONLY extracted media file to drone folder in sub dir \TV\releasename\release.mkv, resumes torrent for seeding ->
- NTM sends EPISODESCAN command to sonarr ->
- Sonarr finds extracted media file in dir \TV\releasename\ ->
- Sonarr MOVES release to final media destination.
- COMPLETE: Release is extracted, sent to Media DIR, Drone DIR is clean, original .rar torrent is still being seeding in finished DIR
Note 1: See the bottom of this guide for what the NTM LOG should look like after process is completed
Note 2: NTM Will “evaluate” ALL torrents once completed downloading because the Auto-Extract plugin does NOT allow you to execute based on labels. However, NTM will ONLY process those completed downloads which match the label you set forth in the NTM config file. By design, you will/should see this line in the NTM logs for every completed download that DOES NOT meet the processing criteria:
2017-07-20 06:52:10 ERROR ::MAIN: Category:[LABEL] is not defined or is not enabled. Please rename it or ensure it is enabled for the appropriate section in your autoProcessMedia.cfg and try again.
2017-07-20 06:52:10 ERROR ::MAIN: A problem was reported in the C:\nzbToMedia\TorrentToMedia.py script.
7. Configuration
-
Sonarr:
Sonarr settings -> Download client ->
- ENABLE completed download handling
- ENABLE Remove
- Redownload Failed download: I recommend YES
- Remove Failed downloads: I recommend YES
- DISABLE: Drone factory
- Remote Path Mappings:
- Host: IP address of host you entered for your download client
- Remote Path: location to an EMPTY Dir, Local Path: same
(exmaple: I created an empty dir on my seedbox at d:\downloads\empty and the same on the server (Sonarr)).
- Optional: configure your NZB downloader in the same manner
- Deluge (Or your preferred torrent client):
- Preferences -> Downloads
- Download to: Incomplete DIR (DO NOT TOUCH ANYTHING ELSE. (The autoadd plugin will handle the rest))
- Preferences -> Plug-ins:
- Enable: LABEL, Create Labels from deluge main screen: (TV, Movies, etc.).
- Enable: AutoAdd, add “Watch Folder” path for each label you created from above (Screenshot)
- Path Options:
- Watch folder d:\downloads\Incomplete\TV
- Set download location: d:\downloads\Incomplete\TV
- Move completed: d:\downloads\TV\torrents (this will be your Completed DIR)
- Label: TV
- Enable: Execute
- Add Command:
- Event: Torrent Complete,
- Comand: C:\NZBtoMedia\torrenttomedia.bat
(We will create this file later when we clone the NTM repo)
- NZBtoMedia
Note 1: Make sure the NTM config is completed/executed on the box that is running your torrent client aka seedbox)
Note 2: I recommend first browsing to the NTM repo and read the wiki so you can understand how this works. There might be some dependant software you need to install (7-zip)
Note 3: NTM does A LOT more than just auto-extract. As you see in the below .cfg file there are MANY options to fully encode, meta-data, etc. For this purpose, I will only cover the basic code that needs to be in place for this extractions and auto-import process to work.
- Clone Repo from https://github.com/clinton-hall/nzbToMedia
- Command Prompt ->
- “CD C:\”
- “Git clone https://github.com/clinton-hall/nzbToMedia”
- This will create a folder at C:\NZBtoMedia with all the files from the remote repo.
- CD to c:\NZBtoMedia
- Create a .txt file in c:\nzbtoMedia\torrenttomedia.txt (This will be the script DELUGE executes when a torrent is completed downloading
- Edit torrenttomedia.txt ->
- Insert the text: “ C:\nzbToMedia\torrenttomedia.py %1 %2 %3 " (One single line, without quotes).
- save .txt file.
- Rename torrenttomedia.txt to torrenttomedia.bat
- Copy and rename AutoProcessmeda.cfg.spec to autoprocessmedia.cfg
- Edit \NZBtoMedia\autoproecssmedia.cfg with the following values:
[General]
# Enable/Disable update notifications
version_notify = 1
# Enable/Disable automatic updates
auto_update = 1
# Set to the full path to the git executable
git_path = ""
# GitHUB user for repo
git_user = ""
# GitHUB branch for repo
git_branch = ""
# Enable/Disable forceful cleaning of leftover files following postprocess
force_clean = 1
# Enable/Disable logging debug messages to nzbtomedia.log
log_debug = 0
# Enable/Disable logging database messages to nzbtomedia.log
log_db = 1
# Enable/Disable logging environment variables to debug nzbtomedia.log (helpful to track down errors calling external tools.)
log_env = 1
# Enable/Disable logging git output to debug nzbtomedia.log (helpful to track down update failures.)
log_git = 1
# Set to the directory where your ffmpeg/ffprobe executables are located
ffmpeg_path = ""
# Enable/Disable media file checking using ffprobe.
check_media = 0
# Enable/Disable a safety check to ensure we don't process all downloads in the default_downloadDirectories by mistake.
safe_mode = 1
# Turn this on to disable additional extraction attempts for failed downloads. Default = 0 will attempt to extract and verify if media is present.
no_extract_failed = 0
[NzbDrone]
#### autoProcessing for TV Series
#### ndCategory - category that gets called for post-processing with NzbDrone
[[tv]]
enabled = 1
apikey = SonarrAPIKey
host = IP Address of Sonarr
port = 38084
username = ""
password = ""
###### ADVANCED USE - ONLY EDIT IF YOU KNOW WHAT YOU'RE DOING ######
web_root = /sonarr
ssl = 0
delete_failed = 0
# Enable/Disable linking for Torrents
Torrent_NoLink = 0
keep_archive = 1
extract = 1
nzbExtractionBy = Downloader
wait_for = 2
# Set this to minimum required size to consider a media file valid (in MB)
minSize = 0
# Enable/Disable deleting ignored files (samples and invalid media files)
delete_ignored = 1
##### Enable if NzbDrone is on a remote server for this category
remote_path = 1 (You would enable this if the machine this file resides on (your seebox) is different from the box Sonarr runs on)
##### Set to path where download client places completed downloads locally for this category
watch_dir = C:\Media\PlexDownloads\TV\torrents
[Torrent]
###### clientAgent - Supported clients: utorrent, transmission, deluge, rtorrent, vuze, other
clientAgent = deluge
###### useLink - Set to hard for physical links, sym for symbolic links, move to move, move-sym to move and link back, and no to not use links (copy)
useLink = hard
###### outputDirectory - Default output directory (categories will be appended as sub directory to outputDirectory)
outputDirectory = C:\Media\PlexDownloads\drone
###### Enter the default path to your default download directory (non-category downloads). this directory is protected by safe_mode.
default_downloadDirectory = C:\Media\PlexDownloads\tv\torrents
###### Other categories/labels defined for your downloader. Does not include CouchPotato, SickBeard, HeadPhones, Mylar categories.
categories = tv
###### A list of categories that you don't want to be flattened (i.e preserve the directory structure when copying/linking.
noFlatten = ""
#### Deluge (You must edit this if your using TorrentToMedia.py with deluge. Note that the host/port is for the deluge daemon, not the webui)
DelugeHost = IP.Address
DelugePort = 58846
DelugeUSR =
DelugePWD =
###### ADVANCED USE - ONLY EDIT IF YOU KNOW WHAT YOU'RE DOING ######
deleteOriginal = 0 **ENSURE THIS IS 0, if not, once Sonarr post-processes the release, the torrent will fail.
chmodDirectory = 0
resume = 1
resumeOnFailure = 1
[Extensions]
compressedExtensions = .zip, .rar, .7z, .gz, .bz, .tar, .arj, .1, .01, .001
mediaExtensions = .mkv, .avi, .divx, .xvid, .mov, .wmv, .mp4, .mpg, .mpeg, .vob, .iso, .m4v, .ts
audioExtensions = ""
metaExtensions = ""
9.Ensure the above file is saved and file permissions are set appropriately.
10.You must now edit the repo code.
11. Within the NTM local repo, find the file: c:\NZBtoMedia\core\autoProcess\autoProcesstv.py
12. Make a backup of the original file, rename it to autoProcesstv.py.default
13. Copy autoProcesstv.py.default, rename to autoProcesstv.py
14. Use a text editor to CHANGE the following lines 251 - 256: Reference this issue as to why you need to edit the repo code.
if remote_path:
logger.debug("remote_path: {0}".format(remoteDir(dirName)), section)
data = {"name": "DownloadedEpisodesScan", "path": remoteDir(dirName), "downloadClientId": download_id, "importMode": "Move"}
else:
logger.debug("path: {0}".format(dirName), section)
data = {"name": "DownloadedEpisodesScan", "path": dirName, "downloadClientId": download_id, "importMode": "Move"}
15.Save autoProcesstv.py
16. TEST the torrenttomedia.bat (ensure your torrent client is OFFLINE) -> If configured correctly, you’ll see a CMD shell, and should disappear.
17. Obviously, it shouldn’t need to process anything, however, look at the logs for obvious errors.
18 Check the logs at c:\nzbdtomedia\logs
19. I would also recommend to TEST the entire process by manually downloading a .RARed .torrent, append the TV label, and see if it auto extracts to your drone folder
20. Once you verify that the Deluge is executing the script, and NTM is extracting and processing files correctly, I would then test the entire process starting by manually downloading a release for a monitored episode from within the Sonarr WebUI and let 'er rip.
PROFIT!
Huge shout out to @TALOTH for taking flack from all of us entitled users (well, a couple of us) who got all triggered when the decision was made to do away with the drone factory. Also, @Clinton-Hall for his dope code that makes this shit all work.
Cheers.
NOTE: Below is what a successful NTM job LOG should look like for both .RARed and Non-RARed releases:
-
RARed Release:
Result: Release EXTRACTED, copied to drone folder, post-processing completed by Sonarr, Drone DIR clean. BOOM!
2017-07-20 09:37:22 DB ::MAIN: nzbtomedia.db: UPDATE downloads SET status = ?, client_agent = ?, last_update = ?, input_id = ?, input_hash = ?, input_name = ? WHERE input_directory = ? with args [0, u'deluge', 736530, u'd927ed42a4332c62c65b18179879d7c82dd9eb3b', u'd927ed42a4332c62c65b18179879d7c82dd9eb3b', u'Suits.S07E02.1080p.WEB.x264-TBS', u'C:\\Media\\Plexdownloads\\tv\\Torrents']
2017-07-20 09:37:22 INFO ::MAIN: SEARCH: Found torrent directory Suits.S07E02.1080p.WEB.x264-TBS in input directory directory C:\Media\Plexdownloads\tv\Torrents
2017-07-20 09:37:22 INFO ::MAIN: SEARCH: Setting inputDirectory to C:\Media\Plexdownloads\tv\Torrents\Suits.S07E02.1080p.WEB.x264-TBS
2017-07-20 09:37:22 INFO ::MAIN: Auto-detected SECTION:NzbDrone
2017-07-20 09:37:27 INFO ::MAIN: Output directory set to: C:\Media\PlexDownloads\drone\tv\Suits.S07E02.1080p.WEB.x264-TBS
2017-07-20 09:37:27 INFO ::MAIN: Extracting C:\Media\Plexdownloads\tv\Torrents\Suits.S07E02.1080p.WEB.x264-TBS\suits.s07e02.1080p.web.x264-tbs.rar to C:\Media\PlexDownloads\drone\tv\Suits.S07E02.1080p.WEB.x264-TBS
2017-07-20 09:37:34 INFO ::MAIN: EXTRACTOR: Extraction was successful for C:\Media\Plexdownloads\tv\Torrents\Suits.S07E02.1080p.WEB.x264-TBS\suits.s07e02.1080p.web.x264-tbs.rar to C:\Media\PlexDownloads\drone\tv\Suits.S07E02.1080p.WEB.x264-TBS
2017-07-20 09:37:38 INFO ::MAIN: FLATTEN: Flattening directory: C:\Media\PlexDownloads\drone\tv\Suits.S07E02.1080p.WEB.x264-TBS
2017-07-20 09:37:39 INFO ::MAIN: Found 1 media files in C:\Media\PlexDownloads\drone\tv\Suits.S07E02.1080p.WEB.x264-TBS
2017-07-20 09:37:39 INFO ::MAIN: Calling NzbDrone:tv to post-process:Suits.S07E02.1080p.WEB.x264-TBS
2017-07-20 09:37:39 INFO ::MAIN: Attempting to verify tv fork
2017-07-20 09:37:39 INFO ::MAIN: NzbDrone:tv fork set to default
2017-07-20 09:37:40 INFO ::MAIN: FLATTEN: Flattening directory: C:\Media\PlexDownloads\drone\tv\Suits.S07E02.1080p.WEB.x264-TBS
2017-07-20 09:37:40 POSTPROCESS::NZBDRONE: SUCCESS: The download succeeded, sending a post-process request
2017-07-20 09:38:00 DB ::DB: Updating status of our download Suits.S07E02.1080p.WEB.x264-TBS in the DB to 1
2017-07-20 09:38:00 DB ::MAIN: nzbtomedia.db: UPDATE downloads SET status=?, last_update=? WHERE input_name=? with args [1, 736530, u'Suits.S07E02.1080p.WEB.x264-TBS']
2017-07-20 09:38:06 INFO ::CLEANDIRS: Directory C:\Media\PlexDownloads\drone\tv\Suits.S07E02.1080p.WEB.x264-TBS has been processed, removing ...
2017-07-20 09:38:06 INFO ::MAIN: The C:\nzbToMedia\TorrentToMedia.py script completed successfully.
-
Non-RARed Release:
Result: Release COPIED to drone folder, post-processing completed by Sonarr, Drone DIR clean. BOOM!
2017-07-19 06:49:16 INFO ::MAIN: #########################################################
2017-07-19 06:49:16 INFO ::MAIN: ## ..::[TorrentToMedia.py]::.. ##
2017-07-19 06:49:16 INFO ::MAIN: #########################################################
2017-07-19 06:49:16 DB ::MAIN: nzbtomedia.db: UPDATE downloads SET status = ?, client_agent = ?, last_update = ?, input_id = ?, input_hash = ?, input_name = ? WHERE input_directory = ? with args [0, u'deluge', 736529, u'3756fdc9bab0c77fe3a79918f7b70293c1cc29f0', u'3756fdc9bab0c77fe3a79918f7b70293c1cc29f0', u'The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv', u'C:\\Media\\Plexdownloads\\tv\\Torrents']
2017-07-19 06:49:16 INFO ::MAIN: SEARCH: Found torrent file The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv in input directory directory C:\Media\Plexdownloads\tv\Torrents
2017-07-19 06:49:16 INFO ::MAIN: SEARCH: Setting inputDirectory to C:\Media\Plexdownloads\tv\Torrents\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:16 INFO ::MAIN: Auto-detected SECTION:NzbDrone
2017-07-19 06:49:21 INFO ::MAIN: Output directory set to: C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:21 INFO ::COPYLINK: MEDIAFILE: [The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv]
2017-07-19 06:49:21 INFO ::COPYLINK: SOURCE FOLDER: [C:\Media\Plexdownloads\tv\Torrents]
2017-07-19 06:49:21 INFO ::COPYLINK: TARGET FOLDER: [C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv]
2017-07-19 06:49:21 INFO ::COPYLINK: Hard linking SOURCE MEDIAFILE -> TARGET FOLDER
2017-07-19 06:49:21 WARNING ::COPYLINK: Error: Incorrect function.
, copying instead ...
2017-07-19 06:49:21 INFO ::COPYLINK: Copying SOURCE MEDIAFILE -> TARGET FOLDER
2017-07-19 06:49:30 INFO ::MAIN: FLATTEN: Flattening directory: C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:30 INFO ::MAIN: Found 1 media files in C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:30 INFO ::MAIN: Calling NzbDrone:tv to post-process:The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:30 INFO ::MAIN: Attempting to verify tv fork
2017-07-19 06:49:30 INFO ::MAIN: NzbDrone:tv fork set to default
2017-07-19 06:49:30 INFO ::MAIN: FLATTEN: Flattening directory: C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:30 POSTPROCESS::NZBDRONE: SUCCESS: The download succeeded, sending a post-process request
2017-07-19 06:49:50 DB ::DB: Updating status of our download The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv in the DB to 1
2017-07-19 06:49:50 DB ::MAIN: nzbtomedia.db: UPDATE downloads SET status=?, last_update=? WHERE input_name=? with args [1, 736529, u'The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv']
2017-07-19 06:49:55 INFO ::CLEANDIRS: Directory C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv has been processed, removing ...
2017-07-19 06:49:55 INFO ::MAIN: The C:\nzbToMedia\TorrentToMedia.py script completed successfully.