Importing extracted files without Drone Factory [unrar]


#21

Also @Taloth -

Do you have an ETA when you guys are going to discontinue the drone factory?


#22

That should largely be a non-issue, you shouldn’t mix the output folder for you nzb and torrent client. In that case it doesn’t use the mapping nor nzbToMedia.

So no ETA, but as mentioned on the wiki, it won’t automatically disappear since v3 includes a collection of backwards incompatible changes and will be a manual update.


#23

@Taloth

HOLY SHIT. I think we got it.

Doing some final testing.


#24

YAS! It worked! (logs below)

Tested with a .rar release and non-.rar release.

  • non-Rar release flow:
    Sonarr sends release to deluge -> Deluge downloads, begins seeding in finished DIR-> Sonarr doesn’t import because remote path is EMPTY. -> NTM gets finished flag from DELUGE, COPIES to “drone/temp” dir -> NTM sends episodescan command to sonarr -> Sonarr finds media in “temp/drone folder” -> Sonarr MOVES release to final destination . Temp DIR is clean, original .rar torrent is still being seeding in it’s original finished dir.

  • Rar-Release:
    Sonarr send release (rar) to deluge -> Deluge downloads, sends .RARs to “finished/seeding dir” begins seeding -> NTM extracts release, sends extract media file it to “temp/drone folder” -> NTM sends episodescan command to sonarr -> Sonarr finds extracted media file in “temp/drone folder” -> Sonarr MOVES release to final destination . Temp DIR is clean, original .rar torrent is still being seeding in it’s original finished dir.


So to recap:

1. Sonarr settings -> Download Client -> disable drone factory

2. Sonarr settings -> Download Client -> Enable Remote & local paths, however, ensure these directories exist on disk, but are empty

3. Config NzbToMedia autoprocessmedia.cfg to the following:


[NzbDrone]

[[tv]]
    enabled = 1
    apikey = KEY
    host = sonarrhost
    port = sonarrport
    username = ""
    password = ""
    ###### ADVANCED USE - ONLY EDIT IF YOU KNOW WHAT YOU'RE DOING ######
    web_root = /sonarr
    ssl = 0
    delete_failed = 0
    # Enable/Disable linking for Torrents
    Torrent_NoLink = 0
    keep_archive = 1
    extract = 1
    nzbExtractionBy = Downloader
    wait_for = 2
    # Set this to minimum required size to consider a media file valid (in MB)
    minSize = 0
    # Enable/Disable deleting ignored files (samples and invalid media files)
    delete_ignored = 1
    ##### Enable if NzbDrone is on a remote server for this category
    remote_
[Torrent]
   
    clientAgent = deluge
   
    useLink = hard
   
    outputDirectory = C:\Media\PlexDownloads\drone (This is the old "Drone factory folder / temp folder)
    
    default_downloadDirectory = C:\Media\PlexDownloads\tv\torrents (Finished DIR / Seeding DIR)
   
    categories = tv
  
    noFlatten = ""
   
    DelugeHost = IP
    DelugePort = PORT
    DelugeUSR = admin
    DelugePWD = PW
    ###### ADVANCED USE - ONLY EDIT IF YOU KNOW WHAT YOU'RE DOING ######
    deleteOriginal = 0
    chmodDirectory = 0
    resume = 1
    resumeOnFailure = 1

[Extensions]
    compressedExtensions = .zip, .rar, .7z, .gz, .bz, .tar, .arj, .1, .01, .001
    mediaExtensions = .mkv, .avi, .divx, .xvid, .mov, .wmv, .mp4, .mpg, .mpeg, .vob, .iso, .m4v, .ts
    audioExtensions = ""
    metaExtensions = ""

4. Change the \core\NzbToMedia autoProcesstv.py lines 251 - 256 to this:

            if remote_path:
                logger.debug("remote_path: {0}".format(remoteDir(dirName)), section)
                data = {"name": "DownloadedEpisodesScan", "path": remoteDir(dirName), "downloadClientId": download_id, "importMode": "Move"}
            else:
                logger.debug("path: {0}".format(dirName), section)
                data = {"name": "DownloadedEpisodesScan", "path": dirName, "downloadClientId": download_id, "importMode": "Move"}

I’ll write this up in another post to make it all pretty, but does this sound right @taloth ?


Reference logs:

** RARed release / no modification to  .py**
Result:  Release extracted, post-processing complete, orphaned in drone folder



2017-07-19 05:06:40 DB      ::MAIN: nzbtomedia.db: UPDATE downloads SET status = ?, client_agent = ?, last_update = ?, input_id = ?, input_hash = ?, input_name = ? WHERE input_directory = ? with args [0, u'deluge', 736529, u'edde03257c2712865a888e18349313dc083327a9', u'edde03257c2712865a888e18349313dc083327a9', u'Tosh.0.S09E16.1080p.WEB.x264-TBS', u'C:\\Media\\Plexdownloads\\tv\\Torrents']
2017-07-19 05:06:40 INFO    ::MAIN: SEARCH: Found torrent directory Tosh.0.S09E16.1080p.WEB.x264-TBS in input directory directory C:\Media\Plexdownloads\tv\Torrents
2017-07-19 05:06:40 INFO    ::MAIN: SEARCH: Setting inputDirectory to C:\Media\Plexdownloads\tv\Torrents\Tosh.0.S09E16.1080p.WEB.x264-TBS
2017-07-19 05:06:40 INFO    ::MAIN: Auto-detected SECTION:NzbDrone
2017-07-19 05:06:45 INFO    ::MAIN: Output directory set to: C:\Media\PlexDownloads\drone\tv\Tosh.0.S09E16.1080p.WEB.x264-TBS
2017-07-19 05:06:46 INFO    ::MAIN: Extracting C:\Media\Plexdownloads\tv\Torrents\Tosh.0.S09E16.1080p.WEB.x264-TBS\tosh.0.s09e16.1080p.web.x264-tbs.rar to C:\Media\PlexDownloads\drone\tv\Tosh.0.S09E16.1080p.WEB.x264-TBS
2017-07-19 05:06:48 INFO    ::MAIN: EXTRACTOR: Extraction was successful for C:\Media\Plexdownloads\tv\Torrents\Tosh.0.S09E16.1080p.WEB.x264-TBS\tosh.0.s09e16.1080p.web.x264-tbs.rar to C:\Media\PlexDownloads\drone\tv\Tosh.0.S09E16.1080p.WEB.x264-TBS
2017-07-19 05:06:51 INFO    ::MAIN: FLATTEN: Flattening directory: C:\Media\PlexDownloads\drone\tv\Tosh.0.S09E16.1080p.WEB.x264-TBS
2017-07-19 05:06:51 INFO    ::MAIN: Found 1 media files in C:\Media\PlexDownloads\drone\tv\Tosh.0.S09E16.1080p.WEB.x264-TBS
2017-07-19 05:06:51 INFO    ::MAIN: Calling NzbDrone:tv to post-process:Tosh.0.S09E16.1080p.WEB.x264-TBS
2017-07-19 05:06:51 INFO    ::MAIN: Attempting to verify tv fork
2017-07-19 05:06:51 INFO    ::MAIN: NzbDrone:tv fork set to default
2017-07-19 05:06:51 INFO    ::MAIN: FLATTEN: Flattening directory: C:\Media\PlexDownloads\drone\tv\Tosh.0.S09E16.1080p.WEB.x264-TBS
2017-07-19 05:06:51 POSTPROCESS::NZBDRONE: SUCCESS: The download succeeded, sending a post-process request
2017-07-19 05:07:11 DB      ::DB: Updating status of our download Tosh.0.S09E16.1080p.WEB.x264-TBS in the DB to 1
2017-07-19 05:07:11 DB      ::MAIN: nzbtomedia.db: UPDATE downloads SET status=?, last_update=? WHERE input_name=? with args [1, 736529, u'Tosh.0.S09E16.1080p.WEB.x264-TBS']
2017-07-19 05:07:16 INFO    ::CLEANDIRS: Directory C:\Media\PlexDownloads\drone\tv\Tosh.0.S09E16.1080p.WEB.x264-TBS still contains 1 unprocessed file(s), skipping ...
2017-07-19 05:07:16 INFO    ::MAIN: The C:\nzbToMedia\TorrentToMedia.py script completed successfully.




----------

*** non-RARed release // Altered autoProcesstv.py file:**

Result:  Release is copied to drone folder and orphaned.  



2017-07-19 05:37:49 INFO    ::MAIN: #########################################################
2017-07-19 05:37:49 INFO    ::MAIN: ## ..::[TorrentToMedia.py]::.. ##
2017-07-19 05:37:49 INFO    ::MAIN: #########################################################
2017-07-19 05:37:49 DB      ::MAIN: nzbtomedia.db: UPDATE downloads SET status = ?, client_agent = ?, last_update = ?, input_id = ?, input_hash = ?, input_name = ? WHERE input_directory = ? with args [0, u'deluge', 736529, u'd4d6a85fa26ac6e6ab22c2b638ad6bda4fde80e4', u'd4d6a85fa26ac6e6ab22c2b638ad6bda4fde80e4', u'Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv', u'C:\\Media\\Plexdownloads\\tv\\Torrents']
2017-07-19 05:37:50 INFO    ::MAIN: SEARCH: Found torrent file Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv in input directory directory C:\Media\Plexdownloads\tv\Torrents
2017-07-19 05:37:50 INFO    ::MAIN: SEARCH: Setting inputDirectory to C:\Media\Plexdownloads\tv\Torrents\Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv
2017-07-19 05:37:50 INFO    ::MAIN: Auto-detected SECTION:NzbDrone
2017-07-19 05:37:55 INFO    ::MAIN: Output directory set to: C:\Media\PlexDownloads\drone\tv\Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv
2017-07-19 05:37:55 INFO    ::COPYLINK: MEDIAFILE: [Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv]
2017-07-19 05:37:55 INFO    ::COPYLINK: SOURCE FOLDER: [C:\Media\Plexdownloads\tv\Torrents]
2017-07-19 05:37:55 INFO    ::COPYLINK: TARGET FOLDER: [C:\Media\PlexDownloads\drone\tv\Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv]
2017-07-19 05:37:55 INFO    ::COPYLINK: Hard linking SOURCE MEDIAFILE -> TARGET FOLDER
2017-07-19 05:37:55 WARNING ::COPYLINK: Error: Incorrect function.
, copying instead ... 
2017-07-19 05:37:55 INFO    ::COPYLINK: Copying SOURCE MEDIAFILE -> TARGET FOLDER
2017-07-19 05:38:23 INFO    ::MAIN: FLATTEN: Flattening directory: C:\Media\PlexDownloads\drone\tv\Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv
2017-07-19 05:38:23 INFO    ::MAIN: Found 1 media files in C:\Media\PlexDownloads\drone\tv\Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv
2017-07-19 05:38:23 INFO    ::MAIN: Calling NzbDrone:tv to post-process:Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv
2017-07-19 05:38:23 INFO    ::MAIN: Attempting to verify tv fork
2017-07-19 05:38:23 INFO    ::MAIN: NzbDrone:tv fork set to default
2017-07-19 05:38:23 INFO    ::MAIN: FLATTEN: Flattening directory: C:\Media\PlexDownloads\drone\tv\Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv
2017-07-19 05:38:23 POSTPROCESS::NZBDRONE: SUCCESS: The download succeeded, sending a post-process request
2017-07-19 05:38:43 DB      ::DB: Updating status of our download Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv in the DB to 1
2017-07-19 05:38:43 DB      ::MAIN: nzbtomedia.db: UPDATE downloads SET status=?, last_update=? WHERE input_name=? with args [1, 736529, u'Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv']
2017-07-19 05:38:48 INFO    ::CLEANDIRS: Directory C:\Media\PlexDownloads\drone\tv\Frontline.S2017E14.Life.on.Parole.1080p.iT.WEB-DL.AAC2.0.H.264-QUEENS.mkv has been processed, removing ...
2017-07-19 05:38:48 INFO    ::MAIN: The C:\nzbToMedia\TorrentToMedia.py script completed successfully.



----------

*** non Rared file, modified .py, remote path removed**

result:  copied to drone folder, PP completed by Sonarr, DIR clean.  BOOM!




2017-07-19 06:49:16 INFO    ::MAIN: #########################################################
2017-07-19 06:49:16 INFO    ::MAIN: ## ..::[TorrentToMedia.py]::.. ##
2017-07-19 06:49:16 INFO    ::MAIN: #########################################################
2017-07-19 06:49:16 DB      ::MAIN: nzbtomedia.db: UPDATE downloads SET status = ?, client_agent = ?, last_update = ?, input_id = ?, input_hash = ?, input_name = ? WHERE input_directory = ? with args [0, u'deluge', 736529, u'3756fdc9bab0c77fe3a79918f7b70293c1cc29f0', u'3756fdc9bab0c77fe3a79918f7b70293c1cc29f0', u'The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv', u'C:\\Media\\Plexdownloads\\tv\\Torrents']
2017-07-19 06:49:16 INFO    ::MAIN: SEARCH: Found torrent file The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv in input directory directory C:\Media\Plexdownloads\tv\Torrents
2017-07-19 06:49:16 INFO    ::MAIN: SEARCH: Setting inputDirectory to C:\Media\Plexdownloads\tv\Torrents\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:16 INFO    ::MAIN: Auto-detected SECTION:NzbDrone
2017-07-19 06:49:21 INFO    ::MAIN: Output directory set to: C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:21 INFO    ::COPYLINK: MEDIAFILE: [The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv]
2017-07-19 06:49:21 INFO    ::COPYLINK: SOURCE FOLDER: [C:\Media\Plexdownloads\tv\Torrents]
2017-07-19 06:49:21 INFO    ::COPYLINK: TARGET FOLDER: [C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv]
2017-07-19 06:49:21 INFO    ::COPYLINK: Hard linking SOURCE MEDIAFILE -> TARGET FOLDER
2017-07-19 06:49:21 WARNING ::COPYLINK: Error: Incorrect function.
, copying instead ... 
2017-07-19 06:49:21 INFO    ::COPYLINK: Copying SOURCE MEDIAFILE -> TARGET FOLDER
2017-07-19 06:49:30 INFO    ::MAIN: FLATTEN: Flattening directory: C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:30 INFO    ::MAIN: Found 1 media files in C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:30 INFO    ::MAIN: Calling NzbDrone:tv to post-process:The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:30 INFO    ::MAIN: Attempting to verify tv fork
2017-07-19 06:49:30 INFO    ::MAIN: NzbDrone:tv fork set to default
2017-07-19 06:49:30 INFO    ::MAIN: FLATTEN: Flattening directory: C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv
2017-07-19 06:49:30 POSTPROCESS::NZBDRONE: SUCCESS: The download succeeded, sending a post-process request
2017-07-19 06:49:50 DB      ::DB: Updating status of our download The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv in the DB to 1
2017-07-19 06:49:50 DB      ::MAIN: nzbtomedia.db: UPDATE downloads SET status=?, last_update=? WHERE input_name=? with args [1, 736529, u'The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv']
2017-07-19 06:49:55 INFO    ::CLEANDIRS: Directory C:\Media\PlexDownloads\drone\tv\The.Jim.Jefferies.Show.S01E05.1080p.WEB.x264-TBS.mkv has been processed, removing ...
2017-07-19 06:49:55 INFO    ::MAIN: The C:\nzbToMedia\TorrentToMedia.py script completed successfully.





#25

At 2) Instructions seem to be a bit unclear. The Remote Path Mapping must have an appropriate Remote & Local path specified. I assume you meant to say that the Local Directory specified is empty on disk.

At 3) Some cleanup might be a good idea. the block unintentionally ends with ‘remote_’.

At 4) you’re not actually replacing any code. Might want to update in the issue you posted for clinton as well.

Otherwise I’d say it looks correct, I can’t comment on all the nzbToMedia specific settings but I’ve seen nothing weird.
However, you might want to make it more obvious which fields a user is supposed to update, paths, usernames/pw, etc.


#26

@Taloth

"you’re not actually replacing any code. Might want to update in the issue you posted for clinton as well."
we aren’t?? I’m confused. I thought we had to edit that autoprocesstv.py file?

I’ll clean up the instructions for sure and start a new thread. Gimme a day or so.

I didn’t mess with any of the NZB settings either. Only using this as an extractor tool. This is pretty simple 10-minute setup that’s a very good resolution for b***h of a problem.

I’m glad we got it done . Thanks for your patience and willingness to help us out!


#27

re: code… I meant you accidentally copy-pasted the unedited version. it’s missing the ‘importMode’ part.

And thank you for sticking with it, I know it’s not particularly easy (without built in unrar support).


#28

Ahh, sorry, long night…too much jameson. Corrected.

No problem man, Appreciate you .

I’ll write this up in the next 24 hours and blast it out.

Feel free to correct it if it’s lacking something somewhere.


#29

Draft is here:


#30

If you say, “but bro, just download a Linux docker and you’ll be golden bro.” I’m going to find you and poop in your beer.

It wouldn’t be better/easier anyway. How about a Windows Container? ** ducks **

Pretty thorough document Sean, I like how you’re clear about which directories are involved and their purpose.

I wish I could do something about that dummy Remote Path Mapping coz it’s a bit of a hack, I had an idea earlier to do that fairly easily but that would apply to both usenet and torrent clients and we don’t want that.


#31

ha, RIGHT?!

I mean, I intended this guide to cover the most complicated setup and then be backward compitible. If you don’t use usenet, simply remove it from this guide. Ya know?

I should be done in a couple hours.


#32

I have updated these changes in NTM nightly branch.

Thanks for all the testing.

I now see the remote path settings relate to Sonarr and not NTM… so that clarifies that one.
I still think you should be able to use hard-link in NTM to save processing time and disk usage for non-rared archives… Not sure what this is falling back to copy.


#34

Hey guys. I’m trying to following the methods that @seanvree laid out in his guide.
I’m planning on posting some comments in that thread as well because for all the work he put into it there are some issues with clarity…but I’ll cover that later.

The main thing I want to check on here is this whole importmode:move that is stated must be updated in the NTM source. If I’m understanding this all correctly this is only needed when using a separate system as a seedbox, correct?

Without the “move”, then the call NTM makes to sonarr tells it just to hardlink…but it can’t because it is on a different drive/system, so it defaults to copying. AND, the reason we don’t want that is because it leaves us with two copies of the same file (one on the seedbox and one on the media box). We are ok with two copies if the source is archived because we obviously need an archived one to seed (at least temporarily) and a non archived one to watch.

By setting the importMode to “move”, NTM tells sonarr not to try to hardlink(copy), but just to move all the files. Is this accurate?

What I am still confused about on the single box solution here (which seems to be working for me right now without the importMode changes to the NTM code) is once NTM sends the DownloadedEpisodesScan to sonarr, how/why is Sonarr moving from my temp/“drone” directory to the proper season folder? I see something like this in my logs"

17-8-30 22:06:04.5|Debug|EpisodeFileMovingService|Moving episode file: D:\queue\pickup\tv\xxxx.S02E01.avi\xxxx.avi to E:\Data\Media\Video\TV\xxxx\Season 02\xxxx - 2x01 - episode.avi
17-8-30 22:06:04.5|Debug|DiskTransferService|Move [D:\queue\pickup\tv\xxxx.S02E01.avi\xxxx.S02E01.avi] > [E:\Data\Media\Video\TV\xxxx\Season 02\xxxx - 2x01 - episode.avi]

How is sonarr making the decision to move instead of attempting a hardlink or copy? In the hidden Media Management settings I see “Use Hardlinks instead of Copy” (which I have enabled), so I would assume it would either link or copy…not move. Now don’t get me wrong…Move is my desired behavior here because otherwise I would have a link (or worse a copy) in my temp/“drone” folder… I just don’t understand how it makes that call.

thanks

@Taloth - on a related note you might want to consider putting in a configurable switch in settings for DownloadedEpisodesScan to always do one or the other regardless of what is sent in the DownloadedEpisodesScan command. You could also add an extra option to DownloadedEpisodesScan command to override that configurable setting. This way the user can control what behavior they want without having to modify code. AND, if a dev wants they can override that behavior because they know 100% they need to. You may have a better idea in mind, but thought I would mention it.

Me again…wanted to get some more clarification on these things, and why they are needed particularly (I’ve read all the referenced threads).

The issue here is really only a problem if you still want to seed a torrent after completion, correct? I realize that a lot of the hoops to jump through here are precisely because we want to continue to do that and not have any file locks while we do. But, let’s assume I’m not continuing to seed after completion (e.g. you pay $$ for private trackers because you don’t/can’t use the additional bandwidth to seed). Moralizing on the no-seeding aside, this remote file hack wouldn’t be necessary because you could either:

  1. Move the files with the NTM script so they are no longer in the seed directory.
  2. As soon as sonarr imports the file it can reach out to the client and remove the download (if a seed limit of 0 has been reached and the torrent is auto managed, etc).

Am I understanding this correctly?

Assuming we don’t use the remote path hack, what exactly is the danger of Sonarr trying to import the file twice?


#35

hey @bengalih I replied to the main thread:

https://forums.sonarr.tv/t/unrar-win-new-guide-to-auto-process-rar-torrents-via-sonarr-w-o-drone-factory-edit-01-sep-17


#36

thanks sean…saw your post there although it doesn’t actually answer all my questions.
I’m going to assume that all my assumptions above were correct (if not please correct).

What I still need answered specifically:

and…

thanks guys