[UNRAR] [WIN]: **NEW** Guide to auto-process .rar torrents via Sonarr w/o Drone Factory [edit: 01 SEP 17]


#21

So just to clarify some points/responses:

  1. The reason I am making the distinction is because as you state there is no built-in solution for archives. So everyone who uses torrents (or maybe even nzbs without a good post-processor) is going to need something to do the job. I came across your other thread first in searching for good solutions and then this one which led me to NTM - which I never had heard of before. I think it is a good solution because it isn’t another process running (like ExtractNow) and does all the processing and notifications back to sonarr. It also handles the failed downloads and monitoring/link of the downloads in sonarr, which not every other solution (like a simple drone folder) can do. So, in short the NTM solution is a good one but this guide is overly complicated if you just want to do that. For instance you don’t need the empty remote folder, you don’t need to make the changes to the NTM source code, etc if you don’t have all the issues around seeding and/or seedbox/remote share. If a built-in autoextract feature was coming out soon then you may be right and only people with your specific challenges will need this (although NTM still might be more flexible). However just in the meantime my point was that it took me a while to figure out exactly what/why you were doing certain things which made the guide overly complex for what I needed. Really, in the end the best thing I got out of it was learning about NTM.

  2. My main issue here (and maybe others…) is that I just wasn’t familiar with NTM at all. So it took me quite some time reading up on that to just understand how it is solving the problem that you are using it for. So with a big config file it is hard to pick out just those options that are important and need changing. For example you set “force_clean = 1”… but I don’t know why that would be because it wasn’t the default. Once you have more details on how NTM actually processes you can make this decision. In short your guide may be able to get followed in 15 minutes if you don’t care to know how it works AND you don’t run into a single issue. Otherwise, this was a much bigger time investment even for people familiar with these topics.

  3. We can research this more because somewhere in the NTM docs/wiki it says it is optional. What it supposedly does is if safe_mode=1 is not process anything in that directory. I don’t think it has anything to do with re-extraction, it simply prevents extraction/processing at all. And when we call the script normally it passes a specific torrent directory/file so it never even sees/knows about any other folder. If you run it manually without passing all the correct arguments then yes, it does seem to process everything in the folder you run it against. However you likely will be running it against your xxx\tv folder and never against your xxx folder root. That being said you setting it isn’t a bad thing per-se…just I think probably unnecessary which brings me back to all the points I mention in #5.

  4. Or just move it…which is what I’m doing.

  5. If you are just using your torrent client for radarr/sonarr then yes you would have no need. But, if you use your torrent client for anything else it will also trigger the execute script on any torrent you finish. You can change my example by using OR statements so that it covers both your tv and movies folders and anything else you torrent won’t execute the NTM script.
    One other correction on this topic. You state:

I think you meant Auto-Execute (or simply Execute) plugin here.

As far as me getting this working. I’m not a heavy user and I won’t know 100% until I actually get some files that are archived to know 100%. But I have done extensive testing including creating my own test torrents and loading them up into the appropriate labels manually and NTM processes them all and sonarr picks them up. The only thing not tested really is the monitoring from sonarr to deluge (i.e. activity), but I expect that to work as normal. I also have NTM setup to do my subtitle downloads. As I was setting NTM up I was planning on looking into how I could have it call another script to run sublminal when I realized it had it built-in!

Once again, my single box scenario isn’t as complex as yours and only time will tell if I experience file locks, etc. I like the idea of the NTM script and I am upgrading my SabNZB finally and am going to switch to that from the sickbeard script I think. I also plan to try and use NTM for processing downloads from other clients like maybe IRC and/or jdownloader2… so I like the idea of a single post-processing script that can work with anything as long as you can feed it the parameters it needs. I’m 90% sure at this point that the TorrentToMedia.py script doesn’t actually have any reliance on any torrent client…it will still do all the post-processing and simply ignore the pieces that have to do with torrents (pause/resume/etc).

I’ll be sure to chime back in with any issues or if I discover something worthwhile.

thanks


#22

So I’m sorry to say things aren’t working very well. I’m hoping you or @Taloth may be able to shed some light.

I was seeing success when I was running the TorrentToMedia.py script manually, like this:


TorrentToMedia.py 234004f6071ce63e5e912201d9a999885d59103a My.Show.S01E19.HDTV.x264 D:\queue\processing\tv\torrent

This cause the script to look for the directory D:\queue\processing\tv\torrent\My.Show.S01E19.HDTV.x264 and copy/extract it to the defined output directory which I have defined as D:\queue\pickup\tv.

This worked, and after extraction (or just copy in the case of no rar) it sent the API call to Sonarr. Sonarr then did 3 things:

  1. Imported from the D:\queue\pickup\tv
  2. Removed the files from D:\queue\pickup\tv (because #1 was successful)
  3. Removed the download from deluge, which also served to delete D:\queue\processing\tv\torrent\My.Show.S01E19.HDTV.x264

So the net effect was file got into my media folder through Sonarr and all other copies were deleted. Perfect!

However, when running an actual download through Sonarr, the problem arises where Sonarr will actually scan the D:\queue\processing\tv\torrent\My.Show.S01E19.HDTV.x264, seeing how that is the information it has been passed to it by deluge.

In the case of a rar file this shouldn’t cause a problem (I don’t think) because it won’t find anything to import there and so the call from NTM should trigger the import from the pickup directory.

However, for a file that isn’t archived what happens is that even thought the file gets copied from D:\queue\processing\tv\torrent\My.Show.S01E19.HDTV.x264 to the pickup folder, Sonarr uses the deluge path and grabs the copy from the former. This leaves the latter directory there with an extra copy of the data.

At this point I thought that I probably needed to implement the remote path hack that @taloth mentioned. But, it doesn’t appear to work when everything is on the same box. To be clear, I added “10.10.10.102” and “d:\empty” to both path locations. But, the logs show it always still going out to pickup from the processing directory.

I think I can probably fudge this by setting force_clean = 1 in the NTM script. But this doesn’t seem like a graceful solution, and I’m not sure how safe it is either.

Pretty discouraged that there is no standardized way to do this and the only real option is to use the drone folder which we are discouraged from doing.

:sleeping:need sleep

UPDATE:

Ok, so I got this working again though I don’t know how ideal it is.
It turns out I have some of the same issues as you do with your data on a separate seed box because my download location and final media location are actually on separate drives on the same box - which means hard-linking won’t work.

What this means is that the intended pickup directory (what you called “drone”) will still keep a copy of the files in them unless you either edit the NTM code to add the "importMode": "Move" option or use force_clean = 1. The “Move” is probably safer, because what is unclear is this:

When NTM sends the process command to sonarr, how long does it wait? Does it actually wait for a successful response that Sonarr has copied/moved/linked/whatever the file? Or does NTM just wait a certain time after the process command is send. If the latter case, and NTM just continues on processing after so many seconds, then force_clean = 1 could be very dangerous as it might wipe out a file (or at least try and then fail…depending on locks) before Sonarr fully gets it.

So I’m a little suspicious of just using force_clean = 1 by itself. However I must use it because as I am on a single box, this hack with the remote path mapping doesn’t seem to work at all. This means that Sonarr will always scan the final torrent location (after move) and get the copy from there. Since it will have already processed the movie (or at least begun to) by the time NTM sends its notification, Sonarr will not then try to process it from the NTM pickup location. This means that copy of the data will simply remain unless you use force clean.

I’m considering changing my label to move the finished torrent directory to the same drive the final media (seasons) are on. I have a concern though that Sonarr may start scanning that location before the file finishes copying. My guess is that Sonarr doesn’t know that a “move completed to” is taking place when it checks to see if the torrent is complete…so it might start processing before a move to another drive has a chance to happen. I hope @taloth or maybe @markus101 can chime in on that front. I’m going to open another thread maybe on this question because I see another weird issue that might relate to a race condition here.

In the end, I think this will work…but it is kludgey and I’m not crazy about it. I’ll have to let it run for a few weeks and see how clean it is.


#23

I’m back from vacation, so a bit rusty.

You should at least use ImportMode=Move, yes. force_clean is on optional addition. NzbtoMedia polls the Sonarr Command api to find out if Sonarr finished processing but it probably won’t do that forever.
Remote Path mapping should work, even locally, but I’ll need more details about the path and settings you used to be able to tell where it’s wrong.

The torrent state is indeed a bit tricky when it comes to ‘move completed to’, torrent clients have different behavior and I don’t know which does what and whether they still have the ‘downloading’ state while moving.


#24

hey ive made another topic but i’ll try to write here, I have another windows setup that extracts files to an import folder but i dont know enough about all this coding to make a script that tells sonarr to import, can i just use the import bit from this and make it work with another setup i would like to just call it from a bat file

hope you can help


#25

HUGE UPDATE!

THis can now be done VERY easily with a BRAND NEW deluge plugin. See here:

( I will keep this topic open, but I will no longer be actively developing it)

Thanks all to helped especially @Taloth !!


#26

you can use this.


#27

Found an awesome script on reddit a few weeks back that seems to handle this along with radarr support. works for Qbittorrent and Utorrent