What is the real benefit/use case of CDH "remove"?

[bengalih]

So I opened a feature request here:

Allow CDH "Remove" on a per-download client basis (i.e.: don't remove my sabNZBd history)

So when a torrent is done downloading/seeding, most people remove it from their torrent client and it is end of story. With NZBs however I believe it is a different story. A client like sabNZBs shows a history of all processed downloads and this history is a good log to refer to (especially for issues that were encountered). When an NZB is completed (success or error) there is really no reason to remove it from the history in the client. You aren’t removing files, or stopping something from …

It questions the value of using CDH “remove” feature on an NZB client like SAB because it is valuable to have the info in the “history” log, which the “remove” feature strips out.

But, that got me thinking what exactly CDH “remove” benefits anyone? Even in a torrent client?

My understanding is that when your download completes and the torrent client sends the sonarr API call, sonarr will then initiate the CDH and copy/move/link your show to the proper season folder. If you have the “remove” option on, Sonarr will also issue an API call back to the torrent client to remove the item from your queue. However it will only do this if the following criteria are met:

1. The torrent is set to “Auto Managed” (through the label).
   AND
2. The torrent has met its seeding ratio.

If we make the assumption (as a good torrenting community should) that we are going to seed to a 1:1 ratio, then it is extremely unlikely that we will have met criteria #2 by the time Sonarr imports the file and evaluates/sends its remove request.

To the best of my knowledge, once Sonarr has imported the show it is no longer in the activity queue and therefore Sonarr does not make repeated attempts to remove it. Even if it did, it is likely that it takes days or weeks for an older torrent to reach a seeding ratio.

I think it is safe to say that the “remove” function will rarely ever remove anything from the torrent queue.

So - I have to think/ask…what is really the benefit/driving purpose of this feature? I feel like I must be missing something.

[markus101]

To the best of my knowledge, once Sonarr has imported the show it is no longer in the activity queue and therefore Sonarr does not make repeated attempts to remove it. Even if it did, it is likely that it takes days or weeks for an older torrent to reach a seeding ratio.

It’s not shown, but Sonarr still tracks it.

If we make the assumption (as a good torrenting community should) that we are going to seed to a 1:1 ratio, then it is extremely unlikely that we will have met criteria #2 by the time Sonarr imports the file and evaluates/sends its remove request.

Sonarr doesn’t import the second it’s complete, it could be minutes later and with a fast seed box that torrent could be removed (although I can’t see someone with that fast of a setup not seeding). There are also going to be cases where Sonarr can’t import things automatically (issues with the filename, quality, scene numbering, etc) and you wouldn’t want your torrent client nuking it before Sonarr processes it.

[bengalih]

It’s not shown, but Sonarr still tracks it.

So you are saying that even after it is gone from activity, Sonarr is still reaching out on a (what is the?) interval to check to see if seeding ratio is reached and will remove it from the client? Even if it is weeks later it should get removed?

Sonarr doesn’t import the second it’s complete

No, but in my testing so far it generally takes ~1 minute. I would think that was only negatively influenced the the sheer load a particular client might be under.

There are also going to be cases where Sonarr can’t import things automatically (issues with the filename, quality, scene numbering, etc) and you wouldn’t want your torrent client nuking it before Sonarr processes it.

Ofc… but what is this statement apropos to? If it can’t be imported then sonarr would never try to nuke it from the client…so I’m missing the point of this statement.

thanks

[markus101]

So you are saying that even after it is gone from activity, Sonarr is still reaching out on a (what is the?) interval to check to see if seeding ratio is reached and will remove it from the client? Even if it is weeks later it should get removed?

Yes, it still comes back from the download client (until it’s removed from the client) and Sonarr processes it like anything else, just filtered for the benefit of the queue.

No, but in my testing so far it generally takes ~1 minute. I would think that was only negatively influenced the the sheer load a particular client might be under.

Yeah, it’s about 60-90 seconds depending on the job scheduler (assuming other things aren’t taking it’s time), searches, RSS sync, previous imports (if importing across a slow WAN from a remote downloader).

Ofc… but what is this statement apropos to? If it can’t be imported then sonarr would never try to nuke it from the client…so I’m missing the point of this statement.

Perhaps I misunderstood part of your comment, but I originally read it that you were advocating using the torrent client’s auto-remove options when the seeding goal is met “The torrent is set to “Auto Managed” (through the label).” but re-reading it reads more that you’re referring to how it works for CDH remove to operate, so disregard.

[bengalih]

Yes, it still comes back from the download client (until it’s removed from the client) and Sonarr processes it like anything else, just filtered for the benefit of the queue.

Sorry to beat this horse…I just want total clarity on this. You say here “it still comes back from the download client…”. What still comes back? My understanding is that the client (e.g. deluge) is never reaching out to Sonarr…only responding. So it would be Sonarr’s job to keep asking deluge what the seed status is and if it is met (even weeks later), Sonarr will sent an API call to remove it from the client’s queue. Is this accurate?

Perhaps I misunderstood part of your comment, but I originally read it that you were advocating using the torrent client’s auto-remove options when the seeding goal is met “The torrent is set to “Auto Managed” (through the label).” but re-reading it reads more that you’re referring to how it works for CDH remove to operate, so disregard.

Gotcha. Right, well it needs to be “Auto Managed” as you know for “remove” to work. Further more AFAIK deluge (and maybe other clients) don’t have a built in option to automatically remove torrent with data. There are plugins that could do this, but I think the idea is that absent some other program that guarantees it took the data (like Sonarr), no one wants to tell their torrent client to delete their data automatically

So, to go back to the original question. Is it fair to say that CDH “remove” was mostly meant for torrents? It seems that (in my use scenario) the NZB data gets cleaned up even without this on. Since the nzb downloader deletes the archives after extraction and Sonarr moves the files over and cleans up the empty directory…there is really no need for this option on NZB clients unless you want to actually remove the history entry from the client. Note I use SAB, so i don’t know if Get has some other interesting integration here.

And, if that is fair/accurate to say then the main reason to use “remove” for torrents is to not just remove it from the queue, but to actually delete the data once we are sure it has been imported into your media folders?

[markus101]

Sorry to beat this horse…I just want total clarity on this. You say here “it still comes back from the download client…”. What still comes back? My understanding is that the client (e.g. deluge) is never reaching out to Sonarr…only responding. So it would be Sonarr’s job to keep asking deluge what the seed status is and if it is met (even weeks later), Sonarr will sent an API call to remove it from the client’s queue. Is this accurate?

Yes, that’s correct, Sonarr contacts the client, not the other way around. “what it gets back” is the list of items; the response from the download client after Sonarr’s request.

So, to go back to the original question. Is it fair to say that CDH “remove” was mostly meant for torrents?

Yeah, I’d guess that most user’s with it enabled want it for that purpose. With some wanting it to clear out the history in the client (NZBGet has an active history and a “deleted” history of sorts).

And, if that is fair/accurate to say then the main reason to use “remove” for torrents is to not just remove it from the queue, but to actually delete the data once we are sure it has been imported into your media folders?

Yeah, recover the diskspace if copy is used instead of hardlinking or multiple qualities were downloaded and you don’t need every copy still stored.

[bengalih]

Yes, that’s correct, Sonarr contacts the client, not the other way around. “what it gets back” is the list of items; the response from the download client after Sonarr’s request.

Thanks. One more related question on this.

When Sonarr sees the download is completed and reaches out to the folder the job has completed in. What happens if it doesn’t find a valid movie file? I ask specifically in the case of archives.
If a task is set to unextract the archives to the same folder, what happens if:

1. The file is still extracting when Sonarr sees the folder. Won’t it try to grab it and fail because it isn’t fully extracted yet?
2. The file isn’t extracted at all yet. Will Sonarr continue to reach out to that folder until it sees a valid video file? Or will it just give up since there was nothing there when it was told the d/l was complete?

Also - I though that one big reason for removing the drone folder was the complaint that it scanned the disc frequently. If indeed Sonnar does all this “continuing calls” to check for files, etc…it would seem that the underlying idea is the same (assuming it does).

Thanks.

[Thirrian]

I though that one big reason for removing the drone folder was the complaint that it scanned the disc frequently.

There’s plenty of other reasons.

Why the Drone Factory is bad or just plain evil • r/sonarr

[markus101]

What happens if it doesn’t find a valid movie file?

Sonarr tracks that it didn’t see a file and doesn’t import anything.

The file is still extracting when Sonarr sees the folder. Won’t it try to grab it and fail because it isn’t fully extracted yet?

Depends, it may import a partial file, unless the file is locked which prevents that.

The file isn’t extracted at all yet. Will Sonarr continue to reach out to that folder until it sees a valid video file? Or will it just give up since there was nothing there when it was told the d/l was complete?

It’ll try again and again and again.

I though that one big reason for removing the drone folder was the complaint that it scanned the disc frequently. If indeed Sonnar does all this “continuing calls” to check for files, etc…it would seem that the underlying idea is the same (assuming it does).

Except that Drone Factory blindly checks the disk, CDH only does so when the client says it’s complete. For usenet clients this is great because they handle unraring and don’t tell Sonarr it’s complete until after it’s unrared. For torrent clients it’s somewhat the same issue, since it could find a partially extracted file to import since the torrent clients don’t support unrarring as part of their process so complete = download complete. Sonarr would need to handle unraring to improve that process, or clients would need to improve.

[bengalih]

I though that one big reason for removing the drone folder was the complaint that it scanned the disc frequently.

There’s plenty of other reasons.

Yes I’ve re-read that again and it seems every single issue listed is basically “we don’t know the state of the files and so might import too early.” Every single one of those is resolvable by whatever script you are running placing the files in an alternate directory until they are done processing/syncing/whatever. Then, at the end of whatever script you move all the files into the drone folder and they are ready to go.

Yes, the non-drone approach is a bit more elegant in some respects as it is awaiting a “push” to tell it to jump into action instead of polling a pull all the time - but many of the same concerns still exist with it - especially with torrent client. Their is still an outstanding issue of Sonarr pulling from a completed torrent when the torrent client (or a script called from it) still needs to process something on the download.

[Thirrian]

[Taloth]

Yes I’ve re-read that again and it seems every single issue listed is basically “we don’t know the state of the files and so might import too early.” Every single one of those is resolvable by whatever script you are running placing the files in an alternate directory until they are done processing/syncing/whatever. Then, at the end of whatever script you move all the files into the drone folder and they are ready to go.

And yet we encountered countless users complaining about partial import issues, exactly because they didn’t know that you have to extract to an alternate directory. In that sense the Drone Factory was too convenient because it doesn’t force people to realise that their workflow is problematic. And that script that deposits it in the Drone Factory folder can just as easily call the Sonarr api and do it properly.

Torrent clients are indeed an issue because they do not have a post-processing/extracting state. In contrast to nzb clients, since those do have an extracting & pp state which Sonarr waits for to complete before it starts doing it’s thing.
We do intent to make the torrent client workflow a bit more reliable, perhaps in the form of pre-processing scripts managed by Sonarr rather than the client (because they can’t do it themselves), but we’re not sure yet if that’s the right approach for that particular issue.

Your original question was with regards to the ‘Remove’ function. For nzbget removed items become ‘hidden’ and still have partial history available, sabnzbd doesn’t have that. But Remove is important because Sonarr queries the client api to get a list of history items in order to determine what’s ready for import. Having many items in history makes this process slower. For sab that hardly matters, coz we simply limit the number of items sab is allowed to return, but nzbget returns the full list and becomes unacceptably slow after a while.