Multiple repeat downloads

MRobi · February 24, 2019, 10:09pm

Sonarr version (exact version): 3.0.1.378
Mono version (if Sonarr is not running on Windows): 5.10.1.57
OS: FreeNAS
Debug logs:
https://drive.google.com/file/d/1Z-sLHLC1cDt0Vg0bbEGZjjqifKyzUPCC/view?usp=sharing
Description of issue: I’m getting multiple copies of downloads that aren’t processing. I cleared out my downloads folder on Wednesday and as of writing this it has 104 unprocessed episodes sitting there. Many are multiple versions of the same episode.
For Example:
The Orville - twice
The Blacklist - 4 times
Hawaii Five-O - 5 times
The list goes on and on and on.

Following the download history of these items, to the best of my knowledge this seems to be related to slow torrent downloads and upgrades. It appears as if the torrent download starts (which are almost always slow to download), while it’s downloading it will find another version of the episode and start that one too, then it will find a version on a newsgroup service which downloads at lightning speed and completed it. By the time the multiple versions that are being downloaded by torrent are completed, there’s another version that’s already been processed so these just sit there clogging up the downloads folder.

Here’s an example. Episode downloaded and imported successfully. Then it started 2 upgraded versions on transmission, and 1 with sabnzbd. The sabnzbd episode get’s processed replacing the original download, but the other 2 episodes now just sit there completed in my download directory.

MRobi · February 25, 2019, 10:05pm

Log added and an example from today that can be seen in the log.
Capture

Original file downloaded and processed.
Upgraded file grabbed and sent to Transmission.
Upgraded file grabbed and sent to SabNZBD. This file is processed.
Another file grabbed and sent to SabNZBD.

I now have 2 unprocessed files sitting in my downloaded directory because they are not upgrades to the existing file.

markus101 · February 25, 2019, 10:24pm

Are you using preferred words?

MRobi · February 26, 2019, 2:55am

Yes I am using them

markus101 · February 26, 2019, 4:09am

Is something in The.Walking.Dead.S09E11.Bounty.720p.AMZN.WEB-DL.DD+5.1.H.264[eztv] preferred?

It looks like the other WEB release is grabbed, then a couple hours later Sonarr sees the AMZN WEB DL release and grabs it because it’s more preferred, but your logs don’t cover the time when it was grabbed so it’s just an educated guess.

MRobi · February 26, 2019, 1:26pm

Yes. DD+5.1 and DD5.1 are both preferred words with a weight of 20.
X265 is there with a weight of 10.

So I get why the X265 and the first AMZN releases were grabbed Still unsure why the 2nd AMZN release was grabbed as it’s basically the same file and this is the first time I’ve seen it like that.

The main issue I’m having is around the timing of the x265 download. Before that download completes, it finds another more preferred release and grabs that one. But then it does nothing with the x265 release. I would expect that when a more preferred release is found, as part of the post processing routine it would not only delete the previously processed file, but also have it cancel any other current downloads of lower preference that are running for that episode.

MRobi · February 26, 2019, 1:53pm

Here’s a few more that I’m looking at.

And here’s my preferred words
Capture3

markus101 · February 26, 2019, 7:59pm

No, Sonarr does not touch other queued downloads.

There is

This is a tricky problem though, because of seeding requirements for some trackers.

Looks like Sonarr is doing what it should be doing, but due to the slow downloads it’s abandoning some releases. Setting up a delay profile is probably your best path forward here.

MRobi · February 26, 2019, 11:01pm

I’ve already got a delay of 12 hours setup for torrents. And in the example of “Bull” above, you can see the original download was on Feb 18th, an upgrade on the 18th, then the 19th, 21st and finally 25th. Right now I have 3 extra copies of this episode sitting in my download directory. Unfortunately I don’t think a delay profile would help with this unless I’m setting a delay of a week on torrents which is just unreasonable.

Having to manually import or delete literally 100s of duplicate episodes per week completely takes out the automated aspect of using Sonarr. It’s as much work sorting through all of these as it is manually sourcing the files.

beardman · February 26, 2019, 11:45pm

How so? if you have the download client set up to remove the torrent once it has completed seeding to whatever ratio you have chosen, then it will be gone in a fairly reasonable timeframe.

Also, you might want to collapse these into regular expressions, otherwise you are going to end up with doubled scores or even tripled scores.

for instance a release with both HEVC and x265 in it will get +35.
a DD+5.1 will get +80 (20 each for DD,5.1, DD+, and DD+5.1)

MRobi · February 27, 2019, 12:40am

The issue isn’t with the torrent staying active in the download client. I have a script that handles that. The issue is the now lower quality release file is staying in my downloaded directory because while this file was downloading, Sonarr found a more preferred copy and processed it first.

Thanks for this tip. It’s got me looking and seeing regex is an option, so /\b([xh].?265|hevc)\b/i would cover HEVC, x265 and h265 and not give a double score.

markus101 · February 27, 2019, 12:52am

Did the upgrade from the 18th not finish until after the one from the 19th? Assuming the filename is the same as the release name grabbed it should have been imported as an upgrade.

I see the same thing on the 21st and then today, the one from the 21st was never imported, I assume it didn’t take 4-5 days to download and import, the question is why didn’t it import, which debug logs should have the answer to.

beardman · February 27, 2019, 1:01am

Right, but why is that an issue? I have 4 releases of the last episode of The Walking Dead sitting in my downloads directory. All are seeding. All will be deleted when they hit the correct ratio.

yep. That is exactly what I use

MRobi · February 27, 2019, 1:15am

It’s possible the episode from the 18th didn’t finish until after the one from the 19th. I wasn’t really watching it then. The one from the 21st was definitely finished long before the one today. So yes, that is a great question. I enabled the debug logs, but only did that yesterday. So I’ll keep my eyes open for another example moving forward.

I’ve redone my preferred terms. Stupid question here, do I need the / before the expression for sonarr to recognize it as a regex expression? And do I need to add a /i for case insensitive?

markus101 · February 27, 2019, 1:16am

Yes and yes.

MRobi · March 2, 2019, 1:47pm

OK, using regex expressions in the preferred terms has drastically reduced the number of multiple downloads (only 23), but I’ve still got an issue going on here.

Both x265 files are in my downloads directory unprocessed. Both downloaded via transmission while the other files were all downloaded using sabNZBD. Looking at all the other files in my downloads directory, every one of them were downloaded via transmission. My issue could be that torrent files aren’t being processed.

I’m looking at the debug log files, but they don’t go back to yesterday. I have 49 debug log files with the oldest one being from 9 hours ago. Is there a way to extend the debug logs?

MRobi · March 2, 2019, 2:44pm

Found one that’s still in the logs.
I have both x265 episodes in my downloads directory unprocessed. 1 from yesterday. 1 from 1:16am this morning.

The only mention in the 49 log files is below:

19-3-2 01:16:10.6|Debug|DownloadDecisionMaker|Processing release 'How.to.Get.Away.with.Murder.S05E15.720p.WEB.x265-MiNX[eztv]' from 'ExtraTorrent'
19-3-2 01:16:10.6|Debug|Parser|Parsing string 'How.to.Get.Away.with.Murder.S05E15.720p.WEB.x265-MiNX[eztv]'
19-3-2 01:16:10.6|Debug|Parser|Episode Parsed. How to Get Away with Murder - S05E15 
19-3-2 01:16:10.6|Debug|Parser|Language parsed: English
19-3-2 01:16:10.6|Debug|QualityParser|Trying to parse quality for How.to.Get.Away.with.Murder.S05E15.720p.WEB.x265-MiNX[eztv]
19-3-2 01:16:10.6|Debug|Parser|Quality parsed: WEBDL-720p v1
19-3-2 01:16:10.6|Debug|Parser|Release Group parsed: MiNX
19-3-2 01:16:10.6|Debug|AlreadyImportedSpecification|Performing already imported check on report
19-3-2 01:16:10.6|Debug|UpgradeAllowedSpecification|Comparing file quality and language with report. Existing file is WEBDL-720p v1 - English
19-3-2 01:16:10.6|Debug|AcceptableSizeSpecification|Beginning size check for: How.to.Get.Away.with.Murder.S05E15.720p.WEB.x265-MiNX[eztv]
19-3-2 01:16:10.6|Debug|AcceptableSizeSpecification|Possible double episode, doubling allowed size.
19-3-2 01:16:10.6|Debug|AcceptableSizeSpecification|Item: How.to.Get.Away.with.Murder.S05E15.720p.WEB.x265-MiNX[eztv], meets size constraints.
19-3-2 01:16:10.6|Debug|CutoffSpecification|Comparing file quality and language with report. Existing file is WEBDL-720p v1 - English
19-3-2 01:16:10.6|Debug|MaximumSizeSpecification|Maximum size is not set.
19-3-2 01:16:10.6|Debug|LanguageSpecification|Checking if report meets language requirements. English
19-3-2 01:16:10.6|Debug|ReleaseRestrictionsSpecification|Checking if release meets restrictions: How.to.Get.Away.with.Murder.S05E15.720p.WEB.x265-MiNX[eztv]
19-3-2 01:16:10.6|Debug|ReleaseRestrictionsSpecification|[How.to.Get.Away.with.Murder.S05E15.720p.WEB.x265-MiNX[eztv]] No restrictions apply, allowing
19-3-2 01:16:10.6|Debug|QualityAllowedByProfileSpecification|Checking if report meets quality requirements. WEBDL-720p v1
19-3-2 01:16:10.6|Debug|MinimumAgeSpecification|Not checking minimum age requirement for non-usenet report
19-3-2 01:16:10.6|Debug|RetentionSpecification|Not checking retention requirement for non-usenet report
19-3-2 01:16:10.7|Debug|UpgradeDiskSpecification|Comparing file quality and language with report. Existing file is WEBDL-720p v1 - English
19-3-2 01:16:10.7|Debug|UpgradableSpecification|Existing item has better quality, skipping
19-3-2 01:16:10.7|Debug|HistorySpecification|Performing history status check on report
19-3-2 01:16:10.7|Debug|HistorySpecification|Checking current status of episode [51110] in history
19-3-2 01:16:10.7|Debug|UpgradableSpecification|Existing item meets cut-off. skipping.
19-3-2 01:16:10.7|Debug|UpgradableSpecification|Existing item has better quality, skipping
19-3-2 01:16:10.7|Debug|UpgradableSpecification|Existing item has a better preferred word score, skipping

It’s saying the existing item has better quality and a better preferred word score. However x265 is a preferred word with a weight of 20, x264 is not at all.

Here are my preferred words as they are currently set.

MRobi · March 2, 2019, 10:50pm

Another example. All upgrades downloaded via SabNZBD were processed. The latest upgrade downloaded via transmission was not. Although I have to wonder why Star.Trek.Discovery.S02E07.INTERNAL.HDR.1080p.WEB.H265-AMRAP was considered an upgrade. H265 scores 15. The file it replaced is Star.Trek.Discovery.S02E07.Light.and.Shadows.1080p.AMZN.WEB-DL.DD+5.1.H.264-AJP69 and DD+5.1 should be scored at 20.

Unfortunately the debug logs for this morning have been replaced already because the system only keeps a few hours worth of debug logs.

earthscab · March 3, 2019, 2:08pm

I actually have a case where it’s working correctly this morning. I put an 8 hour delay on all downloads from sabnzbd. When I first checked this morning, a download for SNL from the TBS group was pending download due to the delay. Now, an hour later a higher preferred word rated version, the MeGusta x265 version, is now pending download with no sign of the TBS version left over.

MRobi · March 4, 2019, 2:07pm

For me downloads and upgrades from sabnzbd are processing after completing. Although I do wonder how some files are being considered an upgrade when they’re replacing a file that should have a higher preferred word score. My issue seems to be any upgraded file that’s being downloaded through transmission, either not processing at all or no longer being seen as an upgrade even though it has a higher score.

This morning as a test I’ve disabled my script in transmission that clears completed torent files. My thought process is that maybe it’s clearing the torent before sonarr is signaled that the download is complete. I’m running out of ideas on other things to try.