Including custom formats as part of filename format

rhom · March 17, 2024, 11:49am

Sonarr version (exact version): 4.0.1.929
OS: docker / linuxserver.io

ive got a custom format that contains multiple streaming service names in it that i prefer (scored at 2).

my original filename format didnt include these so when the filename on disk was scored against what was found by the indexer it scored less so the same file was grabbed endlessly (via usenet)

to fix that i wanted to put the custom format into the file name format (like i had previously done with release group) but when i do this its adding the name of the custom format into the filename and not the value from the grabbed file that matched whats in the custom format.

to make it work i had to put the name of the custom format into the custom format itself so it would match on that - but that sort of makes my file names look stupid (and technically im losing information should i ever want to remove one of them).

is there a way to do it properly so the original value gets added and not the custom formats name? - and why doesnt sonarr remember the score of the previous download instead of rescoring based off the current filename every time?

rhom · March 19, 2024, 10:49am

seems like theres also a bug with usenet downloads not adding the custom formats tags into the filename.

after setting up the custom format properly and including it in the filename its only getting added when downloading a torrent. when downloading via usenet its not added so it keeps downloading it over and over until a torrrent comes along.

you can see the score is always a +2 so it shouldnt be an upgrade - but it is because the filename isnt getting the custom format added when downloaded via usenet so is scored as 0, its not until the torrent is downloaded that the custom format is finally included in the filename

markus101 · March 19, 2024, 2:25pm

Custom Formats always uses the Custom Format name, you can’t have it use the value.

Without logs from the import it’s impossible to know why it’s not seeing the imported file as containing that custom format (and we also don’t know what that CF is actually checking for either).

rhom · March 20, 2024, 8:57am

the issue its that when downloaded via usenet the custom format is not included in the filename sonarr creates in the first place.

you can see from the screenshots that the filename (that gets deleted) ends in webdl-ntb for all the usenet downloads but ends in webdl.strm-ntb for the torrent - even though you can also see the custom format “strm” was found on each of the usenet downloads it doesnt end up in the final filename.

regarding the custom format, its included below. the final entry for “strm” (the custom formats name) is included so that it will score the on disk filename the same way as it would the original values that are not available there.

{
  "name": "strm",
  "includeCustomFormatWhenRenaming": true,
  "specifications": [
    {
      "name": "Amazon",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": false,
      "fields": {
        "value": "\\b(amzn|amazon)\\b"
      }
    },
    {
      "name": "Apple",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": false,
      "fields": {
        "value": "\\b(atvp|aptv|Apple TV\\+)\\b"
      }
    },
    {
      "name": "HBO",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": false,
      "fields": {
        "value": "\\b(hbo|hmax|hbom|hbo[ ._-]max)\\b"
      }
    },
    {
      "name": "Disney",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": false,
      "fields": {
        "value": "\\b(dsnp|dsny|disney|Disney\\+)\\b"
      }
    },
    {
      "name": "Hulu",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": false,
      "fields": {
        "value": "\\b(hulu)\\b"
      }
    },
    {
      "name": "Netflix",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": false,
      "fields": {
        "value": "\\b(nf|netflix)\\b"
      }
    },
    {
      "name": "Paramount",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": false,
      "fields": {
        "value": "\\b(pmtp|Paramount\\\\+)\\b"
      }
    },
    {
      "name": "Peacock",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": false,
      "fields": {
        "value": "\\b(pcok|Peacock TV)\\b"
      }
    },
    {
      "name": "ShowTime",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": false,
      "fields": {
        "value": "\\b(sho|showtime)\\b"
      }
    },
    {
      "name": "Stan",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": false,
      "fields": {
        "value": "\\b(stan)\\b"
      }
    },
    {
      "name": "custom format name",
      "implementation": "ReleaseTitleSpecification",
      "negate": false,
      "required": false,
      "fields": {
        "value": "\\b(strm)\\b"
      }
    }
  ]
}

standard episode format being used is {series.cleantitle}.s{season:00}e{episode:00}{.mediainfo.videocodec}{.quality.title}{.custom formats}{-release group}

small chunk of logs to show the naming issue when downloaded via usenet

3rd line shows it seeing the strm format as an upgrade
last lines showing the resulting filename does not contain the strm custom format in it

2024-03-20 19:40:53.2|Debug|VideoFileInfoReader|Getting media info from /media/downloads/sabnzbd/ready/sonarr/Palm_Royale_S01E01_Pilot_720p_ATVP_WEB-DL_DDP5_1_H_264-NTb/Palm.Royale.S01E01.Pilot.720p.ATVP.WEB-DL.DDP5.1.H.264-NTb/Palm.Royale.S01E01.Pilot.720p.ATVP.WEB-DL.DDP5.1.H.264-NTb.mkv
2024-03-20 19:40:53.3|Debug|DetectSample|[/media/downloads/sabnzbd/ready/sonarr/Palm_Royale_S01E01_Pilot_720p_ATVP_WEB-DL_DDP5_1_H_264-NTb/Palm.Royale.S01E01.Pilot.720p.ATVP.WEB-DL.DDP5.1.H.264-NTb/Palm.Royale.S01E01.Pilot.720p.ATVP.WEB-DL.DDP5.1.H.264-NTb.mkv] does not appear to be a sample. Runtime 00:49:56.5770000 seconds is more than minimum of 300 seconds
2024-03-20 19:40:53.3|Debug|UpgradeSpecification|New item's custom formats [strm] (2) improve on [] (0), accepting
2024-03-20 19:40:53.3|Debug|ImportDecisionMaker|File accepted
2024-03-20 19:40:53.3|Debug|UpgradeMediaFileService|Removing existing episode file: [378373] season 01/palm.royale.s01e01.h264.720p.webdl-ntb.mkv
2024-03-20 19:40:53.3|Debug|RecycleBinProvider|Attempting to send '/media/tv/palm royale (2024)/season 01/palm.royale.s01e01.h264.720p.webdl-ntb.mkv' to recycling bin
2024-03-20 19:40:53.3|Debug|RecycleBinProvider|Creating folder /media/sonarr-trash/palm royale (2024)/season 01
2024-03-20 19:40:53.3|Debug|RecycleBinProvider|Moving '/media/tv/palm royale (2024)/season 01/palm.royale.s01e01.h264.720p.webdl-ntb.mkv' to '/media/sonarr-trash/palm royale (2024)/season 01/palm.royale.s01e01.h264.720p.webdl-ntb_3.mkv'
2024-03-20 19:40:53.3|Debug|DiskTransferService|Move [/media/tv/palm royale (2024)/season 01/palm.royale.s01e01.h264.720p.webdl-ntb.mkv] > [/media/sonarr-trash/palm royale (2024)/season 01/palm.royale.s01e01.h264.720p.webdl-ntb_3.mkv]
2024-03-20 19:40:53.3|Debug|RecycleBinProvider|File has been moved to the recycling bin: /media/sonarr-trash/palm royale (2024)/season 01/palm.royale.s01e01.h264.720p.webdl-ntb_3.mkv
2024-03-20 19:40:53.3|Debug|EpisodeService|Detaching episode 94656 from file.
2024-03-20 19:40:53.6|Debug|SubtitleFileService|Deleting Extra from database for episode file: [378373] season 01/palm.royale.s01e01.h264.720p.webdl-ntb.mkv
2024-03-20 19:40:53.6|Debug|OtherExtraFileService|Deleting Extra from database for episode file: [378373] season 01/palm.royale.s01e01.h264.720p.webdl-ntb.mkv
2024-03-20 19:40:53.6|Debug|MetadataFileService|Deleting Extra from database for episode file: [378373] season 01/palm.royale.s01e01.h264.720p.webdl-ntb.mkv
2024-03-20 19:40:53.6|Debug|EpisodeFileMovingService|Moving episode file: /media/downloads/sabnzbd/ready/sonarr/Palm_Royale_S01E01_Pilot_720p_ATVP_WEB-DL_DDP5_1_H_264-NTb/Palm.Royale.S01E01.Pilot.720p.ATVP.WEB-DL.DDP5.1.H.264-NTb/Palm.Royale.S01E01.Pilot.720p.ATVP.WEB-DL.DDP5.1.H.264-NTb.mkv to /media/tv/palm royale (2024)/season 01/palm.royale.s01e01.h264.720p.webdl-ntb.mkv
2024-03-20 19:40:53.6|Debug|DiskTransferService|Move [/media/downloads/sabnzbd/ready/sonarr/Palm_Royale_S01E01_Pilot_720p_ATVP_WEB-DL_DDP5_1_H_264-NTb/Palm.Royale.S01E01.Pilot.720p.ATVP.WEB-DL.DDP5.1.H.264-NTb/Palm.Royale.S01E01.Pilot.720p.ATVP.WEB-DL.DDP5.1.H.264-NTb.mkv] > [/media/tv/palm royale (2024)/season 01/palm.royale.s01e01.h264.720p.webdl-ntb.mkv]

markus101 · March 21, 2024, 3:43am

For one of the torrent imports that did add the CF properly, please post the source and destination paths from history.

rhom · March 21, 2024, 4:59am

torrent

Name
Below Deck S11E07 Cats Out of the Bag 720p AMZN WEB-DL DDP2 0 H 264-NTb[TGx]

Source
/media/downloads/transmission/complete/sonarr/Below Deck S11E07 Cats Out of the Bag 720p AMZN WEB-DL DDP2 0 H 264-NTb[TGx]/Below.Deck.S11E07.Cats.Out.of.the.Bag.720p.AMZN.WEB-DL.DDP2.0.H.264-NTb/Below.Deck.S11E07.Cats.Out.of.the.Bag.720p.AMZN.WEB-DL.DDP2.0.H.264-NTb.mkv

Imported To
/media/tv/below deck (2013)/season 11/below.deck.s11e07.h264.720p.webdl.strm-ntb.mkv

Custom Format Score
+2

usenet

Name
Below_Deck_S11E07_Cats_Out_of_the_Bag_720p_AMZN_WEB-DL_DDP2_0_H_264-NTb

Source
/media/downloads/sabnzbd/ready/sonarr/Below_Deck_S11E07_Cats_Out_of_the_Bag_720p_AMZN_WEB-DL_DDP2_0_H_264-NTb/Below.Deck.S11E07.Cats.Out.of.the.Bag.720p.AMZN.WEB-DL.DDP2.0.H.264-NTb/Below.Deck.S11E07.Cats.Out.of.the.Bag.720p.AMZN.WEB-DL.DDP2.0.H.264-NTb.mkv

Imported To
/media/tv/below deck (2013)/season 11/below.deck.s11e07.h264.720p.webdl-ntb.mkv

Custom Format Score
+2

markus101 · March 23, 2024, 4:45am

I haven’t been able to pinpoint exactly what might be causing this, trace logs of an usenet import would be helpful.

rhom · March 23, 2024, 8:00am

trace logs for a manual search and grab for below.deck.s11e07.h264.720p.webdl-ntb.mkv

https://pub.microbin.eu/upload/ape-raven-pony

markus101 · March 24, 2024, 1:57am

Thanks, that helps.

2024-03-23 18:52:47.7|Trace|CustomFormatCalculationService|Using scene name for release title: Below_Deck_S11E07_Cats_Out_of_the_Bag_720p_AMZN_WEB-DL_DDP2_0_H_264-NTb

\b in \b(amzn|amazon)\b doesn’t match _ because it’s not treated as a word boundary character, unlike . or space.

Should be able to fix this, but there could be some edge cases with that value being stored and the regex being used.

rhom · March 24, 2024, 7:59am

the underscore not matching \b makes sense but where does scene name come from that its checking against? i cant seem to find it anywhere in what was downloaded.

would it be simpler to just change the regex to (\b|_)(amzn|amazon)(\b|_)

edit - fixed the regex as [\b_] doesnt work but (\b|_) does

markus101 · March 25, 2024, 3:45am

It’s from the download client’s download name.

rhom · March 30, 2024, 12:37am

ok, so for an auto import it uses the download name so long as its a single episode. i presume (partial) season packs default to using the filename?

manual imports always use the filename?

what about the potential for loss or alteration of data between the indexer → download → file name? (eg season packs but also download client (eg sab) replacing spaces with underscores is another)

would it be possible for sonarr to test the formats against both (or all three?) names (release title) instead of just the one?

ie for each format test it against the download name and if it matches return true, else test it against the file name and return true it it matches, else test it against the indexer name and return true if it matches, else return false?

repeat the test against however many name sources there are available, in whatever order is more relevant.

for manual imports parse the parent folder name, if its in a valid format sonarr likes then include that as a name source to test against as well - most manual imports still have the job name as the parent folder so its worth checking.

markus101 · April 1, 2024, 11:36pm

Imports with a known item from the download client prefer the name from the download client, unless it’s a pack release.

A manual import could use the folder name, but depends on the naming/how many files there are.

That’s one reason for preferring the name in the download client, but different indexers can vary wildly.

Possibly, but that doesn’t necessarily make things easier to make a decision on which one to use and combining them from different sources is going to be messy if they report different things.

rhom · April 2, 2024, 1:02pm

i was thinking more for just the release title matching - as far as im aware any release title custom formats dont match against actual file properties so mismatches at that level shouldnt matter?

is “release title” the whole name? i expected the “known” bits (series, release group, quality, etc) to have been removed from the name to generate the release title but i cant find its actual definition anywhere so i could be very wrong on that.

trying to work out what to do automatically when theres a mismatch is going to cause someone grief no matter what you go with. it probably safer to just shift it to “manual intervention required” and let the user sort it out themselves.

markus101 · April 4, 2024, 12:16am

Yes, it uses the whole name, not likely an issue in most cases except when common terms are used to match.

rhom · April 7, 2024, 10:40am

are there any plans to fix the issue with season packs? the filenames dont always match the grab or download torrent name so lose the data that was in those.

worse, somehow it kept the release group even though its not part of the filename at all - so obviously some data it matched/retained off either the grab or download name was used as part of the final filename - thats not meant to happen, but if it can then it begs the question why other data isnt included from there as well?.

i mean if its using the whole name then wheres the harm in adding the two values, download client name and filename, together and then parsing that? at least that way we’d never lose data and i cant see a downside.

i dont have logs for this just the episode history, but that should be enough to show the issue. ie release group is not in the downloaded filename from the season pack, yet ended up in the filename sonarr renamed it to.

eg. got pack (because it was a season pack) but didnt get strm (NF wasnt in the filename)

grab

Name: Parasyte The Grey S01 720p x264 DUAL NF-BadRips
Indexer: TorrentDownloads
Release Group: BadRips
Custom Format Score: +33
Series Match Type: Title
Info URL: https://torrentdownloads.unblockit.africa/torrent/1704661000/Parasyte-The-Grey-S01-720p-x264-DUAL-NF-BadRips
Download Client: transmission
Grab ID: 8E537F7850F8AB48B9B25B22DA0A799867BDEE39
Age (when grabbed): 0 minutes
Published Date: 06/04/2024 08:42:43

import

Name: Parasyte.The.Grey.S01E06.720p.x264.DUAL
Source: /media/downloads/transmission/complete/sonarr/Parasyte.The.Grey.S01.720p.x264.DUAL.NF-BadRips/Parasyte.The.Grey.S01E06.720p.x264.DUAL.mkv
Imported To: /media/tv/parasyte the grey (2024)/season 01/parasyte.the.grey.s01e06.x264.720p.hdtv.pack-badrips.mkv
Custom Format Score: +32

this inconsistency causes download loop issues, again. and yes, it did end up grabbing the same torrent and sending it to the download client again (ive unmonitored the season to stop it).

note - considering sonarr keeps track of imported torrent hashes and wont import any of them again, could it check before sending it to the download client so that if the hash has already been imported it wont send it?

i know it wont know the hash until after the grab but once grabbed and it knows the hash so why send that to the download client knowing it wont ever get imported?