Possible Synology indexer notification bug

Hi,

I’m not yet using the Synology Indexer notification because I’m running the Master branch, but I had a bit of a look at the code on github. If I’m reading it correctly there are a couple of things that may result in undesirable or unintended behavior.

1. The directories in the file path are never indexed or confirmed as having been indexed.

The Synology Indexer database stores an absolute path to each sub-directory in the file path in a “directory” table. As well as an absolute path to the video file itself in the “video” table. If any sub-directory in the path is not in the “directory” table, the file will be inaccessible.

Example Scenario: User adds a new show telling Sonarr to “Create empty series folders”. Sonarr creates the folders, adds the show, downloads the episodes and places them in appropriate folders. Synology indexer is notified to add each episode file to the database.

Expected Result: User can access all episodes from chosen media player/device.

Actual Result: Entire series remains unlisted on media server.

2. Renaming triggers a re-indexing of the entire series folder.

This I assume is the intended behavior. However it has some side effects that may be unintended:

  • If the series has many seasons and/or episodes this can take a long time even when only a small number of files have been renamed, as Synology Indexer will re-index all of them anyway.
  • All episodes are effectively removed and re-added, this causes ALL the episodes in the series to be unavailable during this time.

Re-indexing is generally very time consuming on a Synology NAS. As an alternative Synology Indexer provides renaming options in the form:

-N new_dirpath old_dirpath
    rename a dir
-n new_filepath old_filepath
    rename a file

Finally I have a handful of general notes about Synology Indexer quirks.

  • I noticed you have an “UpdateLibrary()” function that runs " /usr/syno/bin/synoindex -R video". Please don’t ever use this for anything. For anyone with any kind of substantial collection this will cause days if not weeks of indexer activity and general mayhem as their series and episodes pop back in.

  • The obvious fix for #1 above is to run your “AddFolder” function on every sub-directory in the file path every time you add a file. Doing this however causes another problem. That is, any time a new file is downloaded it will re-add all the sub-directories to the index causing anyone who is watching ANY file in those directories to be disconnected. (This is the same problem as re-indexing the entire directory.)

  • Finally, Synology Indexer used to have an option for checking if a file or directory has already been indexed using the “-g” flag. However in recent versions this has been removed. An alternative to this is to query the database directly, I have my own solution for this, as I’m sure others do. If this interests you at all just let me know and I’d be happy to send it your way.

Apologies for the long winded post. I hope all of the above was clear. If you require any more information or clarification on the above points please let me know. Also as I noted at the top, I haven’t actually used your implementation of this as I’m still on the Master branch, so if I’ve misread/misinterpreted or otherwise made an error in the above, I apologise.

If you got this far, thank you for reading.

Cheers,

Igor.

#1 sounds like a show stopper in some ways, if things are really this bad (with the Syno Indexer) I’m not sure its something we will be able to support properly.

Issue #2 is a more complex problem to solve because it requires a reworking of how we send the notification.

//cc @Taloth @NMe

It’s mostly just a bit limited, it’s not so bad in practice as long as you don’t re-index large sets of files all the time.

Having said that if you guys don’t have the time or resources to put extra effort into this, a great alternative is allowing us to fire off custom scripts after a download. This is the way I was handling it on my old SAB, Sickbeard setup, but back then SAB was handling the series/episode sorting and running the script too.

The synoindex database could be queried for #1 to see if the directory is already present or not, and add it if required?

Yep, that works fine but you must query for both the series folder and the season folder. Also you have to query the database directly as synoindex itself no longer provides an interface to do this, but this is essentially the same as it is still just a single command.

Something like:

/usr/syno/pgsql/bin/psql mediaserver admin -c "select count(path) from directory where path = 'dirPath';"

Then you parse the result, and repeat for each sub-directory.

Issue #1 is not an issue. I’ve just doublechecked and there are plenty of shows that didn’t have a directory the last time I actually looked at the SynoIndex which are included just fine right now. My episodes end up in /volume1/video/series/{genre}/{series title}/{season}/, so I think that fits your description. I don’t update the index on intervals, only Sonarr touches it, so whatever you thought you noticed in the code apparently doesn’t work like you feared.

I think #2 might be done that way intentionally because of multi-episode files.

@NMe Are your files being renamed at all? If so they might be triggering the series directory re-indexing which DOES add all sub-directories to the index. Alternatively SynoIndex may have fixed that in a recent build, I haven’t updated DSM in a month or two. That’s good news if it’s working though.

For #2, what do you mean because of multi-episode files? I don’t understand how this is related.

Couple of things:

  1. The UpdateLibrary method on the proxy is actually never used.

  2. When a new file is added (and renamed in the process) AfterRename is NOT triggered. It just calls the individual remove/add methods, so it’s lightweight.

  3. The AfterRename full scan of the Series directory is primarily because we don’t get the details of the rename in that part of the process, we just know that at least 1 file in that series was renamed.
    It’s possible to implement the individual renames, but the rename process is triggered manually so it didn’t seem like a problem to do it like this.
    In addition, if you change the naming scheme, and then trigger a rename. It’s likely all of your files will have to be renamed. So it’s actually more efficient to let synoindex run through the dir and update everything as opposed to calling synoindex 2000 times to remove the old ones and then add the new ones.

Although I’m open to improvements (of course) I would first like to know if it’s actually a problem. Atm it looks like both scenarios are handled properly.

I find the #1 scenario very very unlikely, so I tested it for ya on some old DSM 4.1 VM:

  1. Your root tv series folder is /volume1/test/TV
  2. Ensure this folder is in the synoindex config (Did that via configuration panel)
  3. Put some video file /volume1/test/TV/MySeries/Season1/MySeries.S01E01.mkv
  4. Invoke synoindex -a /volume1/test/TV/MySeries/Season1/MySeries.S01E01.mkv
  5. After a bit check Videostation, and voila there it is.

Only thing is that the root parent /volume1/test/TV needs to be listen in the indexer settings, something the user should already have done, and not something we’re going to override. Then regardless of subdirectory any file in there will be indexed.

@Taloth Thanks for taking the time to reply. I’ll reply point by point for clarity. Apologies in advance for the long post.

I figured that, I was just mentioning that it should probably stay unused forever. :smile:

As you say, if this is not being triggered for renames on import it likely won’t be a problem as it will rarely be used. So all seems well on that front.

My setup is a little different.
My root folder is /volume1/video/TV Shows.
But the folder in the synoindex config is /volume1/video, because I also have a /volume1/video/Movies.

So let’s take your example of a video file in /volume1/video/TV Shows/MySeries/Season1/MySeries.S01E01.mkv.
Let’s assume this series is new and we do not yet have directories for it.

As far as I know synoindex only monitors files automatically that have been transferred or otherwise manipulated by the Web GUI / Samba / AFP. As such I will be using the shell to make directories, copy files, etc. in order to simulate what Sonarr would be doing.

First we run a test to see the correct output for a working (properly indexed) file.

Known indexed file: '/volume1/video/TV Shows/WorkingSeries/S01/WorkingFile.S01E01.mkv

/usr/syno/pgsql/bin/psql mediaserver admin -c "select title from video where path = '/volume1/video/TV Shows/WorkingSeries/S01/WorkingFile.S01E01.mkv';"

                         title
-------------------------------------------------------
 WorkingFile.S01E01
(1 row)



/usr/syno/pgsql/bin/psql mediaserver admin -c "select title from directory where path = '/volume1/video/TV Shows/WorkingSeries/S01';"

 title
-------
 S01
(1 row)


/usr/syno/pgsql/bin/psql mediaserver admin -c "select title from directory where path = '/volume1/video/TV Shows/WorkingSeries';"

 title
-------
 WorkingSeries
(1 row)


/usr/syno/pgsql/bin/psql mediaserver admin -c "select title from directory where path = '/volume1/video/TV Shows';"

title
-------
 TV Shows
(1 row)


/usr/syno/pgsql/bin/psql mediaserver admin -c "select title from directory where path = '/volume1/video';"

 title
-------
 video
(1 row)

Now the test.

Test File: /volume1/video/TV Shows/MySeries/Season1/MySeries.S01E01.mkv

First we create the directories.

mkdir "/volume1/video/TV Shows/MySeries"
mkdir "/volume1/video/TV Shows/MySeries/Season1"

Then we copy or move the new file to the new directory. In this case copy.

cp MySeries.S01E01.mkv "/volume1/video/TV Shows/MySeries/Season1/MySeries.S01E01.mkv"

Now we run synoindexer.

synoindex -a "/volume1/video/TV Shows/MySeries/Season1/MySeries.S01E01.mkv"

Now we run the above steps to check for proper indexing.
First the file itself.

/usr/syno/pgsql/bin/psql mediaserver admin -c "select title from video where path = '/volume1/video/TV Shows/MySeries/Season1/MySeries.S01E01.mkv';"

                         title
-------------------------------------------------------
 MySeries.S01E01
(1 row)

This is exactly correct

Now the directories.

/usr/syno/pgsql/bin/psql mediaserver admin -c "select title from directory where path = '/volume1/video/TV Shows/MySeries/Season1';"

 title
-------
 (0 rows)


/usr/syno/pgsql/bin/psql mediaserver admin -c "select title from directory where path = '/volume1/video/TV Shows/MySeries';"

 title
-------
(0 rows)

etc...(I omitted the others as they were already indexed and therefore are still working.)

This is the problem. The file is indexed but the folders are not. When devices list the available folders, the relevant ones are missing, and even though the file is there, you can’t get to it.

I actually just noticed that you noted you were checking the results in Video Station. Video Station seems to keep it’s own index, and indeed my test file does appear there. The media server index for DLNA is the one I’m querying above.

Video Station only works with a computer (via Web GUI), AirPlay devices, iOS/Android devices, and Windows Phone. The media server is for most everything else. In my case the device I’m playing this on is a Playstation 3.

All of the above was tested on DSM 5.1.

Please note again, that I haven’t actually run this version of Sonarr as I’m still on the master branch, so apologies if I am wasting your time on a non-issue.
This is just something I ran into on my old Sickbeard and SABnzbd setup. There SAB would sort a file when it downloaded and then run synoindex -a "/volume1/video/TV Shows/MySeries/Season1/MySeries.S01E01.mkv". This caused the above issue.

Anyway, thanks for reading. Sorry again for the long post, just wanted to show you how I got there.

Just a bit of an update on this issue now that the SynoIndex notifications are part of the Master branch.

I tested for the issue described above (#1) with the latest Master release of Sonarr. I have confirmed that the bug is indeed present. Tested on DSM 5.1.

To reproduce do as above, or simply add a new show to Sonarr and allow it to download/process at least 1 episode. Attempt to play with DLNA compatible device.

I also confirmed that Video Station does indeed use a different database to Media Server, therefore this bug does not apply to Video Station.

Okay, added a card