Not scanning all folders and files

[steelrat]

Have Sonarr running on Windows server machine and love it. Am now in process of moving everything over to my Ubuntu headless server, V12.04.5. After many hours I now have Sonarr up and running but I’ve hit a major problem.

I’ve started importing my existing TV series into the program but, firstly Sonarr is not listing all of the folders in the tv series directory. I can add a folder by hand though. But then not all the files within the folders are seen! So while having complete seasons only a handful of episodes are being found.

Details of Sonarr and mono are:
Version 2.0.0.3154
Mono Version 4.0.1 (tarball Tue May 12 15:40:13 UTC 2015)

Can anyone help? I’m not a genius with Linux so don’t baffle me with too many linux based questions!

Thanks.

[markus101]

Sounds like a permissions issue, you’ll need to make sure all the files and folders are readable and writable by the user running Sonarr.

[steelrat]

I’ve taken some listings and screenshots to show what I mean.

Directory from CLI
total 148
drwxr-xr-x 39 robert robert 4096 May 31 11:34 .
drwxr-xr-x 6 root root 99 May 29 22:30 …
drwxr-xr-x 4 robert robert 4096 May 14 13:56 8 Out of 10 Cats
drwxr-xr-x 5 robert robert 4096 May 26 20:56 8 Out of 10 Cats Does Countdown
drwxr-xr-x 2 robert robert 4096 Jun 8 2014 Alcatraz
drwxr-xr-x 2 robert robert 4096 Nov 1 2013 An Idiot Abroad
drwxr-xr-x 4 robert robert 4096 May 29 22:54 Arrow
drwxr-xr-x 2 robert robert 4096 Dec 23 17:58 Ascension
drwxr-xr-x 2 robert robert 4096 Jun 25 2014 Believe
drwxr-xr-x 4 robert robert 8192 May 16 10:02 Bones
drwxr-xr-x 2 robert robert 4096 Jun 7 2014 Cosmos A Spacetime Odyssey
drwxr-xr-x 2 robert robert 4096 Feb 1 10:29 Cowboy Builders
drwxr-xr-x 2 robert robert 4096 Jan 18 03:49 David Attenborough’s Conquest of the Skies
drwxr-xr-x 2 robert robert 4096 Feb 14 19:32 Diy Sos
drwxr-xr-x 2 robert robert 4096 Sep 22 2014 Extant
drwxr-xr-x 2 robert robert 4096 Mar 25 11:24 Fifth Gear
drwxr-xr-x 7 robert robert 81 May 29 12:28 Game of Thrones
drwxr-xr-x 3 robert robert 21 May 6 08:37 Gotham
drwxr-xr-x 4 robert robert 112 May 19 22:18 Grimm
drwxr-xr-x 2 robert robert 4096 Jun 8 2014 Guitar Center Sessions
drwxr-xr-x 4 robert robert 114 May 29 22:43 Have I Got News for You
drwxr-xr-x 2 robert robert 4096 Nov 12 2013 How To Build A Planet
drwxr-xr-x 2 robert robert 4096 Jun 30 2014 Intelligence (2014)
drwxr-xr-x 3 robert robert 21 Mar 14 11:06 James May’s Cars of the People
drwxr-xr-x 2 robert robert 4096 Nov 19 2013 Karl Pilkington The Moaning Of Life
drwxr-xr-x 4 robert robert 8192 May 13 09:32 Marvel’s Agents of S.H.I.E.L.D
drwxr-xr-x 6 robert robert 4096 Feb 22 14:48 Never Mind the Buzzcocks
drwxr-xr-x 3 robert robert 29 May 27 00:12 Newzoids
drwxr-xr-x 5 robert robert 4096 Apr 24 03:32 Perception
drwxr-xr-x 4 robert robert 4096 Feb 16 09:18 Plebs
drwxr-xr-x 2 robert robert 4096 May 22 2014 Revolution
drwxr-xr-x 4 robert robert 4096 Feb 24 17:55 Sleepy Hollow
drwxr-xr-x 4 robert robert 4096 May 30 14:00 The 100
drwxr-xr-x 2 robert robert 74 Nov 15 2013 The Moaning Of Life
drwxr-xr-x 2 robert robert 4096 May 6 2014 The Tomorrow People (US)
drwxr-xr-x 2 robert robert 4096 Dec 24 03:46 The Wrong Mans
drwxr-xr-x 10 robert robert 4096 Mar 9 00:25 Top Gear
drwxr-xr-x 2 robert robert 4096 Dec 23 17:55 Trollied
drwxr-xr-x 2 robert robert 4096 Nov 2 2013 True Justice

What Sonarr sees

Example directory contents
total 1300
drwxr-xr-x 4 robert robert 4096 May 29 22:54 .
drwxr-xr-x 39 robert robert 4096 May 31 11:34 …
-rw-r–r-- 1 robert robert 40494 Nov 7 2013 Arrow S02 E05 League of Assassins.mkv.cover.jpg
-rw-r–r-- 1 robert robert 29404 Nov 15 2013 Arrow S02 E06 Keep Your Enemies Closer.mkv.cover.jpg
-rw-r–r-- 1 robert robert 31272 Nov 21 2013 Arrow S02 E07 State v. Queen.mkv.cover.jpg
-rw-r–r-- 1 robert robert 41630 Dec 12 00:17 Arrow S03 E09 The Climb.mkv.cover.jpg
-rwxr–r-- 1 robert robert 700503 Oct 27 2013 backdrop.jpg
-rw-r–r-- 1 robert robert 221698 Nov 1 2013 cover.jpg
-rwxr–r-- 1 robert robert 221698 Oct 27 2013 folder.jpg
drwxr-xr-x 2 robert robert 4096 Feb 22 12:36 Season 2
drwxr-xr-x 2 robert robert 4096 May 14 10:28 Season 3
-rwxr–r-- 1 robert robert 974 Oct 27 2013 series.xml

total 35767984
drwxr-xr-x 2 robert robert 4096 Feb 22 12:36 .
drwxr-xr-x 4 robert robert 4096 May 31 19:42 …
-rwxr-xr-x 1 robert robert 1824646799 Oct 10 2013 Arrow - S02 E01 - City of Heroes WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1750442254 Oct 17 2013 Arrow - S02 E02 - Identity WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1763963354 Oct 24 2013 Arrow - S02 E03 - Broken Dolls WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1681066812 Oct 31 2013 Arrow - S02 E04 - Crucible WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1192764523 Nov 6 2013 Arrow - S02 E05 - League of Assassins HDTV-720p.mkv
-rwxr-xr-x 1 robert robert 1224538172 Nov 14 2013 Arrow - S02 E06 - Keep Your Enemies Closer HDTV-720p.mkv
-rwxr-xr-x 1 robert robert 1206253168 Nov 20 2013 Arrow - S02 E07 - State V. Queen HDTV-720p.mkv
-rwxr-xr-x 1 robert robert 1613716211 Dec 5 2013 Arrow - S02 E08 - The Scientist WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1569440490 Dec 12 2013 Arrow - S02 E09 - Three Ghosts WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1662024605 Jan 17 2014 Arrow - S02 E10 - Blast Radius WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1636204056 Jan 23 2014 Arrow - S02 E11 - Blind Spot WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1606723864 Jan 31 2014 Arrow - S02 E12 - Tremors WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1636778919 Feb 6 2014 Arrow - S02 E13 - Heir to the Demon WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1647329301 Feb 28 2014 Arrow - S02 E14 - Time of Death WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1651233794 Mar 6 2014 Arrow - S02 E15 - The Promise WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1648684046 Mar 28 2014 Arrow - S02 E16 - Suicide Squad WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1639702992 Mar 27 2014 Arrow - S02 E17 - Birds of Prey WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1609559978 Apr 3 2014 Arrow - S02 E18 - Deathstroke WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1656682547 Apr 17 2014 Arrow - S02 E19 - The Man Under the Hood WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1594551662 Apr 24 2014 Arrow - S02 E20 - Seeing Red WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1550780550 May 1 2014 Arrow - S02 E21 - City of Blood WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1614183579 May 8 2014 Arrow - S02 E22 - Streets of Fire WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1645082254 May 15 2014 Arrow - S02 E23 - Unthinkable WEBDL-1080p.mkv

total 39405076
drwxr-xr-x 2 robert robert 4096 May 14 10:28 .
drwxr-xr-x 4 robert robert 4096 May 31 19:42 …
-rwxr-xr-x 1 robert robert 1688888582 Oct 9 2014 Arrow - S03 E01 - The Calm WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1602722318 Oct 16 2014 Arrow - S03 E02 - Sara WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1637290442 Oct 23 2014 Arrow - S03 E03 - Corto Maltese WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1602341504 Oct 30 2014 Arrow - S03 E04 - The Magician WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1666828007 Nov 6 2014 Arrow - S03 E05 - The Secret Origin of Felicity Smoak WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1651577160 Nov 13 2014 Arrow - S03 E06 - Guilty WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1655114166 Nov 20 2014 Arrow - S03 E07 - Draw Back Your Bow WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1621945356 Dec 7 20:30 Arrow - S03 E08 - The Brave and the Bold (2) WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1422767526 Dec 11 15:30 Arrow - S03 E09 - The Climb HDTV-720p.mkv
-rwxr-xr-x 1 robert robert 1735823330 Jan 22 07:43 Arrow - S03 E10 - Left Behind WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1720408323 Jan 29 09:42 Arrow - S03 E11 - Midnight City WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1814983990 Feb 5 09:20 Arrow - S03 E12 - Uprising WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1751123962 Feb 12 15:26 Arrow - S03 E13 - Canaries WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 3792565109 Feb 18 21:28 Arrow - S03 E14 - The Return HDTV-1080p.mkv
-rwxr-xr-x 1 robert robert 1384015874 Feb 26 15:56 Arrow - S03 E15 - Nanda Parbat WEBDL-720p.mkv
-rwxr-xr-x 1 robert robert 1749318922 Mar 19 12:46 Arrow - S03 E16 - The Offer WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1713880284 Mar 26 14:12 Arrow - S03 E17 - Suicidal Tendencies WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1873336877 Apr 2 16:25 Arrow - S03 E18 - Public Enemy WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1730573324 Apr 16 04:13 Arrow - S03 E19 - Broken Arrow WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1711674848 Apr 23 21:21 Arrow - S03 E20 - The Fallen WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1676840912 Apr 30 20:26 Arrow - S03 E21 - Al Sah-him WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1715517908 May 7 04:31 Arrow - S03 E22 - This Is Your Sword WEBDL-1080p.mkv
-rwxr-xr-x 1 robert robert 1431196757 May 14 10:22 Arrow - S03 E23 - My Name Is Oliver Queen WEBDL-720p.mkv

and what Sonarr tells me there is

I wonder if this may help. I still have Sonarr running on a Windows Server and that sees everything just fine.

[markus101]

Which user is running Sonarr?

[steelrat]

I autostart it from a script where I setuid to robert.

[markus101]

Please hastebin some trace logs of a refresh of Arrow (clear the existing logs), that way we’ll see what files it actually saw and why they weren’t added.

[steelrat]

Here you go…

15-6-2 20:35:33.1|Info|SceneMappingService|Updating Scene mappings
15-6-2 20:35:35.0|Info|RefreshSeriesService|Updating Info for Arrow
15-6-2 20:35:35.4|Info|RefreshEpisodeService|Starting episode info refresh for: [257655][Arrow]
15-6-2 20:35:35.6|Info|RefreshEpisodeService|Finished episode refresh for series: [257655][Arrow].
15-6-2 20:35:35.6|Info|DiskScanService|Scanning disk for Arrow
15-6-2 20:35:35.7|Info|DiskScanService|Completed scanning disk for Arrow
15-6-2 20:36:59.8|Info|Database|Vacuuming Log database
15-6-2 20:36:59.9|Info|Database|Log database compressed
15-6-2 20:37:14.8|Debug|QualityUpgradableSpecification|Existing item meets cut-off. skipping.
15-6-2 20:37:14.8|Debug|QualityUpgradableSpecification|Existing item meets cut-off. skipping.
15-6-2 20:37:14.8|Debug|QualityUpgradableSpecification|Existing item meets cut-off. skipping.
15-6-2 20:37:14.8|Debug|QualityUpgradableSpecification|Existing item meets cut-off. skipping.
15-6-2 20:37:14.8|Debug|QualityUpgradableSpecification|Existing item meets cut-off. skipping.
15-6-2 20:37:16.7|Info|SceneMappingService|Updating Scene mappings
15-6-2 20:37:17.1|Debug|XemProxy|Fetching alternate names
15-6-2 20:37:18.1|Debug|XemProxy|Fetching Series IDs from
15-6-2 20:37:18.6|Info|RefreshSeriesService|Updating Info for Arrow
15-6-2 20:37:19.0|Info|RefreshEpisodeService|Starting episode info refresh for: [257655][Arrow]
15-6-2 20:37:19.2|Debug|QualityUpgradableSpecification|Existing item meets cut-off. skipping.
15-6-2 20:37:19.2|Debug|QualityUpgradableSpecification|Existing item meets cut-off. skipping.
15-6-2 20:37:19.2|Debug|QualityUpgradableSpecification|Existing item meets cut-off. skipping.
15-6-2 20:37:19.2|Debug|QualityUpgradableSpecification|Existing item meets cut-off. skipping.
15-6-2 20:37:19.2|Debug|QualityUpgradableSpecification|Existing item meets cut-off. skipping.
15-6-2 20:37:19.3|Info|RefreshEpisodeService|Finished episode refresh for series: [257655][Arrow].
15-6-2 20:37:19.4|Debug|RefreshSeriesService|Finished series refresh for Arrow
15-6-2 20:37:19.4|Debug|XemService|Scene numbering is not available for Arrow [257655]
15-6-2 20:37:19.4|Info|DiskScanService|Scanning disk for Arrow
15-6-2 20:37:19.4|Debug|DiskScanService|Scanning ‘/media/Video/TV Series/Arrow’ for video files
15-6-2 20:37:19.4|Debug|DiskScanService|5 video files were found in /media/Video/TV Series/Arrow
15-6-2 20:37:19.4|Debug|ImportDecisionMaker|Analyzing 0/5 files.
15-6-2 20:37:19.4|Info|DiskScanService|Completed scanning disk for Arrow
15-6-2 20:37:19.5|Debug|ExistingMetadataService|Looking for existing metadata in /media/Video/TV Series/Arrow
15-6-2 20:37:19.5|Debug|Parser|Parsing string 'backdrop.jpg’
15-6-2 20:37:19.5|Debug|Parser|Unable to parse backdrop.jpg
15-6-2 20:37:19.5|Debug|Parser|Parsing string 'backdrop.jpg’
15-6-2 20:37:19.5|Debug|Parser|Unable to parse backdrop.jpg
15-6-2 20:37:19.5|Debug|Parser|Parsing string 'backdrop.jpg’
15-6-2 20:37:19.5|Debug|Parser|Unable to parse backdrop.jpg
15-6-2 20:37:19.5|Debug|Parser|Parsing string 'cover.jpg’
15-6-2 20:37:19.5|Debug|Parser|Unable to parse cover.jpg
15-6-2 20:37:19.5|Debug|Parser|Parsing string 'cover.jpg’
15-6-2 20:37:19.5|Debug|Parser|Unable to parse cover.jpg
15-6-2 20:37:19.5|Debug|Parser|Parsing string 'cover.jpg’
15-6-2 20:37:19.5|Debug|Parser|Unable to parse cover.jpg
15-6-2 20:37:19.5|Debug|CleanMetadataService|Cleaning missing metadata files for series: Arrow

[steelrat]

Are there any more thoughts on this? I’ve gone through all settings and checked all permissions so many times. The version of Sonarr running on my Windows server sees everything, the version on Ubuntu still doesn’t see any more than before. If I can list everything from the cli, there must be a problem in nzbdrone with reading of directories and files, surely?

[markus101]

It still sounds like a permissions issue, but I can’t point out what it might be.

When Sonarr is running what do you get back if you run ps aux?

@steelrat

If I can list everything from the cli, there must be a problem in nzbdrone with reading of directories and files, surely?

Which user are you running ls with? Sonarr asks the OS for the list of files, then filters out anything thats not a video file, since those files don’t show up on Ubuntu, but do on Windows they aren’t be returned.

[steelrat]

I assume you only need the output from ps aux for Sonarr so here it is.

robert 29266 1.2 2.5 155960 105576 ? Ssl Jun05 18:47 mono --debug /opt/NzbDrone/NzbDrone.exe

I run ls -la as robert, which is what I log in with using PuTTy.

The odd thing is that running ls on any of them returns everything that’s there. What I don’t seem to be able to get my head around is that the user/group permissions are identical on all the files and folders so how is it possible to see some but not others? It’s very confusing!

I tried uninstalling Sonarr and re-installing, and same with mono but I am getting exactly the same results. If I run Sonarr as root I still get exactly the same result.

Thank you for continuing to help out.

[markus101]

Could you try with mono 3.10? It should be available in apt-get through our repo or mono’s (4.0.1 should also be assuming its supported on 12.04).

[steelrat]

OK, well that turned into a mission! I managed to remove all traces of mono 4.01 eventually, after many, many problems. Cleaned out everything and started from scratch.

I now have Sonarr Version 2.0.0.3212 installed, running under Mono Version
3.12.1 (tarball Fri Mar 6 19:14:08 UTC 2015). I have changed ownership of all directories and files to robert, and all groups to nogroup. All directories have permissions of 0755 set, all files have 0644 set. NzbDrone is now up and running again and I am experiencing exactly the same problems.

[steelrat]

I guess this is just not going to work then. Oh well, back to Sickbeard, I suppose.