MediaInfo errors for a lot of files [dev 2.0.0.1773]

Help & Support
1

Hi, I noticed a lot of MediaInfo errors lately, preventing files from being imported.
I’m on the latest develop version (2.0.0.1773), running on a Synology NAS.

Marr.Data.DataMappingException: The DataMapper was unable to load the following field: ‘MediaInfo’ value: ‘{
  “videoCodec”: “AVC”,
  “videoBitrate”: 3846856,
  “width”: 1280,
  “height”: 720,
  “audioFormat”: “AC-3”,
  “audioBitrate”: 384000,
  “runTime”: “00:42:58.9960000”,
  “audioStreamCount”: 1,
  “audioChannels”: 6,
  “audioProfile”: “”,
  “videoFps”: 00.000,
  “audioLanguages”: “English”,
  “subtitles”: “English”,
  “scanType”: “Progressive”
}’. Additional unparsable characters are at the end of the string. —> System.FormatException: Additional unparsable characters are at the end of the string.
  at System.Convert.ConvertFromBase64 (System.String value, Int32 fromBase, Boolean unsigned) [0x00000] in :0
  at System.Convert.ToInt64 (System.String value, Int32 fromBase) [0x00000] in :0
  at Newtonsoft.Json.JsonTextReader.ParseNumber () [0x00000] in :0
  at Newtonsoft.Json.JsonTextReader.ParseValue () [0x00000] in :0
  at Newtonsoft.Json.JsonTextReader.ReadInternal () [0x00000] in :0
  at Newtonsoft.Json.JsonReader.ReadAsDecimalInternal () [0x00000] in :0
  at Newtonsoft.Json.JsonTextReader.ReadAsDecimal () [0x00000] in :0
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType (Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonContract contract, Boolean hasConverter) [0x00000] in :0
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject (System.Object newObject, Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty member, System.String id) [0x00000] in :0
  — End of inner exception stack trace —
  at Marr.Data.Mapping.MappingHelper.LoadExistingEntity (Marr.Data.Mapping.ColumnMapCollection mappings, System.Data.Common.DbDataReader reader, System.Object ent, Boolean useAltName) [0x00000] in :0
  at Marr.Data.Mapping.MappingHelper.CreateAndLoadEntity (System.Type entityType, Marr.Data.Mapping.ColumnMapCollection mappings, System.Data.Common.DbDataReader reader, Boolean useAltName) [0x00000] in :0
  at Marr.Data.Mapping.MappingHelper.CreateAndLoadEntity[EpisodeFile] (Marr.Data.Mapping.ColumnMapCollection mappings, System.Data.Common.DbDataReader reader, Boolean useAltName) [0x00000] in :0
  at Marr.Data.DataMapper.Query[EpisodeFile] (System.String sql, ICollection1 entityList, Boolean useAltName) [0x00000] in <filename unknown>:0   at Marr.Data.QGen.QueryBuilder1[NzbDrone.Core.MediaFiles.EpisodeFile].ToList () [0x00000] in :0
  at Marr.Data.QGen.QueryBuilder1[NzbDrone.Core.MediaFiles.EpisodeFile].System.Collections.Generic.IEnumerable<T>.GetEnumerator () [0x00000] in <filename unknown>:0   at System.Linq.Enumerable.Single[EpisodeFile] (IEnumerable1 source, System.Func2 predicate, Fallback fallback) [0x00000] in <filename unknown>:0   at System.Linq.Enumerable.SingleOrDefault[EpisodeFile] (IEnumerable1 source, System.Func2 predicate) [0x00000] in <filename unknown>:0   at NzbDrone.Core.Datastore.Extensions.RelationshipExtensions+<>c__DisplayClass32[NzbDrone.Core.Tv.Episode,NzbDrone.Core.MediaFiles.EpisodeFile].b__0 (IDataMapper db, NzbDrone.Core.Tv.Episode parent) [0x00000] in :0
  at Marr.Data.LazyLoaded2[NzbDrone.Core.Tv.Episode,NzbDrone.Core.MediaFiles.EpisodeFile].LazyLoad () [0x00000] in <filename unknown>:0   at Marr.Data.LazyLoaded1[NzbDrone.Core.MediaFiles.EpisodeFile].get_Value () [0x00000] in :0
  at NzbDrone.Core.MediaFiles.EpisodeImport.Specifications.UpgradeSpecification+<>c__DisplayClass1.b__0 (NzbDrone.Core.Tv.Episode e) [0x00000] in :0
  at System.Linq.Enumerable.Any[Episode] (IEnumerable1 source, System.Func2 predicate) [0x00000] in :0
  at NzbDrone.Core.MediaFiles.EpisodeImport.Specifications.UpgradeSpecification.IsSatisfiedBy (NzbDrone.Core.Parser.Model.LocalEpisode localEpisode) [0x00000] in :0
  at NzbDrone.Core.MediaFiles.EpisodeImport.ImportDecisionMaker.EvaluateSpec (IRejectWithReason spec, NzbDrone.Core.Parser.Model.LocalEpisode localEpisode) [0x00000] in :0

The file in this example plays perfectly fine.
If I wait long enough, I start seeing the same errors over and over again for files that were downloaded a long time ago as well. Once again, I can see nothing wrong (I’m not an expert) with those files, as they play fine etc.

2

We recently started adding mediainfo details to the database for each episodefile. That information is needed to allow adding Codec details to the renaming format. (And other possible future features)

I’ve copied the text you provided and tested it locally. I get the same error.
Looks like a bug in the Json deserializer.
Gives an error on “videoFps”: 00.000
but not on “videoFps”: 0.000
How odd is that?

I’ll try to determine why it’s not parsing properly.
In your case it actually might actually prevent drone from doing stuff like update existing files.

On a sidenote, I wonder why the fps is 0. should be something like 23976.

3

Hi Taloth, now that you mention it, it’s so obvious.
All the files that throw these errors have 00.000 videoFps in the drone logs, which indeed doesn’t make sense. I checked a couple, and in VLC’s media info the correct framerate is shown.

I suppose the “worst” from and end-user point of view is that files are not moved/renamed once they’re downloaded if they have this MediaInfo bug. Other stuff that fails is CutoffModule, MissingModule, EpisodeModule, CalendarModule…

4

Could you provide me with one of the Release names? So I can reproduce the mediainfo being generated.
And actually, it’s a bit more complicated than

@markus101 Need your help here. This is a total mess.
In the center is ‘decimal’, decimal preserves significant digits. This causes a bit of a problem with the VideoFileInfoReader parsing the string, this is producing the ‘decimal’ that, when formatted, results in 00.000. (Why, I dunno, I couldn’t repro that)
Second is Newtonsoft.Json, which can’t parse it back again coz a leading zero on a second digit (‘00’ , ‘01’) is perceived as octal int, not float.

As I see it we need to fix how the VideoFileInfoReader produces decimals, to prevent new occurrences.
But we probably need Db migration to purge the error from the db.

5

Decimal.TryParse doesn’t add a leading zero when trying to parse “00.000”, the result is 0.000. At least in windows anyways. Could have gone wrong in a number of places, so I’d be curious to see the results of your testing.

@Thirrian what version if mono are you running? mono --version should show it.

6

Hi markus,

DiskStation> ./mono --version
Mono JIT compiler version 3.4.0 (tarball Wed May 28 16:13:54 CEST 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       normal
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen
DiskStation>

7

@markus101
mediainfo on windows is returning 23.976. But we forgot InvariantCulture so it gets parsed as 23976.0 on my regional settings. So that’s another bug.
I dunno how the formatter replaced the digits with zeroes. Could be a mono thing.

@Thirrian
Don’t know if the mediainfo cmdline is available on your syno, but could you do 'mediainfo ’ and post the results?

8

@Taloth that’s not something that’s available on syno out of the box, but I installed mediainfo from http://packages.missilehugger.com/

Output on one of the “offending” files:

DiskStation> mediainfo bla.mkv
General
Unique ID                                : 189293128264047351743160096471959847340 (0x8E6887C83B5D864C8184C16D7C91C1AC)
Complete name                            : bla.mkv
Format                                   : Matroska
Format version                           : Version 4 / Version 2
File size                                : 1.18 GiB
Duration                                 : 42mn 40s
Overall bit rate                         : 3 953 Kbps
Encoded date                             : UTC 2014-08-08 01:07:20
Writing application                      : mkvmerge v7.0.0 (‘Where We Going’) 64bit built on Jun 22 2014 08:19:39
Writing library                          : libebml v1.3.0 + libmatroska v1.4.1

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 5 frames
Codec ID                                 : V_MPEG4/ISO/AVC
Duration                                 : 42mn 40s
Bit rate                                 : 3 490 Kbps
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 23.976 fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.158
Stream size                              : 1.04 GiB (88%)
Writing library                          : x264 core 142 r2453 ea0ca51
Encoding settings                        : cabac=1 / ref=5 / deblock=1:0:0 / analyse=0x3:0x113 / me=umh / subme=8 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=18 / lookahead_threads=3 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=23 / scenecut=40 / intra_refresh=0 / rc_lookahead=50 / rc=crf / mbtree=1 / crf=18.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Language                                 : English
Default                                  : Yes
Forced                                   : No
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : A_AC3
Duration                                 : 42mn 40s
Bit rate mode                            : Constant
Bit rate                                 : 384 Kbps
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 KHz
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 117 MiB (10%)
Default                                  : Yes
Forced                                   : No

9

Okay, quite possibly this is another version, so not really representative. but thx for the effort.

10

I tried to reproduce it but no luck.

Do you have time and are you willing to help investigate this issue on the syno itself?

I’ve made a small test application that basically calls the mediainfo library and serializes it like nzbdrone does without all the other fuss. (https://www.cubbyusercontent.com/pli/test-mediainfo.exe/_ad1a30d3f7a14244ada4847c657c1dde)
It should be copied to the directory containing NzbDrone.exe.
And then executed from the commandline like:
mono test-mediainfo.exe

If i run it on my ubuntu server I get:

Culture: en-US
{"VideoCodec":"AVC","VideoBitrate":3489955,"Width":1280,"Height":720,"AudioFormat":"AC-3","AudioBitrate":384000,"RunTime":"00:42:40.9780000","AudioStreamCount":1,"AudioChannels":6,"AudioProfile":"","VideoFps":23.976,"AudioLanguages":"English","Subtitles":"","ScanType":"Progressive"}
11

I had to complicate the command a bit…
It’s based on the start/stop script that the syno package uses to start/stop drone. It’s entirely the same command line found in there, but with NzbDrone.exe substituted by test-mediainfo.exe. This was necessary to avoid errors about not finding libmediainfo.so (to be found in nzbdrone/lib).

DiskStation> env PATH=/usr/local/mono/bin:${PATH} LD_LIBRARY_PATH=/usr/local/nzbdrone/lib /usr/local/mono/bin/mono /usr/local/nzbdrone/share/NzbDrone/test-mediainfo.exe /usr/local/nzbdrone/share/NzbDrone/bla.mkv
Culture: en-US
{“VideoCodec”:“AVC”,“VideoBitrate”:3489955,“Width”:1280,“Height”:720,“AudioFormat”:“AC-3”,“AudioBitrate”:384000,“RunTime”:“00:42:40.9780000”,“AudioStreamCount”:1,“AudioChannels”:6,“AudioProfile”:“”,“VideoFps”:23.976,“AudioLanguages”:“English”,“Subtitles”:“”,“ScanType”:“Progressive”}

So once again this looks fine :confused:

Edit: just a heads-up, I won’t have a lot of time during the day for the next 4 days, and I’m not sure yet about internet connectivity in the evening, so my replies might take a little longer from now on. Still absolutely willing to invest time on this :slight_smile:

12

Now I’m suspecting that the file we tried wasn’t the one that triggered the bug.

Next one:
https://www.cubbyusercontent.com/pli/test-mediainfo2.exe/_7fc7c4a737fc4ca9a00a4f5497d8ca4b

This one needs the nzbdrone.db file as argument.
It will query the db, get all EpisodeFile entries with the 00.000 and run the analysis on all of those.
Time for drastic measures.

Example: mono test-mediainfo2.exe /zstore/mediadb/tv/nzbdrone.db
(you can pipe it to a file, if needed)

You don’t have to post all of them, just a couple would be fine. Obviously I’m interested in the ones producing 00.000.
Since I would like the releasenames to be included, feel free to PM me the results so it doesn’t end up public.

13

Hi Taloth, I’ll run this tonight and send you the results. I’m having decent internet access again.

14

Okay, well, I thought this might take ages to check each file in the db, but it stops after just one, so here’s the results already. FYI, I also ran this when drone was stopped to be sure there’s no conflict with access to nzbdrone.db, but there was no difference.

Interesting story, when I want to open that show in drone, it gives me

[GET] Internal Server Error: /api/episodefile/?seriesId=54

Other shows don’t seem to have this issue. I just took a couple of “samples” by clicking a few random shows.
Note sure if it’s related or I’m getting sidetracked, just giving you all information I got…

15

It’s the same cause, drone can’t read the episodes from that series in the db due to that field not being parable so reports an error to the UI.

But I’m pretty pissed off that the test resulted in 23.976 again. Pretty much hit a deadend now.

16

Should I just edit the db to get rid of the 00.000 values then? Then observe if it happens again when drone reads mediainfo for new episodes?

17

yes, I think that’s the best thing to do atm.

18

Hi @Taloth, I took drone offline, ran a query over the database, and actually found only a single file (the one in the pastebin) which had the framerate as 00.000, very strange. Drone was generating multiple log files per day and throwing this error about so many other files as well all the time… :open_mouth:

I corrected the one entry to 23.976 and started drone again, and now it looks all fine. I’ll observe it for a while and let you know if things go wrong again… I just found out that I’m missing entire seasons of a couple of shows in better quality, those are all downloading now and importing fine so far.

Makes no sense to me, but okay, I’m glad it works again, and thanks for all the effort! :slight_smile:

19

Not that strange at all. The errors “about so many other files” were not about those files, instead every time drone accessed the database it would run into the corrupted one.

It’s like a splinter in your foot, you can’t walk or stand, and before you know it you can’t work, type, eat, sit either.

But it’s good to know it’s continuing business as usual.