I’ve got a setup where I’m using 4 indexers: a private one, Spotweb, AnimeZB and Fanzub. Every time Drone checks the RSS feeds it has to check 325 reports, and they’re processed pretty slow, often at least 5 seconds per report.
It got me thinking: what if we could use certain indexers only for manual searches? That way I could keep Drone feeling snappy while at the same time still having all the indexers at my disposal if something fails to be picked up, which should be rare if your primary indexers are any good.
Euh, My rss feed parses around 300 reports on my develop setup in just a couple of seconds tops… total.
I’m really wondering where those 5 seconds/report comes from.Sounds like you have a huge performance hog somewhere.
Could you first confirm by comparing the timestamps on the “Starting RSS Sync” and “RSS Sync Completed. Reports found: xxx, Reports downloaded: 0” log lines?
If the duration is confirmed, please enable Trace loglevel, trigger a RSS Sync and gather the relevant log file lines. Maybe that will give an indication of what’s making it so terribly slow.
A bit later it goes a bit faster, but that seems to be because Fanzub screwed something up:
14-8-6 08:52:05.6|Info|RssSyncService|Starting RSS Sync
14-8-6 08:52:05.9|Error|FetchFeedService|An error occurred while processing feed. https://fanzub.com/rss/?cat=anime&max=100
System.Xml.XmlException: Text node cannot appear in this state. Line 1, position 7.
at Mono.Xml2.XmlTextReader.ReadText (Boolean notWhitespace) [0x00000] in :0
at Mono.Xml2.XmlTextReader.ReadContent () [0x00000] in :0
at Mono.Xml2.XmlTextReader.Read () [0x00000] in :0
at System.Xml.XmlTextReader.Read () [0x00000] in :0
at Mono.Xml.XmlFilterReader.Read () [0x00000] in :0
at Mono.Xml.XmlFilterReader.Read () [0x00000] in :0
at System.Xml.Linq.XElement.LoadCore (System.Xml.XmlReader r, LoadOptions options) [0x00000] in :0
at System.Xml.Linq.XNode.ReadFrom (System.Xml.XmlReader r, LoadOptions options) [0x00000] in :0
at System.Xml.Linq.XContainer.ReadContentFrom (System.Xml.XmlReader reader, LoadOptions options) [0x00000] in :0
at System.Xml.Linq.XDocument.ReadContent (System.Xml.XmlReader reader, LoadOptions options) [0x00000] in :0
at System.Xml.Linq.XDocument.LoadCore (System.Xml.XmlReader reader, LoadOptions options) [0x00000] in :0
at System.Xml.Linq.XDocument.Load (System.Xml.XmlReader reader, LoadOptions options) [0x00000] in :0
at System.Xml.Linq.XDocument.Load (System.Xml.XmlReader reader) [0x00000] in :0
at NzbDrone.Core.Indexers.RssParserBase.Process (System.String xml, System.String url) [0x00000] in :0
at NzbDrone.Core.Indexers.FetchFeedService.Fetch (IIndexer indexer, IEnumerable`1 urls) [0x00000] in :0
14-8-6 08:52:07.2|Info|DownloadDecisionMaker|Processing 225 reports
14-8-6 09:01:03.2|Info|RssSyncService|RSS Sync Completed. Reports found: 225, Reports grabbed: 0
I mean, Drone is running on a NAS and as such it hasn’t got that much hardware available (Intel Atom D2700 @ 2.13GHz with 3GB of RAM), but during processing I never see my CPU usage go above 40% (usually it’s steady around 30%), nor is my RAM anywhere near full. Most of the time I don’t hear my hard drives working either.
The following are the results of top while not syncing and while syncing, in that order:
The biggest difference between your setup and mine will most likely be this:
Could you pastebin me the complete tracelog for one rss sync?
It definitely might have to do with the number of monitored shows. I believe we could still optimize that a bit, but only if a significant portion of those rss releases are about series you have added.
Edit: I should add that not all of the reports take seconds to process, sometimes it races through a bunch of report in the wink of an eye, only to get stuck on one afterwards. Most reports do seem to take longer than a few seconds each though.
There are 6 pages in my blacklist. I turned FDH off, turned Trace logging back on and ran another sync, no noticeable difference in processing time: log file.
Let me know if there is more info I can give you, or poke me on IRC if you’d like direct access to my setup.
Even i had this problem of slow rss syncs’ but i thought it was because of the number of shows I had, Now after upgrading to the lastest develop version the rss sync is so fast. Thanks man