Logic behind timings of various background processes (DB vacuuming, scene map updating, etc)

Help & Support
1

Hi,

I’ve recently started using NZBDrone as a replacement of SickBeard and so far I’m very happy with it!
But since I use NZBDrone, the harddisks in my Synology NAS no longer go into hibernation as ND keeps executing processes on various moments. (FYI: my disk idle time limit is set to 10mins).

In my SickBeard/CouchPotato/Sabnzbd setup, I let everything run at the same time, so when there is nothing to be downloaded or processed, my disks would power down after 10 minutes, and spin up again about 50 minutes later for the next search. I’m not able to achieve this with ND (yet).

I’m close to letting everything (ND/CP) run at the same moment, except I can’t make sense of the database vacuuming or scene map updates timings. Is there a way to influence the time when these processes should be executed?
I’ve manually restarted ND on the time I like, and manually started RSS updates and library updates to make them all start around the same time for, as far as I know, that’s the only way to influence it.

Example of my log:

Database         Database Compressed                                        11:34
Database         Vacuuming database                                         11:34
RssSyncService RSS Sync Completed. Reports found: 600, Reports grabbed: 0   11:08
DownloadDecisionMaker    Processing 600 reports                             11:08
RssSyncService    Starting RSS Sync                                         11:08
Database          Database Compressed                                       10:33
Database          Vacuuming database                                        10:33
Database          Database Compressed                                       10:16
Database          Vacuuming database                                        10:16
HousekeepingService    Running housecleaning tasks                          10:16
BackupService          Starting Backup                                      10:12
RssSyncService RSS Sync Completed. Reports found: 494, Reports grabbed: 0   10:08
DownloadDecisionMaker    Processing 494 reports                             10:08
RssSyncService         Starting RSS Sync                                    10:08
Database          Database Compressed                                       09:33
Database          Vacuuming database                                        09:33
SceneMappingService    Updating Scene mappings                              09:27
Database          Database Compressed                                       09:13
Database          Vacuuming database                                        09:13
HousekeepingService    Running housecleaning tasks                          09:13
SceneMappingService    Updating Scene mappings                              09:13
LifecycleService    Restart requested.                                      09:12

There are two things that are not yet properly aligned for me: the database compression (why twice within 20 minutes, except for the last hour?) and the scene mappings update. Is there a way for me to schedule everything better so my disks can idle down?

Would the following be a feasable/logical order:
(- Update Scene mappings before fetching RSS feeds - not needed every hour, therefor in brackets)
(- Update library/episode refresh - not needed every hour, therefor in brackets)

  • Fetch RSS feeds
  • Compress DB

What if all of this is executed one after the other (the end of one process automatically starts the next one instead of using ‘fixed’ timeframes for every process), using the time ND was (re)started as starting point for the first process?

Thanks for reading, and keep up the good work!

2

Trim logs (and compressing the logs database happens every hour) as well as on startup, which will require the disk that drone is on + the disk with the databases stored on it to be only. Scene mappings are updated when a series (or all series) are refreshed, or every 3 hours.

They are not executed in a particular order because they have no relation to each other, they are independent tasks that all need to be executed at some point.

3

Thanks for the info, much appreciated!

I think it would still benefit some of us (and therein probably lies the issue: not many people would need this option or find this important enough) to have some control of when the tasks are executed. As a developer, you could program to be executed sequentially instead of every xx minutes based on some arbitrary start time, but I understand why the fixed time frame is chosen.

If ND could handle those independant tasks sequentially when wanted and/or in a shorter timeframe, HDD’s could stay longer in hibernate, positively influencing energy-consumption and wear&tear. (Also, let’s say you have a NAS/system which is a bit slower, you could really schedule when different apps need to run to optimize processor runtime.

For example: at xx:05 NZBDrone works it magic (all tasks), and at xx:10 CouchPotato starts (all tasks). If nothing interesting is found, the disks go to sleep around xx:20, only to awake again at xx:05, so in one hour time, the disks are only active for 15mins instead of the randomness that occurs now, often resulting in no hibernation at all.

Cheers!

4

They are executed according to their interval, after the last run’s end time, so if a task started at 00:00 and ran for 10 minutes, with an interval of 1 hour it would run again at 01:10.

These jobs are intentionally not scheduled, if drone scheduled all the Series update jobs to occur at 1am then everyone running it would execute it at the same time and hammer trakt (drone uses UTC for dates and times so around the globe it would execute at the same time). Additionally, if they were scheduled and depending on the amount of data drone needed to process it took almost the interval time it would try to run the job again almost immediately, which would be pointless. I do realize they could all have a random minute offset, but it adds a complexity that doesn’t need to be there.

If multiple jobs are required to be triggered at the same time (if drone was off then this would stack them up), then they do run sequentially, but since almost every job has a different interval there is not really a way that they can always run sequentially, unless you store the last run time and still run things off an interval, which is what drone does already, it just doesn’t need to group multiple jobs to execute them in batch. The job “scheduler” runs every 30 seconds looking for anything that needs to run, if there is only one job, that job is executed, if more than one job is ready to be run they are executed sequentially, but in no specified order.

There are some improvements that we’re going to make, cleaning up some jobs that are triggered via other means or don’t need to run very often, but we’re not making plans to overhaul the execution of these scheduled jobs.