High (100%) CPU usage on Linux

Hey guys,

I just moved over from SickBeard and really like NZBDrone. Great job!

I installed NZBDrone on my Ubuntu 12.04 LTS box flawlessly, but have noticed that it’s running with 100% cpu usage all the time (mono is #1 entry on ‘top’). If I restart it, it will sit at 0.2% CPU for a while until it fetches a release, etc., then it will just sit at 100% again.

Has anyone else experienced this? Any clues on how to fix/debug this?

Thanks!

I’ve noticed mine will occasionally climb, but I haven’t seen 100% in several releases. Honestly, I think I’ve had to restart it more due to updates than climbing cpu usage of late.

Yeah I updated yday but still have this issue. Everything fine on startup but as soon as it downloads/processes a release, mono pegs at 100% CPU until nzbdrone is restarted.

Bit of a deal breaker for switching from sickbeard.

Please enable trace logging and grab the log file from start to 100% (delete the log file then start drone). Want to see if anything weird is happening.

I’m getting the same issue on linux running v2.0.0.855

This is from the logs:

13-11-20 20:32:34.2|Info|EpisodeSearchService|Episode search completed. 1 reports downloaded.
13-11-20 20:36:58.6|Fatal|GlobalExceptionHandlers|EPIC FAIL: Object reference not set to an instance of an object

System.NullReferenceException: Object reference not set to an instance of an object
at Microsoft.AspNet.SignalR.Infrastructure.Connection.ProcessCommand (Microsoft.AspNet.SignalR.Messaging.Command command) [0x00000] in :0
at Microsoft.AspNet.SignalR.Infrastructure.Connection.b__8 (System.Object state, Microsoft.AspNet.SignalR.Messaging.Message message) [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.MessageBusExtensions.Enumerate[Object] (IList1 messages, System.Func2 filter, System.Action2 onMessage, System.Object state) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.ProcessResults (MessageResult result) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.GetResponse (MessageResult result) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection+ReceiveContext.InvokeCallback (MessageResult result) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.MessageBusCallback (MessageResult result, System.Object state) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.<Receive>b__4 (MessageResult result, System.Object s) [0x00000] in <filename unknown>:0 at (wrapper delegate-invoke) System.Func3<Microsoft.AspNet.SignalR.Messaging.MessageResult, object, System.Threading.Tasks.Task1<bool>>:invoke_TResult__this___T1_T2 (Microsoft.AspNet.SignalR.Messaging.MessageResult,object) at Microsoft.AspNet.SignalR.Messaging.Subscription.Invoke (MessageResult result, System.Action1 beforeInvoke, System.Object state) [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.Subscription.WorkImpl (System.Threading.Tasks.TaskCompletionSource1 taskCompletionSource) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Messaging.Subscription.Work () [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Messaging.MessageBroker.PumpImpl (System.Threading.Tasks.TaskCompletionSource1 taskCompletionSource, ISubscription subscription) [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.MessageBroker.PumpAsync () [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.MessageBroker.ProcessWork (System.Object state) [0x00000] in :0

13-11-20 20:41:12.5|Info|RssSyncService|Starting RSS Sync
13-11-20 20:41:16.1|Warn|IgnoreCertErrorPolicy|[System.Net.HttpWebRequest] RemoteCertificateChainErrors
13-11-20 20:41:16.3|Warn|IgnoreCertErrorPolicy|[System.Net.HttpWebRequest] RemoteCertificateChainErrors
13-11-20 20:41:16.5|Warn|IgnoreCertErrorPolicy|[System.Net.HttpWebRequest] RemoteCertificateChainErrors
13-11-20 20:41:32.6|Info|DownloadDecisionMaker|Processing 300 reports
13-11-20 20:41:43.7|Info|RssSyncService|RSS Sync Completed. Reports found: 300, Reports downloaded: 0

Here’s mine:

[Info] Router: Application mode: Interactive
[Info] OwinHostController: starting server on http://*:8989/
[Info] NancyBootstrapper: Starting NzbDrone API
[Info] SceneMappingService: Updating Scene mapping
[Info] HousekeepingService: Running housecleaning tasks
[Warn] GlobalExceptionHandlers: SignalR Heartbeat error.
EPIC FAIL: System.NullReferenceException: Object reference not set to an instance of an object
at Microsoft.AspNet.SignalR.Infrastructure.Connection.ProcessCommand (Microsoft.AspNet.SignalR.Messaging.Command command) [0x00000] in :0
at Microsoft.AspNet.SignalR.Infrastructure.Connection.b__8 (System.Object state, Microsoft.AspNet.SignalR.Messaging.Message message) [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.MessageBusExtensions.Enumerate[Object] (IList1 messages, System.Func2 filter, System.Action2 onMessage, System.Object state) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.ProcessResults (MessageResult result) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.GetResponse (MessageResult result) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection+ReceiveContext.InvokeCallback (MessageResult result) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.MessageBusCallback (MessageResult result, System.Object state) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.<Receive>b__4 (MessageResult result, System.Object s) [0x00000] in <filename unknown>:0 at (wrapper delegate-invoke) <Module>:invoke_Task1__this___MessageResult_object (Microsoft.AspNet.SignalR.Messaging.MessageResult,object)
at Microsoft.AspNet.SignalR.Messaging.Subscription.Invoke (MessageResult result, System.Action1 beforeInvoke, System.Object state) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Messaging.Subscription.WorkImpl (System.Threading.Tasks.TaskCompletionSource1 taskCompletionSource) [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.Subscription.Work () [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.MessageBroker.PumpImpl (System.Threading.Tasks.TaskCompletionSource`1 taskCompletionSource, ISubscription subscription) [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.MessageBroker.PumpAsync () [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.MessageBroker.ProcessWork (System.Object state) [0x00000] in :0
[Fatal] GlobalExceptionHandlers: EPIC FAIL: Object reference not set to an instance of an object

System.NullReferenceException: Object reference not set to an instance of an object
at Microsoft.AspNet.SignalR.Infrastructure.Connection.ProcessCommand (Microsoft.AspNet.SignalR.Messaging.Command command) [0x00000] in :0
at Microsoft.AspNet.SignalR.Infrastructure.Connection.b__8 (System.Object state, Microsoft.AspNet.SignalR.Messaging.Message message) [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.MessageBusExtensions.Enumerate[Object] (IList1 messages, System.Func2 filter, System.Action2 onMessage, System.Object state) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.ProcessResults (MessageResult result) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.GetResponse (MessageResult result) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection+ReceiveContext.InvokeCallback (MessageResult result) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.MessageBusCallback (MessageResult result, System.Object state) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Infrastructure.Connection.<Receive>b__4 (MessageResult result, System.Object s) [0x00000] in <filename unknown>:0 at (wrapper delegate-invoke) <Module>:invoke_Task1__this___MessageResult_object (Microsoft.AspNet.SignalR.Messaging.MessageResult,object)
at Microsoft.AspNet.SignalR.Messaging.Subscription.Invoke (MessageResult result, System.Action1 beforeInvoke, System.Object state) [0x00000] in <filename unknown>:0 at Microsoft.AspNet.SignalR.Messaging.Subscription.WorkImpl (System.Threading.Tasks.TaskCompletionSource1 taskCompletionSource) [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.Subscription.Work () [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.MessageBroker.PumpImpl (System.Threading.Tasks.TaskCompletionSource`1 taskCompletionSource, ISubscription subscription) [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.MessageBroker.PumpAsync () [0x00000] in :0
at Microsoft.AspNet.SignalR.Messaging.MessageBroker.ProcessWork (System.Object state) [0x00000] in :0

Does the CPU spike up after that error? Haven’t seen that error cause any issues, it just logs periodically when running under mono and continues on. Though @r0b07 's error looks like its actually two.

@dmbkiwi - do you have the same 100% CPU issue?

Need some more information as well:
What version of drone?
What version of Ubuntu?
What version of mono?

For next time:
Its much easier to see logs that are posted on pastebin as opposed to the body of the reply, it also keeps the length of the thread down.

@markus101 Yes, I have the same CPU issue.

I’m running nzbdrone v2.0.0.855 on OpenSuse 12.3. This is the mono version:

Mono JIT compiler version 3.0.4 (tarball Thu Mar 7 10:35:21 UTC 2013)
Copyright © 2002-2012 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: x86
Disabled: none
Misc: softdebug
LLVM: supported, not enabled.
GC: Included Boehm (with typed GC and Parallel Mark)

I’ve run into this problem on ubuntu 13.10 with mono v2 and compiled from source v3.2.3 and nzbdrone 740 through 854 (i update a lot since i don’t mind bugs or work in progress).

When i had some spare time i ran a profiler on the process to see what was up during the spike, here are the results (I forget how long of a time period i had the profiler running): http://pastebin.com/Nx2DvSkD

So i went and checked dispose for a signalr subscription: https://github.com/SignalR/SignalR/blob/master/src/Microsoft.AspNet.SignalR.Core/Messaging/Subscription.cs

I’m not totally sure how SignalR -> nzbdrone works yet (haven’t had the time to look through it all), but hopefully this can give you a starting point as to what’s wrong.

Either way, i recompiled signalr.core.dll with a thread.sleep(100) instead of the spinwait and haven’t had an issue in 24hours:
http://imgur.com/R94Ani2 (7day graph, those spikes to the left are earlier in week the mono nzbdrone process at 100%)

My setup is a bit out of the ordinary because i have nzbdrone and sabnzbd behind an nginx server, so this may have been part of the cause due to timeout settings on the passthrough and thought the cpu @ 100% was just me.

Will continue to keep an eye on it and notify back with more info if needed.

@Moffisaurus thanks alot for the info, I’m fairly certain you found the issue. looking at the code, and I just want to shake my head.

we are gonna fork signalr to fix this issue and some other issues we are seeing in *nix environments.

Awesome, look forward to having an updated NZBDrone with thread.sleep();

Moffisaurus, in the meantime, is there any way to get the updated signalr dll from you? Have been trying to build my own, but not at all familiar with C# and NuGet is driving me a bit bonkers.

this issue should be fixed in 2.0.0.880 or later, let us know if you still see it happening.

Thanks to @Moffisaurus tracking it down.

Awesome. Superfast turnaround, I love it.

Another donation coming your way.

Just downloaded latest release, and issue seems to be fixed. Thanks very much.