Another update: The Visual Studio debugger didn’t give me more insights, but I moved on to low level debugging with the native debugger, then stuff started to get interesting.
There’s a huge amount of memory fragmentation (one of the things we knew could cause OOM exceptions, but didn’t expect it to actually be the case).
Nerd Warning: Technical content ahead
Here’s the memory allocation from a locally running Sonarr on my own machine:
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 141 6ed9f000 ( 1.732 GB) 86.61%
Image 640 a22b000 ( 162.168 MB) 59.12% 7.92%
<unknown> 575 38f2000 ( 56.945 MB) 20.76% 2.78%
MappedFile 26 1de3000 ( 29.887 MB) 10.89% 1.46%
Stack 60 1340000 ( 19.250 MB) 7.02% 0.94%
Heap 20 598000 ( 5.594 MB) 2.04% 0.27%
Other 7 44000 ( 272.000 kB) 0.10% 0.01%
TEB 20 32000 ( 200.000 kB) 0.07% 0.01%
PEB 1 3000 ( 12.000 kB) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free f4aa000 41f76000 ( 1.031 GB)
Image 51773000 e7c000 ( 14.484 MB)
<unknown> 3c8d000 f63000 ( 15.387 MB)
MappedFile 1814000 113c000 ( 17.234 MB)
Stack d10000 fd000 (1012.000 kB)
Heap 7b33000 13c000 ( 1.234 MB)
Other 7ef40000 23000 ( 140.000 kB)
TEB 9c4000 3000 ( 12.000 kB)
PEB 9c1000 3000 ( 12.000 kB)
Two things: Usage Summary->Free
is 1.7GB, so there’s alot of free space (divided amoung 141 continuous regions). And Largest Region by Usage>Free
is 1.031GB, so the largest piece of available free memory is over a gigabyte.
Now, this is from your memory dump:
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 25784 5e16c000 ( 1.470 GB) 73.51%
<unknown> 26277 e0a6000 ( 224.648 MB) 41.41% 10.97%
Image 967 bc94000 ( 188.578 MB) 34.76% 9.21%
Heap 579 6f76000 ( 111.461 MB) 20.55% 5.44%
Stack 57 1180000 ( 17.500 MB) 3.23% 0.85%
Other 9 40000 ( 256.000 kB) 0.05% 0.01%
TEB 19 13000 ( 76.000 kB) 0.01% 0.00%
PEB 1 1000 ( 4.000 kB) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 0 10000 ( 64.000 kB)
<unknown> 2170000 1498000 ( 20.594 MB)
Image 64d93000 e7b000 ( 14.480 MB)
Heap 6a12d000 be3000 ( 11.887 MB)
Stack 7520000 fc000 (1008.000 kB)
Other 7f770000 23000 ( 140.000 kB)
TEB 7f549000 1000 ( 4.000 kB)
PEB 7f79f000 1000 ( 4.000 kB)
Same two statistics: Usage Summary->Free
is 1.47GB, so there’s alot of free space, but it’s divided amount a whopping 25784 regions. And Largest Region by Usage>Free
is 64 KB, so the largest piece of available free memory is virtually nothing.
This is what’s causing the Out of Memory exception, it’s not really running out of memory… it just doesn’t have a big enough continuous piece of it.
So lets dive a little bit deeper, a small piece of the entire memory allocation table:
+ 70f40000 70f41000 1000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE <unknown> [................]
+ 70f41000 70f50000 f000 MEM_FREE PAGE_NOACCESS Free
+ 70f50000 70f51000 1000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE <unknown> [................]
+ 70f51000 70f60000 f000 MEM_FREE PAGE_NOACCESS Free
+ 70f60000 70f61000 1000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE <unknown> [................]
+ 70f61000 70f70000 f000 MEM_FREE PAGE_NOACCESS Free
+ 70f70000 70f71000 1000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE <unknown> [................]
+ 70f71000 70f80000 f000 MEM_FREE PAGE_NOACCESS Free
+ 70f80000 70f81000 1000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE <unknown> [................]
+ 70f81000 70f90000 f000 MEM_FREE PAGE_NOACCESS Free
+ 70f90000 70f91000 1000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE <unknown> [................]
+ 70f91000 70fa0000 f000 MEM_FREE PAGE_NOACCESS Free
As you can see, every 0x10000 (64kbyte) there’s a 0x1000 (4kbyte) block of memory allocated (MEM_COMMIT) for ‘something’.
It’s kind of like throwing needles all over the floor, it won’t cover every inch of the floor, but no matter where you plant your foot, you’re gonna step on one of em.
Anyway, that ‘something’ could be bitdefender, or something else. It’s possible to find out what allocated that region of memory, but not from the dump I have atm. (By default, memory allocations aren’t ‘tracked’)
Btw. those 4kbyte blocks of memory consist of all zeroes.
It’s definitely not Sonarr, but could be a native library Sonarr uses, my bet is still on bitdefender though.