Strangest thing happened to my drone install yesterday

sdjaime · September 28, 2014, 3:10pm

Running Ubuntu 14.04 with develop branch; mono 3.8. Been running smoothly with this new mono version for a few weeks now.

Yesterday the autoupdate process to 2051 kicked off around 2pm. Log below.

Database	Database Compressed	6:18pm
Database	Vacuuming database	6:18pm
Database	Database Compressed	2:05pm
Database	Vacuuming database	2:05pm
InstallUpdateService	NzbDrone will restart shortly.	2:05pm
InstallUpdateService	Starting update client /tmp/nzbdrone_update/NzbDrone.Update.exe	2:05pm
InstallUpdateService	Preparing client	2:05pm
BackupService	Starting Backup	2:05pm
InstallUpdateService	Update package extracted successfully	2:05pm
InstallUpdateService	Extracting Update package	2:05pm
InstallUpdateService	Update package verified successfully	2:05pm
InstallUpdateService	Verifying update package	2:05pm
InstallUpdateService	Downloading update 2.0.0.2051	2:04pm

But around 6pm, when I went to connect to drone, it wasn’t running. Tried to restart the service, but when I looked in /opt/NzbDrone, it was empty, minus 2 folders (UI and Update, I think). Tried to run an apt-get install, but it said drone was already installed. So I moved my config folder, did a remove and then an install, chmod 777 on the /opt/NzbDrone directory (else updates won’t work–on an aside, is there a better way to go about this?), launched drone, updated to newest develop, then copied my config folder back.

All seems well again, but it appears that the 2pm update failed somehow. Any guesses as to what may have caused it? I did have to reboot the server yesterday, but the chance of that happening during the short autoupdate window seems small.

markus101 · September 28, 2014, 3:41pm

If drone is running via upstart it may have tried to restart it and messed up the update process. What does the update log from that time contain?

sdjaime · September 28, 2014, 5:07pm

Ah! Didn’t even know there was an update log. Makes sense.

Here is the portion that looks like the culprit:

14-9-27 14:05:04.6|Info|BackupAndRestore|Attempting to rollback upgrade    14-9-27 14:05:04.6|Debug|DiskProviderBase|Copy /tmp/nzbdrone_update/nzbdrone_backup/ -> /opt/NzbDrone
    14-9-27 14:05:04.6|Debug|DiskProviderBase|Copy /tmp/nzbdrone_update/nzbdrone_backup/NzbDrone.Update -> /opt/NzbDrone/NzbDrone.Update
    14-9-27 14:05:04.6|Debug|DiskProviderBase|Copy /tmp/nzbdrone_update/nzbdrone_backup/NzbDrone.Update/Exceptron.Client.dll -> /opt/NzbDrone/NzbDrone.Update/Exceptron.Client.dll
    14-9-27 14:05:04.6|Info|StartNzbDrone|Starting NzbDrone
    14-9-27 14:05:04.6|Info|StartNzbDrone|Starting NzbDrone.exe
    14-9-27 14:05:04.6|Debug|ProcessProvider|Starting mono /opt/NzbDrone/NzbDrone.exe --nobrowser
    14-9-27 14:05:04.6|Fatal|UpdateApp|An error has occurred while applying update package.
    
    System.UnauthorizedAccessException: Access to the path "/opt/NzbDrone/NzbDrone.Update/Exceptron.Client.dll" is denied.
      at System.IO.File.Delete (System.String path) [0x00000] in <filename unknown>:0 
      at System.IO.FileInfo.CopyTo (System.String destFileName, Boolean overwrite) [0x00000] in <filename unknown>:0 
      at (wrapper remoting-invoke-with-check) System.IO.FileInfo:CopyTo (string,bool)
      at NzbDrone.Common.Disk.DiskProviderBase.TransferFolder (System.String source, System.String target, TransferAction transferAction) [0x00000] in <filename unknown>:0 
      at NzbDrone.Common.Disk.DiskProviderBase.TransferFolder (System.String source, System.String target, TransferAction transferAction) [0x00000] in <filename unknown>:0 
      at NzbDrone.Common.Disk.DiskProviderBase.CopyFolder (System.String source, System.String destination) [0x00000] in <filename unknown>:0 
      at NzbDrone.Update.UpdateEngine.BackupAndRestore.Restore (System.String target) [0x00000] in <filename unknown>:0 
      at NzbDrone.Update.UpdateEngine.InstallUpdateService.Start (System.String installationFolder, Int32 processId) [0x00000] in <filename unknown>:0 
      at NzbDrone.Update.UpdateApp.Start (System.String[] args) [0x00000] in <filename unknown>:0 
      at NzbDrone.Update.UpdateApp.Main (System.String[] args) [0x00000] in <filename unknown>:0

sdjaime · September 28, 2014, 5:10pm

Forgot to mention… yes, I am running via upstart.

markus101 · September 28, 2014, 7:21pm

Yeah, that looks like the file was in-use, upstart being too good at its job. The only work around right now is to use apt-get to install and not use drone’s built in updater.

sdjaime · September 29, 2014, 2:43am

No problem. It’s only happened once to me, so I might just take my chances keeping things the way they are.

Curious though, is there a better way to recover from that situation in the future?

Also, is there a way to be on the develop branch using the apt-get method?

Thanks!

zero_day · September 29, 2014, 5:34am

same was happened here when I discovered this morning that my whole folder was empty.
also in combination with the upstart …

markus101 · September 29, 2014, 6:14am

Yeah, change the sources URL to develop instead of master. You’ll have to edit sources.list (you can get the full path from the wik, see the installation page).

Dennis · September 29, 2014, 11:50am

Could that also be the reason why the folder gets empty?
I run development
I have the master in apt-get and I noticed that the master was updated although I was running develop already.
seems wierd.

I probably will update the sources list cause using 2 versions mixed is asking for troubles (I think).

markus101 · September 29, 2014, 1:37pm

Could what? Upstart? Upstart definitely doesn’t like the way drone updates, but without being able to sudo and stop the running version of drone, update then start it again there isn’t really a good way to do it at this time.

sdjaime · October 5, 2014, 8:07am

Well, it happened again tonight. So I’ve decided to disable the automatic updates, change the branch in the sources list to develop, and upgrade via apt-get. Shame, because I loved the push updates, but a non-self destruct drone is better

markus101 · October 5, 2014, 8:17am

You could script the update, drone has the ability to run a script instead of it’s built-in updater, but it would need to be able to stop drone via upstart, so would need to be able to sudo.

sdjaime · October 5, 2014, 3:13pm

Any downside to using the init script method instead of upstart?

markus101 · October 5, 2014, 4:26pm

No automatic recovery if it crashes, but then again it won’t try to recover during an update.

sdjaime · October 5, 2014, 7:22pm

Ehhhh, nothing but problems. Tried to switch over to the Ubuntu init.d script on the wiki, but I kept running into issues (pid issues, bad syntax on the home directory, etc.). Even trying to tweak the script to fix an error as it popped up didn’t seem to help much.

Then I tried the Debian script. For whatever reason, every time I started up drone, it completely wiped out the nzbdrone config directory (even the auto backups… everything). And I hadn’t manually backed it up recently, so I had to start from a very old 2 month backup. Ugh. Then it started snatching a bunch of (what it thought was) missing episodes before drone had a chance to update its library. What a mess.

I’ll just stick to apt-get.