Ubiqiti UniFi recovery

When I noticed by UniFi controller had gone offline I knew this would be an adventure. As I dug into the Ubuntu server I had it running on it became apparent that the hard drive that it was installed on was failing.

As is the case too often this was the exact moment I started wishing I had done a recent backup. Well not to worry, after fixing the mongodb using this guide from Ubiquiti I was able to get the Unifi controller running again. The system and hard drive were limping along but functional enough to try and offload the configuration.

Now the frustration began. I attempted to create the backup via the UniFi GUI. Ubiquiti guide on this. But all attempts resulted in no backups being created, and no download of the backup starting. Despite the friendly message stating that the "download would begin shortly".

By following steps to increase logging I was able to see a successful backup was in fact being created

/usr/lib/unifi/logs/server.log
[2022-02-25T12:59:05,595] <webapi-20> DEBUG api    - /api/s/default/cmd/backup finished (3 handling, 0 rendering)                                                     
[2022-02-25T12:59:27,460] <webapi-29> DEBUG api    - /api/s/default/cmd/backup finished (16 handling, 1 rendering)                                                    
[2022-02-25T12:59:35,307] <webapi-20> DEBUG api    - /api/s/default/cmd/backup finished (1 handling, 1 rendering)                                                     
[2022-02-25T12:59:35,406] <backup> INFO  system - [server backup] started  
[2022-02-25T12:59:35,409] <backup> INFO  db     - Connecting to mongodb://localhost:27117                                                                             
[2022-02-25T12:59:35,419] <backup> DEBUG db     - export ...heatmap          
[2022-02-25T12:59:35,431] <backup> DEBUG db     - export ...voucher              
[2022-02-25T12:59:35,433] <backup> DEBUG db     - export ...networkconf
[2022-02-25T12:59:35,436] <backup> DEBUG db     - export ...rogueknown
[2022-02-25T12:59:35,437] <backup> DEBUG db     - export ...traffic_rule
..... Snip
[2022-02-25T12:59:35,601] <backup> DEBUG db     - export ...stat_daily
[2022-02-25T12:59:35,611] <backup> DEBUG system - zipDir: timestamp
[2022-02-25T12:59:35,612] <backup> DEBUG system - zipDir: format
[2022-02-25T12:59:35,613] <backup> DEBUG system - zipDir: db_stat.gz
[2022-02-25T12:59:35,613] <backup> DEBUG system - zipDir: db.gz
[2022-02-25T12:59:35,620] <backup> DEBUG system - zipDir: sites/
[2022-02-25T12:59:35,622] <backup> DEBUG system - zipDir: sites/default/
[2022-02-25T12:59:35,622] <backup> DEBUG system - zipDir: sites/default/config.gateway.json
[2022-02-25T12:59:35,623] <backup> DEBUG system - zipDir: system.properties
[2022-02-25T12:59:35,624] <backup> DEBUG system - zipDir: version
[2022-02-25T12:59:35,631] <backup> INFO  system - [server backup][BACKUP] end
[2022-02-25T12:59:35,634] <backup> INFO  event  - [event] Backup has been created

But WHERE is the backup being created. It is certainly not getting downloaded in the browser, but could it be on disk somewhere? I ran a find to look for files with new modified times around the time of the backup. Using find to look for modified files in the last 120 minutes proved to be the ticket. I excluded the db path since this is the working directory for mongodb. Many false positives are in that folder since the DB is continuously updating.

 find /usr /var  -mmin -120  -not -path "/var/lib/unifi/db/*" 2> /dev/null 
/usr/lib/unifi
/usr/lib/unifi/replay_pid6141.log
/usr/lib/unifi/replay_pid7807.log
/var/lib/unifi
/var/lib/unifi/firmware.json
/var/lib/unifi/backup/6.5.55.unf
/var/lib/unifi/backup/meta.json
/var/lib/unifi/db
/var/lib/unifi/model_lifecycles.json
/var/lib/unifi/system.properties
/var/lib/unifi/system.properties.bk

sure enough the /var/lib/unifi/backup/ had several files that looked like backups.

I decided it would be best to start fresh on a new system for my UniFi controller. Once setting up the new controller I was given the option of creating a new site, or restoring from a backup. Fingers Crossed.

I uploaded the backup file and nothing happened. GRRRRR. OK maybe its dumb and needs the .unifi extension. This did not work either. What is going on here, no error, nothing. (meant to screenshot this be forgot to).

Chrome inspector tools to the rescue. The javascript XHR request for uploading the backup file returned a 40* error. Says clearly the file is "too large". Its disappointing that Ubiquiti would not render this error in the webpage.

No fear. I realized I used the "last 7 Days" option when I generated that backup

I re-ran the backup using "settings only" and pulled it from the backup folder on the mostly broken controller

This time the new controller took the file and successfully restored all my settings.

Interestingly enough even on the new controller I can still not get the download backup button to work. If I cared enough I would start looking into the API a bit more. It looks like the backup is triggered via an API call

post to /api/s/default/cmd/backup

{cmd: "async-backup", days: 0}
cmd: "async-backup"
days: 0

My new system in vitalized so I suppose my backup will be in the form of regular snapshots and archival of said snapshots on a separate server.