Starting afresh

3/9/2017

Yesterday upon restarting this server everything went to hell. OOM killer began killing processes regardless of the >1gb ram available and no in-bound or out-bound connections were working. I don't know at what point I screwed things up and the uptime on the system was >200 days. Instead of hunting down the problem, I decided reinstalling the operating system was the best choice.

I spent the entirety of yesterday setting up my email server, IRC, ZNC etc and only today am getting around to doing the small things.

This is the first time I've ever needed a backup, and thankfullt my prior preperations for a disaster scenario paid off.

Things lost
a few hours of email
a few hours of IRC logs
months of Slack logs

Argh, the Slack logs, this annoys me the most - seeing as it was a somewhat recent addition to my backup routine I hadn't put them in a directory that is included in my system backup (I don't let the backup script backup the backups, recursion to the Nth degree) and this is where I stored my MongoDB dumps. I feel like an idiot, but, in the scheme of things this is a very minor loss. I manged to get my entire email collection and IRC log restored, I am very glad about that.

This was an amazing learning experience and allowed me to test my backup system in a realistic scenario, and all things considered, it worked really well. Now to improve it.

I do have the raw /data/db MongoDB directory however, if I can learn how to restore it from these files, I will be a happy man.

Later edit: Well, i'm dumb, the mongo databases were never located in /data/db, so I considered them lost. Today I thought that maybe my default db location was different, and indeed it was. So now I have everything and have put my logging script back on hourly. I lost an entire month of logs, but oh well.