Time for another post about Tableau Server and how to get the best out of it in a large-scale, enterprise deployment situation.
Today we are focusing on how to PROPERLY back up your Tableau Server installation.
Like many aspects of enterprise services, this is a simple concept, but one that if you get wrong, can spell disaster. It always amazes me how many people / organisations don’t do this properly or even at all.
You know how annoyed you get when your mum tells you she isn’t backing up all her family photos – well that’s what I get like when I see IT systems neglecting backups.
Note this post refers to a standalone Tableau installation with a manual failover to DR. We don’t yet have a clustered environment. I’ll update the post with considerations when we implement that.
What’s a backup?
Seems a simple question, and there are a number of different types of backups that you can take, each useful in different situations. Here’s what I’ve got in place:
Full System Backup
This is a complete dump of the server filesystems to disk (or tape – there’s still plenty of tape backup infra out there). Most likely it will be one of the big vendor products that look like the mothership from Close Encounters of the Third Kind.
Your full system backup should be set up by your server team when you get your machine. However, the principle of “trust no-one” applies here as always and it’s up to you to check the following:
- Have the backups been set up at all?
- Are they backing up all the filesystems? – Many times I’ve seen that only operating system partition backups have been set up, and I’ve had to request the application partitions be included.
- Have the backups been succeeding? – Get your backup team to send you a report every month of backup completion. They don’t always succeed and you probably won’t be told that there has been a failure.
- If you need to perform a restore, do you know the process and how long does it take?
If you get the okay on that then you’re good. But only as an insurance policy. Full system backups can take a long time to restore, and may only be weekly so you could end up losing data even if these are in place. It’s up to you to ensure you’re covered rather than rely on other teams doing things correctly.
Nightly Tableau Backup
There’s no excuse for not having this in place. It’s easy to set up and it is a case of when rather than if it saves your ass.
The tabadmin backup command gets Tableau Server to dump all content & configuration to a single .tsbak file. You don’t have to stop the server to do this and it doesn’t seem to impact performance too much while it is running so this should be the first backup you configure.
A simple script like this will do the job.
@echo OFF set Binpath="D:\Program Files\Tableau\Tableau Server\9.0\bin" set Backuppath="D:\Program Files\Tableau\Backups\nightly" echo %date% %time%: *** Housekeeping started *** tabadmin backup %Backuppath%\ts_backup_ -d timeout 5 tabadmin cleanup move "D:\Program Files\Tableau\Backups\nightly\*" \\\tableau_shr\backups\nightly\ echo %date% %time%: *** Housekeeping completed ***
The tabadmin backup command does the actual work here, dumping everything to a file. Always a good idea to run tabadmin cleanup afterwards to remove logs etc.
We run this script at a quiet time for the server (not that there is one in my global environment). We use the Windows Scheduler on the server but I’d recommend using a decent scheduler like Autosys or whatever your enterprise standard is as WTS is pretty poor.
IMPORTANT: You may have noticed the move command at the end there. That takes our newly created backup file and moves it OFF THE SERVER to a share drive accessible by my backup server. Why? Well what happens if you lose the server and your backup file is on it? You may as well have no backup. So move it somewhere else.
Update – this tip actually saved my ass this week when we lost our entire VM cluster (er.. hardware team – *cough* – what’s going on??) . We were able to failover to the backup server successfully. Going forward we will be soon implementing Tableau’s High Availability capability.
Do make sure you rotate your backup files with a script that deletes the old files or your share drive will fill up. I keep 4 days worth, just in case the current file is somehow corrupted – rare but can happen.
You may know I’m not a fan of running enterprise apps on Windows. I prefer Linux for a number of reasons that I’m not going to go into here. I know many users want Tableau Server on Linux, and the amazing Tamas Foldi has only gone and written it himself – so one day we may see it.
Anyway, with Windows apps I always build in a weekly application restart. In our case every Saturday morning. That involves a server reboot (to clean out any OS related temp stuff), application restart and a tabadmin cleanup. The tabadmin cleanup with the server stopped has the added bonus of clearing out the temp files (doesn’t happen when the server is running). These files can get pretty big so worth clearing out.
Virtual Machine Snapshots
If you’re running on a VM then you may be able to utilise the VM snapshot facility. Contact your VM admins for details. I’ve not needed to implement this but I know some that do. VM snapshots are super handy.
Do be aware that Tableau don’t seem to support this though..
Config File Backup
Sometimes it’s handy to just back up your Tableau Server config. I’ve got a script that grabs all the .yml and template files in my Server directory, zips them up and moves them off the server. Pretty useful to refer back to old config settings if you need to. Make sure you include workgroup.yml.
If you’re being really good then you’ll be checking your config files into a revision control repo like SVN.
Site Specific Backups
Tableau Server allows you to backup per site. This doesn’t give me much extra but I know in orgs that have lots of sites, or a site per team / business unit it can be very handy.
One thing that isn’t great about exporting a site is that the site is locked and inaccessible as the export is taking place. See Toby Erkson’s blog for more info on exporting a site.
Backup File Size & Duration
As your environment grows you’ll need to be mindful of the size of your backup file. Mine is around 16GB and takes well over an hour to write. Takes about 25 mins to restore. You’ll need to understand those numbers as your system matures.
Another variable that can affect backup time is the specification of your primary server. If your primary is low spec then you’re gonna get a longer time to write a backup. I don’t have any stats on that but I know it is true. Contact Jeff Mills of Tableau if you want more info on Weak Primaries & backup times.
Backup Your Logs
Less important this, but handy to do on a weekly basis is to zip up your logs. We have a much better solution for logfile management using Splunk – you’ll see a blog about that in the future.
The Most Important Bit – TEST YOUR BACKUPS
OK so you’re backing up like a man / woman possessed? Fine. You’re only as good as your last restore. So TEST your backups periodically. Files get corrupted and you don’t want to be discovering that your only backup is broken when you need it.
OK that’s it. Backups can save your life – don’t ignore them. Paranoia is king in IT!