The Three Levels of Tableau Support

Posted on January 12, 2017 by paulbanoub

Hi all,

Let’s talk a bit more about how to build a top Tableau support team. This post focuses on the support my team provides to our user base. At the moment we have just over 1000 Tableau Desktop users, and approximately 8000 active users on the server every month – that’s a lot of demand for our services.

Now users can be a pretty demanding bunch, with myriad questions, queries and problems. And we are busy. So how do we ensure that users get the level of support they need? Well we provide 3 levels of main support, with the objective being to ensure that the type of user query / issue is directed to a channel that gives it the appropriate level of attention. This ensures an efficient use of my team’s valuable time, and critically it cuts down the traffic to our email inbox which is always a good thing.

Some of the support options for our users

Level 1 – Man down!

Red alert! Something is busted and it needs to be looked at now! For this we need any incident to be logged in a trouble ticket system, with appropriate priority and detail. We use Service Now for this (many other tools available).

So if users think Tableau is broken or they need some immediate help then they log a ticket. This is mandatory. We need to track and log the progress, and the data is audited regularly. No ticket, no fix. We obviously don’t wait stubbornly for the ticket though, if there’s a big issue we investigate while the incident is logged.

Once the ticket is logged it flows through our regular support flow. First my Level 1 team will take a look and see if there’s an easy fix. If they can’t fix it then it’s an escalation to my more skilled Level 2 team, and then a potential escalation to my main Level 3 team for the trickier issues. There may be a future post coming about effective incident management, so I won’t go into detail here.

Some users don’t like us mandating that they raise an incident ticket. But it’s the only way to ensure traceability of problems.

Level 2 – It can wait

Sometimes users have problems or requests for assistance that are not so time sensitive. Maybe a development dashboard has broken, or someone needs help from the team to perfect that Pareto chart, or hey – maybe they just wanna talk about how much they love Tableau (it happens!)?

Book your appointment with a Tableau Dr.

That’s where a Tableau Dr. Session is needed. We dedicate 3 half days a week to Tableau Dr. Sessions. Users log onto our community page and can book their session from a list of available slots. If the next slot is in a couple of days then they have to wait to be seen. Providing this structure to the sessions is critical as it allows my team to keep control. Before we implemented the structured sessions we were getting peppered with do-it-now requests for Dr. Sessions. That meant my team was context switching all over the place and other projects were being impacted.

Providing structure also makes users understand this is a finite resource and thus they are more appreciative of this dedicated time with my Tableau experts.

Level 3 – Let’s talk about Tableau, baby

Next level of support is for general chat. Could be a question about functionality, or a point about performance, or a geeky joke, or someone just wanting to ask a question about our upgrade strategy – could be anything really.

That’s where our Lync Group Chat comes in. We’ve generally got a couple of hundred users on the chat channel at any one time so it’s a decent forum for such questions and banter. It’s great for my support team to see a question get asked, and then before we have a chance to pick it up, another user has provided the answer – a self healing community – IT support nirvana!

Wanna chat Tableau? Use our Group Chat

What’s in it for me?

These support options ensure that each query gets an appropriate response. If it has all hit the fan, then we act quickly. If it needs more care and detail, then we book that time, and if it just needs someone to talk to then we’ve got a community of people ready to give that data hug. It also means we get hardly any emails. And email is a dreadful means of logging an issue, as there’s no traceability or feedback. Users only get annoyed when they feel a query is being ignored, and ensuring the correct channel for a query means users get feedback as appropriate and aren’t left wondering where that email question went to.

Also my support team can plan their work and aren’t constantly context switching, one of the biggest enemies of productivity.

So that’s it. Pretty simple to implement but mightily effective. As always, ping me if you want a more detailed run-through.

Happy vizzing, Paul

Tableau on Tour Keynote Speakers – Some Suggestions

Gallery

Posted on December 13, 2016 by paulbanoub

This gallery contains 32 photos.

Hi all, I love the Tableau Conference. But I also have a lot of fun at the smaller “Tableau On Tour” events. In particular I love the keynote speeches. We’ve had some crackers recently, with particular recent favourites being Tim … Continue reading →

Empowering Your Tableau Users With Makeovers & Proactive Support

Posted on November 21, 2016 by paulbanoub

Hi all,

More on building that dream Tableau Centre of Excellence function. I’ve previously posted about how to structure your support team and ways to build user engagement with “Tableau Champions”, this post focuses on how you can use Tableau’s introspection capabilities to deliver a more proactive support function.

What is proactivity?

The traditional definition of proactive is as follows. To me it means means seeing into the future and getting to an issue before it even happens. In the world of IT Support, proactivity really is the Holy Grail, meaning the difference between a good support function and an amazing one. But it’s super-hard to achieve, especially in the complex enterprise level setups that have multiple break points. You can almost never prevent something from breaking, no matter how good your monitoring is.

What you can do is add some proactivity into the way your team operates by identifying when your users are not getting the best from your service. In Tableau Server world we have the ability to spot the following and much more.

Slow Tableau visualisations
Consistently failing extracts
Stale content

I won’t go into how to achieve this, it’s the subject of a future post. But I’ll point you in the direction of these 2 posts that should get you on the way. Go check out Custom Admin Views by Mark Jackson and Why are my Extracts Failing – by Matt Francis.

I get my team to scan our admin views, to identify those users that in our opinion are not getting the best experience they can from Tableau. If we see someone who might be experiencing consistently slow visualisations, or have regularly failing extracts then we give them a call. Often the users won’t even have a complaint. But our message is “We think you’re not getting the best experience possible, and we want to make that happen”.

The initial reaction is often surprise. “I’m ok, I didn’t raise an issue” will be a common response. But then once we’ve worked with the user, and improved their experience, you’ll find they are blown away. You may even get a call from their management!

You’ll find this kind of service is very rare in most organisations so if you can deliver it, even sporadically, then you’ll be regarded very highly.

Makeovers

This is pretty simple, if a little time-consuming. Browse the Tableau content on your server. Spot something that doesn’t look great – it might be slow, not compliant with your best practices, or just fugly. Download that content, and give it a makeover. Make it look great, maybe add some improved functionality, make it nail best practices.

This is one of my team’s favourite activities due to the reaction of the user / client. They LOVE it. It really creates a sense of engagement, the user feels that your team actually really cares about them. We’ve also had our Tableau Champions participate in Makeovers which is even better as it saves my team some cycles.

Be careful though, some user content might be confidential and the user may not appreciate an admin poking around in their data. Also, remember that by doing this you are implying a criticism of their work, so handle the communication with care and sensitivity.

Also ensure that you don’t just change stuff and then drop it back on their laps. In a self-serve model like mine users develop and support their own content so it is crucial the user knows what you’ve changed, how you’ve changed it and what benefits you feel the modification brings. Pull them and their manager into a call, run through what you’ve done and then hand it back over to them to run with it.

These have been very successful in my organisation. Users truly appreciate the help and my team has fun doing it.

So there you are, a couple of tips for adding that gloss to your Tableau support service.

Cheers, Paul

Tableau Server – all about the… Backgrounder

Posted on May 5, 2016 by paulbanoub

Hello everyone.

You may know me as a Tableau Centre of Excellence manager. That can involve a lot of paperclip pushing skills, with the real work being done by my excellent team (thx @jakesviz & The Information Lab). But I do try and get down and dirty with my lovely Tableau Server environment to keep my skills fresh. Obviously I don’t mess with it – @jakesviz gets pretty protective about his Server.

This series of posts is my attempt to shed some light on the internals of Server. Note there are many more experts in this field than me (Craig Bloodworth, Mark Jackson, Jen Vaughan, Tamas Foldi, Mike Roberts, Angie Greenhaw – to name but a few) so please do comment if anything is incorrect here. Maybe you guys could help me evolve this post?

What is the backgrounder?

The backgrounder is a process that runs as part of the Tableau Server application. As the name suggests, it handles background tasks such as refreshing extracts, running subscriptions and also processes tasks initiated from tabcmd.

Here are the backgrounder processes. The .exe file and the .war file. The WAR file is a Web Application Archive, and contains all the necessary components and resources needed for a Web Application such as Backgrounder.

On a clustered environment you’ll find these files in D:\Program Files\Tableau\Tableau Server\worker.1\bin (may vary slightly with your installation).

Other files related to the backgrounder.

Template (.templ) files – These files TBD

There are also a few .rb files in D:\Program Files\Tableau\Tableau Server\worker.1\tabmigrate\db\migrate.

And we also have a .properties file which contains all the config entries relevant to the backgrounder. It also has almost all of the other stuff that you’d find in the main workgroup.yml file which is odd. I’d have expected it to be just the backgrounder config.

Here is the location of the backgrounder log files

Here are 2 instances of backgrounder.exe running on my server (from Task Manager).

Can I mess with it?

Backgrounder can be configured. There are several settings present in both workgroup.yml and backgrounder.properties. Workgroup.yml is the master config file, and it populates the backgrounder.properties (and other .properties files) when a ‘tabadmin configure’ is run.

I don’t know what all of these do (yet) and the only one I’ve ever edited is ‘backgrounder.extra_timeout_in_seconds‘ which sets the max time in seconds that a backgrounder session can run for. Tableau kills off the session if this threshold is reached. Useful for forcing users to optimise their extract times!

I also pay attention to the ‘backgrounder.vmopts‘ parameter, as this defines the size of the java heap space for this component. All components have a vmopts setting and I’ve had to increase them on occasion due to out of memory problems.

You may also want to change the ‘backgrounder.log.level‘ if you need more debug info, although Tableau logs are chatty enough for me.

If there’s a golden parameter in this lot that you get value from then let me know in the comments.

backgrounder.deploy.dir: D:/Program Files/Tableau/Tableau Server/data/tabsvc/backgrounder
backgrounder.external_cache.concurrency_limit: 10
backgrounder.external_cache.enabled: true
backgrounder.external_cache.num_connections: 1
backgrounder.external_native_query_cache_disable: true
backgrounder.extra_timeout_in_seconds: 1800
backgrounder.failure_threshold_for_run_prevention: -1
backgrounder.jdbc.wg.connections: 8
backgrounder.jdbc.wg.idle_connections: 4
backgrounder.log.dir: D:/Program Files/Tableau/Tableau Server/data/tabsvc/logs/backgrounder
backgrounder.log.level: info
backgrounder.native_api.log.level: info
backgrounder.out_of_date_schedule_minutes: 240
backgrounder.ping_dataengine.millis_to_wait: 5000
backgrounder.ping_dataengine.num_retries: 24
backgrounder.ping_services.num_retries: 10000000
backgrounder.ping_services.time_to_wait: 5000
backgrounder.purge_directories.directories: ""
backgrounder.querylimit: 7200
backgrounder.restart_interval_in_minutes: 480
backgrounder.restrict_serial_collections_to_site_level: true
backgrounder.search_index_verification.enabled: true
backgrounder.sheet_image_api.max_age: 240
backgrounder.sleep_interval: 10
backgrounder.sort_jobs_by_run_time_history_observable_hours: -1
backgrounder.sort_jobs_by_type_schedule_boundary_heuristics_milliSeconds: -1
backgrounder.timeout_tasks: refresh_extracts, increment_extracts, subscription_notify, single_subscription_notify
backgrounder.tomcat.threads: 4
backgrounder.urlprefix: backgrounder
backgrounder.vmopts: -XX:+UseConcMarkSweepGC -Xmx512m

Note that Tableau Support don’t like you to edit config files manually, they recommend that you use the tabadmin set commands to change any parameters. They might have to change that recommendation when we see Tableau Server on Linux.

For more about .templ, Ruby & properties files check out this from Tamas Foldi.

What are the problems with the backgrounder?

Here are some of the things that can be problematic with the backgrounder.

Single Threaded – This means the backgrounder process can only run one thread at a time, a thread being a set of executable instructions that a process can perform. The upshot of this is that your backgrounder works through a queue of tasks one-by-one.

Latency – Due to the single threaded nature of backgrounder, you may see delays or ‘latency’. For example, if you have one backgrounder, and 2 tasks for it to perform at 2am, then task 2 will have to wait until task 1 has finished. If task 1 takes an hour then task 2 won’t start until 3am. This can be annoying for users that expect their data to be refreshed by a certain time.

Resource Intensive – The backgrounder can consume a significant amount of processing power (CPU) and input / output (I/O) on your server. This is dependent on the type of task it is performing. It’s not uncommon to see a backgrounder node consuming 100% CPU.

Other Stuff – The backgrounder process also does other stuff on Tableau Server that isn’t concerned with extract refreshes. For example – reaping extracts, checking disk space, synching Active Directory groups, rebuilding the search index etc. Bear that in mind when building your system. In reality these tasks don’t take up too much resource but they do take some and you should be aware.

Isolating the backgrounders

A common configuration in a clustered environment is to dedicate one of your worker nodes to the backgrounder processes. This means you can dial up the number of backgrounders and let them do their stuff without worrying about any impact on other processes. This is one of the most common performance recommendations from Tableau support.

You can also get a lot of info out of the Postgres DB relating to backgrounder usage and performance. Nelson Davis has posted a guide to getting started here.

Improvements I’d like to see

Ok so here are a bunch of improvements I’d like to see to the topic of backgrounders and extract management. I know some of this will drop in upcoming releases and some of these problems have been solved with custom solutions at some customer sites.

Alerting – Tableau Server doesn’t alert (email / IM / SMS) when a task fails. This means you’ll need to set up external monitoring to detect issues. I know Tableau are on this though so expect to see it in an upcoming version. Some people in the community have also coded their own solutions to this problem but it really should be native functionality.

Control per site – We segregate our dev / test / prod user environments using sites , all on the same server. We run 8 backgrounder processes on that server, which are shared across the tasks on all sites. As an administrator I’d really like to be able to bind backgrounder processes to specific sites. For example, 2 backgrounders on each of dev & test sites, then the other 4 dedicated to the production site. That would ensure production tasks always have enough resource to be able to execute on time.

Control per process – I’d like to be able to stop / pause / mess with individual backgrounder processes easily. It is possible – see this from Toby Erkson, but it would be good to have this as part of an administrator console or something.

Control per type / size / pattern of extracts – It would be good if I could dedicate specific backgrounder processes to particular extracts based on their characteristics. In particular I’d like to allocate one backgrounder to all the extracts that take less than 1 minute to complete. Or even use this to reward users that show diligence with their extract management by dynamically prioritizing incremental refreshes or extracts that have a low failure rate.

Better metrics – I’d like to see exactly how much CPU is taken up by a particular backgrounder process or task / schedule or per project. This would be useful for chargeback.

Dynamic reprioritisation – I love all my users. But in particular the ones that take good care of their extract refreshes. I’d like Tableau to be able to dynamically increase the priority of tasks that complete quickly, are incremental and that have a low failure rate. The message being, if you want your stuff to get the best slice of available resource then help us out with best practice.65

Disable run now – We’ve had some issues with the “run now” option that allows users to kick off an extract refresh on-demand using the UI. In particular we’ve seen some trigger-happy users bring our server down by hammering on the run now option multiple times. I’d like to disable that or maybe throttle it somehow.

Better guidelines from Tableau – The documentation from Tableau isn’t great in this area.

I have an 8 core 128GB server and run 8 backgrounders with no capacity issues. And I know of other organisations running way more than that. According to this doc I should be running between 2-4. That would be some serious under-utilization of the server. I’d like to see some clearer recommendations, maybe taking into account the variety of use cases that I’ve seen in other big enterprise deployments.

OK that’s all for now. This post could have been more detailed but I figure that I’ll get some valuable inputs from the community that will help me expand it. Actually in the time I was writing this, Mike Roberts was doing the same – check his post for some excellent info.