Not everyone accepts that the reason we put so much work into creating relational models is so we don't end up with a giant table Frankenstein'd together with each and every attribute an entity can have. Not easy to make this clear to folks who use Excel for their data needs and can get away with it .. to a point.
I find it's the same with dashboards in BI tools. It's going to perform incredibly slowly if we throw in every column that 'might' be needed to drill down into.
👋 Hey there, I am Waylon Walker
I am a Husband, Father of two beautiful children, Senior Python Developer currently working in the Data Engineering platform space. I am a continuous learner, and sha
I don't understand BI dashboards and the reason behind how they work. I have seen them take 30s to display 12 points on a line chart. A real example, I rebuilt it with vanilla js and a rest API in about an hour.... Under 100ms refresh with no optimization and very little effort.
I get that they make it really easy to click together a dashboard. They are great at that, and there is a place for it. I just don't get how they can fall over so easy in some cases.
Re: 'Why is it so slow you're stealing my life, I hate this tool!'
The dashboard is only one piece of the puzzle. When you're dragging and dropping fields around, the tool is calling the database behind the scenes to execute a SQL query.
It could be slow because ...
All the queue slots are taken in your database and you're waiting in a queue.
The database is being slammed by other queries outside of the BI tool world.
There are limits in place to how many rows can be returned.
There are timeout limits on the webserver side (depending on the tool)
You've dragged and dropped something that doesn't have anything to join on and it's trying to do a cross join.
Your report hasn't been cached so each time you drag and drop it's executing the query all over again.
BI dashboards and self service is tricky to set up since everyone wants everything 'just in case', tricky to manage and keep archiving off the old stuff and tricky to keep everyone educated.
In some cases it would be quicker for the Analyst team just to do each report rather than empower users with self-service.
I could talk about this all day but @alanhylands
does a better job at explaining it than I do
👋 Hey there, I am Waylon Walker
I am a Husband, Father of two beautiful children, Senior Python Developer currently working in the Data Engineering platform space. I am a continuous learner, and sha
very well said. I like that "self-service" "just in case". It seems like a very interesting approach. The real question is, is it worth weeks of setup from the Analyst for self service when the analyst could run it ad-hoc in a few hours. I often find my organization trying to be over prepared to click something through the dashboard for the meeting with the big boss. But in reality most of the ones going to the big boss end up "pre-rendered" into PowerPoint anyways.
Thanks for the recommended article, that was just too well written!
Oh, and before mentioned example used the same exact table as the BI tool. Felt like there was too much hidden behind the BI tool curtain to really point to what the issue really was.
I can see why Self-Service was the magic bullet. Free up the Analyst to do more 'value add' work and leave the 'give me a number' type requests that are easy to get to the Tool. The problem is the end user needs to:
be able to use it from a functional point of view
know where to go to get the numbers in the masses of reports
know the filters to add to make sure it is actually the right number
be surrounded by data governance to make sure nothing sensitive is shared or that crazy numbers aren't shared to the wide world ... or just the big boss and that URGENT power point presentation
The reality is there's a lot of work that goes into building models and educating the user. That is if they want to be educated at all.
The tool we use at my workplace has an API, it just hasn't been switched on yet. It will be interesting to see if that speeds things up and allows our end users to hit the tables directly .. sort of.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Not everyone accepts that the reason we put so much work into creating relational models is so we don't end up with a giant table Frankenstein'd together with each and every attribute an entity can have. Not easy to make this clear to folks who use Excel for their data needs and can get away with it .. to a point.
I find it's the same with dashboards in BI tools. It's going to perform incredibly slowly if we throw in every column that 'might' be needed to drill down into.
I don't understand BI dashboards and the reason behind how they work. I have seen them take 30s to display 12 points on a line chart. A real example, I rebuilt it with vanilla js and a rest API in about an hour.... Under 100ms refresh with no optimization and very little effort.
I get that they make it really easy to click together a dashboard. They are great at that, and there is a place for it. I just don't get how they can fall over so easy in some cases.
Re: 'Why is it so slow you're stealing my life, I hate this tool!'
The dashboard is only one piece of the puzzle. When you're dragging and dropping fields around, the tool is calling the database behind the scenes to execute a SQL query.
It could be slow because ...
BI dashboards and self service is tricky to set up since everyone wants everything 'just in case', tricky to manage and keep archiving off the old stuff and tricky to keep everyone educated.
In some cases it would be quicker for the Analyst team just to do each report rather than empower users with self-service.
I could talk about this all day but @alanhylands does a better job at explaining it than I do
Fighting The Fallacy That Everyone Wants (And Needs) Self Service BI
Alan Hylands ・ Oct 9 '18 ・ 4 min read
very well said. I like that "self-service" "just in case". It seems like a very interesting approach. The real question is, is it worth weeks of setup from the Analyst for self service when the analyst could run it ad-hoc in a few hours. I often find my organization trying to be over prepared to click something through the dashboard for the meeting with the big boss. But in reality most of the ones going to the big boss end up "pre-rendered" into PowerPoint anyways.
Thanks for the recommended article, that was just too well written!
Oh, and before mentioned example used the same exact table as the BI tool. Felt like there was too much hidden behind the BI tool curtain to really point to what the issue really was.
I can see why Self-Service was the magic bullet. Free up the Analyst to do more 'value add' work and leave the 'give me a number' type requests that are easy to get to the Tool. The problem is the end user needs to:
The reality is there's a lot of work that goes into building models and educating the user. That is if they want to be educated at all.
The tool we use at my workplace has an API, it just hasn't been switched on yet. It will be interesting to see if that speeds things up and allows our end users to hit the tables directly .. sort of.