Positioning Amazon Quicksight in the BI Stack : A point of view (Part -1)
When i started my career in BI more than a decade back, reporting consisted of two major players SAP Business Objects and IBM Cognos. ‘Reporting’ was the general phrase and ‘Dashboards’ and ‘Stories’ weren't much in use. It was usually connecting to Oracle or SQL Server and building reports, publishing to repositories or scheduling them to end users’ inbox.
Centralized Governance in the BI space was limited to account usage and was done the by the ‘Reporting admin’ team. They took care of all the installations, upgrades, migration etc.
Years later BI & Portfolio managers started facing a common problem. There was just too many reports in the system. Reporting migration projects usually had about 8k-9k reports to be migrated! There were frequent report refresh issues, out of memory issues, reporting portal down issues. The ungoverned,too many adhoc reporting,publishing in the portal were taking their toll. I still vividly remember many scheduled reports running to users’ inbox every day for years even after the users had left the organisation!
IT solution providers came up with solutions to govern the reports in the system. They came up with accelerators that could classify the reports into Hot,Warm & Cold based on their usage and retire the cold ones. I was part of the development of one such accelerator which my BI architect designed to identify users who were dormant and remove their access from the system, releasing the unused Licenses.
With the arrival of TIBCO Spotfire, Qlik, Tableau lot of glamour walked into the reporting space. ‘Dashboards’,’ Stories’, ‘Self-service’,’Analytics’ were the buzzwords. The data space also saw the rise of in-memory , nosql, big data systems. The traditional canned reports, with occasional bar ,line and pie charts were seen as legacy, there was more demand for Bubble charts, Tree maps, Heat Maps, Waterfall charts, Sanky diagram etc. Lot of the new BI tools had in-memory processing capacity which catered to these new gen visualizations. Such was the buzz for these fancy viz that in one of my BI interviews after seeing i had implemented Waterfall and Bubble charts in SAP BO, the interviewer just asked the scenario behind it and how i had implemented. Once he was confirmed that i had the hands on exp with these tools , he directly asked me to take the HR interview bypassing the next 2 rounds!
Along with new report development projects, there were a lot of projects requiring to migrate the reports to these new tools which in most cases led to redeveloping them.
Although a lot of processing and glamour had made the BI world more shiny, the issues of governance, report refresh issues, portal downs could still be seen. Shortage of resources in these tools was a big show stopper.
Past few years have almost seen an explosion in the BI space. With the arrival of open source tools such as Python,R,D3 and their libraries, users are now spoilt for choices! There is increasing need to show more in less space (per my experience on most occasions they dont convey much though!) Dynamic dashboards which are based on streaming source are increasingly seen as the symbols of technical excellence. Users increasingly ask for action driven dashboards where they can take actions through buttons from the visualizations. Integration of Programming frameworks in BI tools is now a new norm.
Let us face it, Pictures convey a thousand words, Visualizations are seen as a barometer of success of any BI or Big Data Analytics implementation, undermining the complex data integrations and transformations. They are therefore the ones that can make or break in a new implementation especially if it is in the POC stages. Analytics consultants therefore go the extra mile to make the visuals fancier, trying to accommodate a lot, making the BI tools like a genie which can simply yield to just about any request. Couple this with ‘Agile’ where users can now review the visuals as they are built and ask for changes, has made the life of developers very interesting!
This trend has led to many Analytics projects running the risk of overshooting timelines, not making the cut beyond the POCs or the initial analysis phase. The change creeps are now as much as 60%-70% and generally this leads to reducing the number of visuals than was planned earlier to somehow wrap up the projects and go-live!
In this visual chaos of increased tool options, needing more for less, lack of governance, dearth of multi-skilled resources and increasingly costly implementations, it is interesting to see how the AWS visual offering ‘Quicksight’ fits. Does it compound the chaos or make it clearer….wait for my next part of this article!
