A Big Data Spark engineer spends on an average only 40% on actual data or ml pipeline development activity. Most of their time is often devoted to managing the clusters or optimizing the spark application variables. As an alternative to this monolith set up of clusters is the cloud based…

With increasing number of organizations adopting a Multi-Cloud Strategy, data movement has been a point of contention for the Analytics teams. Data sprawl and the egress costs of the host cloud providers are some serious side effects of this. Independent Software Vendors (ISVs) who develop a multi-tenant Data Platform for…

Disclaimer: Views, thoughts, and opinions expressed in the blog belong solely to the author, and not necessarily to the author’s employer, organisation, committee or other group or individual Organizations increasingly look at BigQuery as a single source of truth for their data. BigQuery is increasingly becoming the metaphor for a…

Disclaimer: Views, thoughts, and opinions expressed in the blog belong solely to the author, and not necessarily to the author’s employer, organisation, committee or other group or individual Denormalization is a key feature of BigQuery. It reduces table joins and speeds us the query execution. Some call it a table…

Spark on EMR Data Pipeline with Apache Airlfow Large enterprises with big on-premise footprint of Hadoop clusters powering Spark, prefer Amazon EMR as their cloud migration solution. This lift and shift approach ensures, the developers and the end user community do not get hazzled by new ways of doing things…

Launch Apache Airflow without provisioning or managing servers on AWS in less than 30 mins! (This artcile assumes you have a working knoweldge of Apache Airflow. Keep a sample / hello world kind of Airflow Dag ready before proceeding with the setup) While Apache Airflow is perhaps the most used…

Best Practices for designing & developing Data Pipelines using Apache Airflow Apache Airflow is a thing of beauty when it comes to designing and developing Data Pipelines especially for Big Data loads. Data Engineering has now transcended to Data Ops and this space has more customized tool offerings for just…

( This article assumes readers have working knowledge of Apache Airflow and Dockers) When it comes to building data pipelines Apache Airflow works like a charm. With its robust set of operators Data Engineers can now integrate a wide variety of external data sources to their Internal data systems. …

Renaissance in Indian IT Education Every now and then we come across those studies and reports which find only about 10% of Indian Engineering Graduates to be having employable skills. …