General best practices: Project Structure: Generally, each developer in a team has his own preferences of project structure. Which makes it difficult to understand the project for other developers and the same structure for all projects in account brings the consistency, helps fast onboarding new developers into the team. So, it is always…

Many developers believe writing tests will be time consuming and not worth it but I have a different perception here, yes I agree at the initial time of project development will be time consuming when compared to without writing tests but imagine over time the project becomes bigger and bigger…

In spark, reading a json file is pretty straightforward but constructing a schema for complex json data is challenging especially for newbies in spark. In this blog I will try to provide some tips to make it simple. How to create schema: In spark, Dataframe schema is constructed using a struct object. A struct…

Python is a popular programming language for Apache Spark users, and Google Bigquery is one of the most powerful serverless analytical database service. In this blog, I share a step by step approach to setup Pyspark to connect with Google Bigquery. System Requirements Java 11.0.8 or Java 1.8 Python 3.7.7 Pyspark 3.1.1 …