Data Skipping logic constitutes an integral part of Advanced Table formats storing huge data sets. Read this blog to understand how Z order optimization helps these formats to achieve it effectively when predicate patterns are not known prior. — Data skipping at source is very important for executing queries against large data tables based on advanced table formats, such as Delta and Iceberg. …

Skewed Joins lead to stragglers in a Spark Job bringing down the overall efficiency of the Job. Here are the five exclusive tips to address Skewed Joins in different situations. — Joins are the one of the most fundamental transformations in a typical data processing routine. A Join operator makes it possible to correlate, enrich and filter across two input datasets. The two input datasets are generally classified as a left dataset and a right dataset based on their placing with…

Use of Coalesce in Spark applications is set to increase with the default enablement of ‘Dynamic Coalescing’ in Spark 3.0. Now, you don’t need to do manual adjustments of partitions for shuffles any more, nor you would feel restricted from ‘spark.sql.shuffle.partitions’ value. Read this story to know more about it.. — Importance of Partitioning: Right set of partitions is like a holy grail for optimum execution of a Spark application. A Spark application achieves optimum efficiency when each of its constituent stages execute optimally. This in turn implies that each of the stage should run on optimum number of partitions which…

Linearizability And Serializability together constitute the gold standard for consistency in distributed databases. Based on this gold standard, one can easily evaluate the consistency promises of various distributed database solutions. — Before we look Linearizability And Serializability together, Let us first understand these two concepts in isolation: Linearizability: A data storage system depicts linearizability when for a set of operations, being performed on a single data object, two conditions are satisfied. First condition states that the execution is atomic for each…

Spark has provided two very important transformations, viz., Map and MapPartitions, for developers, to accomplish certain custom data processing scenarios. However, due to overlap in their capabilities, there always remains some ambiguity, among the developers, to rightly choose between Map and MapPartitions… — Map and MapPartitions, both, fall in the category of narrow transformations as there is one to one mapping between output and input partitions when both gets executed. Also, both the transformations allow developers to code custom data processing logic on typed records. However, from functionality perspective:

Stragglers in your Spark Application affect the overall application performance and waste premium resources. — Stragglers are detrimental to the overall performance of Spark applications and lead to resource wastages on the underlying cluster. Therefore, it is important to identify potential stragglers in your Spark Job, identify the root cause behind them, and put required fixes or provide preventive measures.

Filtering a Spark Dataset against a collection of data values is commonly encountered in many data analytics flows. This particular story would explain four different ways to achieve the same. — Let us assume there is a very large Dataset ‘A’ with the following schema: root: | — empId: Integer | — sal: Integer | — name: String | — address: String | — dept: Integer The Dataset ‘A’ needs to be filtered against a set of employee IDs (empIds), ‘B’…

This story is exclusively dedicated to the Join operation in Apache Spark, giving you an overall perspective of the foundation on which Spark Join technology is built upon. — Join operations are often used in a typical data analytics flow in order to correlate two data sets. Apache Spark, being a unified analytics engine, has also provided a solid foundation to execute a wide variety of Join scenarios.

Partitioning is one of the basic building blocks on which the Apache Spark framework has been built. Just setting the right partitioning across various stages, a lot of spark programs can be optimized right away. — I encountered Apache Spark around 4 years back, and since then, I have been architecting Spark applications that are meant for executing complex data processing flow on massively sized multiple data sets.