Posts
My blog
Cancel

Unified Streaming Pipeline: Intelligent Multi-Source Deduplication with APPLY CHANGES

One of the most challenging problems I’ve solved recently was figuring out how to collate events arriving from multiple source systems, each with its own delivery pattern, format, and reliability m...

Performance Issues with Parallel Branching in Delta Live Tables

I’ve worked with Databricks for more than a decade now, and it amazes me how it evolved exponentially from a simple Spark workspace into a full platform for data engineering, ML, analytics, and now...

Trip to Paris

The first non-weekend trip while staying in Amsterdam was to the City of Lights, Paris!! An early morning Thalys from Amsterdam Central dropped us in over 3 hours at the Gare du Nord station. Our ...

Effects

Effects Any program that does anything useful has some side-effect. Well, the whole point of a program is to have some side-effect. Examples of side-effects are reading from console, writing to db,...

Stateful processing with Kafka

Kafka is a distributed streaming platform storing data in topics and providing scalablity with topic partitions. Even though Kafka is extensively used for connecting our microservices through publi...

Implementing a Health Check Mechanism for Kafka

Kafka is a well known distributed streaming platform that stores data in topics and provides scalability with topic partitions. At Veon, we use Kafka extensively to publish and consume events from...

State Monads

Scala is a OOP language with full support for functional programming and beautifully mixes them together to bring the best from both the worlds. Pure Functions In functional paradigm, all your fun...

Variance in Scala

There are different types of polymorphism in Scala. Inheritance, Parametric polymorphism (Generics in Java), etc. We are concerned about Parametric Polymorphism in this post and a classic example w...

Diving into Scala's Typeclasses

Typeclasses are one of the most beautiful ways of extending existing classes and building new features and functionality over them. It is always considered a good practise to evolve programs/syste...