1). “The irony of data: built for speed, stuck in traffic.”

“Data’s a trend-chaser. First a warehouse, then a lake, now a lakehouse. But behind the buzzwords, it’s always chasing the same thing: `CLARITY`.”

Data’s like that seasoned pro who’s seen every fad come and go, warehouse, lake, lakehouse, whatever’s next.

It doesn’t need a rebrand to stay relevant because it’s still solving the same timeless puzzle: making chaos make sense.

Humans, queries, entire companies still rely on it to turn raw noise into reason.

So , it’s old, but like vinyl or good coffee, that’s exactly why it never goes out of style.

2). The problem: today’s data chaos hiding inside modern warehouses.

“Modern context” in very easy words means how something fits or makes sense in today’s world, not in the past.

When we say “in the modern context,” it means we are looking at something based on how people live, think, and use new ideas and technology today. It’s about understanding how things fit in our present time, not how they were in the past.

Example:

In the past, very few women worked outside the home. But today, in the modern context, women have jobs and build careers. So society’s idea of women’s roles has changed to fit modern life.

Cloud warehouses made data cheap and fast → so people just started to dump everything in them.

The magnitude of data has exploded ; especially after 2020.

In 2020, the world generated around 64.2 zettabytes of data that’s 64.2 trillion gigabytes. Since then, global data creation has surged exponentially, fueled by the rise of AI, IoT, remote work, and digital media. By 2025, projections from DemandSage and Seagate estimate total data generation will reach 181 zettabytes nearly three times what it was in 2020, growing at an average rate of 22–26% per year.

Cloud data warehouses made storing all this information cheap, fast, and scalable, leading many organizations to adopt a “dump everything” mindset pouring every dataset into the warehouse without much structure or strategy. While this worked at first, it quickly created chaos: hundreds of tables, unclear relationships, and multiple conflicting versions of key metrics like “revenue.”

Without proper cleaning, validation, or governance, data becomes cluttered, slow to query, and hard to trust.

It’s like blending all your groceries together and calling it dinner technically everything’s there, but nobody wants to touch it.

The result is data overload, slower performance, and unreliable insights proving that even in an age of infinite storage, organized data still wins.

Graphical representation:

Emphasize the pain:

• Analysts don’t trust dashboards because the data behind them is messy or wrong, leading to incorrect insights.

• Metrics don’t match across reports or tables, making it impossible to know which numbers are true.

• Queries take forever to run since the system is overloaded with unorganized, duplicated, or conflicting data.

Humour:

“If you’ve ever joined five tables just to figure out who bought what, welcome to the world of data wrestling , where dimensional modeling is the referee that stops the chaos and declares a clear winner!”

3). The promise: A way to make data actually make sense.

Dimensional modeling is the proven way to structure data so anyone, not just engineers, can navigate it confidently and get clear, reliable answers to real business questions.

It is based on two core ideas:

1. Facts: These are the important numbers or measurements in your business, like sales amount, number of units sold, or revenue. Facts represent what you want to measure or analyze.

2. Dimensions: These are the descriptive details or context about the facts, such as time (day, month, year), product (name, category), customer (age, location), or store. Dimensions help you answer questions like “When was it sold?” or “Who bought it?” or “Where was it sold?”

The fact tables hold the measurable data, while the dimension tables hold the descriptive information.

By organizing data into facts and dimensions, dimensional modeling improves data retrieval speed, clarity, and usability in data warehouses

Simple visual example student_attendence

Captures the measurable event: a student attending a class.

✅ Fact table = the event itself (who attended what, when, and for how long). It stores numeric or measurable data.

Dimension Table – students

Describes who attended.

📚 Dimension Table – courses

Describes what they attended.

🔗 Relationships:

• student_attendance.student_id → students.student_id

• student_attendance.course_id → courses.course_id

In simple terms:

• Fact table = the measurable event (attendance).

• Dimension tables = the descriptive context (who the student is, what course they attended).

“Trying to analyze data without dimensional modeling is like trying to untangle Christmas lights frustrating, time-consuming, and making you wonder if you should just buy a new set!”

4. “Why it still matters ; even in the age of dbt and the modern data stack”.

Because tools like dbt, Airflow, and the rest of the modern data stack are powerful , but they don’t replace the need for good modeling, they just automate and scale it.

Dimensional modeling still matters because:

• Supports modern tools: dbt, metrics layers, and BI platforms still rely on a solid data model underneath to work effectively.

• Scales with complexity: As data grows and evolves, a dimensional model keeps queries simple and performance predictable.

• Timeless foundation: Technologies change, but the need for clarity, consistency, and context in analytics never does.

It’s the difference between following a clear path and getting lost without a clue where to go.

5. The myth: “Isn’t a data lake enough?”

“Sure, you can run analytics directly on raw JSON files in S3, if you’re okay with messy queries, slow results, and a lot of confusion every time you need an answer.”

Just storing all your data in one place (like a data lake in S3) doesn’t mean it’s ready to use. That’s like having all the ingredients for a cake but no recipe, the pieces are there, but they’re not useful yet.

Sure, you can run analytics directly on messy raw JSON files in S3, but you’ll end up with confusing queries, slow results, and numbers that don’t match from one dashboard to another.

That’s where modeling comes in. It’s the process of organizing and structuring your data so it’s clear, consistent, and easy to work with.

Once you model your data:

• Metrics stay consistent — everyone agrees on what “active user” or “revenue” means.

• Queries run faster — because the data is organized and optimized.

• New people can understand the data quickly — because there’s a clear structure.

Key idea: Tools will always change, but the basic principle of organizing your data so humans (and machines) can understand it will never stop being important.

6. Example: online retail store

“Think of an online retail store managing thousands of daily orders. The fact table records each sale, what was sold, who bought it, where, and when. The dimension tables add meaning: products describe what was sold, customers tell who bought it, stores show where it happened, and time shows when. With this setup, the business can quickly ask questions like ‘Which region generated the most revenue this month?’ or ‘Which product categories are gaining popularity over time?

Step 1: Create Tables

Step 2: Insert Sample Data

Step 3: Query to Generate Result

Then output:

This complete query sequence, from start to end models customer purchases cleanly, showcasing how the fact table logs each order and the dimension tables describe who, what, and when.

Show how this model powers metrics:

• “Want monthly revenue by region? Easy aggregate sales by time and region dimensions.”

• “Want retention by customer type? Join facts to the customer dimension.”

1. Monthly Revenue by Region:

Output:

Graph:

2. Retention by Customer Type:

Output:

Graph:

“Add a key takeaway”:

“Good modeling turns technical structure into business clarity; everyone speaks the same data language.”

7.“Final reflection: What’s Next”

Key Takeaways:

• Dimensional modeling is all about clarity.

• It streamlines SQL, strengthens metric accuracy, and turns dashboards into tools you’re actually proud to share.

Structured data models do more than organize information they build trust and consistency across every report. Facts capture what happens; dimensions explain why and where it happens. Together, they turn raw data into clear, actionable insight.

Up Next:

In our next blog, we’ll meet the true stars of the analytics world facts and dimensions tables and learn how to tell them apart.

1. Introduction:

SQL: not just SELECT, meet the power tools you forgot existed

SQL isn’t just about pulling rows with a simple SELECT *. It’s a language packed with hidden features that can make your queries sharper, faster, and far more expressive. Beyond the basics, SQL offers advanced capabilities like Grouping Sets, Rollups, and Self Joins , tools that many developers overlook, but which can completely change how you approach reporting and analysis. Once you know how to use them, you’ll find yourself solving problems more effectively and writing queries that do far more with less code.

2. Foundations: why not just GROUP BY?

Before we get into the advanced stuff, let’s take a quick refresher. GROUP BY is where most SQL users start when they need summaries.It collapses rows, applies aggregates, and gets you answers fast, totals by region, averages by customer, counts by category.

The problem

GROUP BY answers one question at a time.

if you need totals by region, by product, and the overall total, you end up writing (and maintaining) three queries, or gluing them with UNION ALL.

That’s where Grouping Sets, Rollups, and Self Joins come in. They’re designed to move you past single-level aggregations, so you can capture multiple perspectives, richer breakdowns, and even relational insights, all without writing query after query.

3. Grouping sets: multiple summaries in one query

Grouping Sets are like having multiple GROUP BY clauses rolled into a single query. Instead of writing and unioning separate aggregations, you tell SQL exactly which “buckets” you care about, and it returns them all at once.

Example: one query instead of three

Say you want sales totals:

1. By country

2. By product

3. Overall total

With plain GROUP BY, you’d have to write three separate queries and glue them with UNION ALL:

With Grouping Sets, it’s one clean query:

output:

✅ Result: You get all three aggregations in a single result set, no unions, no duplicate queries.

When to use grouping sets:

• Dashboards → where you want country-level, product-level, and global KPIs all pulled at once.

• Reports → monthly or quarterly summaries that combine different aggregation levels side by side.

• Multi-level summaries → subtotals, grand totals, and detailed breakdowns generated in one pass.

  In short:

  Anytime you find yourself writing the same SUM or COUNT three different ways, Grouping Sets can probably collapse it into a single, elegant query.

4. Rollups

What Are rollups?

A Rollup is a GROUP BY shortcut that creates progressive subtotals along a hierarchy. You give SQL an ordered list of columns, and it automatically returns results at every level, from the most detailed grouping down to the grand total.

Example: website traffic by date

Suppose you’re analyzing website visits and want to see:

• Visits by year and month

• Visits by year

• The overall total

You can do this with a single query using ROLLUP:

output:

👉 So with a single query using ROLLUP, you’ve got:

• year → Year of the visit(s).

• month → Month of the visit(s).

• total_visits → How many visits happened in that year/month.

• row_type → Label telling you whether the row is monthly detail, a yearly subtotal, or the overall grand total.

  How rollups compare to grouping sets:

  Both Grouping Sets and Rollups let you collapse multiple aggregations into one query, the difference is in control vs convenience.

• Grouping Sets → You spell out each grouping you want. Perfect when you need a custom mix (e.g. totals by country, by product, but not both together).

• Rollups → You hand SQL a hierarchy, and it builds every subtotal and the grand

  total for you. Less typing, cleaner code, but the shape is fixed.

In short:

• Use Grouping Sets when you want custom, non-hierarchical views.

• Use Rollups when you want hierarchical roll-ups with minimal effort.

Analogy

“Rollups are like elevator stops: you can get off at the city floor, the state floor, the country floor, or ride all the way up to the grand total.”

It conveys the hierarchical nature (each level builds on the last), but in a way that’s easy to visualize.

5. Self Joins:

A self join is simply joining a table to itself. Instead of pulling data from two different tables, you treat the same table as if it were two, usually by giving each instance an alias.

Real-world examples:

• Comparing rows:

  • In a weather table, join each city’s record to another city’s record on the same date to compare conditions (e.g., New York’s temperature vs. Los Angeles’ temperature on June

• Time-series analysis:

  • In the same weather table, join today’s weather record for a city to yesterday’s record to track day-over-day changes in temperature, rainfall, or humidity.

Output:

✅ In one query, you get:

• Row comparison → city vs. city, same date.

• Time-series → today vs. yesterday, same city.

• Changes over time in temperature, rainfall, humidity.

Why self joins matter in analytics:

• Comparisons → how one row stacks up against another (city vs. city, product vs. similar product).

• Trends → how a row evolves over time (today vs. yesterday’s weather, last month vs. this month’s reading).

• Hierarchies → how rows connect in a structure (employee vs. manager, category vs. subcategory).

In analytics, this means you’re not limited to totals and averages. A self join shows patterns, progressions, and relationships, the deeper layer of meaning that makes raw data actionable.

Analogy:

“It’s like comparing today’s selfie to yesterday’s, same subject, new context.”

6. Putting it all together

Scenario: team productivity — dept → manager rollups

Problem:

In most organizations, productivity metrics are stored at the employee level (e.g., hours worked, tasks completed, revenue generated). But business leaders don’t just need to see this information for individuals, they need rollups across the hierarchy:

• A manager wants to see how their team performed.

• A department head wants to see all their managers’ totals, plus comparisons across them.

• The company wants to see aggregated totals across departments.

Goal:

• i-level metrics by department × manager × month

• Compare employees under each manager (self-join to the manager)

• Get manager subtotals, department subtotals, company totals, all in one result

Assumed tables (map to your schema):

• employees(employee_id, full_name, manager_id, department)

• work_log(employee_id, work_date) one row per day an employee did work (or any activity event)

Query:

Output:

7. “Challenges & pro tips”

1. Performance pitfalls:

Queries with self-joins, grouping sets, and rollups can scan huge volumes of data, making them slow and resource-intensive on large datasets.

2. Aggregate placeholders in rollups and grouping sets:

When you use ROLLUP or GROUPING SETS, SQL adds extra rows for subtotals and grand totals. In those rows, the grouped columns are filled with NULL , not because the data is missing, but as a marker that this row represents an aggregate.

3. Where self-joins fall short:

• Performance overhead → Joining a large table to itself often doubles the data scanned and can make queries very slow.

• Complexity → Queries become harder to read and maintain, especially with multiple conditions across the same table.

• Scalability issues → On very large datasets, self-joins may time out or require excessive compute compared to equivalent window function solutions.

Pro tips :

1. Speed up queries

• Summarize data first (like user–month).

• Add indexes on important columns.

• Only check key retention periods (Month+1, Month+2).

2. Read totals clearly

• Use GROUPING() to know which rows are subtotals.

• Change NULLs into labels like ALL MONTHS or ALL YEARS.

3. Use window functions

• Try LAG, LEAD, or ROW_NUMBER instead of self-joins.

• They are faster, cleaner, and scale better.

8. The big picture: why It’s useful

At the end of the blog, ROLLUP and GROUPING SETS aren’t obscure SQL tricks, they’re shortcuts to thinking and working like an analyst. They turn three or four repetitive queries into one clear, maintainable query that scales with your questions. Fewer queries mean fewer errors, faster answers, and cleaner code.

So here’s the takeaway:

Next time you catch yourself writing multiple GROUP BY queries in a row, stop and ask, could this be one query with ROLLUP or GROUPING SETS?

Introduction:

In Today’s analytics, raw data alone isn’t enough, what matters is how flexibly and deeply you can explore it. When analyzing data, it’s often important to calculate values like running totals, rankings, or comparisons between rows, without losing the details of each row. That’s where SQL window functions come in.

They let you perform powerful calculations across related rows while keeping every row in your results. This makes it easier to spot trends, compare values, and build more insightful reports. Whether you're just starting with SQL or looking to improve your analysis skills, understanding window functions is a valuable step forward.

SQL Window Functions Core Concepts:

Each window function uses the OVER clause to define its scope. You can refine this window with:

• PARTITION BY to divide data into groups for independent calculations,

• ORDER BY to set row order within each group for operations like rankings or running totals.

Window functions fall into several key categories:

• Aggregate functions (SUM(), AVG(), COUNT()) for running totals or moving averages.

• Ranking functions (ROW_NUMBER(), RANK(), DENSE_RANK()) to assign positions or ranks within partitions.

• Value functions (LAG(), LEAD(), FIRST_VALUE(), LAST_VALUE()) to access values from other rows in the window.

  Example:

This example calculates a cumulative total of duration_seconds ordered by start_time, without grouping or removing rows.

This example calculates a cumulative total of duration_seconds ordered by start_time, without grouping or removing rows.

Primary SQL Ranking Functions:

Ranking functions in SQL are specialized window functions that assign ranks to rows based on their ordering within a dataset. The primary ranking functions are ROW_NUMBER(), RANK(), DENSE_RANK(), and NTILE().

• ROW_NUMBER() - ROW_NUMBER() is a window function in SQL that assigns a unique sequential integer to rows within a result set, based on the order you specify.

• Output:

  The query would produce:

• RANK(): Assigns ranks to rows, but if rows have identical values per the ORDER BY clause, they receive the same rank. This may create gaps in the rankings after ties (i.e., if two rows are ranked 1, the next row is ranked 3).

• Output:

  The query would produce:

• DENSE_RANK(): Functions similarly to RANK(), but does not leave gaps in the ranking sequence after ties. If two rows are tied for rank 1, the next row is assigned rank 2.

• Output:

  The query would produce:

• NTILE(n): Divides the data into n equal or nearly equal groups and assigns each row a group number.

• Output :

  (rows split into 4 nearly equal buckets, highest values in quartile 1)

📦 Key Navigation Functions

Navigation functions are a subset of SQL window functions that allow you to access values from rows adjacent to the current row within a defined window. These adjacent rows can be from before (LAG) or after (LEAD) the current row. Other navigation functions include FIRST_VALUE() and LAST_VALUE() which retrieve boundary values within each window.

The navigation summary for the LEAD() , LAG(), FIRST_VALUE() and LAST_VALUE() functions from the SQL time-series window functions is as follows:

LEAD and LAG

• Purpose: The LEAD and LAG functions enable access to data from rows after (LEAD) or before (LAG) the current row within a result set ordered by a specific column, making them ideal for time-series and sequential data analysis.

• Functionality: LEAD looks "into the future" (next rows), while LAG looks "into the past" (previous rows). They are frequently used to calculate differences between consecutive data points, such as monthly stock prices or moving averages.

FIRST_VALUE() and LAST_VALUE()

• Purpose:

  FIRST_VALUE() and LAST_VALUE() return the first and last value within the ordered window of each partition. They're ideal when you need to reference the starting or ending state of a sequence, like the first purchase amount, the initial score, or the last known status.

• Functionality:

  • FIRST_VALUE() captures the earliest value in the ordered set.

  • LAST_VALUE() captures the latest value, but note: its behavior depends on the window frame definition. Without correct frame specification, it may return the current row’s value instead of the true “last” row.

Example:

Imagine a Scenario : Analyzing Employee Promotion Patterns in an organisation

Business Use Case:

An HR analytics team wants to understand employee promotion trends over time. Specifically, they want to know:

• When each employee got promoted.

• What was their previous job title (LAG()).

• What is their next job title (LEAD()).

• What was their first-ever title in the company (FIRST_VALUE()).

• What is their current/latest title (LAST_VALUE()).

SQL query:

✅ Output of query on that data

What is an Aggregate Function?

An aggregate function in SQL is a special type of function that takes multiple rows of data and condenses them into a single value based on a calculation you specify.

They’re used when you want summarized information, such as totals, averages, or counts, rather than listing individual rows.

Common SQL aggregate functions include:

• COUNT(): Returns the number of rows in a set, or the number of non-NULL values in a specified column.

• SUM(): Calculates the total sum of values in a numeric column.

• MIN(): Retrieves the minimum value from a specified column.

• MAX(): Retrieves the maximum value from a specified column.

📊 Business Use Cases for Aggregation

🛍 Retail & E-Commerce:

• Sales Reporting: Calculate SUM(sales_amount) to find total revenue per day, month, or store.

• Top-Selling Products: Use COUNT(product_id) grouped by product to identify best performers.

🏥 Healthcare

• Patient Volume: Use COUNT(patient_id) to track inflow by department.

• Treatment Success Rate: Aggregate outcomes with SUM(successful_cases) / SUM(total_cases).

🎓 Education

• Performance Tracking: AVG(marks) per subject or class.

• Top Scores: Identify MAX(score) for award consideration.

Advanced patterns of SQL Window Functions:

1. Row-to-Row Comparisons

• Functions Used: LAG() and LEAD()

• Purpose: Compare a row with its preceding or following row within the same partition.

• Typical Applications:

  • Finding trends or changes over time

  • Calculating deltas between consecutive events

  • Detecting anomalies in sequential data

(Conceptual Example):

2. Rolling and Moving Calculations

• Functions Used: AVG(), SUM(), etc., combined with window frames

• Purpose: Compute metrics like rolling sums, running averages, or cumulative totals.

• Applications:

  • Smoothing time-series

  • Spotting moving trends

  • Monitoring performance metrics

4. Conditional Insights

• Approach: Use CASE inside a window function to apply conditional aggregations.

• Applications:

  • Aggregating only certain categories of data while still operating in a partition

  • Segmenting by logic without breaking partitions

  

  Performance Principles:

  • Indexes & Partitions

    • Index columns that are used in PARTITION BY and ORDER BY.

    • Consider table partitioning to reduce the data each query scans.

  • Window Frame Efficiency

    • Use specific frames (e.g., ROWS BETWEEN 3 PRECEDING AND CURRENT ROW) instead of broad ones like UNBOUNDED PRECEDING when possible. Narrow windows run faster.

  • Reduce Complexity

    • Keep only necessary columns in the OVER() clause.

    • Simplify expressions inside window functions; push heavy calculations outside the window if possible.

  • Benchmark Queries

    • Run queries on real or representative datasets.

    • Use EXPLAIN / execution plans or profiling tools to spot slow steps.

  ---

  Quick best practices

  • Use only when needed. If a GROUP BY works, use that.

  • Set PARTITION BY and ORDER BY carefully, add a tiebreaker (e.g., ORDER BY ts, id).

  • Filter early, select fewer columns before the window step.

  • Use tight frames (ROWS BETWEEN N PRECEDING AND CURRENT ROW) instead of unbounded history.

  • Reuse results, don’t compute the same window twice (or use QUALIFY where available).

  • Handle NULLs in LAG/LEAD with a default value.

  • Watch performance: check the plan/profile; index/partition on the columns you partition/order by.

  Conclusion

  Window functions add trends, ranks, and rolling metrics without losing row detail. Keep them predictable and fast with precise ordering, small frames, and smaller inputs, and always verify with the execution plan.

1. Sqlmesh delivers 22x faster data transformation and 10x cost savings vs dbt Core™ on Snowflake.

Experience lightning-fast transformations with Sqlmesh, 22x speed improvements and 10x lower Snowflake spend vs dbt Core™.

SQLMesh achieves 22x faster execution and 10x lower Snowflake costs, not through tweaks, but through architectural reinvention. Virtual environments eliminate wasteful rebuilds. Column-level lineage prevents cascade updates. State-aware deployments reuse computation instead of repeating it. over $1M saved annually and 311 hours of dev time reclaimed each month.

https://www.tobikodata.com/blog/sqlmesh-delivers-22x-faster-data-transformation-and-10x-cost-savings-vs-dbt-core-tm-on-snowflake

2. "Michael, dlt, and the art of unbreakable API pipelines."

Building unbreakable API pipelines with Michael and dlt.

Michael Shoemaker shows how to build API pipelines that survive change, automatically. Using dlt, BigQuery, and Cloud Functions, he creates a zero-touch setup that handles new fields, schema shifts, and daily runs without manual fixes or duct tape.

https://dlthub.com/blog/michael-dlt-tutorials

3. A practitioner’s guide to LLM-native pipeline building with dltHub workspace.

LLM pipelines made easy with dltHub workspace.

Data engineering is shifting, from manual coding to AI-assisted assembly. This piece introduces LLM-native scaffolds in dltHub Workspace: structured, reusable blueprints that let language models generate reliable pipelines in minutes. Human judgment ensures accuracy, while the framework handles the complexity.

https://dlthub.com/blog/practitioners-guide-dlthub-workspace

4. You're doing quarterly planning wrong

Rethinking quarterly planning

Quarterly planning doesn’t need to be perfect, just useful. This post shares hard-earned insights from coordinating 26+ teams, why async prep beats long meetings, how to focus on what actually ships, and why true ownership belongs with engineers.

https://posthog.com/newsletter/quarterly-planning-mistakes

Note: I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Do you have a project or idea?

Feel free to drop me a line. If it’s interesting, let’s chat. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now

1. What’s new with Azure Databricks: unified governance, open formats, and AI-native workloads

The future of data on Azure: Databricks brings unified governance and AI to the lakehouse

Azure Databricks is evolving into the intelligent core of the modern data stack. With updates like Databricks One, Lakebase, Lakeflow, and enhanced Unity Catalog, it now offers a unified, open, and secure experience for building, governing, and scaling AI-native workloads on Azure.

https://www.databricks.com/blog/whats-new-azure-databricks-unified-governance-open-formats-and-ai-native-workloads

2. Netflix Tudum architecture: from CQRS with Kafka to CQRS with RAW Hollow

How Netflix Tudum evolved: from Kafka CQRS to Raw Hollow efficiency

Netflix transformed Tudum’s architecture, moving from a Kafka-based CQRS setup to a RAW Hollow-powered model. The shift brought leaner infrastructure, faster updates, and a smoother experience at scale. An example of how streamlining infrastructure can unlock both speed and reliability.

https://netflixtechblog.com/netflix-tudum-architecture-from-cqrs-with-kafka-to-cqrs-with-raw-hollow-86d141b72e52

3.Sync data from MySQL to Iceberg

Syncing data from MySQL to Apache Iceberg can be seamless using BladePipe, a lightweight alternative to traditional Kafka-based pipelines. With real-time ingestion, minimal latency, and a simplified setup, it offers an efficient path to building a modern lakehouse.

https://doc.bladepipe.com/blog/tech_share/mysql_iceberg_sync

4. Making sense of Apache Iceberg statistics

Understanding Apache Iceberg table statistics

Guy Yasoor breaks down the key layers of Apache Iceberg’s statistics, from Parquet min/max to Puffin files and Bloom filters. These aren’t just backend details, they’re what enable faster queries, better planning, and lower compute costs.

https://www.ryft.io/blog/making-sense-of-apache-iceberg-statistics

5. The Apache Iceberg avalanche: how the open table format changes the face of data lakes

The Apache Iceberg shift: transforming data lakes with open table formats

Apache Iceberg bridges the gap between flexible data lakes and high-performance warehouses. By decoupling compute, storage, and metadata, it gives modern data teams freedom, interoperability, and control. Iceberg is rapidly becoming a foundational piece in next-gen data platforms, powering open, efficient, and future-proof lakehouses.

https://www.snowflake.com/en/blog/apache-iceberg-data-lakehouse-architecture/

Note: I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Do you have a project or idea?

Feel free to drop me a line. If it’s interesting, let’s chat. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now

1. Driving content delivery efficiency through classifying cache misses

Optimizing content delivery by tracking and categorizing cache misses

Netflix tries to stream from the closest server for speed and quality. But sometimes it misses: that’s a cache miss. When that happens, Netflix uses real-time data, health checks, and predictions to quickly reroute your stream. That’s how it keeps things running smoothly.

https://netflixtechblog.com/driving-content-delivery-efficiency-through-classifying-cache-misses-ffcf08026b6c

2.Querying Apache Iceberg with Sub-Second performance

Apache Iceberg wasn’t built for low latency, but Firebolt makes it possible. With smart metadata caching, Parquet optimizations, and subresult reuse, interactive queries run in under a second, without moving data. For anyone working with large datasets, this changes what’s possible.

https://www.firebolt.io/blog/querying-apache-iceberg-with-sub-second-performance

3. All about Iceberg partitioning and partitioning writing strategies

Optimizing data layout with Iceberg partitioning and write strategies

Apache Iceberg brings precision to partitioning with rich metadata that enables manifest pruning and file skipping, dramatically improving query performance. PartitionSpec and manifest lists let query engines bypass irrelevant data files with ease, avoiding costly scans.The blog explores four file writing strategies tailored for different ingestion needs, from high-throughput, unordered streaming to memory-efficient, sorted batch writes. Each approach balances schema evolution, memory usage, and write patterns.

https://olake.io/iceberg/iceberg-partitioning-and-writing-strategies

4. OLake + Glue + Snowflake - A deep dive into modern data partitioning

Building an open, efficient lakehouse with OLake, Glue & Snowflake

Modern data ingestion doesn’t have to be fragile, expensive, or vendor-locked. With OLake, Apache Iceberg, AWS Glue, and Snowflake, you get a fully open, scalable, and efficient pipeline from databases to your lakehouse, no copy-paste ETL nightmares

https://olake.io/iceberg/olake-glue-snowflake

5. How Unity Catalog managed tables automate performance at scale

Smarter tables, less work: Unity Catalog’s secret to hands-free optimization.

Unity Catalog managed tables bring AI-driven optimizations, like auto-clustering, smart vacuuming, and metadata caching, that reduce storage costs by 50%+ and boost query speeds by up to 20x. No tuning, no scripts, just better performance at scale.

https://www.databricks.com/blog/how-unity-catalog-managed-tables-automate-performance-scale

Note: I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Do you have a project or idea?

Feel free to drop me a line. If it’s interesting, let’s chat. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now

Imagine you're a 5-year-old with a massive toy collection that keeps growing every day. You have toys everywhere, and it takes you over an hour just to find the specific toy you want to play with. That's exactly what was happening to LY Corporation, one of Japan's biggest tech companies!

What is LY Corporation?

LY Corporation is like a giant digital playground. Think of it as the company that runs:

• LINE - the messaging app that almost everyone in Japan uses (like WhatsApp but way more popular)

• Yahoo! JAPAN - the website where people search for things and read news

• PayPay - the app people use to pay for things with their phones

They have over 320 million users across Asia and handle more data than you can imagine - we're talking about 1.1 exabytes, which is like having 1.1 billion huge toy boxes full of information!

How big is their data?

To understand how massive LY Corporation's data is, let's use simple numbers:

• They have over 100,000 tables (think of each table as a different type of toy box)

• Every single day, they process 1.2 trillion records (that's like counting to 1.2 followed by 12 zeros!)

• Their data is growing really, really fast because millions of people use their apps every day

What was their old data system like?

Before Apache Iceberg, LY Corporation was using something called Hive (think of it as their old toy organization system). Here's how it worked:

The old way (Hive System)

• Data collection: People using LINE, Yahoo, and PayPay would create data (like sending messages, searching, making payments)

• Data journey: This data would travel through something called Kafka (like a conveyor belt) to Flink (a processing machine) and finally land in HDFS storage (the toy warehouse)

• Format conversion: The data would start as AVRO and Protocol Buffer files and then get converted to ORC format every hour

• The big wait: It took 1 to 1.5 hours before anyone could actually use this data

The problems they faced

Remember when your room was so messy you couldn't find your favorite toy car under a pile of socks and Legos? LY Corporation had the same problem, just with, you know, terabytes of data.

1. Super slow data access

Getting fresh data took them over an hour. Yep, an hour.

In a world where modern businesses need data in minutes, not “go-make-a-coffee-and-come-back” hours, this was a big problem. Their machine learning models and real-time services were basically stuck waiting around, tapping their feet.

2. The tiny files chaos

Imagine having a million tiny toy boxes instead of a few big, organized ones. LY Corp’s system (HDFS) was drowning in these tiny files, each demanding attention.

And the “brain” of their system (the namenode) was getting so overloaded trying to keep track that it was running out of memory, like your laptop when you have 47 Chrome tabs open.

3. Deleting data was a nightmare

When someone requested their personal data be deleted (think GDPR and privacy laws), it wasn’t just a quick “delete” button. Nope.

They had to rewrite huge files just to remove a single person’s data. Imagine having to rewrite your entire diary just to erase one sentence. Painful.

4. Schema changes were a disaster

Adding or removing data fields? It was like having to reorganize your entire Lego collection every time you bought a new set.

Why did they choose Apache Iceberg?

Apache Iceberg is like a super-smart toy organization system. Here's why LY Corporation fell in love with it:

1. Delete specific things easily

• Instead of emptying entire toy boxes to remove one toy, Iceberg can point to exactly what needs to be deleted

• Perfect for when people ask to have their personal data removed (GDPR compliance)

2. Make data available super fast

• Reduced their data delay from 1.5 hours to just 5 minutes

• That's 10-12 times faster!

3. Easy changes without big mess

• Adding new data fields is like adding new compartments to your toy box without emptying everything out

• Schema evolution made it simple to adapt as their services grew

4. Better organization

• Iceberg manages metadata (information about information) much more efficiently

• Reduces pressure on their central database (Hive Metastore)

Think of Apache Iceberg as having a magical catalog (Iceberg’s manifest and metadata approach) that knows exactly where every piece of data is stored and can retrieve it when needed.

How Did they actually make the switch?

The migration strategy

LY Corporation was smart - they didn't try to change everything at once! Here's what they did:

Gradual migration approach

• Started with new tables that needed Iceberg's special features

• Migrated 8,000+ tables to Iceberg by 2024

• Still have 90,000+ tables on the old Hive system

• Only migrate tables that actually benefit from Iceberg's features

The new technical setup

New Data pipeline:

• Kafka (data conveyor belt) → Flink (processing) → Apache Iceberg (smart storage)

• Data gets flushed every 5 minutes instead of every hour

• Files are stored in ORC format (efficient storage format)

• Everything runs on Kubernetes (modern container platform)

What the table optimizer does

Think of LY’s Table optimizer as an automated janitor for Iceberg tables, keeping them fast, tidy, and cost-efficient without manual babysitting.

Coordinator

Acts like a manager, watching all tables to see which need maintenance. It keeps a smart queue, prioritizing based on file size, file count, or table health.

Workers

Workers are the system’s hands, they execute tasks from the coordinator, like merging files and cleaning data, all in parallel for speed and efficiency.

Automatic compaction

Automatic compaction merges many small files into larger ones, using smart strategies like bin-packing or sorting, to speed up queries and reduce storage waste.

Snapshot management

Iceberg’s time travel creates snapshots with every change. The optimizer prunes old snapshots and cleans up orphan files to save space and keep metadata lean.

Spark jobs

File rewrites and data merges are handled by distributed Spark jobs, scheduled and assigned by the coordinator for efficient processing at scale.

How It’s built

• Microservices on Kubernetes: Fully distributed, scalable, and resilient, with coordinators and workers scaled independently.

• Automated scheduling: Maintenance runs on autopilot, prioritizing urgent tasks and balancing workload.

• Monitoring & health checks: Continuously tracks table health, file counts, and performance, alerting or auto-fixing as needed.

• Conflict prevention: Prevents overlapping maintenance and user operations on the same table, ensuring safe, conflict-free optimizations.

In short:
The Table Optimizer keeps LY’s Iceberg tables clean and fast, automatically handling compaction, cleanup, and snapshot pruning, so the team can focus on building, not babysitting tables.

What challenges did they face?

Even with a great plan, moving this much data wasn't easy:

Migration challenges

1. Deciding what to migrate

• With 100,000+ tables, they had to carefully choose which ones would benefit from Iceberg

• Tables that were small or didn't need Iceberg's features could stay on Hive

2. User training

• Data engineers and analysts needed to learn new tools and concepts

• Not everyone understood Iceberg's advanced features initially

3. Data copying between systems

• Their old backup strategy (DistCP) needed to be redesigned for Iceberg

• Moving data between different storage systems became more complex

Ongoing challenges

Even with their big clean-up, LY Corp still has a few chores to keep things running smoothly.

1. Table housekeeping

Iceberg tables don’t magically stay fast and efficient. They need regular “housekeeping” to keep them tidy and optimized, kind of like sweeping the floor before the dust bunnies take over.

Sure, their Table Optimizer helps a lot, but it’s still an ongoing chore they can’t ignore.

2. File size juggling

Too many tiny files? Bad for performance. Super large files? Harder to manage.

It’s a balancing act, like pouring cereal without spilling. Automatic compaction steps in to help, merging those tiny files into more manageable sizes to keep everything humming along.

What goals did they achieve?

LY Corporation hit some amazing targets:

Performance improvements

1. Massive speed boost

• Data latency: From 1.5 hours to 5 minutes (10-12x faster)

• Query performance: Much faster due to better file organization

• Real-time capabilities: Better support for online machine learning

2. Operational efficiency

• Automated maintenance: Less manual work for data engineers

• Better resource utilization: More efficient use of computing power

• Reduced complexity: Simpler data pipeline management

Business benefits

1. Compliance made easy

• GDPR compliance: Can efficiently delete user data when requested

• Data governance: Better control over data lifecycle

• Audit capabilities: Clear tracking of data changes

2. Business agility

• Faster insights: Data available for analysis much sooner

• Better ML/AI: More timely data for machine learning models

• Competitive advantage: Faster response to business needs

Cost savings and benefits

1️⃣ Direct cost reductions
Efficient file organization and compression lower compute and storage costs.

2️⃣ Operational savings
Automation reduces manual maintenance and pipeline failures, saving engineering time.

3️⃣ Infrastructure efficiency
Makes better use of your existing hardware while easing pressure on Hive Metastore and HDFS.

Indirect Benefits

1️⃣ Faster Time-to-Market
Teams get fresh data faster, enabling quicker feature rollouts and business decisions.

2️⃣ Improved user experience
Real-time features and personalization improve with fresher data and faster response to user behavior.

What's special about their implementation?

Innovation: the table optimizer

LY Corporation didn't just use Iceberg - they innovated on top of it. Their Table Optimizer is like having a smart assistant that:

• Monitors thousands of tables automatically

• Predicts when maintenance is needed

• Runs optimization jobs without human intervention

• Prevents conflicts between different operations

• Provides detailed metrics and monitoring

Hybrid approach

They took a smart approach by not migrating everything at once. Stable, simple tables stayed on Hive (no need to fix what isn’t broken), while complex, dynamic tables moved to Iceberg, focusing effort where Iceberg’s features would bring the most benefit.

Why this matters for other companies

LY Corporation's journey shows that:

1. Scale matters: At their data volumes, small improvements have huge impacts
2. Gradual migration works: You don't need to change everything at once
3. Automation is key: Building tools like Table Optimizer pays off
4. Compliance is critical: Modern data platforms must support privacy regulations

The Future

LY Corporation is continuing to:

• Expand Iceberg adoption to more suitable tables

• Improve their Table Optimizer with more intelligence

• Explore new Iceberg features like branching and tagging

• Share their learnings with the broader Apache Iceberg community

What makes this special

This isn't just a story about switching technology - it's about transforming how a massive company handles data. LY Corporation proved that even with 1.1 exabytes of data and 100,000+ tables, you can make smart, gradual changes that deliver huge benefits.

They turned their data platform from a slow, rigid system into a fast, flexible, compliance-ready powerhouse that can adapt to whatever the future brings. And the best part? They did it without disrupting their services to 320 million users!

Concluding thoughts

Sometimes the biggest improvements come from making your data smarter, not just bigger. LY Corporation's Apache Iceberg journey is a perfect example of how the right technology choices can transform not just your data platform, but your entire business capability.

Got a project or idea?

Drop me a line. If it’s interesting, I’m in. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now

1.Announcing Lakebase Public Preview

Lakebase public preview, a SQL-first lakehouse built for performance.

Traditional databases struggle with scale, speed, and modern AI workflows. They sit siloed from analytics, making intelligent application development harder than it should be. Lakebase is a serverless, Postgres-compatible database, integrated directly into the lakehouse. Built for real-time feature serving, model interaction, and unified governance , it marks a quiet but significant shift in how data apps are developed.

https://www.databricks.com/blog/announcing-lakebase-public-preview

2. Change Takes More Than a Megaphone: Communicate, Experiment and Educate to Drive Transformation

Transformation demands more than speaking up: it requires communication, experimentation, and education.

Real AI adoption doesn’t go viral, it builds through strong ties, repeated exposure, and shared success. When people see value, they follow. The organizations winning with AI are the ones educating, experimenting, and elevating others.

https://www.snowflake.com/en/blog/change-transformation-communication-education-ai/

3. SQLMesh delivers 22x faster data transformation and 10x cost savings vs dbt Core™ on Snowflake

Sqlmesh outperforms dbt Core on Snowflake with 22x faster transformations and 10x lower costs.

SQLMesh reimagines how data is transformed on Snowflake. By replacing physical rebuilds with virtual environments, tracking changes at the column level, and eliminating redundant computation, it saves over $1M annually and frees up 311 hours of developer time each month.

https://www.tobikodata.com/blog/sqlmesh-delivers-22x-faster-data-transformation-and-10x-cost-savings-vs-dbt-core-tm-on-snowflake

4. The complete stream processing journey on FlinkSQL

The stream processing lifecycle, powered by FlinkSQL.

Grab rebuilt its stream processing stack, from Zeppelin notebooks to a shared FlinkSQL gateway, cutting query times by 80% and enabling pipeline deployment in under 10 minutes. A compelling look at how stream processing can be fast, collaborative, and production-ready.

https://engineering.grab.com/the-complete-stream-processing-journey-on-flinksql

5.Macroeconomics and the data industry

Understanding the economic undercurrents driving data work today.

Venture capital flows, shifting interest rates, and economic cycles ripple through every layer of the data world, reshaping tools, budgets, and even career paths. Understanding these forces is key to navigating rapid change and making smarter decisions as a data professional.

The Analytics Engineering Roundup

Macroeconomics and the data industry

New podcast episode! Julia and I spoke with Kevin Hu of Metaplane. Whether you’re mapping the emerging data observability landscape or just love a good physics metaphor, give it a listen…

Read more

4 years ago · 19 likes · Tristan Handy

Note: I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Do you have a project or idea?

Feel free to drop me a line. If it’s interesting, let’s chat. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now

1. Leveraging Agentic AI in Games

Agentic AI turn your game into a living, breathing world.

Agentic AI brings a new layer of intelligence to games, NPCs that remember your actions, QA bots that adapt as the game evolves, and support systems that respond with real understanding. It’s not about replacing creative teams, but giving them tools to build deeper, more personal, and more responsive experiences.

https://www.databricks.com/blog/leveraging-agentic-ai-games

2. Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

Netflix unify petabytes of data across teams, tools, and use cases, without breaking everything

Netflix is solving one of the hardest problems in modern data systems: fragmented, duplicated, inconsistent models across platforms. Instead of forcing uniformity, they’ve built a semantic knowledge graph that lets teams define core concepts once, then reuse, map, and project them everywhere. UDA is quietly transforming how data connects, scales, and stays human-readable in a world of microservices and massive content growth.

https://netflixtechblog.com/uda-unified-data-architecture-6a6aee261d8d

3. Making Every Search Rewarding: How Ibotta Transformed Offer Discovery With Databricks.

Ibotta rebuilt its offer discovery engine using Databricks to personalize, scale, and delight millions of users.

The authors in this post show how Ibotta went from prototype to production, transforming search with fine-tuned models and Databricks Vector Search. It's a story of precision, iteration, and making every query count, reducing zero-result searches by 70% and unlocking meaningful engagement at scale.

https://www.databricks.com/blog/making-every-search-rewarding-how-ibotta-transformed-offer-discovery-databricks

4.The Apache Iceberg v3 Table Spec: Celebrating the Open Source Community’s Shared Success

Apache Iceberg v3 strengthen the open data ecosystem, and what role has the open source community played in shaping its new table spec

Apache Iceberg’s new v3 table spec is more than just a technical update, it’s a story of community, collaboration, and open standards shaping the future of data. If you care about how vendor-neutral innovation happens in real time, powered by hundreds of contributors across companies. From default values to deletion vectors and geospatial types, it offers a deep dive into how open source is building scalable, flexible data infrastructure.

https://www.snowflake.com/en/blog/apache-iceberg-v3-table-spec-oss-shared-success/

5. Load Testing with Impulse at Airbnb

Airbnb usesImpulse to simulate millions of requests and ensure their systems scale reliably under peak load conditions.

Airbnb’s Impulse isn’t just a load testing tool, it’s a powerful framework that helps engineers simulate real-world traffic, mock complex dependencies, and test asynchronous flows, all within CI/CD. This post is a must-read if you care about building resilient systems at scale , it shows how decentralized, self-service load testing can catch the hard problems.

https://medium.com/airbnb-engineering/load-testing-with-impulse-at-airbnb-f466874d03d2

I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Subscribe now

1. Snowflake Intelligence: Talk to Your Data, Unlock Real Business Insights

Are you ready to talk to your data and unlock real business insights with Snowflake Intelligence?

This article explores how Snowflake Intelligence empowers business users to ask complex questions in natural language, and get instant, trusted answers from structured and unstructured data. It’s not just search; it’s reasoning across emails, documents, CRMs, and more.

https://www.snowflake.com/en/blog/intelligence-snowflake-summit-2025/

2. Introducing Cortex AISQL: Reimagining SQL into AI Query Language for Multimodal Data

How is SQL evolving to handle multimodal data in the age of AI?

In this article Cortex AISQL is transforming SQL into a language for AI-native, multimodal analytics. You can now process images, text, and soon audio—all inside Snowflake using just SQL. This post shows how AI pipelines become simple SQL queries, unlocking deep insights without complex tools.

https://www.snowflake.com/en/blog/ai-sql-query-language/

3. The shift to Grey Box Engineering and Outcome Engineering

"How are Grey Box Engineering and Outcome Engineering reshaping the future of software development and AI systems?

In this post, Adrian introduces Grey Box Engineering, a paradigm where code is accessible but often irrelevant, and where success is measured not by implementation, but by verification.

https://dlthub.com/blog/grey-box

4. Behind Viewer Retention Analytics at Scale

What does it really take to understand and improve viewer retention at scale?

In this blog, the author explores how Vimeo deciphers what keeps viewers engaged, or makes them drop off. Using ClickHouse for high-speed analytics and LLMs for AI-generated insights, they decode billions of viewing sessions with precision.

https://medium.com/vimeo-engineering-blog/behind-viewer-retention-analytics-at-scale-8dbbb5ae7ae2

5. DuckLake: SQL as a Lakehouse Format

Can SQL itself become your lakehouse format?

The author says that we’ve overcomplicated lakehouses. What if you could drop all the layers, catalogs, and manifest files, and just use SQL? Mark Raasveldt and Hannes Mühleisen introduce DuckLake, a new open table format that puts all metadata into a SQL database while still using Parquet on blob storage. It’s simpler, faster, and more resilient.

https://duckdb.org/2025/05/27/ducklake.html

All rights reserved Den Digital, India. I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Subscribe now

1. Introducing Apache Spark 4.0

"What’s new in Apache Spark 4.0, and how can it supercharge your workloads in Databricks Runtime 17.0?”

The authors present significant advancements in Spark 4.0, including SQL language enhancements, improved Python capabilities, and streaming improvements. These updates aim to make Spark more powerful, ANSI-compliant, and user-friendly, while maintaining compatibility with existing workloads. Discover how these features can elevate your data processing experience.

https://www.databricks.com/blog/introducing-apache-spark-40

2. 5 Reasons Why Azure Databricks is the Best Data + AI Platform on Azure

Is Azure Databricks the Ultimate Data + AI Platform on Azure?

This blog explores how Azure Databricks brings together governance, performance, and integrated AI in one unified platform. With deep Azure integrations, from Power BI to OpenAI, it simplifies your entire data estate while cutting costs and boosting speed.

https://www.databricks.com/blog/5-reasons-why-azure-databricks-best-data-ai-platform-azure

3. Just make it scale: An Aurora DSQL story

Explore how Aurora DSQL delivers seamless performance, serverless scaling, and simplicity, all in one powerful story of data innovation.

In this blog, the author walks us through the origin story of Aurora DSQL: a bold re-architecture of relational databases for the serverless era. It's a story of hitting scale limits, questioning decades-old assumptions, and ultimately rewriting critical components in Rust for performance, safety, and control. A must-read for engineers interested in systems design, distributed databases, and the real-world trade-offs behind big technical bets.

https://www.allthingsdistributed.com/2025/05/just-make-it-scale-an-aurora-dsql-story.html

4. How AI will Disrupt BI As We Know It

“How exactly will AI transform Business Intelligence and reshape the way we make data-driven decisions?”

The author explains that AI, powered by context protocols like MCP, will outpace any BI tool at exploratory data analysis, breaking the “conveyor belt” of report lifecycles and forcing a new model for governance and presentation.it’s fascinating because it explains why every data practitioner (from analysts to executives) stands to gain: faster, more democratized analytics without sacrificing trust or control.

The Analytics Engineering Roundup

How AI will Disrupt BI As We Know It

Business intelligence is on a collision course with AI…

Read more

a year ago · 75 likes · 5 comments · Tristan Handy

5. Iceberg?? Give it a REST!

“What if you could give Apache Iceberg a RESTful interface, how would that transform your data lake workflows?”

In this blog Anders Swanson says that the Iceberg REST Catalog (IRC) abstracts away storage complexity, letting you query and update Iceberg tables from any engine without manual file paths. It’s fascinating because it promises true multi-engine interoperability and removes last-mile friction for analytics teams.

The Analytics Engineering Roundup

Iceberg?? Give it a REST!

The analytics engineering landscape is shifting beneath our feet as our familiar data warehouse coalesces into the data engineer's lake house—all thanks to a powerful new abstraction. For us SQL lovers, the future paradoxically resembles both the present and past, yet the opportunity ahead is simply too compelling to ignore…

Read more

a year ago · 26 likes · Anders Swanson

All rights reserved Den Digital, India. I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Subscribe now

1. Apache Airflow 3.0 Is Here!

"Apache Airflow 3.0 has just landed; what’s the feature you’re most excited about?"

This article explores the latest release's transformative features, including service-oriented architecture, a modern React-based UI, DAG versioning, and enhanced security measures. I'd like you to explore how these advancements can improve your data engineering workflows.

https://www.linkedin.com/pulse/apache-airflow-30-here-fatih-aktas-9p50f/

2. Eliminating Bottlenecks in Real-Time Data Streaming: A Zomato Ads Flink Journey

"How did Zomato leverage Apache Flink to eliminate bottlenecks and streamline real-time data streaming in their ads platform?”

Zomato’s Data Platform team revamped its real-time Ads feedback system by migrating from a legacy Flink Java pipeline to a modern Flink SQL architecture with automated reconciliation. This overhaul reduced state size by over 99% (from 150 GB to 500 MB), cut infrastructure costs by $3,000 per month, and eliminated downtime. Key improvements included handling late events through a separate reconciliation job, publishing incremental counts, and leveraging Flink SQL’s efficiency, which resulted in greater reliability, scalability, and business trust.

https://blog.zomato.com/eliminating-bottlenecks-in-real-time-data-streaming-a-zomato-ads-flink-journey

3. Iceberg Operation Journey: Takeaways for DB & Server Logs

What can the Iceberg operation journey teach us about optimizing database and server log management for better performance and reliability?

This article dives into Kakao’s real-world strategies for partitioning, compressing, and optimizing Iceberg tables for both database and server logs. It’s interesting because it reveals practical experiments, like testing zstd compression levels and file compaction routines, that directly impact performance and storage efficiency.

https://tech.kakao.com/posts/695

4. Deep Dive: How Row-level Concurrency Works Out of the Box

"Ever wondered how databases manage multiple users updating the same row at the same time without chaos?

The authors explore Databricks' innovative approach to row-level concurrency, highlighting how Liquid Clustering and deletion vectors enable automatic conflict resolution during concurrent data modifications. This technique streamlines data workflows, eliminating the need for complex partitioning or retry mechanisms.

https://www.databricks.com/blog/deep-dive-how-row-level-concurrency-works-out-box

5. How Agoda Uses GPT to Optimize SQL Stored Procedures in CI/CD

"How is Agoda leveraging GPT to streamline and optimize SQL stored procedures in their CI/CD pipeline?

In this article, the author explains how Agoda integrated GPT into their CI/CD pipeline, automating performance analysis and recommendations for SQL stored procedures. This approach has significantly decreased manual review times and accelerated merge request approvals.

https://medium.com/agoda-engineering/how-agoda-uses-gpt-to-optimize-sql-stored-procedures-in-ci-cd-29caf730c46c

All rights reserved Den Digital, India. I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Subscribe now

1. SQL Gets Easier: Announcing New Pipe Syntax

What if writing complex SQL was as simple as chaining steps together with a pipe?

Most of us systematically reason through queries, start with data, filter it, and then transform it. But SQL forces you to write that logic in reverse. Databricks' new pipe syntax changes that. It lets you write SQL like you think: one clear, composable step at a time. No more nested subqueries. It's just clean, readable logic.

https://www.databricks.com/blog/sql-gets-easier-announcing-new-pipe-syntax

2. Simplifying Multimodal Data Analysis with Snowflake Cortex AI

"Can Snowflake now handle text, images, audio, and video analysis, all with just SQL?"

The blog explores how Snowflake Cortex AI COMPLETE Multimodal lets you analyze images, text, audio, and video within Snowflake using just SQL. In this blog, you no longer need to move data or juggle tools. It's a quiet but powerful shift toward unified, AI-native analytics that simplifies everything.

https://www.snowflake.com/en/blog/multimodal-data-analysis-cortex-ai/

3. Alexa+ gets us a step closer to ambient interfaces

Is Alexa+ the beginning of truly natural, invisible computing?

Jeff Blankenburg explores how Alexa+ brings us closer to ambient computing, a world where technology becomes invisible and intuitive. With generative AI, Alexa shifts from command-based responses to proper comprehension and conversation. It's a quiet leap toward the Star Trek computer we’ve always imagined.

https://www.allthingsdistributed.com/2025/04/alexa-plus-gets-us-a-step-closer-to-ambient-interfaces.html

4. Data Engineering: Now with 30% More Bullshit

Are we solving real data problems or just drowning in buzzwords and overhyped tools?

This blog looks at the current state of data engineering, where flashy terms like “Data Fabric” and “Zero ETL” often mask old problems in new wrappers. It reminds us that real progress comes not from hype but from clear thinking, solid pipelines, and a deep understanding of the business

https://luminousmen.com/post/data-engineering-now-with-30-more-bullshit

5. Iceberg?? Give it a REST!

Is Apache Iceberg + REST the missing link to true data interoperability?

In this blog, Anders Swanson explores how Apache Iceberg and the Iceberg REST Catalog (IRC) are quietly transforming analytics engineering. The author shows how Iceberg makes data feel more like a traditional database, simple, reliable, and interoperable across engines.

The Analytics Engineering Roundup

Iceberg?? Give it a REST!

The analytics engineering landscape is shifting beneath our feet as our familiar data warehouse coalesces into the data engineer's lake house—all thanks to a powerful new abstraction. For us SQL lovers, the future paradoxically resembles both the present and past, yet the opportunity ahead is simply too compelling to ignore…

Read more

a year ago · 26 likes · Anders Swanson

All rights reserved Den Digital, India. I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Subscribe now

Remote work, once a temporary response to the COVID-19 pandemic, has become a defining element of the modern workforce in 2025. What began as an emergency measure has transformed how people work, live, and interact with cities. This shift profoundly reshapes urban life, altering housing trends, economic activity, infrastructure design, and social behaviors. Let’s explore how step by step.

🧑‍💻 1. The remote work revolution: where we stand today.

By 2025, remote work will no longer be a perk; it will be the norm. Approximately 27% of workers with remote-capable jobs are fully remote, while 53% follow hybrid schedules, which combine home and office work.

The change isn’t just logistical; it’s cultural. A staggering 98% of remote workers want to continue working remotely for the rest of their careers. Companies report stable or improved productivity, and employees show higher engagement levels than those of fully in-office workers. Remote work is no longer the future; it’s the present.

🏘️ 2. Migration and housing patterns: The “Donut Effect”

The “Donut Effect” One of the most visible consequences of remote work is a shift in where people choose to live. Freed from the need to be near their offices, workers are moving away from crowded and expensive city centers to suburbs, smaller cities, and even rural areas.

💸 3. Economic Impact on cities and suburbs

Remote work has reshaped local economies by reducing demand in urban centers while boosting activity in suburban and rural areas. Cities that once thrived on office workers supporting restaurants, transit, and retail are seeing revenue declines, while suburbs benefit from increased spending by residents working from home.

4. Social and environmental implications

Remote work offers undeniable benefits, such as better work-life balance, flexibility, and reduced commute stress. However, it also introduces risks, such as social isolation and blurred boundaries between personal and work life. From a diversity and inclusion standpoint, remote work has opened doors for individuals who previously faced barriers, including those with disabilities and members of underserved communities.

⚠️ 5. Challenges and the road ahead

Despite the positives, challenges remain.

• Urban inequality: Service workers are dependent on downtown foot traffic. Cleaners, food vendors, and retail clerks face job insecurity as cities empty.

• Digital divide: Not everyone has reliable internet access or the digital literacy needed to participate in the remote economy.

• Policy gaps: Cities must act now to support urban and suburban growth. This means investing in broadband, revising zoning laws, and reimagining city design for a hybrid workforce.

Conclusion:

As remote work redefines how and where we live and work, cities and suburbs must adapt quickly to evolving economic and social dynamics. The shift presents opportunities for greater flexibility and inclusivity, as well as challenges that demand thoughtful policy and infrastructure responses. Navigating this transformation will be crucial to shaping resilient, equitable urban futures.

Do you have a project or idea?

Feel free to drop me a line. If it’s interesting, I’m in if it’s weird, even better.

💌 amanguptanalytics@gmail.com

1. How to Pitch Your Boss to Adopt Apache Iceberg?

Want to modernize your data lake? Here's how to pitch Apache Iceberg to your boss.

In this blog, the author discusses that Apache Iceberg is quietly becoming the core of a more open, flexible, and cost-efficient data future. It challenges the status quo of traditional warehouses by decoupling storage from computing, enabling true interoperability without vendor lock-in.

https://blog.det.life/how-to-pitch-your-boss-to-adopt-apache-iceberg-6da93c969f67

2. How AI will disrupt data engineering as we know it

Is AI about to change data engineering forever?

Tristan Handy says AI won’t replace data engineers; it will reshape their work entirely, automating tasks and unlocking new roles in platform engineering, automation, and business enablement. It’s a deep dive into how frameworks like dbt and tools like AI copilots are converging to change everything fast.

https://www.getdbt.com/blog/how-ai-will-disrupt-data-engineering

3. Upskilling data engineers

How do you grow as a data engineer in today’s fast-changing landscape?

Georg Heiler discusses that upskilling in data engineering isn’t about chasing the latest tools but mastering fundamentals, thinking locally, and designing scalable, testable systems. From asset graphs to metadata, mentorship to mindset, this is a practical, honest guide for anyone who wants to grow beyond buzzwords.

https://georgheiler.com/post/learning-data-engineering

4. Foundation Model for Personalized Recommendation

What if a single foundation model could power all of Netflix’s personalized recommendations?

In this article, Netflix applies LLM-style architecture to build a foundation model for personalized recommendations, moving beyond dozens of task-specific models to a unified, scalable system. By tokenizing user interactions and blending metadata with behavioral data, Netflix redefines how deep personalization works globally.

https://netflixtechblog.com/foundation-model-for-personalized-recommendation-1a0bd8e02d39

5 How to ETL at Petabyte-Scale with Trino

How do you run fast, reliable ETL at a petabyte scale without breaking the system or the budget?

The author says that Trino, with the proper tuning and architecture, can power efficient, large-scale ETL pipelines using nothing more than SQL, intelligent compaction, and thoughtful configuration. It’s a rare deep dive into how Salesforce handles petabyte ingestion quickly and clearly.

https://engineering.salesforce.com/how-to-etl-at-petabyte-scale-with-trino-5fe8ac134e36/

All rights reserved Den Digital, India. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Subscribe now

1. 2025 Guide to Architecting an Iceberg Lakehouse

"Is your data architecture ready for Iceberg in 2025?”

The author says that designing an Apache Iceberg lakehouse is not just about tools; it’s about rethinking how you govern, scale, and serve data across teams. With major 2024 shifts (like Databricks acquiring Tabular and Snowflake’s Polaris Catalog), this 2025 guide offers a clear, honest look at how to architect a modern Iceberg lakehouse, from storage and catalogs to ingestion and consumption.

https://medium.com/data-engineering-with-dremio/2025-guide-to-architecting-an-iceberg-lakehouse-9b19ed42c9de

2. Building a Data Lakehouse with Iceberg, Spark, and AWS Glue

"Can Apache Iceberg, Spark, and AWS Glue be the open stack you need for your next-gen data lakehouse?”

Ritam Mukherjee says that yes, you can, by combining Apache Iceberg, Spark, and AWS Glue into a clean, modern architecture. This short and practical guide walks you through every step, from raw S3 data to Iceberg tables and Athena queries, all without managing infrastructure.

https://medium.com/towards-data-engineering/building-a-data-lakehouse-with-iceberg-spark-and-aws-glue-d96e6e081c4f

3. Open Source Data Engineering Landscape 2025

"Is your data stack ready for the open source shift happening in 2025?"

Alireza Sadeghi discusses that open-source data engineering is not just growing; it’s reshaping itself. From zero-disk architectures and native Python libraries to catalog wars and single-node processing, this 2025 landscape maps the most important tools, trends, and tectonic shifts shaping the future of data infrastructure.

https://medium.com/@ApacheDolphinScheduler/open-source-data-engineering-landscape-2025-db53ce18d53d

4. Why Walmart Chose Apache Hudi for Their Lakehouse

"Why did one of the world’s biggest retailers pick Apache Hudi for their lakehouse over Iceberg or Delta?”

Vu Trinh discusses Walmart’s decision to ignore the hype. They chose Apache Hudi for its low-latency upserts, real-time processing, and open-source flexibility across clouds. Through deep benchmarking, they found Hudi best suited to their scale, control needs, and hybrid workloads.

https://blog.det.life/why-walmart-chose-apache-hudi-for-their-lakehouse-c0a3574db0ba

5. Step-by-Step Guide - Replicating PostgreSQL to Iceberg with OLake & AWS Glue

"How do you replicate Postgres data to Iceberg tables using OLake and AWS Glue, step by step?”

Rohan Khameshra explains that Postgres-to-Iceberg replication doesn’t need to be hard or expensive. Using OLake and AWS Glue, he walks you through a step-by-step setup that supports real-time CDC, schema evolution, and cost-efficient analytics at scale.

https://olake.io/iceberg/postgres-to-iceberg-using-glue

All rights reserved Den Digital, India. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Subscribe now

Sure, you've got ChatGPT, Claude, Gemini, and even Copilot lined up like a dream team of robot interns. But if everything the AI spits out sounds robotic and lifeless, you’re not alone. It’s not that the AI is broken. It’s just missing a decent content architect to guide the way.

As Josh Wills might say, with a few strategic tweaks, we can go from meh to marvelous. Here are five ways to improve your AI-generated content.

1. Stop telling your AI model to “just write something” ✍️

If your prompt is "Write about X," don’t be surprised when the result sounds like a high school book report. One-line prompts get you one-dimensional results.

Be specific. Seriously:

• Who’s it for? 🎯

• What’s the tone? 🗣️

• How should it look? 🖼️

Example: “For data engineers. Witty. Three short paras. End with bullets.”

Better yet:

• Show a sample

• Ask for options

The model’s not magic. It’s just really good autocomplete with a PhD. Guide it like you mean it.

For example, compare these two:

• “Write a blog post about prompt engineering.”

• “Write a 300-word post for junior data engineers explaining prompt engineering using a cooking metaphor. Make it witty, and start with a bad example.”

The second gets you usable content. The first gets you oatmeal.

2. Winging it is not a strategy 🛫

Starting without a plan? That’s the quickest way to write 800 words of structured nonsense.

Take five minutes. Make an outline.

• Yes, the AI can help you build one.

• No, it won’t know what matters until you do.

Even a few quick bullets or a rough flow sketch is enough. Then let Claude or ChatGPT do their thing.

Remember:

• Good outlines don’t have to be good. They just have to exist.

• You can fix a rough draft. You can’t fix a blank one.

Plan first. Generate second. Edit last. 🔁

It’s not magic. It’s just a process.

3. One-shot drafts are how bad takes happen 🧩

Asking your AI to write the entire piece in one go? That’s like asking it to cook dinner, do the dishes, and fold your laundry at the same time. It’ll try, but the results will be mushy.

Try this instead:

• Chunk it. Always.

• Don’t dump the whole task on the model at once.

Why?

• That’s how you get generic tone, fuzzy logic, and zero edge.

• Start small: begin with the intro.

• Read it. Fix it.

• Then move on to the next part.

• Section by section. Small asks. Tight feedback loops. 🔄

The more you focus, the better it sounds. Clarity scales with constraint.

Think of it like a relay race, not a solo sprint:

Outline → Section → Review → Iterate

This keeps the narrative tight, and your sanity intact.

4. Don’t skip the edit: POLISH, POLISH, POLISH ✨

Here’s where the magic actually happens: editing.

This is your time to shine: tighten the logic, add a clever turn of phrase, or kill the fluff with precision.

Hit “generate” and walk away? Don’t be shocked when it reads like something written by, well, a robot.

Time to take off autopilot and step in as editor-in-chief. 🧠

Start with:

• Fixing clunky phrasing

• Spotting logic gaps

• Smoothing transitions

• Making sure the tone doesn’t zigzag from Slack joke to corporate keynote

The goal isn’t perfection. It’s flow, with a bit of flavor.

Drop in a metaphor. Chop a paragraph in half. Rebuild a sentence so it actually makes sense.

This is where your voice shows up. It’s also where most AI content quietly crumbles if you skip it.

Golden rule: if it sounds like a machine wrote it, it’s not done. One edit makes it better. Two? That’s when the good stuff hits.

5. Your AI wrote it. Now make it sound like you did. 🎤

Just because AI lent a hand doesn’t mean it has to read like instructions on assembling IKEA furniture.

Bland content is background noise. Your voice is the signal. 📡

So punch it up:

• Crack a joke.

• Toss in a metaphor.

• Mention something your audience actually gets. Bonus points if it’s not from a case study.

Prompt with intent: "Write with the snark of Josh Wills and the clarity of an O’Reilly book." Suddenly, your model has a compass instead of a blindfold.

Still feels off? No problem:

• Ask for three takes in different voices: snappy, serious, playful.

• Feed in writing samples you like: Josh Wills, Lex Fridman, Ali Abdaal, Douglas Adams. Let the model riff.

When one lands, lock in the tone. Apply it section by section.

A single sharp analogy can turn a forgettable paragraph into the line people quote back to you.

If your AI-written blog reads like corporate onboarding docs, it deserves to be ignored. Set the bar higher than “not embarrassing.”

Boring fades. Voice lasts. Make yours count.

6. Bonus: Use Feedback Loops – Don’t create in a vacuum 🔁

Critique your content.

It’s not a common practice, but it’s a best one. Ask AI to critique your content and tell how it scores and what the weak areas are. For example, I use a GPT with books from Seth Godin and Chip Heath in it. And I ask it something like:

"Critique my content according to principles of Seth Godin and Chip Heath. Tell me what works best and what can be improved. Give the current article a score for likely performance."

And then paste the article in.

Simple, right? But very effective. It lets you quickly vet your draft so that poor quality doesn’t creep in.

Great creators seek feedback. After drafting, don’t just hit publish. Put your content through a feedback wringer. Ask a colleague or friend to tear it apart (better them than your readers). Or use the AI itself as a critic: have ChatGPT analyze Claude’s output or vice versa and see what weaknesses they spot.

Often, I’ll prompt the AI with:

"Critique the above and suggest improvements."

– and you’ll get surprisingly useful editing suggestions.

Another pro move: read your content

out loud 🔊

(or use a text-to-speech tool) to catch awkward phrasing or pacing issues. If you stumble while reading, that part needs fixing.

Some folks even record themselves giving feedback and feed that back into the AI to implement changes. Also, leverage your community. Drop a snippet in your team’s Slack channel for quick reactions.

The point is to create a feedback loop between you, AI, and real humans to refine the content. Iteration isn’t a sign your first try was bad. It’s how you make good content great.

Conclusion

AI moves fast, and tools change weekly. What matters is how you use them. Try new prompts, automate the boring parts, and fine-tune models for your writing. Watch what works, double down, and cut the rest.

Mediocre AI content is a choice. With the right workflow, you’re not just creating. You’re compounding. So iterate, measure, and improve. Now stop reading and make something worth sharing. 🚀

Got a project or idea?

Drop me a line. If it’s interesting, I’m in. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now

1. Should Stakeholders be Writing SQL for Self-Service?

"Should business stakeholders write their own SQL, or is that a data team's job?"

This post explores the quiet chaos that unfolds when non-analysts write SQL, from silent bugs and broken logic to rising costs and conflicting truths. With stories from Amazon, eBay, and VMware, Brandon shows how good intentions in self-service can lead to dangerous outcomes.

https://medium.com/@analyticsmentor/should-stakeholders-be-writing-sql-for-self-service-289561fbbbbf

2. Introducing Meta’s Llama 4 on the Databricks Data Intelligence Platform

"Ready to unlock next-gen AI with Meta’s Llama 4 on Databricks?"

This blog explores how Meta’s Llama 4, now live on Databricks, lets you build powerful, multimodal AI agents grounded in your own data. The authors explore how features like Unity Catalog, Mosaic AI, and Test-Time Adaptive Optimization make it possible to scale safely, customize fast, and deploy without infra complexity.

https://www.databricks.com/blog/introducing-metas-llama-4-databricks-data-intelligence-platform

3. Announcing Anthropic Claude 3.7 Sonnet is natively available in Databricks

"What can your enterprise build with Claude 3.7 Sonnet now natively available on Databricks?"

The author says that Claude 3.7 Sonnet, Anthropic’s most advanced model, is now natively available in Databricks, unlocking hybrid reasoning, agentic workflows, and full governance at scale. You can now build intelligent, domain-specific AI agents directly on your enterprise data without managing infrastructure.

https://www.databricks.com/blog/anthropic-claude-37-sonnet-now-natively-available-databricks

4. How to Load Data into Apache Iceberg: A Step-by-Step Tutorial

"Curious how to load data into Apache Iceberg without getting lost in the lake? 🧊

The author says that loading data into Iceberg isn't just about writing files; it's about mastering metadata, schema evolution, and real-time ingestion. Whether you're using Spark, Flink, or Python, this tutorial walks you through it all with clarity and depth. It's a must-read if you care about building future-proof data lakes.

https://estuary.dev/blog/loading-data-into-apache-iceberg/

5. How Apache Iceberg Powers the Data Lake and Trino Makes It Explorable

"Ever wondered how Apache Iceberg turns your data lake into a powerful warehouse, and how Trino lets you query it like a pro?”

Rakesh Rathi says that Apache Iceberg brings order, version control, and safety to your data lake, while Trino gives you the power to query it all with simple SQL. If you’ve ever dreamed of joining S3-based Iceberg tables with live PostgreSQL data without moving anything, this blog shows you how. It's a must-read for anyone building the next-gen data platform.

https://medium.com/@rakesh.open.source/how-apache-iceberg-powers-the-data-lake-and-trino-makes-it-explorable-4315084d1928

6. Bufstream: Stream Kafka Messages to Iceberg Tables in Minutes

"Want to stream Kafka messages into Apache Iceberg tables in just minutes, without the usual complexity?”

In this article, Bufstream reimagines Kafka for the cloud: cheaper, stateless, schema-aware, and natively integrated with Apache Iceberg. It replaces the need for fragile pipelines and lets you stream Kafka messages into Iceberg tables in minutes. If you're serious about simplifying real-time data architecture, this read is for you.

https://blog.det.life/bufstream-stream-kafka-messages-to-iceberg-tables-in-minutes-6c60c470e67f

All rights reserved Den Digital, India. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Subscribe now

1. "Ever wondered how to efficiently load data into Apache Iceberg tables?”

The author describes that efficient data loading is the key to unlocking Apache Iceberg's full potential. If you're wondering how to manage petabyte-scale data with tools like Spark, Flink, and even Python, this tutorial will walk you through it step by step. It's a hands-on, no-fluff guide to building reliable, scalable pipelines for modern data lakehouses.

https://estuary.dev/blog/loading-data-into-apache-iceberg/

2. Apache Paimon vs. Apache Iceberg - A Detailed Comparison

"Apache Paimon or Apache Iceberg, which table format is right for your data stack?”

Priyansh Khodiyar says that choosing the right table format is not just a technical decision; it’s a strategic one. Are you optimizing for real-time insights or batch-scale analytics? This article delivers a deep, no-fluff comparison of Paimon and Iceberg, packed with real-world benchmarks and use cases to guide your decision.

https://olake.io/iceberg/paimon-vs-iceberg

3. Why move to Apache Iceberg - A Practical Guide to Building an Open, Multi-Engine Data Lake?

In this blog, the author describes Apache Iceberg as not just hype; it’s a practical shift for teams managing massive, multi-engine data workloads. But is it right for you? This guide breaks down the real benefits, hidden challenges, and when not to adopt Iceberg. It's a must-read if you're building a data platform for scale and flexibility.

https://olake.io/iceberg/move-to-iceberg

4. The majesty of Apache Flink and Paimon.

"What makes Apache Flink and Paimon a powerful duo for real-time data processing?”

In this article, Giannis Polyzos argues that Apache Flink and Paimon offer more than just performance; they offer a new paradigm for real-time lakehouse architecture. Are you wondering how to simplify streaming, handle CDC, and enable low-latency analytics with fewer moving parts? This piece explains why Paimon is built for Flink and why that matters.

https://medium.com/@ipolyzos_/the-majesty-of-apache-flink-and-paimon-d36e73571fc9

5. Accelerating Large-Scale Test Migration with LLMs

"Can large language models really speed up large-scale test migrations?

Charles Covey-Brandt advocates that large language models aren’t just for chat; they can drive real, production-scale code migrations. Want to see how Airbnb updated 3,500+ test files in weeks instead of years? This post breaks down the strategy, tooling, and lessons learned from their Enzyme-to-RTL migration using LLMs.

All rights reserved by Den Digital, India. I have provided links for informational purposes and do not imply endorsement. All views expressed in this newsletter are my own and do not represent the opinions of any current, former, or future employers.

Subscribe now

1. Merge-on-Read vs Copy-on-Write in Apache Iceberg

What is the difference between merge-on-read and copy-on-write in Apache Iceberg?

The author says that Apache Iceberg’s Merge-on-Read and Copy-on-Write approaches offer a nuanced balance between rapid updates and seamless reads. Discover why this dynamic is not only intellectually compelling but also critical for modern data challenges.

https://olake.io/iceberg/mor-vs-cow

2. Apache Iceberg Copy-On-Write (COW) vs Merge-On-Read (MOR): A Deep Dive

Apache Iceberg: Should you choose copy-on-write or merge-on-read?

In this article, Dani Pálma explores Copy-On-Write (COW) and Merge-On-Read (MOR), two approaches with distinct trade-offs in speed, storage, and efficiency. With PySpark examples and real-world insights, this deep dive helps you pick the best strategy for your data workloads.

https://estuary.dev/blog/apache-iceberg-cow-vs-mor/

3. Iceberg Catalog Showdown: Apache Polaris vs Unity Catalog

How do Apache Polaris and Unity Catalog compare when managing iceberg tables?

In this article, Karen Zhang explores Apache Polaris and Unity Catalog, two powerful data catalogs shaping the future of cloud data management. Polaris offers multi-engine flexibility, while Unity Catalog seamlessly integrates with Databricks, but which one is right for your architecture?

https://estuary.dev/blog/iceberg-catalog-apache-polaris-vs-unity-catalog/

4. Introducing Serverless Batch Inference

How do you run large-scale llm batch inference with zero setup?

This article introduces serverless AI functions, enabling effortless, 10x faster batch inference directly in Databricks. There is no infrastructure to manage, just seamless AI-powered workflows.

https://www.databricks.com/blog/introducing-serverless-batch-inference

5. Title Launch Observability at Netflix Scale

How does Netflix monitor and scale observability for millions of users?

The author, Varun Khaitan, explores how Netflix scales observability for title launches, from real-time monitoring to "Time Travel" simulations that catch issues before they happen. With Kafka queues, Hollow Feeds, and proactive insights, Netflix ensures every story reaches its audience seamlessly.

https://netflixtechblog.com/title-launch-observability-at-netflix-scale-8efe69ebd653

All rights reserved Den Digital, India. I have provided links for informational purposes and do not suggest endorsement. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Subscribe now