1. Sqlmesh delivers 22x faster data transformation and 10x cost savings vs dbt Core™ on Snowflake.

Experience lightning-fast transformations with Sqlmesh, 22x speed improvements and 10x lower Snowflake spend vs dbt Core™.

SQLMesh achieves 22x faster execution and 10x lower Snowflake costs, not through tweaks, but through architectural reinvention. Virtual environments eliminate wasteful rebuilds. Column-level lineage prevents cascade updates. State-aware deployments reuse computation instead of repeating it. over $1M saved annually and 311 hours of dev time reclaimed each month.

https://www.tobikodata.com/blog/sqlmesh-delivers-22x-faster-data-transformation-and-10x-cost-savings-vs-dbt-core-tm-on-snowflake

2. "Michael, dlt, and the art of unbreakable API pipelines."

Building unbreakable API pipelines with Michael and dlt.

Michael Shoemaker shows how to build API pipelines that survive change, automatically. Using dlt, BigQuery, and Cloud Functions, he creates a zero-touch setup that handles new fields, schema shifts, and daily runs without manual fixes or duct tape.

https://dlthub.com/blog/michael-dlt-tutorials

3. A practitioner’s guide to LLM-native pipeline building with dltHub workspace.

LLM pipelines made easy with dltHub workspace.

Data engineering is shifting, from manual coding to AI-assisted assembly. This piece introduces LLM-native scaffolds in dltHub Workspace: structured, reusable blueprints that let language models generate reliable pipelines in minutes. Human judgment ensures accuracy, while the framework handles the complexity.

https://dlthub.com/blog/practitioners-guide-dlthub-workspace

4. You're doing quarterly planning wrong

Rethinking quarterly planning

Quarterly planning doesn’t need to be perfect, just useful. This post shares hard-earned insights from coordinating 26+ teams, why async prep beats long meetings, how to focus on what actually ships, and why true ownership belongs with engineers.

https://posthog.com/newsletter/quarterly-planning-mistakes

Note: I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Do you have a project or idea?

Feel free to drop me a line. If it’s interesting, let’s chat. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now

1. What’s new with Azure Databricks: unified governance, open formats, and AI-native workloads

The future of data on Azure: Databricks brings unified governance and AI to the lakehouse

Azure Databricks is evolving into the intelligent core of the modern data stack. With updates like Databricks One, Lakebase, Lakeflow, and enhanced Unity Catalog, it now offers a unified, open, and secure experience for building, governing, and scaling AI-native workloads on Azure.

https://www.databricks.com/blog/whats-new-azure-databricks-unified-governance-open-formats-and-ai-native-workloads

2. Netflix Tudum architecture: from CQRS with Kafka to CQRS with RAW Hollow

How Netflix Tudum evolved: from Kafka CQRS to Raw Hollow efficiency

Netflix transformed Tudum’s architecture, moving from a Kafka-based CQRS setup to a RAW Hollow-powered model. The shift brought leaner infrastructure, faster updates, and a smoother experience at scale. An example of how streamlining infrastructure can unlock both speed and reliability.

https://netflixtechblog.com/netflix-tudum-architecture-from-cqrs-with-kafka-to-cqrs-with-raw-hollow-86d141b72e52

3.Sync data from MySQL to Iceberg

Syncing data from MySQL to Apache Iceberg can be seamless using BladePipe, a lightweight alternative to traditional Kafka-based pipelines. With real-time ingestion, minimal latency, and a simplified setup, it offers an efficient path to building a modern lakehouse.

https://doc.bladepipe.com/blog/tech_share/mysql_iceberg_sync

4. Making sense of Apache Iceberg statistics

Understanding Apache Iceberg table statistics

Guy Yasoor breaks down the key layers of Apache Iceberg’s statistics, from Parquet min/max to Puffin files and Bloom filters. These aren’t just backend details, they’re what enable faster queries, better planning, and lower compute costs.

https://www.ryft.io/blog/making-sense-of-apache-iceberg-statistics

5. The Apache Iceberg avalanche: how the open table format changes the face of data lakes

The Apache Iceberg shift: transforming data lakes with open table formats

Apache Iceberg bridges the gap between flexible data lakes and high-performance warehouses. By decoupling compute, storage, and metadata, it gives modern data teams freedom, interoperability, and control. Iceberg is rapidly becoming a foundational piece in next-gen data platforms, powering open, efficient, and future-proof lakehouses.

https://www.snowflake.com/en/blog/apache-iceberg-data-lakehouse-architecture/

Note: I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Do you have a project or idea?

Feel free to drop me a line. If it’s interesting, let’s chat. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now

1. Driving content delivery efficiency through classifying cache misses

Optimizing content delivery by tracking and categorizing cache misses

Netflix tries to stream from the closest server for speed and quality. But sometimes it misses: that’s a cache miss. When that happens, Netflix uses real-time data, health checks, and predictions to quickly reroute your stream. That’s how it keeps things running smoothly.

https://netflixtechblog.com/driving-content-delivery-efficiency-through-classifying-cache-misses-ffcf08026b6c

2.Querying Apache Iceberg with Sub-Second performance

Apache Iceberg wasn’t built for low latency, but Firebolt makes it possible. With smart metadata caching, Parquet optimizations, and subresult reuse, interactive queries run in under a second, without moving data. For anyone working with large datasets, this changes what’s possible.

https://www.firebolt.io/blog/querying-apache-iceberg-with-sub-second-performance

3. All about Iceberg partitioning and partitioning writing strategies

Optimizing data layout with Iceberg partitioning and write strategies

Apache Iceberg brings precision to partitioning with rich metadata that enables manifest pruning and file skipping, dramatically improving query performance. PartitionSpec and manifest lists let query engines bypass irrelevant data files with ease, avoiding costly scans.The blog explores four file writing strategies tailored for different ingestion needs, from high-throughput, unordered streaming to memory-efficient, sorted batch writes. Each approach balances schema evolution, memory usage, and write patterns.

https://olake.io/iceberg/iceberg-partitioning-and-writing-strategies

4. OLake + Glue + Snowflake - A deep dive into modern data partitioning

Building an open, efficient lakehouse with OLake, Glue & Snowflake

Modern data ingestion doesn’t have to be fragile, expensive, or vendor-locked. With OLake, Apache Iceberg, AWS Glue, and Snowflake, you get a fully open, scalable, and efficient pipeline from databases to your lakehouse, no copy-paste ETL nightmares

https://olake.io/iceberg/olake-glue-snowflake

5. How Unity Catalog managed tables automate performance at scale

Smarter tables, less work: Unity Catalog’s secret to hands-free optimization.

Unity Catalog managed tables bring AI-driven optimizations, like auto-clustering, smart vacuuming, and metadata caching, that reduce storage costs by 50%+ and boost query speeds by up to 20x. No tuning, no scripts, just better performance at scale.

https://www.databricks.com/blog/how-unity-catalog-managed-tables-automate-performance-scale

Note: I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Do you have a project or idea?

Feel free to drop me a line. If it’s interesting, let’s chat. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now

1.Announcing Lakebase Public Preview

Lakebase public preview, a SQL-first lakehouse built for performance.

Traditional databases struggle with scale, speed, and modern AI workflows. They sit siloed from analytics, making intelligent application development harder than it should be. Lakebase is a serverless, Postgres-compatible database, integrated directly into the lakehouse. Built for real-time feature serving, model interaction, and unified governance , it marks a quiet but significant shift in how data apps are developed.

https://www.databricks.com/blog/announcing-lakebase-public-preview

2. Change Takes More Than a Megaphone: Communicate, Experiment and Educate to Drive Transformation

Transformation demands more than speaking up: it requires communication, experimentation, and education.

Real AI adoption doesn’t go viral, it builds through strong ties, repeated exposure, and shared success. When people see value, they follow. The organizations winning with AI are the ones educating, experimenting, and elevating others.

https://www.snowflake.com/en/blog/change-transformation-communication-education-ai/

3. SQLMesh delivers 22x faster data transformation and 10x cost savings vs dbt Core™ on Snowflake

Sqlmesh outperforms dbt Core on Snowflake with 22x faster transformations and 10x lower costs.

SQLMesh reimagines how data is transformed on Snowflake. By replacing physical rebuilds with virtual environments, tracking changes at the column level, and eliminating redundant computation, it saves over $1M annually and frees up 311 hours of developer time each month.

https://www.tobikodata.com/blog/sqlmesh-delivers-22x-faster-data-transformation-and-10x-cost-savings-vs-dbt-core-tm-on-snowflake

4. The complete stream processing journey on FlinkSQL

The stream processing lifecycle, powered by FlinkSQL.

Grab rebuilt its stream processing stack, from Zeppelin notebooks to a shared FlinkSQL gateway, cutting query times by 80% and enabling pipeline deployment in under 10 minutes. A compelling look at how stream processing can be fast, collaborative, and production-ready.

https://engineering.grab.com/the-complete-stream-processing-journey-on-flinksql

5.Macroeconomics and the data industry

Understanding the economic undercurrents driving data work today.

Venture capital flows, shifting interest rates, and economic cycles ripple through every layer of the data world, reshaping tools, budgets, and even career paths. Understanding these forces is key to navigating rapid change and making smarter decisions as a data professional.

The Analytics Engineering Roundup

Macroeconomics and the data industry

New podcast episode! Julia and I spoke with Kevin Hu of Metaplane. Whether you’re mapping the emerging data observability landscape or just love a good physics metaphor, give it a listen…

Read more

4 years ago · 19 likes · Tristan Handy

Note: I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Do you have a project or idea?

Feel free to drop me a line. If it’s interesting, let’s chat. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now

1. Leveraging Agentic AI in Games

Agentic AI turn your game into a living, breathing world.

Agentic AI brings a new layer of intelligence to games, NPCs that remember your actions, QA bots that adapt as the game evolves, and support systems that respond with real understanding. It’s not about replacing creative teams, but giving them tools to build deeper, more personal, and more responsive experiences.

https://www.databricks.com/blog/leveraging-agentic-ai-games

2. Model Once, Represent Everywhere: UDA (Unified Data Architecture) at Netflix

Netflix unify petabytes of data across teams, tools, and use cases, without breaking everything

Netflix is solving one of the hardest problems in modern data systems: fragmented, duplicated, inconsistent models across platforms. Instead of forcing uniformity, they’ve built a semantic knowledge graph that lets teams define core concepts once, then reuse, map, and project them everywhere. UDA is quietly transforming how data connects, scales, and stays human-readable in a world of microservices and massive content growth.

https://netflixtechblog.com/uda-unified-data-architecture-6a6aee261d8d

3. Making Every Search Rewarding: How Ibotta Transformed Offer Discovery With Databricks.

Ibotta rebuilt its offer discovery engine using Databricks to personalize, scale, and delight millions of users.

The authors in this post show how Ibotta went from prototype to production, transforming search with fine-tuned models and Databricks Vector Search. It's a story of precision, iteration, and making every query count, reducing zero-result searches by 70% and unlocking meaningful engagement at scale.

https://www.databricks.com/blog/making-every-search-rewarding-how-ibotta-transformed-offer-discovery-databricks

4.The Apache Iceberg v3 Table Spec: Celebrating the Open Source Community’s Shared Success

Apache Iceberg v3 strengthen the open data ecosystem, and what role has the open source community played in shaping its new table spec

Apache Iceberg’s new v3 table spec is more than just a technical update, it’s a story of community, collaboration, and open standards shaping the future of data. If you care about how vendor-neutral innovation happens in real time, powered by hundreds of contributors across companies. From default values to deletion vectors and geospatial types, it offers a deep dive into how open source is building scalable, flexible data infrastructure.

https://www.snowflake.com/en/blog/apache-iceberg-v3-table-spec-oss-shared-success/

5. Load Testing with Impulse at Airbnb

Airbnb usesImpulse to simulate millions of requests and ensure their systems scale reliably under peak load conditions.

Airbnb’s Impulse isn’t just a load testing tool, it’s a powerful framework that helps engineers simulate real-world traffic, mock complex dependencies, and test asynchronous flows, all within CI/CD. This post is a must-read if you care about building resilient systems at scale , it shows how decentralized, self-service load testing can catch the hard problems.

https://medium.com/airbnb-engineering/load-testing-with-impulse-at-airbnb-f466874d03d2

I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Subscribe now

1. Snowflake Intelligence: Talk to Your Data, Unlock Real Business Insights

Are you ready to talk to your data and unlock real business insights with Snowflake Intelligence?

This article explores how Snowflake Intelligence empowers business users to ask complex questions in natural language, and get instant, trusted answers from structured and unstructured data. It’s not just search; it’s reasoning across emails, documents, CRMs, and more.

https://www.snowflake.com/en/blog/intelligence-snowflake-summit-2025/

2. Introducing Cortex AISQL: Reimagining SQL into AI Query Language for Multimodal Data

How is SQL evolving to handle multimodal data in the age of AI?

In this article Cortex AISQL is transforming SQL into a language for AI-native, multimodal analytics. You can now process images, text, and soon audio—all inside Snowflake using just SQL. This post shows how AI pipelines become simple SQL queries, unlocking deep insights without complex tools.

https://www.snowflake.com/en/blog/ai-sql-query-language/

3. The shift to Grey Box Engineering and Outcome Engineering

"How are Grey Box Engineering and Outcome Engineering reshaping the future of software development and AI systems?

In this post, Adrian introduces Grey Box Engineering, a paradigm where code is accessible but often irrelevant, and where success is measured not by implementation, but by verification.

https://dlthub.com/blog/grey-box

4. Behind Viewer Retention Analytics at Scale

What does it really take to understand and improve viewer retention at scale?

In this blog, the author explores how Vimeo deciphers what keeps viewers engaged, or makes them drop off. Using ClickHouse for high-speed analytics and LLMs for AI-generated insights, they decode billions of viewing sessions with precision.

https://medium.com/vimeo-engineering-blog/behind-viewer-retention-analytics-at-scale-8dbbb5ae7ae2

5. DuckLake: SQL as a Lakehouse Format

Can SQL itself become your lakehouse format?

The author says that we’ve overcomplicated lakehouses. What if you could drop all the layers, catalogs, and manifest files, and just use SQL? Mark Raasveldt and Hannes Mühleisen introduce DuckLake, a new open table format that puts all metadata into a SQL database while still using Parquet on blob storage. It’s simpler, faster, and more resilient.

https://duckdb.org/2025/05/27/ducklake.html

All rights reserved Den Digital, India. I have provided links for informational purposes and do not suggest endorsement to you. All views expressed in this newsletter are my own and do not represent current, former, or future employer” opinions.

Subscribe now