1). “The irony of data: built for speed, stuck in traffic.”

“Data’s a trend-chaser. First a warehouse, then a lake, now a lakehouse. But behind the buzzwords, it’s always chasing the same thing: `CLARITY`.”

Data’s like that seasoned pro who’s seen every fad come and go, warehouse, lake, lakehouse, whatever’s next.

It doesn’t need a rebrand to stay relevant because it’s still solving the same timeless puzzle: making chaos make sense.

Humans, queries, entire companies still rely on it to turn raw noise into reason.

So , it’s old, but like vinyl or good coffee, that’s exactly why it never goes out of style.

2). The problem: today’s data chaos hiding inside modern warehouses.

“Modern context” in very easy words means how something fits or makes sense in today’s world, not in the past.

When we say “in the modern context,” it means we are looking at something based on how people live, think, and use new ideas and technology today. It’s about understanding how things fit in our present time, not how they were in the past.

Example:

In the past, very few women worked outside the home. But today, in the modern context, women have jobs and build careers. So society’s idea of women’s roles has changed to fit modern life.

Cloud warehouses made data cheap and fast → so people just started to dump everything in them.

The magnitude of data has exploded ; especially after 2020.

In 2020, the world generated around 64.2 zettabytes of data that’s 64.2 trillion gigabytes. Since then, global data creation has surged exponentially, fueled by the rise of AI, IoT, remote work, and digital media. By 2025, projections from DemandSage and Seagate estimate total data generation will reach 181 zettabytes nearly three times what it was in 2020, growing at an average rate of 22–26% per year.

Cloud data warehouses made storing all this information cheap, fast, and scalable, leading many organizations to adopt a “dump everything” mindset pouring every dataset into the warehouse without much structure or strategy. While this worked at first, it quickly created chaos: hundreds of tables, unclear relationships, and multiple conflicting versions of key metrics like “revenue.”

Without proper cleaning, validation, or governance, data becomes cluttered, slow to query, and hard to trust.

It’s like blending all your groceries together and calling it dinner technically everything’s there, but nobody wants to touch it.

The result is data overload, slower performance, and unreliable insights proving that even in an age of infinite storage, organized data still wins.

Graphical representation:

Emphasize the pain:

• Analysts don’t trust dashboards because the data behind them is messy or wrong, leading to incorrect insights.

• Metrics don’t match across reports or tables, making it impossible to know which numbers are true.

• Queries take forever to run since the system is overloaded with unorganized, duplicated, or conflicting data.

Humour:

“If you’ve ever joined five tables just to figure out who bought what, welcome to the world of data wrestling , where dimensional modeling is the referee that stops the chaos and declares a clear winner!”

3). The promise: A way to make data actually make sense.

Dimensional modeling is the proven way to structure data so anyone, not just engineers, can navigate it confidently and get clear, reliable answers to real business questions.

It is based on two core ideas:

1. Facts: These are the important numbers or measurements in your business, like sales amount, number of units sold, or revenue. Facts represent what you want to measure or analyze.

2. Dimensions: These are the descriptive details or context about the facts, such as time (day, month, year), product (name, category), customer (age, location), or store. Dimensions help you answer questions like “When was it sold?” or “Who bought it?” or “Where was it sold?”

The fact tables hold the measurable data, while the dimension tables hold the descriptive information.

By organizing data into facts and dimensions, dimensional modeling improves data retrieval speed, clarity, and usability in data warehouses

Simple visual example student_attendence

Captures the measurable event: a student attending a class.

✅ Fact table = the event itself (who attended what, when, and for how long). It stores numeric or measurable data.

Dimension Table – students

Describes who attended.

📚 Dimension Table – courses

Describes what they attended.

🔗 Relationships:

• student_attendance.student_id → students.student_id

• student_attendance.course_id → courses.course_id

In simple terms:

• Fact table = the measurable event (attendance).

• Dimension tables = the descriptive context (who the student is, what course they attended).

“Trying to analyze data without dimensional modeling is like trying to untangle Christmas lights frustrating, time-consuming, and making you wonder if you should just buy a new set!”

4. “Why it still matters ; even in the age of dbt and the modern data stack”.

Because tools like dbt, Airflow, and the rest of the modern data stack are powerful , but they don’t replace the need for good modeling, they just automate and scale it.

Dimensional modeling still matters because:

• Supports modern tools: dbt, metrics layers, and BI platforms still rely on a solid data model underneath to work effectively.

• Scales with complexity: As data grows and evolves, a dimensional model keeps queries simple and performance predictable.

• Timeless foundation: Technologies change, but the need for clarity, consistency, and context in analytics never does.

It’s the difference between following a clear path and getting lost without a clue where to go.

5. The myth: “Isn’t a data lake enough?”

“Sure, you can run analytics directly on raw JSON files in S3, if you’re okay with messy queries, slow results, and a lot of confusion every time you need an answer.”

Just storing all your data in one place (like a data lake in S3) doesn’t mean it’s ready to use. That’s like having all the ingredients for a cake but no recipe, the pieces are there, but they’re not useful yet.

Sure, you can run analytics directly on messy raw JSON files in S3, but you’ll end up with confusing queries, slow results, and numbers that don’t match from one dashboard to another.

That’s where modeling comes in. It’s the process of organizing and structuring your data so it’s clear, consistent, and easy to work with.

Once you model your data:

• Metrics stay consistent — everyone agrees on what “active user” or “revenue” means.

• Queries run faster — because the data is organized and optimized.

• New people can understand the data quickly — because there’s a clear structure.

Key idea: Tools will always change, but the basic principle of organizing your data so humans (and machines) can understand it will never stop being important.

6. Example: online retail store

“Think of an online retail store managing thousands of daily orders. The fact table records each sale, what was sold, who bought it, where, and when. The dimension tables add meaning: products describe what was sold, customers tell who bought it, stores show where it happened, and time shows when. With this setup, the business can quickly ask questions like ‘Which region generated the most revenue this month?’ or ‘Which product categories are gaining popularity over time?

Step 1: Create Tables

Step 2: Insert Sample Data

Step 3: Query to Generate Result

Then output:

This complete query sequence, from start to end models customer purchases cleanly, showcasing how the fact table logs each order and the dimension tables describe who, what, and when.

Show how this model powers metrics:

• “Want monthly revenue by region? Easy aggregate sales by time and region dimensions.”

• “Want retention by customer type? Join facts to the customer dimension.”

1. Monthly Revenue by Region:

Output:

Graph:

2. Retention by Customer Type:

Output:

Graph:

“Add a key takeaway”:

“Good modeling turns technical structure into business clarity; everyone speaks the same data language.”

7.“Final reflection: What’s Next”

Key Takeaways:

• Dimensional modeling is all about clarity.

• It streamlines SQL, strengthens metric accuracy, and turns dashboards into tools you’re actually proud to share.

Structured data models do more than organize information they build trust and consistency across every report. Facts capture what happens; dimensions explain why and where it happens. Together, they turn raw data into clear, actionable insight.

Up Next:

In our next blog, we’ll meet the true stars of the analytics world facts and dimensions tables and learn how to tell them apart.

1. Introduction:

SQL: not just SELECT, meet the power tools you forgot existed

SQL isn’t just about pulling rows with a simple SELECT *. It’s a language packed with hidden features that can make your queries sharper, faster, and far more expressive. Beyond the basics, SQL offers advanced capabilities like Grouping Sets, Rollups, and Self Joins , tools that many developers overlook, but which can completely change how you approach reporting and analysis. Once you know how to use them, you’ll find yourself solving problems more effectively and writing queries that do far more with less code.

2. Foundations: why not just GROUP BY?

Before we get into the advanced stuff, let’s take a quick refresher. GROUP BY is where most SQL users start when they need summaries.It collapses rows, applies aggregates, and gets you answers fast, totals by region, averages by customer, counts by category.

The problem

GROUP BY answers one question at a time.

if you need totals by region, by product, and the overall total, you end up writing (and maintaining) three queries, or gluing them with UNION ALL.

That’s where Grouping Sets, Rollups, and Self Joins come in. They’re designed to move you past single-level aggregations, so you can capture multiple perspectives, richer breakdowns, and even relational insights, all without writing query after query.

3. Grouping sets: multiple summaries in one query

Grouping Sets are like having multiple GROUP BY clauses rolled into a single query. Instead of writing and unioning separate aggregations, you tell SQL exactly which “buckets” you care about, and it returns them all at once.

Example: one query instead of three

Say you want sales totals:

1. By country

2. By product

3. Overall total

With plain GROUP BY, you’d have to write three separate queries and glue them with UNION ALL:

With Grouping Sets, it’s one clean query:

output:

✅ Result: You get all three aggregations in a single result set, no unions, no duplicate queries.

When to use grouping sets:

• Dashboards → where you want country-level, product-level, and global KPIs all pulled at once.

• Reports → monthly or quarterly summaries that combine different aggregation levels side by side.

• Multi-level summaries → subtotals, grand totals, and detailed breakdowns generated in one pass.

  In short:

  Anytime you find yourself writing the same SUM or COUNT three different ways, Grouping Sets can probably collapse it into a single, elegant query.

4. Rollups

What Are rollups?

A Rollup is a GROUP BY shortcut that creates progressive subtotals along a hierarchy. You give SQL an ordered list of columns, and it automatically returns results at every level, from the most detailed grouping down to the grand total.

Example: website traffic by date

Suppose you’re analyzing website visits and want to see:

• Visits by year and month

• Visits by year

• The overall total

You can do this with a single query using ROLLUP:

output:

👉 So with a single query using ROLLUP, you’ve got:

• year → Year of the visit(s).

• month → Month of the visit(s).

• total_visits → How many visits happened in that year/month.

• row_type → Label telling you whether the row is monthly detail, a yearly subtotal, or the overall grand total.

  How rollups compare to grouping sets:

  Both Grouping Sets and Rollups let you collapse multiple aggregations into one query, the difference is in control vs convenience.

• Grouping Sets → You spell out each grouping you want. Perfect when you need a custom mix (e.g. totals by country, by product, but not both together).

• Rollups → You hand SQL a hierarchy, and it builds every subtotal and the grand

  total for you. Less typing, cleaner code, but the shape is fixed.

In short:

• Use Grouping Sets when you want custom, non-hierarchical views.

• Use Rollups when you want hierarchical roll-ups with minimal effort.

Analogy

“Rollups are like elevator stops: you can get off at the city floor, the state floor, the country floor, or ride all the way up to the grand total.”

It conveys the hierarchical nature (each level builds on the last), but in a way that’s easy to visualize.

5. Self Joins:

A self join is simply joining a table to itself. Instead of pulling data from two different tables, you treat the same table as if it were two, usually by giving each instance an alias.

Real-world examples:

• Comparing rows:

  • In a weather table, join each city’s record to another city’s record on the same date to compare conditions (e.g., New York’s temperature vs. Los Angeles’ temperature on June

• Time-series analysis:

  • In the same weather table, join today’s weather record for a city to yesterday’s record to track day-over-day changes in temperature, rainfall, or humidity.

Output:

✅ In one query, you get:

• Row comparison → city vs. city, same date.

• Time-series → today vs. yesterday, same city.

• Changes over time in temperature, rainfall, humidity.

Why self joins matter in analytics:

• Comparisons → how one row stacks up against another (city vs. city, product vs. similar product).

• Trends → how a row evolves over time (today vs. yesterday’s weather, last month vs. this month’s reading).

• Hierarchies → how rows connect in a structure (employee vs. manager, category vs. subcategory).

In analytics, this means you’re not limited to totals and averages. A self join shows patterns, progressions, and relationships, the deeper layer of meaning that makes raw data actionable.

Analogy:

“It’s like comparing today’s selfie to yesterday’s, same subject, new context.”

6. Putting it all together

Scenario: team productivity — dept → manager rollups

Problem:

In most organizations, productivity metrics are stored at the employee level (e.g., hours worked, tasks completed, revenue generated). But business leaders don’t just need to see this information for individuals, they need rollups across the hierarchy:

• A manager wants to see how their team performed.

• A department head wants to see all their managers’ totals, plus comparisons across them.

• The company wants to see aggregated totals across departments.

Goal:

• i-level metrics by department × manager × month

• Compare employees under each manager (self-join to the manager)

• Get manager subtotals, department subtotals, company totals, all in one result

Assumed tables (map to your schema):

• employees(employee_id, full_name, manager_id, department)

• work_log(employee_id, work_date) one row per day an employee did work (or any activity event)

Query:

Output:

7. “Challenges & pro tips”

1. Performance pitfalls:

Queries with self-joins, grouping sets, and rollups can scan huge volumes of data, making them slow and resource-intensive on large datasets.

2. Aggregate placeholders in rollups and grouping sets:

When you use ROLLUP or GROUPING SETS, SQL adds extra rows for subtotals and grand totals. In those rows, the grouped columns are filled with NULL , not because the data is missing, but as a marker that this row represents an aggregate.

3. Where self-joins fall short:

• Performance overhead → Joining a large table to itself often doubles the data scanned and can make queries very slow.

• Complexity → Queries become harder to read and maintain, especially with multiple conditions across the same table.

• Scalability issues → On very large datasets, self-joins may time out or require excessive compute compared to equivalent window function solutions.

Pro tips :

1. Speed up queries

• Summarize data first (like user–month).

• Add indexes on important columns.

• Only check key retention periods (Month+1, Month+2).

2. Read totals clearly

• Use GROUPING() to know which rows are subtotals.

• Change NULLs into labels like ALL MONTHS or ALL YEARS.

3. Use window functions

• Try LAG, LEAD, or ROW_NUMBER instead of self-joins.

• They are faster, cleaner, and scale better.

8. The big picture: why It’s useful

At the end of the blog, ROLLUP and GROUPING SETS aren’t obscure SQL tricks, they’re shortcuts to thinking and working like an analyst. They turn three or four repetitive queries into one clear, maintainable query that scales with your questions. Fewer queries mean fewer errors, faster answers, and cleaner code.

So here’s the takeaway:

Next time you catch yourself writing multiple GROUP BY queries in a row, stop and ask, could this be one query with ROLLUP or GROUPING SETS?

Introduction:

In Today’s analytics, raw data alone isn’t enough, what matters is how flexibly and deeply you can explore it. When analyzing data, it’s often important to calculate values like running totals, rankings, or comparisons between rows, without losing the details of each row. That’s where SQL window functions come in.

They let you perform powerful calculations across related rows while keeping every row in your results. This makes it easier to spot trends, compare values, and build more insightful reports. Whether you're just starting with SQL or looking to improve your analysis skills, understanding window functions is a valuable step forward.

SQL Window Functions Core Concepts:

Each window function uses the OVER clause to define its scope. You can refine this window with:

• PARTITION BY to divide data into groups for independent calculations,

• ORDER BY to set row order within each group for operations like rankings or running totals.

Window functions fall into several key categories:

• Aggregate functions (SUM(), AVG(), COUNT()) for running totals or moving averages.

• Ranking functions (ROW_NUMBER(), RANK(), DENSE_RANK()) to assign positions or ranks within partitions.

• Value functions (LAG(), LEAD(), FIRST_VALUE(), LAST_VALUE()) to access values from other rows in the window.

  Example:

This example calculates a cumulative total of duration_seconds ordered by start_time, without grouping or removing rows.

This example calculates a cumulative total of duration_seconds ordered by start_time, without grouping or removing rows.

Primary SQL Ranking Functions:

Ranking functions in SQL are specialized window functions that assign ranks to rows based on their ordering within a dataset. The primary ranking functions are ROW_NUMBER(), RANK(), DENSE_RANK(), and NTILE().

• ROW_NUMBER() - ROW_NUMBER() is a window function in SQL that assigns a unique sequential integer to rows within a result set, based on the order you specify.

• Output:

  The query would produce:

• RANK(): Assigns ranks to rows, but if rows have identical values per the ORDER BY clause, they receive the same rank. This may create gaps in the rankings after ties (i.e., if two rows are ranked 1, the next row is ranked 3).

• Output:

  The query would produce:

• DENSE_RANK(): Functions similarly to RANK(), but does not leave gaps in the ranking sequence after ties. If two rows are tied for rank 1, the next row is assigned rank 2.

• Output:

  The query would produce:

• NTILE(n): Divides the data into n equal or nearly equal groups and assigns each row a group number.

• Output :

  (rows split into 4 nearly equal buckets, highest values in quartile 1)

📦 Key Navigation Functions

Navigation functions are a subset of SQL window functions that allow you to access values from rows adjacent to the current row within a defined window. These adjacent rows can be from before (LAG) or after (LEAD) the current row. Other navigation functions include FIRST_VALUE() and LAST_VALUE() which retrieve boundary values within each window.

The navigation summary for the LEAD() , LAG(), FIRST_VALUE() and LAST_VALUE() functions from the SQL time-series window functions is as follows:

LEAD and LAG

• Purpose: The LEAD and LAG functions enable access to data from rows after (LEAD) or before (LAG) the current row within a result set ordered by a specific column, making them ideal for time-series and sequential data analysis.

• Functionality: LEAD looks "into the future" (next rows), while LAG looks "into the past" (previous rows). They are frequently used to calculate differences between consecutive data points, such as monthly stock prices or moving averages.

FIRST_VALUE() and LAST_VALUE()

• Purpose:

  FIRST_VALUE() and LAST_VALUE() return the first and last value within the ordered window of each partition. They're ideal when you need to reference the starting or ending state of a sequence, like the first purchase amount, the initial score, or the last known status.

• Functionality:

  • FIRST_VALUE() captures the earliest value in the ordered set.

  • LAST_VALUE() captures the latest value, but note: its behavior depends on the window frame definition. Without correct frame specification, it may return the current row’s value instead of the true “last” row.

Example:

Imagine a Scenario : Analyzing Employee Promotion Patterns in an organisation

Business Use Case:

An HR analytics team wants to understand employee promotion trends over time. Specifically, they want to know:

• When each employee got promoted.

• What was their previous job title (LAG()).

• What is their next job title (LEAD()).

• What was their first-ever title in the company (FIRST_VALUE()).

• What is their current/latest title (LAST_VALUE()).

SQL query:

✅ Output of query on that data

What is an Aggregate Function?

An aggregate function in SQL is a special type of function that takes multiple rows of data and condenses them into a single value based on a calculation you specify.

They’re used when you want summarized information, such as totals, averages, or counts, rather than listing individual rows.

Common SQL aggregate functions include:

• COUNT(): Returns the number of rows in a set, or the number of non-NULL values in a specified column.

• SUM(): Calculates the total sum of values in a numeric column.

• MIN(): Retrieves the minimum value from a specified column.

• MAX(): Retrieves the maximum value from a specified column.

📊 Business Use Cases for Aggregation

🛍 Retail & E-Commerce:

• Sales Reporting: Calculate SUM(sales_amount) to find total revenue per day, month, or store.

• Top-Selling Products: Use COUNT(product_id) grouped by product to identify best performers.

🏥 Healthcare

• Patient Volume: Use COUNT(patient_id) to track inflow by department.

• Treatment Success Rate: Aggregate outcomes with SUM(successful_cases) / SUM(total_cases).

🎓 Education

• Performance Tracking: AVG(marks) per subject or class.

• Top Scores: Identify MAX(score) for award consideration.

Advanced patterns of SQL Window Functions:

1. Row-to-Row Comparisons

• Functions Used: LAG() and LEAD()

• Purpose: Compare a row with its preceding or following row within the same partition.

• Typical Applications:

  • Finding trends or changes over time

  • Calculating deltas between consecutive events

  • Detecting anomalies in sequential data

(Conceptual Example):

2. Rolling and Moving Calculations

• Functions Used: AVG(), SUM(), etc., combined with window frames

• Purpose: Compute metrics like rolling sums, running averages, or cumulative totals.

• Applications:

  • Smoothing time-series

  • Spotting moving trends

  • Monitoring performance metrics

4. Conditional Insights

• Approach: Use CASE inside a window function to apply conditional aggregations.

• Applications:

  • Aggregating only certain categories of data while still operating in a partition

  • Segmenting by logic without breaking partitions

  

  Performance Principles:

  • Indexes & Partitions

    • Index columns that are used in PARTITION BY and ORDER BY.

    • Consider table partitioning to reduce the data each query scans.

  • Window Frame Efficiency

    • Use specific frames (e.g., ROWS BETWEEN 3 PRECEDING AND CURRENT ROW) instead of broad ones like UNBOUNDED PRECEDING when possible. Narrow windows run faster.

  • Reduce Complexity

    • Keep only necessary columns in the OVER() clause.

    • Simplify expressions inside window functions; push heavy calculations outside the window if possible.

  • Benchmark Queries

    • Run queries on real or representative datasets.

    • Use EXPLAIN / execution plans or profiling tools to spot slow steps.

  ---

  Quick best practices

  • Use only when needed. If a GROUP BY works, use that.

  • Set PARTITION BY and ORDER BY carefully, add a tiebreaker (e.g., ORDER BY ts, id).

  • Filter early, select fewer columns before the window step.

  • Use tight frames (ROWS BETWEEN N PRECEDING AND CURRENT ROW) instead of unbounded history.

  • Reuse results, don’t compute the same window twice (or use QUALIFY where available).

  • Handle NULLs in LAG/LEAD with a default value.

  • Watch performance: check the plan/profile; index/partition on the columns you partition/order by.

  Conclusion

  Window functions add trends, ranks, and rolling metrics without losing row detail. Keep them predictable and fast with precise ordering, small frames, and smaller inputs, and always verify with the execution plan.

Imagine you're a 5-year-old with a massive toy collection that keeps growing every day. You have toys everywhere, and it takes you over an hour just to find the specific toy you want to play with. That's exactly what was happening to LY Corporation, one of Japan's biggest tech companies!

What is LY Corporation?

LY Corporation is like a giant digital playground. Think of it as the company that runs:

• LINE - the messaging app that almost everyone in Japan uses (like WhatsApp but way more popular)

• Yahoo! JAPAN - the website where people search for things and read news

• PayPay - the app people use to pay for things with their phones

They have over 320 million users across Asia and handle more data than you can imagine - we're talking about 1.1 exabytes, which is like having 1.1 billion huge toy boxes full of information!

How big is their data?

To understand how massive LY Corporation's data is, let's use simple numbers:

• They have over 100,000 tables (think of each table as a different type of toy box)

• Every single day, they process 1.2 trillion records (that's like counting to 1.2 followed by 12 zeros!)

• Their data is growing really, really fast because millions of people use their apps every day

What was their old data system like?

Before Apache Iceberg, LY Corporation was using something called Hive (think of it as their old toy organization system). Here's how it worked:

The old way (Hive System)

• Data collection: People using LINE, Yahoo, and PayPay would create data (like sending messages, searching, making payments)

• Data journey: This data would travel through something called Kafka (like a conveyor belt) to Flink (a processing machine) and finally land in HDFS storage (the toy warehouse)

• Format conversion: The data would start as AVRO and Protocol Buffer files and then get converted to ORC format every hour

• The big wait: It took 1 to 1.5 hours before anyone could actually use this data

The problems they faced

Remember when your room was so messy you couldn't find your favorite toy car under a pile of socks and Legos? LY Corporation had the same problem, just with, you know, terabytes of data.

1. Super slow data access

Getting fresh data took them over an hour. Yep, an hour.

In a world where modern businesses need data in minutes, not “go-make-a-coffee-and-come-back” hours, this was a big problem. Their machine learning models and real-time services were basically stuck waiting around, tapping their feet.

2. The tiny files chaos

Imagine having a million tiny toy boxes instead of a few big, organized ones. LY Corp’s system (HDFS) was drowning in these tiny files, each demanding attention.

And the “brain” of their system (the namenode) was getting so overloaded trying to keep track that it was running out of memory, like your laptop when you have 47 Chrome tabs open.

3. Deleting data was a nightmare

When someone requested their personal data be deleted (think GDPR and privacy laws), it wasn’t just a quick “delete” button. Nope.

They had to rewrite huge files just to remove a single person’s data. Imagine having to rewrite your entire diary just to erase one sentence. Painful.

4. Schema changes were a disaster

Adding or removing data fields? It was like having to reorganize your entire Lego collection every time you bought a new set.

Why did they choose Apache Iceberg?

Apache Iceberg is like a super-smart toy organization system. Here's why LY Corporation fell in love with it:

1. Delete specific things easily

• Instead of emptying entire toy boxes to remove one toy, Iceberg can point to exactly what needs to be deleted

• Perfect for when people ask to have their personal data removed (GDPR compliance)

2. Make data available super fast

• Reduced their data delay from 1.5 hours to just 5 minutes

• That's 10-12 times faster!

3. Easy changes without big mess

• Adding new data fields is like adding new compartments to your toy box without emptying everything out

• Schema evolution made it simple to adapt as their services grew

4. Better organization

• Iceberg manages metadata (information about information) much more efficiently

• Reduces pressure on their central database (Hive Metastore)

Think of Apache Iceberg as having a magical catalog (Iceberg’s manifest and metadata approach) that knows exactly where every piece of data is stored and can retrieve it when needed.

How Did they actually make the switch?

The migration strategy

LY Corporation was smart - they didn't try to change everything at once! Here's what they did:

Gradual migration approach

• Started with new tables that needed Iceberg's special features

• Migrated 8,000+ tables to Iceberg by 2024

• Still have 90,000+ tables on the old Hive system

• Only migrate tables that actually benefit from Iceberg's features

The new technical setup

New Data pipeline:

• Kafka (data conveyor belt) → Flink (processing) → Apache Iceberg (smart storage)

• Data gets flushed every 5 minutes instead of every hour

• Files are stored in ORC format (efficient storage format)

• Everything runs on Kubernetes (modern container platform)

What the table optimizer does

Think of LY’s Table optimizer as an automated janitor for Iceberg tables, keeping them fast, tidy, and cost-efficient without manual babysitting.

Coordinator

Acts like a manager, watching all tables to see which need maintenance. It keeps a smart queue, prioritizing based on file size, file count, or table health.

Workers

Workers are the system’s hands, they execute tasks from the coordinator, like merging files and cleaning data, all in parallel for speed and efficiency.

Automatic compaction

Automatic compaction merges many small files into larger ones, using smart strategies like bin-packing or sorting, to speed up queries and reduce storage waste.

Snapshot management

Iceberg’s time travel creates snapshots with every change. The optimizer prunes old snapshots and cleans up orphan files to save space and keep metadata lean.

Spark jobs

File rewrites and data merges are handled by distributed Spark jobs, scheduled and assigned by the coordinator for efficient processing at scale.

How It’s built

• Microservices on Kubernetes: Fully distributed, scalable, and resilient, with coordinators and workers scaled independently.

• Automated scheduling: Maintenance runs on autopilot, prioritizing urgent tasks and balancing workload.

• Monitoring & health checks: Continuously tracks table health, file counts, and performance, alerting or auto-fixing as needed.

• Conflict prevention: Prevents overlapping maintenance and user operations on the same table, ensuring safe, conflict-free optimizations.

In short:
The Table Optimizer keeps LY’s Iceberg tables clean and fast, automatically handling compaction, cleanup, and snapshot pruning, so the team can focus on building, not babysitting tables.

What challenges did they face?

Even with a great plan, moving this much data wasn't easy:

Migration challenges

1. Deciding what to migrate

• With 100,000+ tables, they had to carefully choose which ones would benefit from Iceberg

• Tables that were small or didn't need Iceberg's features could stay on Hive

2. User training

• Data engineers and analysts needed to learn new tools and concepts

• Not everyone understood Iceberg's advanced features initially

3. Data copying between systems

• Their old backup strategy (DistCP) needed to be redesigned for Iceberg

• Moving data between different storage systems became more complex

Ongoing challenges

Even with their big clean-up, LY Corp still has a few chores to keep things running smoothly.

1. Table housekeeping

Iceberg tables don’t magically stay fast and efficient. They need regular “housekeeping” to keep them tidy and optimized, kind of like sweeping the floor before the dust bunnies take over.

Sure, their Table Optimizer helps a lot, but it’s still an ongoing chore they can’t ignore.

2. File size juggling

Too many tiny files? Bad for performance. Super large files? Harder to manage.

It’s a balancing act, like pouring cereal without spilling. Automatic compaction steps in to help, merging those tiny files into more manageable sizes to keep everything humming along.

What goals did they achieve?

LY Corporation hit some amazing targets:

Performance improvements

1. Massive speed boost

• Data latency: From 1.5 hours to 5 minutes (10-12x faster)

• Query performance: Much faster due to better file organization

• Real-time capabilities: Better support for online machine learning

2. Operational efficiency

• Automated maintenance: Less manual work for data engineers

• Better resource utilization: More efficient use of computing power

• Reduced complexity: Simpler data pipeline management

Business benefits

1. Compliance made easy

• GDPR compliance: Can efficiently delete user data when requested

• Data governance: Better control over data lifecycle

• Audit capabilities: Clear tracking of data changes

2. Business agility

• Faster insights: Data available for analysis much sooner

• Better ML/AI: More timely data for machine learning models

• Competitive advantage: Faster response to business needs

Cost savings and benefits

1️⃣ Direct cost reductions
Efficient file organization and compression lower compute and storage costs.

2️⃣ Operational savings
Automation reduces manual maintenance and pipeline failures, saving engineering time.

3️⃣ Infrastructure efficiency
Makes better use of your existing hardware while easing pressure on Hive Metastore and HDFS.

Indirect Benefits

1️⃣ Faster Time-to-Market
Teams get fresh data faster, enabling quicker feature rollouts and business decisions.

2️⃣ Improved user experience
Real-time features and personalization improve with fresher data and faster response to user behavior.

What's special about their implementation?

Innovation: the table optimizer

LY Corporation didn't just use Iceberg - they innovated on top of it. Their Table Optimizer is like having a smart assistant that:

• Monitors thousands of tables automatically

• Predicts when maintenance is needed

• Runs optimization jobs without human intervention

• Prevents conflicts between different operations

• Provides detailed metrics and monitoring

Hybrid approach

They took a smart approach by not migrating everything at once. Stable, simple tables stayed on Hive (no need to fix what isn’t broken), while complex, dynamic tables moved to Iceberg, focusing effort where Iceberg’s features would bring the most benefit.

Why this matters for other companies

LY Corporation's journey shows that:

1. Scale matters: At their data volumes, small improvements have huge impacts
2. Gradual migration works: You don't need to change everything at once
3. Automation is key: Building tools like Table Optimizer pays off
4. Compliance is critical: Modern data platforms must support privacy regulations

The Future

LY Corporation is continuing to:

• Expand Iceberg adoption to more suitable tables

• Improve their Table Optimizer with more intelligence

• Explore new Iceberg features like branching and tagging

• Share their learnings with the broader Apache Iceberg community

What makes this special

This isn't just a story about switching technology - it's about transforming how a massive company handles data. LY Corporation proved that even with 1.1 exabytes of data and 100,000+ tables, you can make smart, gradual changes that deliver huge benefits.

They turned their data platform from a slow, rigid system into a fast, flexible, compliance-ready powerhouse that can adapt to whatever the future brings. And the best part? They did it without disrupting their services to 320 million users!

Concluding thoughts

Sometimes the biggest improvements come from making your data smarter, not just bigger. LY Corporation's Apache Iceberg journey is a perfect example of how the right technology choices can transform not just your data platform, but your entire business capability.

Got a project or idea?

Drop me a line. If it’s interesting, I’m in. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now

Remote work, once a temporary response to the COVID-19 pandemic, has become a defining element of the modern workforce in 2025. What began as an emergency measure has transformed how people work, live, and interact with cities. This shift profoundly reshapes urban life, altering housing trends, economic activity, infrastructure design, and social behaviors. Let’s explore how step by step.

🧑‍💻 1. The remote work revolution: where we stand today.

By 2025, remote work will no longer be a perk; it will be the norm. Approximately 27% of workers with remote-capable jobs are fully remote, while 53% follow hybrid schedules, which combine home and office work.

The change isn’t just logistical; it’s cultural. A staggering 98% of remote workers want to continue working remotely for the rest of their careers. Companies report stable or improved productivity, and employees show higher engagement levels than those of fully in-office workers. Remote work is no longer the future; it’s the present.

🏘️ 2. Migration and housing patterns: The “Donut Effect”

The “Donut Effect” One of the most visible consequences of remote work is a shift in where people choose to live. Freed from the need to be near their offices, workers are moving away from crowded and expensive city centers to suburbs, smaller cities, and even rural areas.

💸 3. Economic Impact on cities and suburbs

Remote work has reshaped local economies by reducing demand in urban centers while boosting activity in suburban and rural areas. Cities that once thrived on office workers supporting restaurants, transit, and retail are seeing revenue declines, while suburbs benefit from increased spending by residents working from home.

4. Social and environmental implications

Remote work offers undeniable benefits, such as better work-life balance, flexibility, and reduced commute stress. However, it also introduces risks, such as social isolation and blurred boundaries between personal and work life. From a diversity and inclusion standpoint, remote work has opened doors for individuals who previously faced barriers, including those with disabilities and members of underserved communities.

⚠️ 5. Challenges and the road ahead

Despite the positives, challenges remain.

• Urban inequality: Service workers are dependent on downtown foot traffic. Cleaners, food vendors, and retail clerks face job insecurity as cities empty.

• Digital divide: Not everyone has reliable internet access or the digital literacy needed to participate in the remote economy.

• Policy gaps: Cities must act now to support urban and suburban growth. This means investing in broadband, revising zoning laws, and reimagining city design for a hybrid workforce.

Conclusion:

As remote work redefines how and where we live and work, cities and suburbs must adapt quickly to evolving economic and social dynamics. The shift presents opportunities for greater flexibility and inclusivity, as well as challenges that demand thoughtful policy and infrastructure responses. Navigating this transformation will be crucial to shaping resilient, equitable urban futures.

Do you have a project or idea?

Feel free to drop me a line. If it’s interesting, I’m in if it’s weird, even better.

💌 amanguptanalytics@gmail.com

Sure, you've got ChatGPT, Claude, Gemini, and even Copilot lined up like a dream team of robot interns. But if everything the AI spits out sounds robotic and lifeless, you’re not alone. It’s not that the AI is broken. It’s just missing a decent content architect to guide the way.

As Josh Wills might say, with a few strategic tweaks, we can go from meh to marvelous. Here are five ways to improve your AI-generated content.

1. Stop telling your AI model to “just write something” ✍️

If your prompt is "Write about X," don’t be surprised when the result sounds like a high school book report. One-line prompts get you one-dimensional results.

Be specific. Seriously:

• Who’s it for? 🎯

• What’s the tone? 🗣️

• How should it look? 🖼️

Example: “For data engineers. Witty. Three short paras. End with bullets.”

Better yet:

• Show a sample

• Ask for options

The model’s not magic. It’s just really good autocomplete with a PhD. Guide it like you mean it.

For example, compare these two:

• “Write a blog post about prompt engineering.”

• “Write a 300-word post for junior data engineers explaining prompt engineering using a cooking metaphor. Make it witty, and start with a bad example.”

The second gets you usable content. The first gets you oatmeal.

2. Winging it is not a strategy 🛫

Starting without a plan? That’s the quickest way to write 800 words of structured nonsense.

Take five minutes. Make an outline.

• Yes, the AI can help you build one.

• No, it won’t know what matters until you do.

Even a few quick bullets or a rough flow sketch is enough. Then let Claude or ChatGPT do their thing.

Remember:

• Good outlines don’t have to be good. They just have to exist.

• You can fix a rough draft. You can’t fix a blank one.

Plan first. Generate second. Edit last. 🔁

It’s not magic. It’s just a process.

3. One-shot drafts are how bad takes happen 🧩

Asking your AI to write the entire piece in one go? That’s like asking it to cook dinner, do the dishes, and fold your laundry at the same time. It’ll try, but the results will be mushy.

Try this instead:

• Chunk it. Always.

• Don’t dump the whole task on the model at once.

Why?

• That’s how you get generic tone, fuzzy logic, and zero edge.

• Start small: begin with the intro.

• Read it. Fix it.

• Then move on to the next part.

• Section by section. Small asks. Tight feedback loops. 🔄

The more you focus, the better it sounds. Clarity scales with constraint.

Think of it like a relay race, not a solo sprint:

Outline → Section → Review → Iterate

This keeps the narrative tight, and your sanity intact.

4. Don’t skip the edit: POLISH, POLISH, POLISH ✨

Here’s where the magic actually happens: editing.

This is your time to shine: tighten the logic, add a clever turn of phrase, or kill the fluff with precision.

Hit “generate” and walk away? Don’t be shocked when it reads like something written by, well, a robot.

Time to take off autopilot and step in as editor-in-chief. 🧠

Start with:

• Fixing clunky phrasing

• Spotting logic gaps

• Smoothing transitions

• Making sure the tone doesn’t zigzag from Slack joke to corporate keynote

The goal isn’t perfection. It’s flow, with a bit of flavor.

Drop in a metaphor. Chop a paragraph in half. Rebuild a sentence so it actually makes sense.

This is where your voice shows up. It’s also where most AI content quietly crumbles if you skip it.

Golden rule: if it sounds like a machine wrote it, it’s not done. One edit makes it better. Two? That’s when the good stuff hits.

5. Your AI wrote it. Now make it sound like you did. 🎤

Just because AI lent a hand doesn’t mean it has to read like instructions on assembling IKEA furniture.

Bland content is background noise. Your voice is the signal. 📡

So punch it up:

• Crack a joke.

• Toss in a metaphor.

• Mention something your audience actually gets. Bonus points if it’s not from a case study.

Prompt with intent: "Write with the snark of Josh Wills and the clarity of an O’Reilly book." Suddenly, your model has a compass instead of a blindfold.

Still feels off? No problem:

• Ask for three takes in different voices: snappy, serious, playful.

• Feed in writing samples you like: Josh Wills, Lex Fridman, Ali Abdaal, Douglas Adams. Let the model riff.

When one lands, lock in the tone. Apply it section by section.

A single sharp analogy can turn a forgettable paragraph into the line people quote back to you.

If your AI-written blog reads like corporate onboarding docs, it deserves to be ignored. Set the bar higher than “not embarrassing.”

Boring fades. Voice lasts. Make yours count.

6. Bonus: Use Feedback Loops – Don’t create in a vacuum 🔁

Critique your content.

It’s not a common practice, but it’s a best one. Ask AI to critique your content and tell how it scores and what the weak areas are. For example, I use a GPT with books from Seth Godin and Chip Heath in it. And I ask it something like:

"Critique my content according to principles of Seth Godin and Chip Heath. Tell me what works best and what can be improved. Give the current article a score for likely performance."

And then paste the article in.

Simple, right? But very effective. It lets you quickly vet your draft so that poor quality doesn’t creep in.

Great creators seek feedback. After drafting, don’t just hit publish. Put your content through a feedback wringer. Ask a colleague or friend to tear it apart (better them than your readers). Or use the AI itself as a critic: have ChatGPT analyze Claude’s output or vice versa and see what weaknesses they spot.

Often, I’ll prompt the AI with:

"Critique the above and suggest improvements."

– and you’ll get surprisingly useful editing suggestions.

Another pro move: read your content

out loud 🔊

(or use a text-to-speech tool) to catch awkward phrasing or pacing issues. If you stumble while reading, that part needs fixing.

Some folks even record themselves giving feedback and feed that back into the AI to implement changes. Also, leverage your community. Drop a snippet in your team’s Slack channel for quick reactions.

The point is to create a feedback loop between you, AI, and real humans to refine the content. Iteration isn’t a sign your first try was bad. It’s how you make good content great.

Conclusion

AI moves fast, and tools change weekly. What matters is how you use them. Try new prompts, automate the boring parts, and fine-tune models for your writing. Watch what works, double down, and cut the rest.

Mediocre AI content is a choice. With the right workflow, you’re not just creating. You’re compounding. So iterate, measure, and improve. Now stop reading and make something worth sharing. 🚀

Got a project or idea?

Drop me a line. If it’s interesting, I’m in. If it’s weird, even better.

💌 amanguptanalytics@gmail.com

Subscribe now