What went wrong with data lakes? A 15-year reality check

It used to be normal for video game playtesting departments to simply film the player playing the game. Things that are important are usually on screen if your game doesn't suck. It may be tempting to record everything that happens in the game world, but if you can't work it out from what's on screen, neither can your prospective players.

I'm building a product and exploring these questions in that domain. I'm thinking about it a little differently. I have a known set of hardware sensors that I'm collecting all the data from. Per sensor the data has a strong schema with a timestamp, what is interesting is the insights that can be gleaned across sensors. So what sensors do you have that you want to record, have a strong schema against that sensor, now have something you can use to tie across sensors, now you can use LLMs to query the data and draw interesting insights

> look at this pile of residential proxies hitting your site

How does it know that?

> The paradigm shift from ETL to ELT is what went wrong with Data Lakes.

It wasn't exactly the shift from ETL to ELT, it was the stated goal of putting "T" on read with no intermediate schema.

In my experience, it's always (since way before data lakes) been a multi-step process and I don't know how it's even possible to do the data lake strategy without re-creating a ton of transforms over and over:

1-Extract

2-Possibly do some transforms early due to specific operational/technical reasons

3-Load - even before "ELT" became a term, load without transform was happening due to constraints, like the transforms requiring data from separate operational systems to even be possible

4-Transform 1 - Foundational stuff like converting unique operational schemas (due to technical or functional reasons) into more natural schemas, easier processing in later steps

5-Transform 2 - Unifying core data that spans multiple operational systems (e.g. order fulfillment txn appears naturally in different systems but for different/overlapping business reasons and with some differences in identifiers and attributes)

6-Transform 3 - First pass at foundational target state

7-Transform 4 - Possibly more, or possibly not

8-Consume valuable data

ive lived etl and elt eras. i would never, ever go back to etl. having your business logic baked into ssis or some god forsaken piece of software was truly awful. being stuck in the rigidity of the integration or package that transformed the data in flight until some poor underpaid engineer is whipped to go make a change to it and it breaks the whole package... having to migrate legacy stuff and pick through an ssis pack age only to find no one remembers why logic is the way that it is makes me wanna find a ledge. etl was a terrible paradigm. maybe people see it through rose tinted glasses because "usable data" was what landed. servicing and scaling over a thousand etl pipelines is the worst ball torture ever, trust me. for the most part i dont think anyone needs to care much about the how/why with data lakes, a competent data strategy involves taking landed elt data and transforming it into usable reporting/integration data that becomes self service. at my last org the whole damn data lake concept freed one small subset of IT that focused on data needs and transformed the organization: all business units could finally hire their own sql analysts because data was consolidated in a warehouse and was being synchronized from many systems and applications and the all those dreams of self-serve reporting came alive. it was a monumental organizational step that changed the face of the org forever.

not much of a dbt nerd myself but sql is at least a language analysts understand, so if they go peep dynamic tables / materialized views on whats transforming the data they can get a wrangle on it. this requires no expertise in file movement and threading and surrounding whatever, the business nerds can stay business nerding and influence and at some orgs have the latitude to control the materialization logic without needing to be an engineer. it's a much better fit since analysts tend to be closer to business needs.

i dont mind blasting data in blob storage. theres a couple use cases where the "lake" thing works out, extending a postgres db to read the same data store via pglake (or duckdb + ducklake) is actually pretty slick. i build some apps that also have report pages that do some pretty heavy analytics. powerbi is effectively dead to me now.

where i think it all went wrong is literally just nomenclature. such stupid namings in the space, self importance galore. datalake is already dumb, theres ten trillion more terribly named things in the "data engineering" space, and this was an era that was largely swamped by linkedin/medium/enshittification (we got snowflake astroturfing their own subreddit trying to get people to pay for their dumb cortex ai) so data engineering is really hard to take seriously sometimes. not to mention the existence of what i consider blip in history saas services like fivetran (still not sure how these guys are still in business). it was a career step towards software for me, but looking back it is a pretty funny space shrouded in too many services and licensing. one finds it hard not to champion for postgres, duckdb, etc.

with that said, data lakes have been ridiculously effective at my last org. moving engineering full time into real time replication lords and having a whole warehouse full of everything you can possibly need enabled even greater heights for us. integrations got to just delta system vs system and true them up on 1-5 minute schedules with literally just a job that runs a select from some view which does the delta compare. doesnt sound like much but when you work at one of those dumb orgs that pays for like 75 different platforms, suddenly it started only taking us days to onboard and wire in a whole platform into the fold at the org and build out middlewares, it's actually hilarious. all of that is possible because of data lakes for the most part, though we didn't always do the "lake" thing and just materialized stuff into tables. being able to make live reporting from just about every system and bring data together acrossed many systems actually insulates us from hemmorhaging way too much money on stupid shit like salesforce and a new erp. you get to buy best in class platforms for whatever function and then your erp is just.. a damn multipage app. all that was possible with our datalake constantly having data dumped into it and going thru the infamous and inevitable end game pipeline: the road to reporting

Bingo. Double points if the infra surrounding this costs more than a maxed out MBP a month in spend

The first time (and all others) I heard "data lake" my brain instantly translated it to "somone elses problem". As in the very name implies a carelessness to just dump whatever and somone else will take care of it.

I have the same feeling about quoting Gartner studies, tbh.

The paper is actually poor in data and mostly requotes other studies.

I'd say Bob Muglia is pretty credible