plot twist: they leave it up, collect telemetry to match and understand the data behind pirating vs. owned emulation, find it useful to open the platform, focus on services, and build their business to 10x on software almost exclusively

not probable, but that would be interesting to see

imo this is the biggest thing deno and bun have going for them. running typescript from the shell is so nice and easy

Finally a competitor to dbt. the world needs this!

There is also sqlmesh (https://sqlmesh.com/). Pretty new as well. It introduces some interesting concepts. For smaller dbt projects it could be a drop-in replacement as it allows importing dbt projects.

I agree. The open source flavor of dbt Core is very well-designed and almost as complete as you would want it, but any competition is good competition

Privacy, longevity, nutrition science, trust, deep space exploration.

I mean I get why everyone wants everything to be fuse-compatible but some things just aren't meant to be done.

> Python’s standard library sucks

Maybe for production ready microservices, but I've always thought Python's std lib was pretty expansive.

And I'm not entirely sure it would be good idea for general purpose language. Knowing the history servers. Which likely will anyway have some critical design or security issue at one and probably many points in their life.

There's the adage that the standard library is where modules go to die, which seems to ring true in Python's case.

Having a big standard library can be great, especially for things and designs that have been settled years ago, but inclusion into the standard library means iterating upon those modules will receive pushback from maintainers looking out for consumers of the standard library who need it to remain stable for potentially decades.

I mean, is anyone really surprised by this? LLMs (as I understand them today) only predict the next token based on previous tokens, so there's no actual logical cohesion to what they produce.

Hey everyone, Jk here. I cofounded Jetty Labs to build data tools. The tl;dr here is that data teams use a lot of tools and understanding how access is configured across the growing stack is complex and exhausting. Jetty is a tiny peek at our broader vision of simplifying data privacy and we'd love you to give it a try and offer any feedback on the product.

I won't be surprised if DE ends up just falling under the "software engineering" umbrella as the jobs grow closer together. With hybrid OLAP/OLTP databases becoming more popular, the skillset delta is definitely smaller than it used to be. Data Engineers are higher leverage assets to an organization than they ever have been before.

I think it's mostly already there, but your big, enterprise houses were late getting the memo. About 12 years ago, I switched to a DE role/title and held it ever since. I worked in a variety of startups doing DE - moving data from over here to over there, with a variety of tools from orchestration frameworks to homegrown code in a variety of languages.

About six years ago, I walked into a local hospital to interview for a DE role and it was very clear that their definition of DE was different than mine. The whole dept worked in nothing but SQL. I thought I was good with SQL, but they absolutely crucified me on SQL and data architecture theories. I ended up getting kicked over to a software engineering role, doing DE in another capacity, which made more sense for me.

Only now I'm hearing that they're migrating to other tools like dbt and requiring their DEs to learn programming languages.

Well my understanding is that a Data Engineer is basically just a DevOps engineer but instead of building infra to run applications they build infra to process, sanitize and normalize data.

Author here - Of course, data engineering involves building infra and being knowledgeable about DevOps practices, but that’s not the only area data engineers should be familiar with. There are many, many more! In my personal experience, sometimes we end up not using DevOps best practices because we spread too thin. That’s why I believe in specialization within data engineering and the surge of “data reliability engineer” or similar

Imho that is absolutely not doing the role justice. For some people that may hold true, but I would expect a data engineer to know everything about distributed systems, database indexes, how different databases work and why you pick them, partitioning, replication, transactions/locking. These are topics a software engineer is typically familiar with. A DevOps engineer wouldn't be.

Or to denormalize data, the distinction of which the data engineer would be most familiar with why and how.

I don't think that's true, in general, just because I think software engineers will largely be focused on creating end user experiences and custom systems, whereas data engineers will be focused on analytics, reporting, ML, and business operations.

There's definitely some overlap, where companies are producing fundamentally data driven user experiences, such as user-visible analytics or live recommendations. But that's niche.

Yeah, maybe this will happen. Where I work (FAANG), I know that DEs get lower compensation than SWEs and SREs.

Funny in my FAANG, it can be the reverse :)

There was an external audit in 2021 – https://research.nccgroup.com/2021/10/27/public-report-whats....