Art & Science
When Datalogue joined Nike, we were part of a series of investments the company made to bring together the arts and sciences. Seeing the marketing juggernaut's concerted investment in data & analytics to elevate the sciences across all its business units was inspiring. After two and half years there, the transformation that had begun years before I joined was still chugging along and will continue to do so for years.
Now that I've gotten a chance to play around with a development stack that lives outside the confines of the enterprise, I find it puzzling that getting fantastic user experiences to production has become trivial with the advent of NextJS, Vercel, and the like. And yet doing anything remotely interesting with data is still a schlep.
The modern data stack either requires that you have a bunch of specialized groups who understand the complexities of data engineering, data science, and data analytics come together to solve interesting problems, or you have a use-case that's vanilla enough to use off-the-shelf point-and-click solutions that are expensive, hard to integrate and hard to grow out of. If you're a solo developer or a small team moving fast to invent the future—good luck doing cool shit with data because the stack built to enable the sciences can't keep up.
Two tailwinds could revolutionize how we build data-intensive applications and level the playing field between the arts and the sciences.
First, the industry has defined standards for the building blocks of data applications. Flows, Data Frames, Metrics & Models are the de facto abstractions for the orchestration, data processing, BI, and AI building blocks of data-intensive applications.
Second is the generative AI frenzy. As it becomes cheaper to automate knowledge work, we'll quickly realize that AI agents making recommendations on the back of shitty data leads to equally poor results as humans making recommendations on the back of shitty data.
The "garbage in, garbage out" problem that we've all come to know is exacerbated by the number of people (and now autonomous agents) that make decisions with garbage data.
The enterprise players will continue to incrementally improve their platforms and curate to the needs of specialized teams of data engineers, data scientists, and analytics engineers. I'm a fan of what Prefect, Databricks, and DBT have done for the enterprise stack and the growing ecosystem of companies that enable enterprise stakeholders (like Metaphor.io, which has built a data catalog that people actually use). But all these players have curated their solutions for an enterprise IT buyer looking for capabilities rather than experiences. Data contracts (opens in a new tab) are finally gaining popularity in unifying disparate pieces of the enterprise stack. All these things make it easier to become data-driven if you have all those specialists on your team.
But what can we do to make building data-intensive applications...dare I say... fun for developers?
You'd need to create a simple experience for developers who want to invent the data-driven future quickly but don't have the luxury to incur the people and time costs of hiring multiple specialists. This approach would need to combine the building blocks of data-intensive applications into a single framework and enable developers to build apps quickly while scaling them effortlessly.
It would require solving two tremendously tricky problems in the data stack (1) creating abstractions that make sense to software engineers and streamline mundane tasks, and (2) the framework should automatically define the infrastructure needed to run the data-intensive application.
I've been incredibly inspired by the work that Guillermo and his team at Vercel have done for the front-end ecosystem.
It's time to build the NextJS for data-intensive apps.
I'll be exploring creating this framework over the next few months. If you're interested in chatting about the space or want to get involved, please reach out!
Just do it, they say...
© Tim Delisle & Belair Delisle Family LLC.RSS