The Work They can’t see

A reflection on engineering leadership, architectural maturity, and why building great systems isn’t enough.

Some time ago, I had a conversation that has stayed with me ever since.

On the surface, it was a fairly ordinary meeting with one of the most recognised companies in the data engineering space. The objective was straightforward: understand where they could help us, whether there was an opportunity to collaborate, and whether their platform could address some of the challenges we were facing. Like many of these conversations, I expected that most of the time would be spent discussing products, capabilities, roadmaps, and the usual questions that come with evaluating a technology partnership.

Instead, the conversation quickly turned into something much more interesting❗️

Rather than talking about products, we started talking about architecture.

Their solution architect wasn’t trying to sell us anything. They were trying to understand how our platform had been designed, asking the kind of questions that another engineer instinctively asks when they want to understand not only what a system does, but why it was built the way it was.

The Questions Engineers Ask

They started with ingestion.

Is it configurable?

It is❗️

We invested in a YAML-driven approach that allows new data sources to be onboarded through configuration rather than code. That decision was deliberate, but it was never really about YAML, or about adopting a particular pattern or framework. The objective was to reduce the engineering effort required every time the business needed to integrate another source, while standardising the onboarding process, reducing the risk of human error, and allowing engineers to spend more of their time solving problems that actually create business value.

Then the next question naturally followed.

“Do you have a reusable data loading framework?”

Again, the answer was yes❗️

Rather than creating bespoke ingestion pipelines for every source system, we had deliberately built a generic framework that standardises how data enters our platform. Every new integration benefits from the same validation, observability, and operational patterns, which means engineers don’t have to repeatedly solve the same technical problem every time another source appears. The framework becomes a piece of infrastructure that allows the organisation to scale its data ecosystem without scaling the engineering effort linearly alongside it.

The conversation then moved naturally to data modelling.

“What are you using for data modelling?”

dbt.

And that was sufficient enough to avoid any further follow-up, because the interesting part of the answer wasn’t really the technology itself. It was the architectural principle behind the decision: keeping the transformation and business logic sufficiently independent from the underlying infrastructure so that the platform could evolve without having to rewrite the logic that gives the data meaning.

That led us into another discussion, this time around the separation of compute and storage. From their perspective, decoupling those concerns creates flexibility and gives organisations more freedom to optimise their infrastructure and manage costs over time.

And they were right‼️

The interesting part was that this wasn’t a recommendation we needed to take away from the meeting. It was already one of the architectural principles that had guided our platform re-architecture.

“What about orchestration?”

The platform provider we use has a top-notch platform, but orchestration has never been its strongest capability. We explained that this was precisely why we had adopted another tool. Again, the decision wasn’t really about the tool itself; it was about keeping orchestration independent from the underlying compute platform, allowing each component of the architecture to evolve independently rather than tying the entire platform to the capabilities and limitations of a single provider.

Finally, they asked a question that I suspect every organisation eventually has to answer.

“What happens if one day you decide to move away from your data platform provider?”

It is exactly the kind of question architects enjoy because there is rarely a perfect answer. There are only trade-offs, and the quality of an architecture is often determined less by whether it eliminates those trade-offs and more by whether they were understood and made deliberately.

Fortunately, it was a question we had already asked ourselves.

By building our transformation layer with dbt, we had deliberately avoided embedding business logic inside a single technology. If business priorities change tomorrow, or if the platform underneath us is no longer the right choice, we have created enough architectural flexibility to move without having to rewrite the platform from scratch.

At that point, something interesting happened😲. The questions stopped.

The solution architect smiled and said something that every engineering team enjoys hearing:

“You’re already in a very good state. You’ve implemented many of the capabilities we normally help customers build. At this stage, I’m not sure there’s much value we can add.”

We agreed to wrap-up the meeting and discuss again in the future. But her statement was an incredibly rewarding moment‼️

Not because external validation defines good engineering, the truth is it doesn’t, but because another experienced engineer had looked beyond the individual technology choices and recognised the thinking behind them. In less than an hour, they had effectively reconstructed years of architectural decisions from a series of questions and answers.

They weren’t simply recognising the technology, but rather they were recognising the thinking behind it.

After the meeting, I shared a summary with the colleague who had introduced us and copied our leadership team to close the loop.

The response surprised me;

“We should discuss this. We’re not there yet!”

At first, I found myself wondering how two groups of intelligent people could look at the same platform and arrive at completely different conclusions.

Then I realised they weren’t actually looking at the same thing.

The Visibility Gap Isn’t Really About Visibility

It would be easy to conclude from this story that engineers understand engineers while leadership simply doesn’t, but I don’t think that is true.

The solution architect and the team had spent an hour discussing architecture, trade-offs, resilience, vendor lock-in, operational maturity, maintainability, and long-term flexibility. Every answer exposed another deliberate engineering decision, and because they had enough experience to understand the implications of those decisions, they could gradually build an accurate mental model of our platform.

Leadership never had that opportunity.

They weren’t present for the conversation, they never heard the questions being asked, and more importantly, they never heard why those questions mattered. They received the conclusion without experiencing the reasoning that led to it.

Expecting them to arrive at exactly the same conclusion would therefore have been unfair.

They weren’t missing technical expertise, They were missing context.

And context changes everything.

This is something I have increasingly come to appreciate as I have moved further into engineering leadership. We often talk about visibility as though it is primarily a communication problem: leadership doesn’t know what engineering is doing, engineers don’t communicate enough, and therefore the solution is to provide more updates, more dashboards, more roadmaps, and more presentations.

I’m not sure it is that simple 😵‍💫

The real problem is that much of the value created by engineering is embedded in decisions that are almost impossible to see from the outside.

You see the platform, while we see the decisions behind the platform. More often than not, those are not the same thing.

Good Engineering Is Designed to Become Invisible

There is something wonderfully frustrating about engineering: when we are doing our jobs well, fewer people notice.

Platforms become more reliable, deployments become routine, onboarding new sources becomes predictable, and the number of incidents gradually decreases. Failures still happen, because software is far too complex for perfection, but they happen less frequently, recovery becomes faster, and the impact on the business becomes significantly smaller.

Ironically, those improvements rarely become visible achievements. Instead, they quietly become expectations.

The organisation gradually forgets how difficult things used to be because today’s reliability becomes tomorrow’s baseline.

A process that once required weeks of engineering effort becomes a button.

A framework that once required months to build becomes something everyone simply assumes exists.

A platform that was deliberately designed to avoid vendor lock-in becomes “the platform”.

And an architecture that allows the organisation to move faster becomes invisible precisely because it allows everything else to move more smoothly.

Success has an unusual side effect.

It hides the engineering that made it possible‼️

This is particularly true for platform engineering. When platform teams do their job well, their best work often manifests itself as the absence of friction. Product teams don’t have to think about the complexity that has been abstracted away from them. Data engineers don’t have to reinvent ingestion patterns. Teams don’t have to worry about the operational characteristics of every individual pipeline because those concerns have already been built into the platform.

The better the platform becomes, the less visible the platform team can become.

And that creates an interesting challenge for engineering leaders.

Because if the value of engineering is increasingly hidden inside the systems we build, then simply building those systems is not enough. Someone has to make that value understandable.

Engineering Leadership Is an Exercise in Translation

This experience made me rethink part of what technical leadership actually is.

For years, I believed the role was primarily about making good architectural decisions, building strong teams, choosing the right technologies, and creating an environment in which engineers could do their best work.

I still believe all of those things matter. But increasingly, I think there is another responsibility that is just as important.

When engineers say, “We use dbt,” another engineer immediately understands that the statement probably carries much more meaning than the name of a tool. They hear portability, separation of concerns, reduced vendor lock-in, and future flexibility because they understand the architectural context in which the decision was made.

Leadership hears the name of another technology. Neither interpretation is wrong❗️

One simply contains years of accumulated engineering context that the other person has never had the opportunity to acquire.

This is where I think technical leadership has a responsibility that is often underestimated. We need to translate that accumulated context into language that the rest of the organisation can understand, not because leadership needs to understand every technical decision, but because they deserve to understand why those decisions matter.

The conversation should never really be about the orchestration tool itself. It should be about resilience and the ability to evolve the platform without being constrained by a single component.

It should never really be about YAML. It should be about reducing the time and effort required to onboard new data sources while improving consistency and reducing operational risk.

It should never really be about dbt. It should be about keeping business logic portable and preserving strategic flexibility while be faster and more cost-effective.

The technology is rarely the story.

The outcomes always are.

This is also why I think technical leaders sometimes get communication wrong. We try to make leadership understand the technology instead of making the technology understandable through its consequences.

We explain the architecture when what we really need to explain is what the architecture enables.

We describe the platform when we should be describing the business capability it creates.

We talk about technical debt when we should be talking about the cost of maintaining the status quo.

Finally we talk about future-proofing when we should be explaining what options we are deliberately preserving for the organisation.

The goal isn’t to make everyone an engineer.

The goal is to make the engineering legible.

The Other Half of the Job

Looking back, I no longer see that meeting as validation of our architecture.

I see it as feedback on my own leadership.

If an external expert could recognise years of engineering work in less than an hour, yet our own leadership still had a very different picture of where the platform stood, then perhaps the platform wasn’t the problem.

Perhaps my communication was😲

As engineering leaders, we often assume that good work speaks for itself. We spend months making architectural decisions, improving reliability, removing technical debt, creating reusable frameworks, and investing in the foundations that will make the organisation faster and more resilient in the future, and then we wonder why those achievements don’t always receive the recognition we believe they deserve.

But good work rarely speaks for itself.

Architecture doesn’t present itself in quarterly reviews.

Thoughtful trade-offs don’t naturally appear on dashboards.

Future flexibility doesn’t show up as a KPI.

And the engineering effort that prevents problems from happening is particularly difficult to demonstrate, because the most successful outcome is often the absence of an incident, an outage, a migration crisis, or a major rewrite.

Someone has to tell that story.

Not by turning every engineering review into a technical deep dive, and certainly not by trying to convince leadership that every technology decision is strategically important, but by connecting the technical decisions to the outcomes that the organisation actually cares about.

Because organisations don’t make decisions based solely on reality.

They make decisions based on the reality they can see.

Perhaps that is one of the most overlooked responsibilities of technical leadership: not only building systems that are reliable, scalable, maintainable, and adaptable, but also making the reasoning behind those systems visible to the people who have to make decisions about their future.

Building great systems is only half the job. Helping others understand why they are great is the other half‼️