One of the most profound shifts I’ve experienced in product engineering is the realization that the “why” behind a problem is more powerful than the “what” or the “how”. In fact, if you deeply understand the why, the what starts to reveal itself almost naturally—and the how becomes a matter of taste, preference, and execution style.
This post is less about a shiny solution and more about the hard-earned clarity that shaped it.
Speed vs. Structure: The Startup Balancing Act
In the early days of Clickpost, our north star was speed. Speed to deliver new products. Speed to ship custom enterprise features. Our products were lean, clean, and purpose-built—and it wasn’t hard to keep tabs on how they performed. Engineering intuition and customer feedback often told us everything we needed to know.
But products scale. Customers grow. And engineering teams evolve.
Soon enough, our once-simple product lines branched out—more features, more sub-products, more interdependencies. If you draw a diagram of our offerings today, it would resemble a tree with multiple levels and deeply connected nodes. And when an issue popped up in a parent node, its ripple effects would cascade across the branches—affecting downstream products in subtle but critical ways.
To keep pace, each team built its own set of monitoring scripts and internal tools. These were bootstrapped, functional, but ultimately fragmented. No customer had visibility into what was happening under the hood.
Eventually, enterprise clients began to build their own internal flows—custom solutions to monitor our system’s performance from their end. This wasn't just problematic—it was dangerous. Each customer had a different, often incorrect, understanding of how our system was supposed to behave. And so they leaned on us.
Suddenly, internal teams were spending valuable time generating ad hoc reports, pulling metrics manually, and translating system behavior into something the customer could understand.
This wasn’t scalable. It was noisy and expensive—especially when you factor in the human cost of repeated firefighting and lost focus.
When performance tracking becomes a bandwidth sink
At one point, we realized that simply tracking performance had become its own mammoth task—requiring dedicated engineering bandwidth.
We were burning cycles explaining ourselves to customers instead of focusing on improving the product itself.
That was the turning point. We needed a proactive, standardized, and self-serve system to monitor and visualize product performance.
And perhaps most importantly—it had to be flexible enough to handle different product requirements without becoming a blocker.
From Ad Hoc to Operational Transparency
The vision was to surface key performance metrics directly to customers through visual dashboards. No more back-and-forth. No more “Can you run a script to check this?” messages on Slack.
But to get there, we had to go back to the fundamentals.
We needed a cultural shift in how we thought about KPIs and metrics. Instead of tackling monitoring later, we needed to integrate it into the design of every new feature and product—from the start.
A Bottom-Up Metric Mindset
Take something like order creation (OC) integration—a common, well-understood flow.
Every new order is a transaction in our system. For each transaction, we can define:
- Dimensions — like enterprise ID, account ID, reference number
- Metrics — like courier partner API latency, status codes
If every team captured these metrics consistently, we could aggregate them effortlessly—daily, weekly, monthly—to understand real-world product performance at scale.
This bottom-up approach doesn’t just help with visibility—it unlocks long-term observability.
Evaluating Tools: Finding the Right Fit
We explored a bunch of stacks. Prometheus with time-series databases, Elasticsearch with Kibana, and more. Our core requirement was that the solution shouldn’t just be about monitoring—it should be extensible for future observability and visibility needs.
Ultimately, we landed on:
- Elasticsearch for data storage and aggregation
- Grafana for beautiful, customizable visualizations
This combination gave us the flexibility and scalability we needed. It checked every box—developer-friendliness, ease of integration, powerful querying and aggregations.
Adoption, Abstraction & The 'PFAL' Approach
Now comes the tricky part: adoption.
We’re currently working on embedding this monitoring mindset into our development workflow. From a code perspective, we’ve built abstractions so that developers don’t have to worry about plumbing. They can simply follow what I like to call the Push first, ask later.
Just send a well-defined payload—with dimensions and metrics—into Elasticsearch, and it shows up on Grafana.
At the moment, product owners are responsible for defining KPIs and metrics in every design document. But in the future, we’ll automate this too—making it a seamless part of our development lifecycle.
In Closing: The Cost of Ignoring the 'Why'
This journey wasn’t sparked by a fancy tech stack or a visionary whiteboard sketch. It started because we were bleeding time, attention, and trust—internally and externally.
By focusing on the why—Why are customers frustrated? Why are we burning bandwidth? Why do issues keep escalating?—we arrived at a solution that was far more robust than anything we could’ve designed with just a “what tech should we use?” mindset.
And that’s the real takeaway.
If you get the why right, everything else starts to fall into place.