Are your data points valid

Not Everything That Can Be Counted Counts, and Not Everything That Counts Can Be Counted

There's debate who actually said the above quote so I will just leave it as that.  Following similar theme to what I have been writing about a lot lately, I wanted to talk about using data to make better decisions.  And this fits similar themes I've been writing about recently--engineering tends to fixate on a single solution to solve problems and does not achieve meaningful results.

I am actually a big advocate of using data to make decisions because then its not an emotional decision, it's a factual decision.  However, collecting the wrong data is as effective has having no data.

Case in point:  A big push at work has been to start gathering metrics in grafana, using dashboards and alerts to elevate system health. On the surface this sounds good.  (And I know the intentions are as well).   

But here is where it is falling apart:  for the most part, the information is meaningless noise.  Between not understanding what a given data point means to the volume of data (and alerts), there is so much chatter it has become meaningless.

We have alerts that "we're working on this" (meaning we don't know if its legit) to "you can just ignore that one" to "this is a problem but I have no more information than before" and so on.  None of this is helpful.  We have invested huge amounts of engineering time, increased our operational costs in both data storage (grafanna) and cpu time.  And now I am asking, for what benefit?

The goal was "system health".  If we are honest with ourselves, that's a very vague goal.  System health for an API end point is not measured the same way you measure health of the database.   System health for invoicing system is not the same as system health for authentication. So you have to have some prior knowledge of what is meaningful to track prior to building observability, otherwise you're just increasing costs and creating noise.  None of which actually helps improve the over all system.

Thoughts?  Please share through my contact page.

PS: I am thinking about opening up comments on this blog.  Lmk what you think of that too.....