Everyone's talking about how to optimize infrastructure.
I want to talk about how to optimize your data value.
In large organizations (and even some of the smaller ones I've consulted to), there was a need for a platform (some endpoint or a pipeline) to collect data. It could be sending metrics to anomaly detection systems, it could be APM monitoring, it could be BI events or messages we collect to "analyze" later.
This poses two points of financial failure:
The provisioners of the platform are not the owners of the data. meaning, asking them why they need more resources would always generate the generic reply - "because our users sends more data to our platform".
Usually (I would guess 99% of the cases) - the value of the data being generated is without any measurable KPI, hence we have no way of knowing if the data we push through our systems generates value.
So, how would we approach it?
Ideally, (as everything in life) - with a smile.
To create a proper engagement with the teams we need collaborations of different stakeholders of the platform. You could start by asking leading questions like
(a) How do you utilize your data? how often is it used/analyzed/processed?
(b) Which measures you've taken to ensure we understand the *value* of the data you use?
(c) How can we define and track this KPI we just discussed automatically?
Some examples we faced of over the years:
Metrics were sent to a platform that charged us per metric, and no one looked at the dashboards or used that information
Debug information sent to a production logs pipeline
APM metrics sent to a platform charging us per metric that was enabled automatically with our provisioning of a new application (without the engagement of any of the developers)
And more. much more.
Be proactive.
Ask the right questions.
Reduce waste - and add value to the data you collect