In short: data teams can cut costs without losing performance by doing five things: using open-source tools, standardising their stack and processes, moving some work back to laptops, tracking costs closely, and getting rid of data nobody uses.
This article was first published in May 2023, when this post was titled "Optimizing Your Modern Data Stack for Economic Challenges". The advice still holds, and we've refreshed the links.
The economic climate looks very different from the low-interest-rate years that came before it. Economic indicators have been showing clear signs of slowdown across sectors, and both governments and organisations are focused on controlling costs.
Stock market chart showing a downward trend
Data teams are no exception. Everyone is convinced of the value of data today. But there has been an almost unchecked rise in spending on data tools and services, which has led to runaway costs. Cloud data warehouses, modern ELT tooling and the growing number of data tools organisations have bought all add to the problem. That was fine in better economic times, but not today.
At the same time, demands from business users are only going to grow. The move towards the modern data stack is inevitable, as more of us discover the value of scalable, resilient data technology. So how do you find the balance? Here are five practical ways for data and analytics teams to cut costs without sacrificing performance.
1. Embrace open-source technology
When anyone asks me the number one thing data teams can do today to cut costs, my answer is: open source. When someone asks me the number one thing data teams can do to drive innovation, my answer is also open source. It's a no-brainer with many benefits.
Open-source tools are cost-effective alternatives to licensed software. A few examples:
- Meltano for data extraction and pipelines
- dbt Core for data transformation
- Apache Superset for data visualisation
With open-source tools, your team gets powerful software without breaking the bank. You're also connected to active developer communities that offer support and ideas. These communities keep building tools that compete with commercial software, and often lead the way on innovation.
2. Standardise your data pipelines and tech stack
Standardising your data pipelines makes teams more efficient and helps them work together. The first step is to build one data stack that covers many user needs. Agreeing on which tools and processes to use goes a long way towards controlling costs.
On top of that, run projects with agile practices, keep all your data engineering code in Git repositories, and keep your technical documentation up to date. Standardising doesn't just streamline your processes. It also gives new team members a consistent experience. When everyone works the same way, you get more out of the modern data stack and manage costs better.
3. Consider local machines for some workloads
Yes, this is a contrarian take, so let me explain. Laptops have become very powerful. Apple's M-series chips and the latest processors from Intel and NVIDIA make it possible to move some workloads back to local machines. Apple's M-series laptops, for example, comfortably handle many data and analytics workloads.
Combine that with fast new analytics databases like DuckDB, and your team can cut cloud costs and work faster. Work out which tasks can move back to local machines, and you can save a lot of money while using your resources more efficiently.
4. Monitor costs and make someone accountable
A system for tracking and monitoring data costs is key to managing them. Set up regular cost monitoring, using cloud cost management platforms or your own monitoring, and make a named person in the data team responsible for costs. That way you keep people accountable and avoid surprise bills. Watching cost trends closely also helps your team make better decisions.
5. Find and remove dark data
"Dark data" is data sitting in your storage systems that nobody uses. It still costs you money to store and process. Run an audit to find it and remove it. You'll free up resources, and your team can focus on getting insights from data that actually matters.
Another very effective way to reduce dark data is to set up a good data catalog, so people can see what data exists and who uses it. There are excellent open-source options for this too.
The bottom line
By following these strategies and sticking to the principles of the modern data stack, your data team can cut costs without losing performance. Balancing cost control with the changing needs of your organisation is essential in today's economy.
FAQ
What's the fastest way for a data team to cut costs? Look at your licensed tools first. Open-source alternatives for extraction (Meltano, Airbyte), transformation (dbt Core) and dashboards (Superset, Metabase) can replace a big part of your software bill.
What is dark data? It's data an organisation collects and stores but never uses. It adds storage and processing costs without adding value.
Can laptops really replace cloud warehouses? Not for everything. But for exploring data, prototyping and working with small-to-medium datasets, tools like DuckDB on a modern laptop are often fast enough, and they cost nothing to run.
Want help finding savings in your data stack? Talk to Newtuple.
