You absolutely can have too much data.
Like when you’re sorting through messages trying to figure out where exactly an API call went wrong. Or when you’re paying for more and more cloud data storage with less and less performance.
DevOps needs data to be successful. It drives the logic telling our automations when and how to fire, and the KPIs telling us if what we’re doing is really working. And there’s never been more of it. As good as we are at collecting data, however, we’re not always as successful at storing, accessing and analyzing it.
Two of our clients — Dickson, a manufacturer of IoT-enabled environmental monitors, and Pronto, a mobile wallet based customer experience platform — were generating huge amounts of data they relied on for daily operations. Before these data piles became unmanageable, we needed to find a system that could keep the data accessible and affordable.
Aiven proved the solution, an open source cloud data infrastructure provider that does exactly what it says on the box, and does it well. With the right place to put the data, and an easier way to access it, we were able to work with both clients on smarter ways to use it both today and going forward.
Data storage in the cloud makes our client’s data scalable
Both Pronto and Dickson deal in data. Dickson provides visualizations for regulated GxP manufacturers, labs and other customers to monitor temperature, humidity, pressure and other environmental factors. All of that data is collected, presented to the end users and stored for them to reference whenever they want.
Pronto operates with a breadth of clientele, working with stadiums, professional sports teams, chain restaurants and other vendors to create a streamlined app or IoT device-enabled experience for customers. That means storing the customer profiles, tracking their actions and transmitting the information seamlessly from person to business and back, and doing that for arenas full of people at once.
With Dickson, we previously stored data in MySQL, a highly regarded open-source cloud database. As the datasets got bigger, we found MySQL struggled under the load of inserting new data and keeping existing data available on demand, something our clients and our clients’ customers rightfully expected. With Pronto, the client needed access to log files to be able to provide rapid support to their customers. It was a problem when those log files were hard to access and distributed across servers.
We needed solutions for both problems, and found one that evaded slowdowns no matter the data size, and made access to log files straightforward and pleasant.
A DevOps metrics framework that’s accessible for the whole team
Access to log files were especially important for Pronto. They are a small team and need to be able to support client issues as efficiently as possible. A solution that required a highly technical background — or that took too long to navigate — would not do. Pronto had clients coming to them for information on how their customer loyalty passes were performing. Moving data across passes, businesses and app stores created multiple possible breaking points. Troubleshooting the entire path takes time. Smarter DevOps metrics could point to exactly where to look.
Pronto wanted a system where it could easily call up an issue using an ID, instead of sorting through a raw log. Beyond that, the team wanted to be able to see overall data for and across clients. With trend data, they could identify problems that were likely to occur, instead of waiting for client feedback, proactively solving problems.
Why DevOps is important to CEOs: Business value and benefits
How Aiven creates a cloud data tool that just works
Like with so many DevOps tools, the most important thing was finding a solution that could work with the systems we already use. The first goal was to find a technology capable of processing and storing time-series data in a scalable and performant way. We liked TimescaleDB, a PostgreSQL extension that layered in the necessary performance and structural enhancements for time-series data. That then led us to Aiven, which supports the plugin out-of-the-box and even powers Timescale Cloud. The pricing was reasonable, the hosting options were excellent, and the customer service was great — we’ve had no complaints.
It’s more than just the storage, though. Aiven supports things like zero-downtime rollouts, so our clients won’t ever have to worry about data going offline, or warn about outages. The team there manages the instances for us, spinning up new servers and down old ones with little downtime so we can minimize interruptions.
How Docker and Kubernetes make infrastructure almost instant
It’s a seamless solution. In a lot of ways, Aiven is the cloud data service for people who don’t like to think about infrastructure. And while we think about infrastructure all day every day, we don’t do it for our health. The goal is to spend that time improving, not maintaining. Aiven helps us do that.
Identifying the DevOps metrics that matter
For Pronto, Aiven is a tool for seeing the forest and the trees — and knowing the difference. As a debugging tool, it gives the team a focused visibility they wouldn’t otherwise have. When something goes wrong, the team checks Aiven to see what came in and went out through its API, knowing right away whether the problem was on the client side, within the Pronto system, or in the Google or Apple app stores.
At this point, it’s just a matter of getting more of that data in a more digestible format, so the Pronto team can spot risks before they happen and prioritize new features and functions.
Enabling better — and more — DevOps data and insights
Data is the right solution for clients who want to make smarter decisions, and Aiven is generally the right solution for clients who have big data stores to grapple with. Aiven has powerful open source tools, the ability to work across clouds, regions and a wide range of plugins and libraries and an expert team to handle any hiccups across them.
As IoT spreads and tracking improves, the data we’re dealing with on a regular basis will only get bigger. Knowing how to efficiently store that data at scale — and with timestamps, user IDs or other navigable contexts — will be the only way to keep that data from becoming more noise than signal.
Accessible data can be made meaningful. Aiven does a great job at making the data accessible and keeping it secure enough for HIPAA compliance and other standards. That leaves it to us to find the meaning.
Do you want more meaningful, accessible data? Contact us.
