Why is Real-Time Data a complex claim?

Why is Real-Time Data a complex claim?

Peter Drucker, the famous austrian-american business author, once coined the phrase “you can’t manage what you can’t measure.” This principle of this phrase has expanded way beyond the business environment. In today’s world, we want to measure absolutely everything.

We measure the quality of our education system through standardized testing, and we’re continuously monitoring body metrics like our heart rate, calories and others, through devices like the Fitbit and the Apple Watch.

But why do we do this? In the end, it’s simply because of our innate desire to have control over our environments and our lives. Humans look for control, period, and studies have shown that it is likely a psychological and biological necessity.

Going back to the business world, where Drucker’s quote was initially relevant, it is no mystery that technology has enabled companies to control every single aspect of their existence. Software integrates easily through things like APIs and Reverse-ETL tools, making disparate data systems more integrated and coherent. Plus, visualizing data has become easier than ever before with the rise of No-Code query builders and other visualization tools.

Unraveling Data: The Modern Data Stack
Data is ubiquitous in todays’ world and with more information, more chances for it to tangle and turn into an unusable mess. The Modern Data Stack is here to solve that problem.

So now that everything is easier and can be measured, are business owners and managers completely satisfied? The simple answer is not really. In their desire for full control, they now want data to be as real-time as possible. They don’t want to miss anything, and want to know exactly what’s happening at that exact moment. Feeling outdated is the worst feeling ever, as you might be missing out on the next growth opportunity or overlooking your biggest threat.

In response to the need of having real-time data, every SaaS tool in the world now claims to be real-time. Of course, in 2022, nobody wants to be that tool that tells a manager they will be looking at old data.  But is your data really in real-time, is that even feasible?

The answer is technically yes, but everything needs to go right, which is not usually the case. In the realm of data syncs, for example, there are a couple of things we must take into account in order to have real-time data. For the sake of simplicity, let’s imagine that we have basic sync from your data warehouse over to a CRM like HubSpot or Salesforce, and you want your GTM teams to see the data in real-time.

How fast does your data change in the source?

First of all, let’s look into the source, your data warehouse. This is probably absorbing the primary data from your product and other places. Well, how fast does your data change in the source? Is it actually real-time? Probably not due to something called latency, which is something every database has and can be simply defined as a time delay in an execution.

In addition, databases occasionally have down-times. You should always aim to have the least amount of downtime, but it is safe to say that they are inevitable. Both due to the latency aspect, and the possibility of a downtime, it’s quite likely that your data won’t be there in real time.

What about your destination?

Now, let’s take a look at the other side of the equation, your destination, in this case, your CRM. There are two things to take into account here. First, is how fast is the destination able to ingest the data. Usually, with established tools such as Salesforce and HubSpot this is not much of an issue, and they are able to ingest data quite fast. That said, there are other CRMs and marketing tools that can take up to 24 hours ingesting synced data.

API Limits and Costs

Simultaneously, we also have the issue of API calls. This is important to remember because every tool has a limit, even though at times it can be hard to find. An API is a software intermediary that allows two systems to communicate with each other. An API call is a message sent to a server asking an API to provide some information.

The World of APIs | HelloGuru
In this article you will get a glimpse into the world of APIs, what they are and where you can learn more about them.

When doing the sync from your warehouse to your CRM using a reverse ETL tool like HelloGuru you are taking advantage of the CRMs API. Take Salesforce for example, usually, an account has 100k API calls that are counted on a rolling 24 hour period. All the systems you integrate to your CRM consume some of these API calls, so make sure you use them well. But how does this impact real-time data? Well, if you want to have real-time data, you’re going to need a lot of API calls, which can be costly.

Latency, destination characteristics and API calls are just some of the things that make real-time data a complex claim. Technically it is possible, but things need to be perfect, and it can be quite expensive. So the next time you read “real-time data” on any tool, be wary of what that means, because near-real time can be as good and much more feasible.