7 Real-Time Data Streaming Databases – Which One Is Right For You?

Photo by Dennis Kummer on Unsplash

In the modern era, everyone expects their data the second it’s updated (if not somehow magically before the data occurs).

Large corporations and Fortune 500 companies depend on this data to be able to predict consumer tastes or estimate where the forces of demand and supply are moving the market.

In turn, many companies are working to modify their batch-style data pipelines into real-time data streams. Real-time data streams provide the ability for analysts, machine learning researchers, and data scientists to develop metrics and models that run as soon as new data is created.

This has become a useful solution for companies that manage manufacturing operations, movie streaming, and detecting issues in system logging.

Real-time analytics is becoming more popular as well as more feasible for companies of all sizes as the cloud provides various tools that can be quickly implemented.

We will be talking about a few of these companies later on but we wanted to reference tools like Kafka, AWS Kinesis, Rockset, Vectorized and Materialize to name a few.

Let’s start with two of the classics.

Data Streaming Solutions

AWS Kinesis

Kinesis is a managed streaming service on AWS. AWS Kinesis being managed provides several advantages compared to some of the other tools on this list. It allows your team to spend less time managing infrastructure components and services and instead focuses more on development. Kinesis allows you to ingest everything from videos, IoT telemetry data, application logs, and just about any other data format live. This means you can run various processes and machine learning models on the data live as it flows through your system, instead of having to go to a traditional database first.

AWS Kinesis also has clear support from companies like Netflix. They use Kinesis to process multiple terabytes of log data every day. This is made easier by the fact that Kinesis is a managed service.

Kafka

The Apache Kafka framework is a distributed publish-subscribe messaging system that receives data streams from disparate source systems.

This software is written in Java and Scala. It’s used for real-time streams of big data that can be used to do real-time analysis. This system isn’t only scalable, fast, and durable but also fault-tolerant.

Owing to its higher reliability and throughput, Kafka is widely used for tracking service calls and IoT sensor data.

So who uses Kafka? Well, it originated with LinkedIn to provide a mechanism to load parallel data in Hadoop systems. Later, in 2011, it became an open-source project under Apache, and now LinkedIn is using it to track operational metrics and activity data. Twitter also uses it — paired with Storm — to build a stream-processing infrastructure.

Kafka is our personal favorite distributed data streaming system because of its operational simplicity. Also for Amazon, a managed-service version of Kafka makes it much easier to implement in your AWS stack.

Newer versions of Kafka not only offer disaster recovery to improve application handling for a client but also reduce the reliance on Java to work on data-streaming analytics. Overall, it feels like the easiest service to manage, personally.

The Real-Time Start-Ups

Going away from more of the classic real-time data solutions we wanted to take a look at some of the newer start-ups that are trying to move into the streaming space. In particular, these real-time streaming solutions offer the ability to easily interact with the data in their streams using SQL. Kafka and Kineses also have ways you can interact with their data using forms of SQL. However, the tools below were developed to be SQL compliant from the get-go.

Materialize

Materialize, is a SQL streaming database startup built on top of the open-source Timely Dataflow project.

It allows users to ask questions of living, streaming data, connecting directly to existing event streaming infrastructure, like Kafka, and to client applications.

Engineers can interact with Materialize using a standard PostgreSQL interface, enabling plug-and-play integration of existing tooling.

When the SQL queries are run they are recast as data flows. This can allow users to perform interactive data exploration and data warehouse-like analytics against live relational data, which is typically not possible.

Under the hood, Materialize uses Timely Dataflow (TDF) as the stream processing engine. This allows Materialize to take advantage of the distributed data-parallel compute engine. The great thing about using TDF is that it has been in open source development since 2014 and has since been battle-tested in production at large Fortune 1000-scale companies.

Narayan co-founder and CEO of Materialize goal for Materialize “is really to help any business to understand streaming data and build intelligent applications without using or needing any specialized skills. Fundamentally what that means is that you’re going to have to go to businesses using the technologies and tools that they understand, which is standard SQL,”.

Materialize also just got another round of funding, so they could be on for bigger and better things shortly.

Rockset

Rockset is a real-time analytics solution that boasts the fact that it provides low latency search, aggregations, and joins on massive semi-structured data, without operational burden. What do all those fancy buzz words mean?

Unlike some of the other real-time databases that are on this list. Rockset is a combination of a database plus a sort of SQL engine that allows you to query across multiple data sources in real-time. For example, you can sit Rockset on top of your DynamoDB, Kafka stream, and MongoDB databases and query/join across all of them.

In real-time.

It automatically indexes your data — structured, semi-structured, geo, and time-series data — for real-time search and analytics at scale.

Also, Rockset provides a great UI for running your queries and several other features that are geared more towards developers.

Like Materialize, Rockset has also received a new round of funding and is currently hiring heavily. All great signs in terms of progress.

Vectorized

Vectorized is still on the newer side of streaming tools as it was just funded in January 2021 with 15.5 million dollars.

The startup’s entry into the crowded data management market is an open-source stream processing platform dubbed Redpanda. It aims to provide an alternative to the industry-standard Apache Kafka engine.

If you want to get a deeper explanation you can hear from the founder of Vectorized Alexander Gallego as he discusses it in the Data Engineering Podcast.

In this podcast, he will discuss how Redpanda was engineered as a drop-in replacement for Kafka. He also shares some of the areas of innovation that they have found to help foster the next wave of streaming applications while working within the constraints of the existing Kafka interfaces.

It’s a great listen if you want to hear about the driving factors for this technology.

Some Open Source Options

There are a lot of options when it comes to picking the right real-time solution. Here are a few others that your team might be interested in. The tools below will require a more technical understanding.

Apache Storm

https://storm.apache.org/

What is Storm?

Storm is a popular distributed real-time computation system that works for big data with a simple-processing model to carry out powerful abstractions. This framework — made an open-source project by Twitter — has been touted as the real-time Hadoop.

It can be used to process new data or to update a database. The distribution function of Storm waits for invocation messages, which upon being received, are computed in a query to construct results.

What is unique about Storm?

This software was developed by Nathan Marz in 2011 to harness higher throughputs while working on multiple nodes in a fraction of seconds.

The Storm software comes with the latency of just a few milliseconds on micro-batch processing, which makes it a reliable data processor. Reliability is a factor that helps Storm stand out as a real-time computation data-processing system.

Apache Storm is based on the phenomenon of “‘fail fast, auto restart” which allows it to restart the process without disturbing the entire operation in case a node fails. The approach makes it fault-tolerant.

Besides the standard configuration of Storm makes it fit instantly for production. This technology is user-friendly and robust which has made it popular among small and medium enterprises along with big-sized organizations.

Flink

https://flink.apache.org/

What is Flink?

Apache Flink is another popular open-source distributed data streaming engine that performs stateful computations over bounded and unbounded data streams. This framework is written in Scala and Java and is ideal for complex data-stream computations.

With continuous stream processing, Flink processes data in the form or in keyed or non-keyed Windows.

What is unique about Flink?

This system is easy to install and can start working with just one command on the command-line interface.

Flink is most popular in the machine learning and data analytics fields, where it’s paired with Gelly to create data-flow programming models. Flink supports timestamping, which makes it convenient to rollback or replay a job.

It uses save points to help in system operations to ensure correct results are provided across failures if a node crashes. This framework processes both real-time and streams data, so it’s ideal for both record and data batches.

Flink is also considered a great alternative to MapReduce — as it’s designed to run stateful streaming for any scale. This framework is independent of Hadoop, but it can be integrated with Hadoop to store, write, or process data.

Which Real-Time Analytics Tool Should You Pick?

Here is the hard part. Which real-time analytics tool should you pick. It’s difficult to provide a concrete answer without knowing your team’s needs and goals.

But I will provide some perspective.

If you’re a small company, then you probably don’t have time or money to migrate your solution if for any reason one of these tools disappears. This is to say, if you were to pick a start-up, you’re at risk of one of those solutions disappearing and then you have to migrate to another tool.

This could be very costly.

So if you do decide to pick a start-up, I would try to get a good deal on your initial rate. Just until there is either enough funding or another company buys them out.

Larger companies can more easily take advantage of some of these start-ups because if the start-up disappears, then they can have a few engineers quickly fix the problem.

At the end of the day, I am sure a few of the start-ups will make it. But you want to make sure you are ready for them to disappear.

Is Streaming Worth It?

Streaming data tools can provide a lot of benefits depending on the use case. They can help provide the ability to manage and process data live.

This can lead to better notifications and decision-making.

Also, the ability to stream and analyze data can allow machine-learning models the ability to provide much better outputs.

Although often these systems are much more difficult to implement compared to daily batch jobs, there are many cases in which the ROI is worth it.

We hope this helped prime you for the different options you have for streaming tools.

Good luck with your development.