Category: data engineering

What Is Change Data Capture

Some data teams need to have their data near real-time for dashboards and reporting. So how can they implement a near real-time data pipeline? One possible choice is a method called change data capture, also known as CDC. I have seen companies employ multiple ways to use CDC or CDC-like approaches to pull data from…
Read more


July 9, 2023 0

7 Data Engineering Projects To Put On Your Resume

Starting new data engineering projects can be challenging. Data engineers can get stuck on finding the right data for their data engineering project or picking the right tools. And many of my Youtube followers agree as they confirmed in a recent poll that starting a new data engineering project was difficult. Here were the key…
Read more


May 21, 2023 0
scaling data teams costs

How To Scale Your Data Team’s Impact Without Scaling Costs 

Photo by Lukas As you increase your analytical processes and abilities, you’ll unavoidably increase costs. But there are definite ways to avoid having your costs grow at an unsustainable rate. This is the topic of a panel at the Modern Data Stack Conference featuring Maura Church, ex-director of data science and data engineering from Patreon.…
Read more


March 14, 2023 0
what is apache airflow

What Is Apache Airflow

  Apache Airflow is an extremely popular tool that data engineers rely on. But why? Why do data engineers like Airflow? Also, what does the Apache Airflow event do? In this article, we will answer questions like: What is Airflow? What is a DAG? Why do people use Apache Airflow? Why do we like Airflow?…
Read more


February 12, 2023 0
data engineering vs software

Do You Need A Data Warehouse – A Quick Guide

Photo by Fotis Fotopoulos on Unsplash Recently several consulting calls started with people asking, “Do we need a data warehouse?” This isn’t a question about whether you need data warehouse consultants, but instead whether you should event start a data warehouse project. Which is a very fair question. Not every company needs a data warehouse.…
Read more


February 2, 2023 0
data engineering consulting

Why You Should Simplify Your Data Infrastructure

Good Design Is Easier to Change Than Bad Design – The Pragmatic Programmer Programming is just one aspect of the difficulties of tech work for data engineers. Creating simple yet robust systems that help manage your data infrastructure is equally important. This challenge of building a simple yet robust data infrastructure remains even with no-code/low-code solutions. Just…
Read more


January 20, 2023 0
data engineering scaling hiring

Improving Your Data Analytics Infrastructure In 2023 – Part 1

Data has been consistently demonstrated to be a valuable asset for businesses of all sizes. Consulting firms, like McKinsey, have found that companies using AI and analytics attribute 20% of their earnings to it. As a consultant, I have personally witnessed how data can uncover new sources of revenue and cost reduction opportunities for clients…
Read more


January 9, 2023 0
data engineering consulting

Should We Get Rid Of ETLs?

AWS has jumped on the bandwagon of removing the need for ETLs. Snowflake announced this both with their hybrid tables and their partnership with Salesforce. Now, I do take a little issue with the naming “Zero ETLs”. Because at the very surface the functionality described is often closer to a zero integration future, which probably…
Read more


December 30, 2022 0
data security consulting

Security And Privacy In The Modern Data World

If you’ve worked as a DBA or data engineer, then you’ve likely had to deal with data access requests. These basic data requests can go from quick requests to endless data access requests that bog down entire data teams. This has been heavily driven by the fact that companies want to be data-driven. Meaning an…
Read more


October 29, 2022 0
data catalog landscape

26 Data Catalogs – From Open Source To Managed

It can be easy to take certain tools for granted when you work for companies with mature infrastructure. One of my favorite tools at Facebook was iData. iData was Facebook’s data discoverability tool. It provided a lot of functionality that I have started to miss. This included the baseline functions you would expect including the…
Read more


August 22, 2022 0