As data increased in volume, velocity, and variety, so, in turn, did the need for tools that could help process and manage those larger data sets coming at us at ever faster speeds. As a result, frameworks such as Apache Spark and Apache Flink became popular due to their abilities to handle big data processing…
Read more
Data has been consistently demonstrated to be a valuable asset for businesses of all sizes. Consulting firms, like McKinsey, have found that companies using AI and analytics attribute 20% of their earnings to it. As a consultant, I have personally witnessed how data can uncover new sources of revenue and cost reduction opportunities for clients…
Read more
Scaling data talent is hard. It’s hard to hire. Hard to grow and just overall hard in terms of managing processes and output. But there are brilliant leaders and managers doing it everyday. I recently interviewed Veronica Zhai, the principal product manager at Fivetran. She has led Fivetran’s analytical team for nearly the last year…
Read more
Photo by Christina @ wocintechchat.com on Unsplash Why does it feel like so many more articles are discussing the data engineering profession? Perhaps it’s because Dice’s 2020 tech jobs report cites data engineering as the fastest-growing field in 2020, increasing by a staggering 50%, while data science roles only increased by 10%? Or maybe it’s just because the Medium…
Read more
Photo by Windows on Unsplash It seems like it might be finally happening. Data engineering is finally getting a little bit of the data lime light. Which makes sense. The data being stored and analyzed is not only becoming more voluminous but its speed, complexity and variety are also increasing. Making it difficult to wrangle. With…
Read more
Photo by NEW DATA SERVICES on Unsplash Many companies treat data as an afterthought, like a chef throwing parsley on a dish and calling it a garnish. That’s because many companies don’t live and die by their data. They don’t eat, breathe and sleep data. Well, I have. Or at least, I have worked for…
Read more
Photo by bruce mars on Unsplash Technology has been speeding forward through 2020. The impacts of Covid have only sped up the adoption of change for companies. Companies small and big, start-ups and corporations have been pushed to innovate and improve processes. For example, small, medium, and large companies are migrating to the cloud to…
Read more
.Photo by Kelly Sikkema on Unsplash As 2020 has comes to a close, we wanted to take a moment to reflect on all the changes in technology as well as look to see where things are going. Whether you are looking at start-ups and their IPOs, improvements in technology or you paid attention to Amazon Re:Invent.…
Read more
Photo by Isaac Smith on Unsplash Analyzing data is a skill required but more than just data analysts and data scientists in 2021. Many roles require some form of data analysis whether it be done in Excel, Jupyter Notebooks, or Tableau. However, data analysis is not generally a course taught in college. It might be…
Read more
When your team needs to create dashboards and metrics using data from external sources like Salesforce and MySQL, they face a difficult problem. How do they get all their data into one place in order to merge it all together? Traditionally you would need to have a large team of data engineers to connect to…
Read more