How To Hire Junior Data Engineers
Photo by Nana Smirnova on Unsplash
With all the recent data events I have put together I inevitably run into new data engineers who are either finishing up college or looking to transition into a data engineer or data scientist position.
In fact I have talked to several newly graduated engineers who are struggling to find work. A few told me how the previous classes from their program were able to find a new job in a few months and now they are far past those few months.
Oddly enough at the same time I have a client that has asked me what they should look for in junior data engineers.
So let’s mash two potatoes with one fork.
In this article I wanted to give some advice to hiring managers looking to hire junior data engineers as well as give advice to new college grads and career switchers in terms of getting their first data engineering role.
Hiring For Juniors
Hiring junior data engineers is tricky. Many data engineering topics aren’t taught in school. So if you ask a newly minted computer science or information systems major to design a data model, they’d likely not know what you were asking for.
So how do you interview junior engineers?
There are several areas that junior engineers should be comfortable with. Those areas in my mind are:
- Programming
- SQL
- Data Modeling ( to a point)
- Desire To Learn
SQL Questions For Junior Data Engineers
Now even here, it’s hard to expect someone who has likely only learned about how to design classic relation data models and probably wrote some light queries to have certain skills in SQL.
For example, at what point in a basic SQL does someone learn about self-joins and how they can relate to window functions. Yet, it’s not uncommon for an interviewer to ask “can you do it without a window function” to an interviewee. This might pose some difficulty for a new engineer.
Here are some question types I do believe they should be able to answer.
- Questions that have them apply inner and/or outer joins
- Questions that demonstrate their ability to run aggregates ( maybe even a case statement inside of an aggregate)
- Questions that let them demonstrate their ability to use conditionals and WHERE clauses
- Questions that involve a sub-query ( I usually don’t see junior data engineers that know what a CTE is)
Overall, they should have a solid foundation in SQL that can be built upon. Maybe you throw one final question in ( if there is time) that is meant to be a little trickier and actually see if they can figure it out.
Programming Questions For Junior Data Engineers
Data engineers require varying levels of programming skills depending on the company they work for as well as a team. That could also be said about traditional software engineers.
However, I believe it is even more true with data engineers. For example, at many companies, data engineers are just software engineers that focus on data problems. In other companies, they are more like the recently minted term “analytics engineer”. Both of these roles require different levels of programming skills.
So step one is to assess what level of programming your team needs. Would the new hire benefit from knowing data structures and algorithms (DSA) or is your company mostly using Python to write Airflow DAGs?
One of these roles would benefit from standard DSA questions. Whereas the other you could probably use what I consider more operational programming questions. Questions that tend to be based on your everyday work. Here are some problems that can kind of replicate some of those problems:
They are less about using dynamic programming or a specific algorithm and more about trying to solve the problem.
At the end of the day, it’s not even all about programming and SQL for junior engineers.
Data Modeling For Junior Data Engineers
Although most graduates from college haven’t had a real class on data modeling, I still believe there should be questions on the subject. After all, if they are interviewing for a data engineering position, they should have done some research.
At the very least they should be able to talk to what are the classic data warehouse models, what different tables are and what they represent and maybe give an example of what a general model could look like.
They don’t have to be perfect, but I’d want to see that they cared enough about the role that they looked into what data engineers actually do.
If not, why are they interviewing?
There are hundreds of articles about how to prepare for a data engineering interview or videos on how to become a data engineer. I know I have made at least 3 ( videos probably another 3 articles).
All of which always covered data modeling. So I wouldn’t expect an amazing response but I’d hope they showed enough curiosity to dig into data modeling.
It’s Not All About Programming
When it comes down to it, programming, system design, and SQL are just the baseline skills required for junior employees. There are a lot of less technical skills that are arguably more important especially when someone is just starting their career. For example here is a list of skills or attributes that you’d. probably want to see:
- Ability and willingness to learn
- Interest in the field ( did they at least try to look into data modeling?)
- Ability to communicate ideas
- Curiosity – ( did they ask questions in the interview about anything?)
- Drive to get things done – ( have they been doing projects outside of school)
Now that you have a general idea of what to test for, let’s talk about why. Why even hire junior engineers?
Why Hire Junior Engineers
Yes, just hiring junior engineers is not a great idea. But once your team has a solid foundation, there are a lot of benefits to hiring junior data engineers.
- It Allows Senior Engineers The Time To Focus On Higher Level Problems – There are a lot of tasks that need to get done in the software world. Some are great teaching opportunities for junior engineers but can become repetitive for senior engineers who have likely solved a similar problem 10,000 times. Hiring junior engineers allows your senior engineers to focus on higher level issues like design and complex infrastructure issues which require more experience.
- Mentorship Grows Your Senior Engineers – Part of growing in any profession is learning how to mentor the next generation. It helps improve communication skills, it forces senior engineers to really think about the why, in terms of how they do things and is generally a good thing. Both junior and senior engineers will benefit in cases with mentorship.
- Too Many Cooks In The Kitchen – When you have multiple senior engineers, eventually there might start being more and more conflict in terms of ideas. There is a fine balance and ratio when it comes to setting up teams and I believe the
does a good job in covering some of these possible ratios. The point is, having a mix of junior, mid-level and senior engineers makes for a healthy balance of work that gets done vs. ideas that need to be implemented.
Some Pet Peeves With Interviews And Final Thoughts For Companies
Un-proctored Pre-Screening
In principle, I dislike un-proctored pre-screenings. That is to say when someone sends you a link to 4-5 problems and you have to solve them on a platform without anyone there from the company. The company loves it because they save employee time but that’s kind of an insult. If you think the candidate you are interviewing is a plausible candidate, it’s kind of slapping them in the face and saying they aren’t worth your company’s real people’s time.
Also, there are often little gotchas that a person might get stuck on that just need a quick nudge. I am a little triggered after watching a friend experience one of the worst pre-screening tools I have ever seen where they had a small text box that allowed you to see maybe 4 rows deep and 30 characters wide in terms of where you put your code. Also, it didn’t let you test your code. It was just bad.
Onsite Vs Homework
I enjoy the onsite process of interviews but I am kind of for the idea of providing the option of either onsite or homework. Some people do well talking in person others do well coding alone. Now there is no avoiding the eventual onsite because the company needs to ensure you’re the one that wrote the code, but perhaps it could be a briefer round that is technical or maybe just focused on going through the code you did write in the homework and asking some follow-ups to make sure you understand what is going on.
DSA and Dynamic Programming Problems
I also believe that trick questions and problem-solving style interviews aren’t always as good for senior engineers who might be spending less time directly coding and more time thinking through high-level design. This isn’t to say you don’t have a coding round or two. It’s more just how many dynamic programming questions are necessary. Sure if the role is dependent on that style of problem-solving, it makes sense. But if not, I’d consider limiting those style problems to people who are just getting out of college because they are more familiar with those questions. It’s kind of like answering geometry problems on the GRE. If you want to feel less intelligent and haven’t taken the GRE I recommend it. It will remind you how much you both knew and temporarily forgot.
Tips For Junior Data Engineers Entering The Market
As I neared the end of my degree, I found myself feeling a lot of stress. I was in my final quarter and had been applying for internships and junior data scientist positions, but I wasn’t having any luck. It was frustrating to feel like I wasn’t making any progress towards my goal of securing a position.
However, things started to turn around early in my last quarter when I was able to land an internship, which made the transition into the workforce a lot smoother. I know firsthand how difficult it can be to get your first job, so I’m sharing some tips that I’ve learned along the way, as well as some advice I received from others that I hope will be helpful to you on your job search journey.
Don’t Get To Stuck On Titles
Many data engineers I have worked with didn’t start as a data engineer. In fact, I’d say it’s pretty common for data analysts and software engineers to move laterally into the position.
Even other roles that are less technical can have the opportunity to move into data engineering by working on projects adjacent to traditional data engineering work.
The data analyst route in particular, I believe works because you will likely be partnered with a data engineer to do much of your work. So don’t get stuck on specifically get a data engineering job title.
Also, to add to that. Forget about job titles in terms of seniority. My very first job at 16 was working in a fine dining kitchen in Seattle called Cafe Juanita. I was a prep cook. All I did was chop shallots, break down rabbits and pick thyme leaves one at a time.
Eventually, after a year or so I wanted to get promoted and I never did. I left that kitchen for another one that gave me the promotion I wanted. Well, when I ended my career I worked for another staple Seattle restaurant called the Herbfarm. They didn’t care about all the other experiences I had in the middle of my short cooking career. They referenced back to Cafe Juanita and told me, we figured if you can hang there you can hang here. Titles don’t matter, experiences (and a little clout) go a long way.
Consider Start-ups And All The None FAANGs
In prior years working for a FAANG and/or big tech( Stripe, Lyft, Uber, Pinterest, Zendesk, etc) was the goal of many. You’d get paid really well and have amazing benefits. Maybe you’d start a Youtube channel and quit because of the clout.
But don’t feel like those are the only companies that exist. There are so many others that might be willing to take chance on a junior data engineer. This includes hospitals ( where I started), insurance companies and start-ups. Many of these organizations can get forgotten due to lack of notoriety. However, you can still pick up skills and experience that may eventually get you noticed by that big tech company you’re dreaming of.
If You Can Get An Internship…GET IT
I have seen time and time again. An internship ends up being the reason someone gets a job. When you’re a junior data engineer. You have nothing. Nothing in terms of work experience and all you can do is put the same example of projects from school that everyone else has on your resume (unless someone you had time to write an app while going through data structures and algorithms).
Point being, internships will give you real experience that you can talk through. They also possibly have return offers attached to them. Yes, many companies are cutting some of these, but it’s still better than nothing. I was lucky enough to get an internship in the last quarter of my college experience and I was also lucky to get a return offer.
This was great because it let me have one less stresser as I was looking for a job I actually wanted (see the tip above).
Referrals
The closest thing to a magic bullet to getting a FAANG role, and probably most roles are referrals. Once someone with the right experience or background gets referred, it might be a few months, but then they constantly get reached out to by said company. Usually, I will say that this is more effective for people who have at least a year of experience. But it’s not a bad idea to keep connected with people who get roles at companies you want to work at. Many companies are biased toward hiring referrals because they have one statistic or another that shows it’s better to hire referrals.
Consider The Future
When you’re looking for your initial job out of college or when you’re trying to change careers, you tend to get tunnel vision. To some degree this is necessary. You need to focus most of your energy on studying, getting the right skills, and putting together a project or two. However, in that moment of tunnel vision, you can forget to think about the future. What are activities you can do now that will not get you this junior job today but will set you up for a staff job or VC investing tomorrow?
Some of this will be networking and creating relationships with people that are both professional and personal and as put by Bethany in the comment above, those people become your co-founders and investors later in life.
Similarly, if you have time, once you do get that job, consider documenting the experience. Whether it be via video, blog, or even just a journal for yourself. The first two can help others both by letting them know they aren’t alone while also possibly giving them advice on how to get their first job in an ever-changing market.
In addition, if you continue that writing, you might eventually become known for it. Writing and doing Youtube videos have opened up far more doors than just getting sponsors for a newsletter. It’s opened up investment opportunities, started conversations, helped me be able to attract crowds to events and so much more. But most of that has been compounding over the past few years.
For now, you should focus on getting your first job. But once you get that first offer, think bigger.
- Where do you want to go?
- What can you do now that could compound later?
- Are there skills you need to attain?
- Who at your company or that you know is where you want to be in 5-10 years?
In the end, once you get your first junior anything position. You’re just getting started.
That’s All For Now
There are a lot of benefits to hiring junior data engineers. When balanced with a good mix of mid-level and senior engineers teams can do well. So don’t be afraid to consider hiring a few for your team if it’s expanding.
As for you new data engineers…if you’re just getting started on your data journey. Good luck! Don’t let the current market bring you down too much. There are plenty of data roles open even if the word “data” and “engineer” are not in them.
Many of us have all struggled to get that first job and we are rooting for you!
If you want to learn more, then check out these articles and videos.
Data Science Consulting – 150+ Firms To Evaluate (2023)
26 Data Catalogs – From Open Source To Managed
Reducing Data Analytics Costs In 2023 – Doing More With Less
Databases Vs Data Warehouses Vs Data Lakes
Onboarding For Data Teams – How to set-up a streamlined onboarding experience