<Embracing Data Engineering: Insights From Three Transformative Years>
Written on
In my career, I've developed a troubling tendency to miss pivotal calls. For instance, I almost let an internship opportunity with NBCUniversal slip away simply because I was napping when the hiring manager reached out. Given that naps were crucial for my survival during 3 a.m. radio production shifts, I have no regrets about that choice. I experienced a brief panic when I returned a call from my previous manager and accidentally reached the reception desk for The Tonight Show instead. After replaying the voicemail a couple of times, I finally managed to connect with him. I nearly repeated this error when a recruiter from my current employer called while I was busy managing a bike rental service. I eventually returned that call, which ultimately led to my current position, while seated on a Muppet-themed bench at Disney's Hollywood Studios.
September 2024 marks three years since that pivotal moment and the start of my data engineering journey. This experience has led me to embrace completing technical tasks and conducting business in the most bizarre settings imaginable. Some memorable instances include delivering a presentation on downtime detection in my nephew's room while a hedgehog exercised nearby, and tackling critical tasks for a company-wide migration from the floor of my wife’s work housing in Chessy, France.
While recognizing three years in a role might seem minor—neither a first anniversary nor a significant milestone such as 5, 10, or 20 years—it's meaningful to me. By the end of September 2024, I will have spent more time in my current role than at my longest previous job with Disney, double the time I spent in graduate school, and half of my entire post-undergraduate career (from a school that appears on my employer's Top 500 Colleges list).
Despite my aversion to math, learning to code only at 22, and having job titles like "boat captain" and "pool towel operator," I've navigated my way into one of the most technically demanding roles in the field. Although I acknowledge a bit of humble bragging here, I hope my unconventional path serves as motivation for those feeling daunted by the learning curve in data science or discouraged by a job market that can best be described as ugh.
I also want to share my thoughts on this anniversary because I attribute much of my motivation to write and share ideas to this platform. My first three published articles were projects I showcased during interviews. After a few months of immersing myself in data engineering, I started this blog with two main goals: to create applicable, professional data engineering content in a space that previously lacked such resources, and to maintain a public notebook documenting my solutions, failures, and lessons learned.
In the spirit of Google’s former motto, “never be evil,” I strive to adhere to my own: “never be boring.” Data engineering and technical writing can easily devolve into dry documentation. If I can't find a way to make a topic I find fascinating and work I enjoy engaging or relevant, I might as well send my journalism degree back to Phoenix (courtesy of The Walter Cronkite School of Journalism at Arizona State University).
This anniversary is bittersweet. Conversations with people in the field struggling to break in have made me aware of how fortunate I was to ride the wave of hiring that followed the 2021-22 COVID-19 pandemic. As companies adjust to inflation and shifting economic conditions, it may take a while before we see a similar hiring surge again. Opportunities still exist, but interest in data engineering has skyrocketed over the past three years, fueled by the AI boom (and the looming concerns of certain futurists) and an increasingly high bar to secure entry-level data science roles. Soft hiring freezes are common, and skepticism towards newcomers is growing in an investment-heavy field where hiring managers prefer proven candidates over those requiring training.
Having once been a novice just three years ago, I'd like to conclude by offering a tip for each stage of a data engineering career:
Breaking In
Don’t Undervalue or Oversell Yourself
I nearly accepted a job that paid $30,000 less than my first salary in data engineering. More importantly, it was a position I was overqualified for. The role was primarily analyst-focused, relying on a basic BI tool like Tableau to create dashboards. While the domain was intriguing, it wouldn't have allowed me to develop in-demand skills such as Python, SQL, cloud technology, or any of the other buzzwords commonly associated with data engineering.
I almost chose that position over my current role because I felt intimidated by the title, the perceived learning curve, and the salary they were offering—a new graduate learning data engineering on the job. My incredibly patient and exceptionally intelligent teammates dedicated hours during my first two months to help me understand the workings of data engineering.
After intensive training in Airflow, advanced SQL, and Google Cloud, I initially worried I had oversold my abilities, as data engineering encompasses much more than basic SELECT queries and represented a complete shift from the dashboards I had previously created and showcased as work samples.
What I truly underestimated was my capacity to learn and implement concepts quickly. Once I recognized I was stepping outside my comfort zone, I invested significant time and effort into coding and learning outside of work to catch up with my colleagues. I also sought and incorporated feedback on my code while documenting my learning and mistakes in a physical notebook.
If you're feeling frustrated with the quality of interview requests you're receiving or feel “stuck” in your application journey, I urge you to assess your skills and experiences honestly and ask yourself: “Am I overselling or undervaluing myself in applications, on my resume, and during interviews?”
Gaining Proficiency
Break, Fix, and Move On
Successfully deploying my first ETL pipeline with minimal oversight, followed by the encouraging feedback “That’s better than we expected,” helped me relax and motivated me to study more urgently. Having joined in Q4 2021, I was cleared to work independently (albeit still with strict PR reviews) just before Thanksgiving.
Initially, I faced two major fears while tackling support tickets: - The possibility of failing to resolve an error, thereby proving my hiring was a mistake. - The risk of damaging production infrastructure, which could also lead to questioning my hiring.
The downside of working in a shared cloud environment is that errors, especially in production, are glaringly public. When something goes wrong, we receive emails or, worse, notifications in a Slack channel shared by stakeholders.
Due to my inexperience, I was the cause of many of these errors, which in turn led to various broken pipelines. My “favorite” recurring mistake? Schema mismatches in SQL loaders.
The upside of encountering numerous errors was that I quickly became familiar with various types and severities of issues—ranging from syntax errors in Python scripts to duplicates in primary data sources. By confronting errors nearly every day, I honed my skills in resolving them.
As I grew more confident in my abilities, I stopped fearing the inevitability of mistakes. By the end of my first year, I was submitting PRs within 15 minutes of an error notification.
During my time in France, I was able to implement fixes before my east coast team was even awake.
Once you're onboarded, gaining proficiency requires hands-on experience with code and processes—and plenty of trial and error.
Advancement
Consistency is Key, Execution is King
My favorite band instructor's saying was, “consistency is key.” To be completely transparent, as I’m not in management, I'm unsure how promotions and advancement work in detail. From what I’ve read, it seems to follow a hierarchical process where your direct manager proposes candidates for promotions and raises during a review cycle to their department head before awaiting a decision.
From anecdotal evidence, I believe my consistent ability to produce quality, reliable work was a significant factor in my promotion. After collaborating with my previous manager for several months, I observed that she always took the time to test and QA outputs before submitting them to higher-ups or putting them into production. A fellow engineer also taught me how to use tools like Docker and virtual environments to create clean development environments that mirrored production.
While I take pride in my code, it’s the final touches in my development process that differentiate success. Successfully deploying on the first attempt means less time debugging and more opportunity to focus on higher-priority tasks. I believe this consistent performance led my former manager to assign me as the lead on an organization-wide migration, which was my flagship project in 2023. To add to the challenge, I was frequently flying between France and the U.S. to visit my wife, who was working temporarily abroad.
While I suspect that in some organizations, a degree of “playing politics” and “getting facetime” with the right people can influence promotions, ideally, I would suggest that consistent performance helps establish you as a reliable asset.
My advice for advancement is to identify a process or framework that promotes repeatable, efficient, accurate, and timely work. Just as consistency is vital, so is repeatability.
Three years of experience is both commendable and somewhat precarious. I think back to high school, where I thought I was as wise as a senior when I was just a junior.
Three years offer the chance for diverse experiences but also present the temptation to rest on past successes and stagnate.
Fortunately, data engineering is a dynamic field where continuous learning and iteration are essential—not just for success, but for survival.
I look forward to engaging in exciting work, discovering and implementing innovative solutions, and I intend to continue documenting, framing, and sharing my learning and experiments with you.
Thank you for being part of this journey for the past three years,
-Zach