A small backstory to explain how I got into my current role
In early 2019, I was five and half years into my job at an online vacation travel booking company. I drove to an office every day as if COVID-19 had never existed (and it hadn’t at this point). But the office I went to was slowly shrinking and the company was migrating all workers to newer, shinier offices. The only problem for me was that this new office was on the opposite end of the city and very far from my home. The commute would be 60-75 minutes each way. Being in my early 40s and having two young kids, I wasn’t willing to sacrifice time every day just to get to and from work at the expense of time I could be spending with my family. So I decided to dust off my resume and go fishing for a new gig. I quickly landed a Sr. Data Engineer position at a “startup” funded by their parent company and even got a 20% raise.
It’s important to know that I chose this job based on what the startup was trying to accomplish, and the challenges I would be helping them overcome. But life is all about managing the changes that are not in your control. A quick 4 months later, the parent company incorporated our entire tech team (myself plus about 8 others). I suddenly found myself having a new boss and a new job that I didn’t apply for. As I often do in times of change, I looked for the positive opportunity in front of me.
My new role had taken me away from the Cloud first approach of the startup and dropped me back into an on-premise platform and a migration effort that was just beginning. There were some familiar things like Hadoop, HDFS, Spark, Python, Linux, etc. But, I also knew that there was a lot for me to learn. I put my head down and went to work. I began learning all the things I knew I would need to know in more depth in order to lead a team and architect a migration solution.
There were lots of areas to learn like:
How best to design a Data Lake?
Which is better Python or Scala?
Should our ETL run on Spark or PrestoSQL?
How do you use Ranger to allow the right authorization to resources?
ORC or Parquet file format?
and many more. One by one, I tackled all of the questions alongside my team and built out our plans, our strategies, and our best practices.
A short 8 months later and COVID-19 was upon us. We began our work-from-home policy and abandoned our awesome workplace inside of a nice WeWork building in downtown Austin.
When it became obvious that we would need to start hiring more talented team members, we engaged a third-party firm that connected us to resumes in LATAM and India. After 18 months of interviewing and making offers, I was now surrounded by a gifted group of women and men from five countries outside of the US. I did a lot of mentoring and educational efforts. I helped explain the WHY and the HOW of what we were doing. I allowed our team to be independent but I also made sure we were all rowing the boat in the same direction.
My personal life was experiencing some change as well. The house that my wife and I had built and moved into at the end of 2017 was now worth a LOT more than we had paid for it. My plan of living there until the kids left for college was no longer the best plan for us. We decided to sell our home and move closer to my wife’s family in Dallas/Fort Worth Texas. After 21 years of being an Austinite, I was gone. You see, even though I like routines, I’ve always embraced change and in order to experience something different, you have to embrace change.
The pandemic has been with us for two years now. I’ve been working from home every day since it started. With the assistance of vaccines and masks and other safety guidelines, I did partake in opportunities for some get-togethers with my team. But the migration project which had moved much slower than I wanted was weighing me down. It was burning me out. I had given so much of my time and energy to the team, to the project, to the company, that I was burning myself out and I needed to find something different. I was not in a hurry to change, I took more of a slow and steady wins the race approach. I had a good job. I had a good team surrounding me. I had peers that were great to work with. But I was just unhappy and I needed to find something different. I wanted something bigger, something that would challenge me in a different way, I wanted to find something that would ignite a renewed energy in me to get to the next level.
So I did what most professionals do, I turned to LinkedIn, Indeed, and Levels.fyi to gather information about what types of jobs people were hiring for and how much those roles were paying. I turned to my friends and former co-workers and talked with them about industry trends and technology. I did not have a timeline for when I needed to make change happen but I had an open mind about letting change happen.
In part 2 of this story of my journey from unhappy to Amazon, I will cover all the ups and downs of preparing for modern-day interviews, the disappointment and sad feelings of rejection, and the ultimate glory and redemption when all of your efforts finally pay off.
