You're Not A Real Engineer Until You Break Production
May 22, 2019
It was about 5:30 PM EST, and the team decided to push a chunk of changes to production for our frontend code base. There were two main goals of the changes: 1. to prevent rendering of an account section, and 2. to add a ton of event tracking for analytics.
Development looked good. Staging looked good too. So we deployed to production. It looked good there too... or so we thought.
"Good work! Go home."
So I went home.
This Morning... (5/21)
When I go into the office, some folks poked fun at me about how I broke the site. I literally had no idea what they were talking about.
I quickly learned that after I left the night before, customers began to complain about form input fields not working. Given that the previous deployment to production was like 95% my changes, I soon realized the obvious. Hellllloooo
Good thing is that
The breakage was reverted within an hour - I have huge respect for the amazing crew that stuck around and made sure that everything was back in working order and that customers' complaints were addressed properly.
I felt so bad. I said sorry for what I did during our morning standup. To my surprise, one of our tech-leads reassured me that no one is to blame, and that the main thing to take away is that we learn how to improve our processes.
Coincidentally, my brother messaged me this afternoon asking, "how's work"? (It's midway through my 4th week at Capsule). I told him that I broke production, to which he replied:
you're not a real engineer until you break production
Well, I've always gone by his works, and I'll continue to do so... I'm an engineer now!
What was the cause?
So we use a (deprecated) library called recompact, and within its
compose() method, I passed react-redux's
getContext(), another method of recompact. We still are unsure why, but this prevented our form (using Formik's
Form) input fields from being filled with any values or text. Perhaps something to do with how Formik handles context.
getContext() as the first argument, and
connect() as the second argument to
compose(), rather than the other way around.
We have a meeting scheduled later this week to talk about this particular incident, which I think will be really informative of how we handle the entire development-testing-QAing-shipping process.
Formik / Redux / Recompact
If you have your own troubles with this particular combo, please talk to me! I'd like to learn more.Tags: reactcapsulecodeproductionbreakrevertincidentformikreduxreact-reduxrecompact
👋 I'm Kevin Wang
🎓 New School for Jazz '13
💻 Software Engineer
🧗🏻♂️ Rock Climber