← Home

You're Not A Real Engineer Until You Break Production

May 22, 2019

This is fine meme

Yesterday... (5/20)

It was about 5:30 PM EST, and the team decided to push a chunk of changes to production for our frontend code base. There were two main goals of the changes: 1. to prevent rendering of an account section, and 2. to add a ton of event tracking for analytics.

Development looked good. Staging looked good too. So we deployed to production. It looked good there too... or so we thought.

"Good work! Go home."

So I went home.

This Morning... (5/21)

When I go into the office, some folks poked fun at me about how I broke the site. I literally had no idea what they were talking about.

The breakage

I quickly learned that after I left the night before, customers began to complain about form input fields not working. Given that the previous deployment to production was like 95% my changes, I soon realized the obvious. Hellllloooo

Good thing is that

The breakage was reverted within an hour - I have huge respect for the amazing crew that stuck around and made sure that everything was back in working order and that customers' complaints were addressed properly.

I felt so bad. I said sorry for what I did during our morning standup. To my surprise, one of our tech-leads reassured me that no one is to blame, and that the main thing to take away is that we learn how to improve our processes.

Coincidentally, my brother messaged me this afternoon asking, "how's work"? (It's midway through my 4th week at Capsule). I told him that I broke production, to which he replied:

you're not a real engineer until you break production

Well, I've always gone by his works, and I'll continue to do so... I'm an engineer now!

What was the cause?

So we use a (deprecated) library called recompact, and within its compose() method, I passed react-redux's connect() "before" getContext(), another method of recompact. We still are unsure why, but this prevented our form (using Formik's Form) input fields from being filled with any values or text. Perhaps something to do with how Formik handles context.

The fix

Just pass getContext() as the first argument, and connect() as the second argument to compose(), rather than the other way around.

Next Steps

We have a meeting scheduled later this week to talk about this particular incident, which I think will be really informative of how we handle the entire development-testing-QAing-shipping process.

Formik / Redux / Recompact

If you have your own troubles with this particular combo, please talk to me! I'd like to learn more.

Tags: reactcapsulecodeproductionbreakrevertincidentformikreduxreact-reduxrecompact
Kevin Wang

👋 I'm Kevin Wang
🎓 New School for Jazz '13
💻 Software Engineer
🧗🏻‍♂️ Rock Climber