Shipped with fear

There is a thing that every de­vel­oper has to do at some point — ship. By ship I re­fer to the act of de­ploy­ing, push­ing the but­ton, pub­lish­ing your work for a wide mass. This act is cen­tral in our in­dus­try: blog posts like this one have been writ­ten, there have been talks on the sub­jects, and method­olo­gies (like Continous Delivery, Continous Integration) in­vented to deal with the pain points as­so­ci­ated with ship­ping. Hell, there are even roles (devops, sysops) for eas­en­ing the bur­den of mov­ing code from one sys­tem to an­other. Why, oh why are we con­scious and some­times afraid of ship­ping our work?

Environments of suc­cess

At Lookback, we try to con­ti­nously ship stuff to pro­duc­tion. We’ve got three en­vi­ron­ments: test­ing, stag­ing, and pro­duc­tion. Nothing un­usual. We use fea­ture branches that we in­de­pen­dently de­ploy to any of these en­vi­ron­ments. We merge to mas­ter, which is con­sid­ered sta­ble, when we’ve con­firmed the bug or fea­ture. This means we can com­fort­ably try out things in test­ing and stag­ing and be some­what sure of that it’ll work in pro­duc­tion, as long as things like en­vi­ron­ment vari­ables and data­base mi­gra­tions have run (note how I ca­su­ally men­tioned those po­ten­tially big fac­tors?). We’ve got a script for rolling back to the ver­sion be­fore the de­ployed bun­dle, in case hell breaks loose and we need to quickly re­store.

That’s the back­ground.

An anec­dote

The thing is: every­body screw up some­time. It might be a pre­ma­ture de­ploy be­fore thor­ough QA, poorly writ­ten code which crash in other en­vi­ron­ments than your own. There are many rea­sons. But the im­por­tant thing is not to let it get to you.

Don’t tell my team mates, but some nights ago I de­ployed code that broke the emoji fea­ture in our pro­duc­tion web app. It was a reg­u­lar ex­pres­sion that went hay­wire for some pro­duc­tion data I had ne­glected, and I quickly rolled back, branched off (this was al­ready merged to mas­ter), and con­tin­ued to push code to the test­ing en­vi­ron­ment (with real data this time).

This has hap­pened many times be­fore, and every time I feel kind of slapped in the face. Brought down to earth again. It’s like some­body’s say­ing Hey, sonny boy! Don’t walk around think­in’ you can ca­su­ally push to pro­duc­tion with­out some­thing go­ing wrong! It’s bound to hap­pen, ya know”. And af­ter every time I tend to be much less lib­eral in my de­ploys to pro­duc­tion (until I for­get about the in­ci­dent).

I of­ten com­pare this with how it was when I was a kid, and went ski­ing down­hill (who am I kid­ding, I still feel like this at grown age). I would be cocky and go down this huge hill and get to­tally pounded. Hey kid! Don’t fuck­ing think you’re gonna be able to go down here ten times with­out falling once! It’s bound to hap­pen, ya know”. I’d then have deep re­spect for the hill, and be afraid to go up there, since I was burned once.

See the sim­i­lar­ity?

What to do

I must never, ever be­come afraid to de­ploy. The best med­i­cin is to do it of­ten: small, atom­i­cal fixes and fea­tures that will get your spirit up again — to con­firm that you’re still okay. The worst thing that can hap­pen (engineering wise) in a fast mov­ing startup is loss of mo­men­tum — that we stop de­ploy­ing to our cus­tomers. That we col­lect these balls of code mud which we push af­ter test­ing them in­ter­nally for some weeks or months. Especially for the web, which is ideal for con­ti­nously dark de­ploy­ing to (transparent changes, no need to up­date any na­tive app code).

But of course: my aim as a de­vel­oper is to be able to write such solid sys­tems that don’t causes ma­jor break­age when de­ployed. I want to be able to trust my­self and the work­flow well enough to be sure and con­fi­dent when de­ploy­ing. Some good recipes for that is the afore­men­tioned en­vi­ron­ment setup (“test first — con­firm on stag­ing — de­ploy to pro­duc­tion”), solid QA, code re­views, and so on.

Developers fear change, new things, refac­tor­ing, de­ploy­ing. But we must con­stantly ex­pose our­selves to this in or­der to grow and win over those fears, and learn how to make the wise de­ci­sion when the day comes to make a re­ally big, im­por­tant de­ci­sion.