San Francisco, California
2017-03-13 to 2017-03-14
Co-presented with Charity Majors
Description
Ask people about their experience rolling out microservices, and one theme dominates: engineering is the easy part, people are super hard! Everybody knows about Conway’s Law, everybody knows they need to make changes to their organization to support a different product model, but what are those changes? How do you know if you’re succeeding or failing, if people are struggling and miserable or just experiencing the discomfort of learning new skills? We’ll talk through real stories of pain and grief as people modernize their team and their stack.
Detailed review by Tanya Reilly
Video recording of talk
Tweets
Join @mipsytipsy & me in Grand Ballroom A to talk microservices at 9:55am! (I won't be here tomorrow but she's keynoting.) #SREcon #SREcon17
— Bridget Kromhout (@bridgetkromhout) March 13, 2017
I'll be holding down Track 3 and livetweeting. Looking forward to dynamic duo of @mipsytipsy & @bridgetkromhout up first! #SREcon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
Keeping calm! #SRECon17 #SREcon17Americas pic.twitter.com/cI4KkphD9k
— Joshua Goldman (@jgoldsre) March 13, 2017
"Computers are easy, A/V is hard." @bridgetkromhout #SREcon17Americas
— Rich Burroughs (@richburroughs) March 13, 2017
.@bridgetkromhout and @mipsytipsy on the human side of microservices: "how to grow up" and get past shiny & new #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: "Pizza teams", "binary up/down", all inexact metrics. "Microservices are about changes" #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Do you have autonomy and manageable scale? What is your culture about communication? #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: humans have the same problems computers do e.g. partitions, eventual consistency, APIs #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: checklists are guidance for moving in a direction, but not mandatory to do all of at once #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: "SWEs should be oncall" -> "so I should fire all ops people" -> "NO DON'T DO IT STUPIDLY" #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: You need to be really good at operations to do microservices. They're more complicated. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
@bridgetkromhout omg yes pic.twitter.com/YnLxUVfTtt
— Joshua Goldman (@jgoldsre) March 13, 2017
.@bridgetkromhout & @mipsytipsy: "Run on a LAMP stack if you can make it work". Don't succumb to buzzword adoption #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Spin things that makes sense out of your monolithic thing, don't decompose everything #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: "Choose boring technology" from last year is still true. Also use SaaS if you can. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Spend as many cycles as you can on your core business, not on random bullshit. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
Microservices shouldn't be a goal, but the result of a need you have. @mipsytipsy & @bridgetkromhout laying down wisdom #SREcon17Americas
— jeremy “typos.af” price (@jermops) March 13, 2017
.@bridgetkromhout & @mipsytipsy: If it's someone else's mission, they will do a better job than you can do half-assing it. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
'Choose boring technology' still true. If it's not core to your business, don't innovate. @mipsytipsy & @bridgetkromhout #SREcon17Americas
— Tanya Reilly (@whereistanya) March 13, 2017
.@bridgetkromhout & @mipsytipsy: "Remember when we used to train our own spam filters, then gmail came along?" #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: If you can't decentralize control, you are not tall enough to ride the microservices ride #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout talking about people practising "Resume-driven development". Hahaha. That's amazing. #SREcon17Americas
— Tanya Reilly (@whereistanya) March 13, 2017
[Beware] engineers practicing resume driven development wrt #microservices @bridgetkromhout #SREcon17Americas
— jeremy “typos.af” price (@jermops) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Teams are services, humans are nodes. Up to a point. Don't restart your humans :P #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: (but there should be enough redundancy that people can take vacations!) #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Management's job is to define and repeat the mission ad nauseum, and do routing/LB/health #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: If you're not passionate about your mission or the money, find another job. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Remember Dunbar's Number - need to shard past 100-150 vs. relying on informal interfaces. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
Thank you for this thread && @bridgetkromhout @mipsytipsy for the (as always) excellent content.
— Matt Weagle (@mweagle) March 13, 2017
.@bridgetkromhout & @mipsytipsy: But don't make managers SPOFs. Deliberately design paths for routing, and chaos monkey. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
@mipsytipsy and @bridgetkromhout on the role of management in tech orgs@SREcon pic.twitter.com/aj3ItPi6rQ
— Connie Lynne (@clynnexx) March 13, 2017
.@bridgetkromhout & @mipsytipsy: The entire point of microservices is you don't need to redeploy *everything*. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Make sure you're tracking that the thing that you shipped *yesterday* still works. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Implicit communication (or lack thereof) is a signal. 10x hidden problems as visible. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: interpersonal blamefulness leads to silenced communication. Working agreements matter. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
Great talk on "Scaling your org with microservices" @bridgetkromhout #SRECon17 pic.twitter.com/rWj93Nj57v
— Taos (@TaosTech) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Flush communications to persistent storage, don't just keep them in peoples' RAM. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Don't just check in the what, check in your why as well. Future-you will thank you. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
"When you make the commit in github, say WHY you made the change, not what the code does."
— Connie Lynne (@clynnexx) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Don't depend upon your managers to have bandwidth to do anything more than routing. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: cross over between TL and M roles every 5 years, don't just stick to one or the other. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Expect SWEs to do ops work the same as you'd expect them to write tests. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: "Designated ops" specialists, not "dedicated ops" --quote from @beerops #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
Thank you @bridgetkromhout & @mipsytipsy we need OPS but OPS need to allow failure. To learn. pic.twitter.com/cxUfpZaaAR
— Chantell Harris (@ITSM_Chantell) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Don't try to abdicate oncall to your ops people; share the responsibility & empowerment. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Oncall shouldn't be hell; get rid of our martyr complex. Spend your error budgets. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
#SREcon17Americas @mipsytipsy @bridgetkromhout: Mgmt = dumb pipe between smart nodes (the engineers) - don't make the network smart
— Kurt Andersen (@drkurta) March 13, 2017
.@bridgetkromhout & @mipsytipsy: If someone says "features are higher priority than oncall", their shipped shit won't work #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
I did computer science so I wouldn't have to talk to people! There's the bait and switch! @bridgetkromhout
— Jamie Wilkinson (@jaqx0r) March 13, 2017
.@bridgetkromhout & @mipsytipsy: You need to be able to debug entire system, not stay within playbooks & compartments. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Trust, but verify. Use auditability, not lockdowns. Otherwise everything is emergency. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
I've worked somewhere that had that first problem @mipsytipsy @bridgetkromhout #srecon17americas pic.twitter.com/Z4T6e7p3H8
— Rich Burroughs (@richburroughs) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Don't promote people to senior SWE unless they are good at ops. Send good signals. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Try to make stateful services less stateful over time. Push your state down in stack. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
"Life is about trading for a better set of problems." @mipsytipsy #SREcon17Americas
— Rich Burroughs (@richburroughs) March 13, 2017
.@bridgetkromhout & @mipsytipsy: If your DBAs and SWEs aren't talking, you have an awful mess from both sides. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Everything is built on observability, whether of humans or svcs. [ed: HAVE SKIP 1:1s FFS] #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: People observability: don't wait for permission. Talk to people. Notice emotions, faces. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: You have a responsibility to your team's well being whether manager or not. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Containers aren't magic and won't fix culture or technology. Get buy-in, talk to others. #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
.@bridgetkromhout & @mipsytipsy: Hard things are hard even when distributed and small. [applause] #srecon17americas
— Liz Fong-Jones (方禮真) (@lizthegrey) March 13, 2017
These ladies where fantastic! @mipsytipsy & @bridgetkromhout make learning fun! #SRECon17 Thought- Don't reinvent too many wheels
— Chantell Harris (@ITSM_Chantell) March 13, 2017
Amazing talk by @bridgetkromhout and @mipsytipsy on the human challenges of microservices. #SREcon17Americas pic.twitter.com/wBqF6N05KH
— Stephen Weinberg (@weinbergs) March 13, 2017
“Empowerment and responsibility go hand in hand” @bridgetkromhout @mipsytipsy at @SREcon 🙌 https://t.co/xQVIfsAqtp
— HumanOps (@humanops) March 14, 2017
I'm not saying "don't answer @bridgetkromhout late night before #SREcon17Americas", just saying you might end up with two new talks to write pic.twitter.com/UmDUpTDcVe
— Charity Majors (@mipsytipsy) March 13, 2017