Talk notes: The only thing that matters is you

The following are my speaking notes from the talk I gave at the December 4th DevOpsTO meetup, titled 'The only thing that matters is you'. I elected to put them up as-is, with minimal editing, because I plan on refining the contents for a future talk - there was just too much content in this one. The abuse of bullet points is because I can't for the life of me figure out indentation in blogger.  The italics, etc are meant to help me emphasize words and phrases, remind me of points, and so forth. 

The Q&A section is interactive, actually getting folks to participate, respond, and think about things.

=================



The following presentation is a compilation of everything I’ve learned on my journey so far - for brevity I’ll be paraphrasing quotes, but the books in the intro slide are my primary source.

Q&A starter

Let’s start by setting the stage in your own minds...
(Point: people matter more than tools or process)

You’ve lost your credit card, maybe it was stolen?...you’re not sure. You call the bank to report it and get their automated system - which option do you prefer? (only get to pick one)

• Punch in card#, then press * to report as lost/stolen, knowing the system will then hang up on you 
• Speak to a person right away 

(poll - hands up if person, hands down if automated system)

Interesting! In an important situation loaded with a sense of urgency - when pressed with risk and consequence - we choose against utilizing tools and process.
     We prefer to talk to a person than utilize a tool, or process.

So, given the deadlines, pressures, and risks of work in IT…it seems natural, when trying to effect change, to be inclined toward, start with, that which we trust most.

Why then, do we choose tools and process over people at work?

Take 15s to think, then start shouting out some ideas...why do we choose tools/process over people at work...when it’s full of risk, pressure, deadlines...
-trust, fear, bias, communication, computers don’t shout at you, risk

Consider the following scenario…
(Point: establishing that you can’t just deal with anyone, you need trust)

Ok, 3 types of people, whom do you trust?
Person1: Passes you in the hall, says ‘how are you doing?’ but never stops.

•      Is this someone you can trust? Why?
•      (They’re not taking the time to really understand)

Person2: Chronically makes promises on behalf of others, oversells & overpromises.

•      Is this someone you can trust? Why?
•      (They’ve no consideration for others)

Person3: Is a CEO who is always able to provide accurate information about deadlines.

•      Is this someone you can trust? Why?
•      (Perhaps a fear of leadership, but at least they do what they say they will)

We trust those around us who demonstrate they are:

• Seeking to understand 
• Thinking outside their own needs 
• Acting with integrity 

Are these desirable traits in co-workers?

Let’s keep going. Here’s another scenario to focus your mind:

Who has ever worked on a team where one person just isn’t cutting it - they’re just not delivering, not showing interest; just punching the clock - anyone?
(hands)
Does this affect your department, your team morale?
What about meeting deadlines with others?

• Do you trust them? Why? 

We don’t trust them because when our reputation and jobs are on the line, the risk is too great. Given the risk in IT no matter our role, we must have trust.

We can see the value of a high-trust environment given those scenarios we just went through, and yet it seems to be lacking wherever we go.

It’s clear that the question becomes, if trust is so important to us working well, improving, managing risks, growing - why don’t we trust more?

Here are some reasons that I’ve run across...

• Feel like I’m treated like a machine 
• My concerns dismissed 
• Fear of reprisal, or fear of being unpopular 
• Hopelessness - See no point, no purpose - nobody’s ever going to change so why bother 
• Sloth - Someone else will surely deal with that, it’s so obvious 
• Silos - no opportunity, we’re encouraged to keep to ourselves 

But surely there are ways to address these trust issues, ways we can tackle ourselves.

•      Are we to do nothing?
•      <well?>
•      (you’re here...so you probably want to do something)

In practice

     Begin here: “Seek first to understand, then be understood.”

Listening first, being genuine - truly hearing what the other is saying, what they are asking, what they need.

Consider these questions when someone brings you their troubles:

• How did they get here? What path have they taken? 
• What’s their story? (everyone has a story) 
• Why are they asking in this way? Angry/sad/hopeless/apathetic 
• What assumption of theirs am I missing? 
• What actually frustrates them here? 
• Is their frustration a downstream event? Look elsewhere in the system? 
• Am I contributing to their frustration? 
• Do I understand the ramifications of the friction here? Am I enabling something undesirable? 
• What can I do to help make things better? 

Once you start asking, you’ll unravel the problems one by one. It will go something like this:

• Because I know that your success is critical to our success, I will seek you out, ask questions, and strive to understand your position and pains. 
• I will actively try to suppress my preconceived notions, mental models, and seek to learn with you, to understand you, to come to a shared understanding, not a dictated solution. 
• You will instinctively want to work with me because I demonstrate true desire to see you succeed - there is no hidden agenda, no dis-in-gen-uity, no false transparency - humans are good at picking these things out. 
• This new shared understanding will uncover what must be done, and we begin the task of iterative implementation (plan-do-check-act). All that is left is follow-through, and continuing the cycle. 

By taking this starting step, our collaboration will lead to success, or in the least, learning something, and the organization will not have gotten just a little better -

•      it will have sustainably improved itself: 
•           that relationship is now established - the next time will be even better, easier. 

The act of seeking to create trust is the start of a positive reinforcing feedback loop -

•      this is something that YOU begin, and it is a tremendously rewarding experience.

I found this advice to hold the key. Before having a conversation, ask yourself this...

• Am I willing to be influenced? 
• Am I genuinely interested in what the other person has to say? 
• Do I believe we can learn together? 

If you can't answer yes to these questions, you should consider what it would take to answer yes prior to having the conversation.

•      What’s holding me back? 
•           Do I have some hidden bias, fear, trust issue? 
•           Have I really thought about why this is?

The key, of course, is yourself.

As many ‘change agents’ will attest (or leaders, as I believe they should be called), you often must be the first to extend the olive branch - trust is a reciprocal act but has to start somewhere - the initiation requires sacrifice on your part.

So YOU must be the lever in your system - act within your sphere of influence, do what you can, invest in mentoring, set an example for others.

•           Demonstrate that change is possible at all.

Whatever requires you, personally, to stretch and be uncomfortable is probably a good first step. Thinking “ehhh, I probably should, but I really would rather not” - that’s a good flag. I was once told: You need to get comfortable with being uncomfortable. (part of why I’m here tonight) Pithy, but effective!

Let’s take it a step further.

If an organization is the product of how its members think and interact, consider:

•      If you can change how YOU think and interact, you by extension change the organization. 

Let me repeat that:
<repeat it>

When we take this perspective, it becomes abundantly clear, I believe, how important our own thinking and behaviour should be. So to each and every one of us here, let’s be clear: The change must start with you.

If you’ve reached that point in your life where you are ready for change, and perhaps I would not be so wrong to suggest that you being here tonight puts you in that box, then push yourself forward! Strive to understand what this means, seek out what it means practically - how can I integrate it into my life?

With change, the only thing that matters is you.

An SIPlay story

Over the past three years the Sports Illustrated Play engineering team has grown, progressed from being told ‘no time for that!’ to being exhorted ‘do more of this!’. The road was not easy, we faced a tough climate.

• a sales- and revenue-driven org “enterprise startup” 
• multiple-acquisition .NET monoliths living in a single ecosystem 
• brand new engineering team, variety of personalities 
• brand new leadership, intent on pushing hard 
• limited development staff 
• no QA staff, no test automation 
• limited operations staff 
• limited budget 
• and in the end...acquisition by a competitor 

...most any excuse you can come up with, we lived. The pressure was intense, a number of folks didn’t make it. Through this climate, with much trial and error, but ultimately continual improvement, we grew to be stronger. A better organization, better team, better people.

Earlier we spoke of reasons why there might not be trust...here’s an example of the friction we ran into:

• There is no time, maybe you can do it after the next feature. 
• That will be too expensive. 
• Why isn’t that out yet? 
• You can’t deploy during business hours! 
• This is high priority. 
• Things are down again, why does this keep happening!? 
• Oh, and this is also high priority. 

What I came to learn was that we could help our organization learn how to move past this, but it would require us to step out of our comfort zone. It would require everything of what we were, and then ten times more - I personally found it to be the biggest challenge of my career.

So there we were there, in a chaotic and often frustrating situation, with the sense that things should be better. We had to personally start fresh (move on from the past), learn from mistakes, and celebrate success.

A number of folks started the change process - started at the beginning, so to speak - and the following is just my individual perspective on the changes - things done and seen in my sphere of influence.

All of this began when we realized that the reason why we had so much friction was due to lack of trust - our desires for change were falling on deaf ears. “We can’t trust them to do the basics, and now they want more time??” The sales and support teams believed we were out to get them - the worst kind of cowboys, perhaps even incompetent - and so were pushing back in the name of revenue and the customer. It was a case of ‘us versus them’, and since I’d been reading all of those authors you saw in the first slide, I started at the beginning - I began speaking with them. Talk transformed into true partner conversations with a direct and simple purpose - address the core issues, the huge problems, their greatest pains, and get us back to a level playing field.

Once we began having these conversations, some amazing insights came out of them (and by amazing, I mean ‘almost comical’), chiefly just how easily addressed the ‘huge problems’ were.

Practically speaking, the talks themselves were very informal and simple - either taking a short slot in a team meeting, or speaking with a few representatives directly, but with each one I stated my intent.

• I am here as a representative of engineering, my intent is to better understand what you need so as to help you succeed, help SIPlay succeed 
• Tell me about your major pain points, consistent problems. What’s hindering you? 
• What would make your daily lives better, ease your pain? How can we measure this? 

From there on, it would be 10-15m discussion ending with “yup, this is a problem” or “ahh, never saw it that way before”, and maybe it was out of scope to fix, but usually it would be something simple and readily addressed with process or tools.

In our case, we had two primary asks from both sales and support:

1. We need to know when something is in production. 
2. When you guys deploy, it takes the system down 

Resolving the “deploy takes the app down” pain was embarrassingly trivial. Here’s how the conversation went:

• Ok, our number one issue is that deploys are blowing up demos! It looks awful in front of customers, and we feel dumb when it happens. 
• That does sound unpleasant. When do you need this to not happen, what kind of time window? 
• I dunno, 9:00am to 10:00pm EST? Actually, maybe even 
• Well, how about we cease all non-critical deploys between 9:30am and 9:00pm EST? 
• Sure, that’s fair, let’s try it. 

Sales and support immediately felt like we cared, and said as much: “thanks for coming and talking to us, this is great!” We even got some shoutouts in the weekly all-call meetings.

The engineering team had meanwhile been transitioning into a new process they called “Dev QB”, and the immediate result these talks had was interesting. The QB was responsible for doing the daily releases, and the restriction was workable, but left us pretty short of time if anything went sideways. Shortly after the release time-restriction was put in place, we had an engineering team meeting with the topic “make release transparent”.

Work began immediately, and after several weeks of work, ended with the ‘canary’ process - basically a process that gave us the ability to deploy without the system going down during demos. We found it so beneficial that we rolled it out to all of our .NET apps. If we have time I’ll show some more detail afterward...the gist is that it made us able to release code whenever we wanted - no demo interruptions.

Other lessons continued being learned on the dev side - they began to hold their ground around issues of quality, building for the future, and ‘are we building the right thing?’.

•      After about a year or so of SIP being a thing, 
•           the engineering team was learning to say no. 

Not everything can be built or fixed at once, and some things should not be built at all. They ran with the newfound power and it made a huge difference - one such result was making code changes to allow for canary deploys to even work at all, and holding to backwards-compatible development practices.

Once the canary work was ready to be rolled out, I had another round of meetings with our sales and support partners where I explained our progress on their deploy woes, and when it would be going live. Following through! Everyone was pleased and the talks shifted to the issue of ‘when is this thing out’. I resolved to make some progress on it before our next meeting. Note - maintaining trust with integrity, transparency.

Perhaps unsurprisingly, we again found a simple way of resolving this and demo-ed it to our sales and support partners, who then called the issue solved - we rolled it out. Conversations from there on out grew to be much less ‘ahhh we have pain!’ and much more ‘you guys are doing a great job, keep it up!’.

A further trust-building mechanism was introduced with semi-annual team health checks (Spotify’s idea, not mine). These were ‘safe space’ 2-3hr-long meetings I facilitated where dev teams were encouraged to consider how they felt things were going at a higher level - mission, learning, code health, fun, process - 11 topics in total. The results encouraged them to continue the self-improvement track they were on and generated really good conversations that otherwise would not have happened. They brought teammates closer together, and gave us some insight into ‘general team health’ across the board. It was especially helpful for the ‘fully remote’ teams - we had an awesome mobile dev team entirely based in Argentina - they greatly appreciated an intentional space where they could talk about those kinds of things.

As mentioned, engineering was looking at other issues with our partners - the dev QB was intended to:

• Provide a means of ‘dev on-call’ without actually calling it that 
• Address support’s need for better/faster developer input/feedback 
• Spread domain knowledge - the engineer with specific domain knowledge always gets specific domain questions, causing bottlenecks 

Additionally, after a raucous blow-up (it was quite something) between support and engineering, a twice-monthly meeting called ‘voice of the customer’ was instituted where eng/product would host a q&a/open-mic session for anyone in the company to show up with ideas/requests/priority-changes. This completely smoothed things over, and is still being done today - it took over from where my initial meetings began.

So now that it was established, this iterative process of ‘meet, understand, deliver’ would continue until we were acquired.

Lessons learned

That was almost a year ago now, and though the acquisition was rough, our lessons and team growth survived, as did our desire to see things become better. So before coming, I polled the dev-qa team, and they wanted to share their big lessons, so here is a “this is what we brought out of our time in SIPlay” list...

• Feature toggles were a core enabler (separating release from deploy - you can deploy with sections of code ‘turned off’) 
• Strangler pattern is doing development on HARD mode and even more so in an environment of multiple acquisitions - normalizing data models from disparate places that have evolved from multiple monoliths. E.g. “what is a team” 
• (Strangler pattern is, essentially, cease developing your monolith, and instead seek to surround it with mini/micro services and abstraction layers) 
• Put your new tech endeavours into stuff outside of your monolith - way easier to manage and learn, limiting the blast radius, so to speak 
• As part of saying no - during a planning or ideation session, say “What would happen if we just didn’t do this?” - it helps everyone understand risk, priority, necessity 

From my perspective, my big lessons were:

• Building trust through partnership is a powerful tool for change, start by just talking to people while trying to understand 
• Our CTO really pushed me to grow, be uncomfortable, reach - it was incredibly hard, but fruitful 
• Consistently taking the SRE approach to pager alerts is a very powerful technique for reducing pager fatigue (SRE approach meaning: the only reason you should get paged is because immediate human hands are required - if it can be dealt with tomorrow, turn off the pager for that alert) 
• The canary process really opened my eyes to taking a results-based focus - seeing crystal clear before/after data proving my change made a difference was super cool 
• How beneficial the team health checks were - almost like group therapy sessions, everyone ate them up, and great things grew out of those simple meetings 

Going forward...

• The SIPlay eng team is helping drive ‘how we want to do dev/qa’ at SportsEngine, and that kinda speaks for itself - it’s really cool working with them. 
• Our canary process has become “how we do Windows” for all of SportsEngine’s Windows-based products. 
• The Team Health Checks continue to yield fruit, now being done for all teams (from 2 teams, now doing 8 teams), again with a lot of positive feedback. 
• I’m also getting to have a lot of fun with stuff like ‘building a learning culture’ and OKRs, so, the future looks good!

Wrapping this all up…

• The most challenging, but most important aspect of an organization is the people 
• The backbone of what helps us work at our best together is trust 
• Sustainable trust requires integrity - do what you say you will do, follow through, repeat 
• This change begins with you - stretch yourself, be uncomfortable, strive 

Jim Collins says, “It’s just as much effort to be great as to be good, so why not try?”. My experience at SIPlay and SportsEngine gave me a taste of this, we truly became a great team, great organization, and now I know no other way.

Folks, I sincerely exhort, encourage, challenge every single one you to walk this path and see where it takes you. When it comes to changing the world around us, the only thing that matters is whether or not you are willing to take the first step.

•      Thank you.

\o/

We are always looking for good people, so if any of this sounds appealing, chat with me after! We are cool with work-from-home and have a fun Oshawa office where darts, cornhole, and a giant connect-four board are a thing. Or you can move to the company headquarters in Minneapolis, which, while pretty amazing as far as offices go, is in Minneapolis.

If anyone has questions tonight or going forward, I am at your service.