Rambletron - Patterns for software development orgs?

Lessons I have learned so far; the naivete of youth allows me to write this down...the wisdom of hindsight allows future me to laugh.  :)

Building portfolio vs. delivering value

This is first because it's ultimately the most important, and hardest, part to get right.
Suspect that things like this are part of it:

• Business Model Generation/Value Proposition Design concepts being applied
• i.e. think hard about what you are doing and why
• Continuous improvement as a cultural centerpiece
• Senior leadership defines the big picture, explicitly calling out continuous improvement as a necessary fact of life is key
• Teaching all levels to apply BMG/VPD concepts to projects
• Really just the act of thinking about something vs. just doing it because you're sure it's the best idea

I cannot comment on how you actually solve/execute on this, but I feel very strongly that this is core to sustainable success & innovation.

Systems thinking

Books like 'Thinking in Systems' and 'The Fifth Discipline' really broadened my perspectives.  I am at best an amateur systems thinker, but have noticed the following:

• While difficult to apply to 'right here, right now' and 'future' states, a systems thinking perspective really highlights issues like 'shifting the burden', and at least allows you to prevent that state from continuing
• Understanding that a reinforcing feedback loop requires leverage at a specific point to change seems simple, but is actually really hard to figure out in real life
• Once you see the system, you cannot just walk away

For an example...

Defeating 'no'

The ability to say 'no' gives incredible power to change your situation.  What you gain by knowing how and when to say no:

• Forces conversation & communication:  Why not?  Maybe we should talk about this.
• Forces prioritization:  This is important, now!  Why?  What will not get done?  Are we ok with that?

I would posit that 'saying no' is a key lever in changing the feature/quality balance.

1. Oh boy, we need this feature-X ASAP!
2. Sorry, we understand business priority to mean that feature-Y is paramount.  Please speak with product (problem#1) and they can help you out.
3. <Menacing thing happens - do you persist?>

So once you say no - 'let your no be no'.  Trust that those in leadership above you can sort it out.

You would also note here that 'transparent corporate strategy' is required.  If the teams have no direction, it's harder to say no.

Continuous improvement, ad-hoc vs. way of life

If you are focused on delivering value to your customers, there will be no limit to your 'scope of tasks' (i.e. non-functional requirements are equally valued, because they bring value).  If you are focused on building a feature/product portfolio, non-functional requirements are immediately 'pains' that always take back seat.

Some additional detriments to the latter viewpoint is constant pain for developers, qa, ops, support, etc, because things that are important to customers (bugs, performance) require non-feature effort.  Avoiding those means you are always working around issues, which leads to accumulation of tech debt, and customers becoming used to a sub-par system.

The obvious counterpoint to this is 'fixing bugs makes us no money compared to new feature-X' - at which point the question should be asked:

• Why are we here?  Making money at the expense of customer experience, or delivering value to our customers?

So if fixing the bug delivers no value, leave it broken, but value should not be equated with profit.

Operations as a repair shop vs. hardware store

• I need help!  Could you do this?
• Ops takes over a task to be performed
• I need help doing this!  Could you enable me?
• Ops builds/buys/writes a tool to be consumed

This is textbook 'shifting the burden' and 'reinforcing negative feedback loop'.

• It's hard for me to do this, I'll just get someone else to do it.
• Oh, they were good with that last task, let's give them this other task.

In the first example 'could you do this?', a task is accepted so that the first party does not have to do it.  But why?

• Because doing that task would prevent them from reaching a goal?
• Why is this a problem?  Is the task not important?
• Because the task is undesirable?
• Why is the task perceived as undesirable?  Is the task not required to achieve other goals?

Some examples of that task:

• Monitoring (monitors, metrics, pattern matching, etc)
• Build/deploy
• Testing (automated & manual, performance, security, etc)
• Verification (metrics, manual or automated verification)
• Troubleshooting (digging through logs)

All of these should be approached:

1. We have a problem, let's discuss
2. We can't figure this problem out (why) because the log files are only visible on the server (why) because we don't have any way of getting them off or storing them somewhere (why) because we don't know how/don't have time (etc etc etc)
3. Ah, so you need ELK (or whatever), we'll provide you with a web interface to view/sort/analyze logs, plus provide you with simple tooling to onboard new applications into this system!

The problem is now gone, the dev/qa team is enabled, and tooling allows future problems to be easily dealt with.

You, of course, require an operations team that is capable of:

• Setting up a reliable system
• Writing code to create onboard tooling
• Taking the time to converse with other teams in partnership

Actioning on-call items

Enforce these rules for your on-call rotations:

1. Resolve the issue ASAP, but try to take time and gather information, or at least identify information sources.
2. Identify one of the following
1. What needs to be fixed to ensure this never happens again?
1. Can I fix it? -> ticket
2. If not, who should fix it? -> ticket
2. Should I be tuning alarm thresholds? -> ticket
3. Should I be removing this alarm entirely? -> ticket

If a ticket is not generated or no decision can be reached alone, it should be raised as a blocker at the next standup meeting.  If no decision can be reached, the product owner should be notified that the alarm will be shut off until action can be taken.

This practice will raise awareness across all teams of the ramifications from design/technical debt/ownership decisions.

Failure to comply will only make life miserable for yourself.

Test automation as a first-class citizen

This comes hand-in-hand with true 'product ownership', your continuous improvement strategy, and 'shifting the burden' patterns.  If you take the stance that it's not my problem, has no value, or 'we will get to it next week' - you will always deal with test automation as a thorn vs. an asset/efficiency multiplier, and this is a common reason why it's hard to adopt.

The case for test automation as a practice is a done deal (context is king), but in feature-focused teams it is always seen as waste/pointless.

Definition of done

The more I talk to people about this, the more I'm convinced it's not just a tool for difficult product/dev team relationships.  I think that the act of establishing an ideal, setting a bar, for what our products are made up of is important because it helps us clearly articulate what we lose when we decide 'push that up to the next sprint'.  It doesn't have to be hard and fast for everything, but it's another conversation point; builds team communication - so it's probably worth the time.

What is really important to us?  Are we cognizant of debt incurred?  To break this, are we certain that the 'priority' action is actually delivering more (timely) value?

Required sacrifices for a remote team environment

A short list based on my brief experience:

• Spending more time (which means allocating, which means not doing features) in cross-team activities.
• Ensuring that you at least have area meet-ups 2-3x a month; face-to-face communication is king
• Sending folk to remote offices periodically allows for ad-hoc conversations, personal growth, opportunities for team learning
• Video calls should be standard for any problem solving activity.  Extended chat discussions can often lead to misunderstandings.
• Any kind of emotional discussion must be held via video - very easy to have text misinterpreted
• "open video channels" have proven to offer many benefits - e.g. one person opens their G2M and a few people join to work on a problem, but the channel just sits open for an extended period of time (hours, even) - provides opportunities for random asides, personal chats while waiting, side comments, etc
• Team health check-like activities are opportunities for groups to gather together
• The act of writing things down is "inefficient", but allows for several benefits: personal aspect of forcing yourself to articulate thoughts, allows people to look at something at their own pace (not everyone thinks quickly on their feet) and comment appropriately
• A public/private channel approach for teams is necessary both for a safe environment but also somewhere to treat as 'home'

I think that ideally, you don't do remote teams.  Human nature is too easily swayed toward avoiding contact, and team learning works best when people are together.  Remote offices can work if you group teams appropriately.  

Never underestimate the value of sitting down to lunch together.

Summary

Well, that was fun.  See you in two years, future me.