“A common example is process as proxy. Good process serves you so you > can serve customers. But if you're not watchful, the process can > become the thing. This can happen very easily in large organizations. > The process becomes the proxy for the result you want. You stop > looking at outcomes and just make sure you're doing the process right. > Gulp.” (Jeff Bezos)

This is a quote from a Jeff Bezos letter to the board, talking about internal metrics and improvements. Sometimes when you add a process to a problem, the process can become the focus and not the outcome.

In PM land we use tools and techniques like burn down charts, sprints, and spikes. You can get obsessed over getting these things right, and fail to ship effective, quality software. The special language used in and around these processes adds to the problem too, as the language ends up feeling like an accomplishment in itself. Too much focus on the pomp and circumstance of a process takes away from actually building great software.

e.g., Let's put a pin in this ...

The real value isn't in the process or lingo, it's in doing things better and making better software within profitable budgets. It's something that's been rolling around my head for a while, and Bezos describes it nicely.

I was talking about basic project management principles with a friend the other day, when I came across this Bezos quote. It sparked a discussion about software development processes, including hitting a few topics that I think are especially distracting in software development.

Do you use sprints, burn down charts (etc.) in software project management?

Yes, these are fairly common software management tools now, and I use some aspects of them. Now of course we don't say “spike” ever, or “tent pole”, but we do plan sprints of flexible durations, and we're always aiming for minimum viable steps and frequent releases.

I think it's a common mistake to set fixed durations with sprints, as it can shift the focus to succeeding at the sprint versus shipping something viable (i.e., both useful in the short and medium term). It can really deflate the team if sprint durations make no sense, as developers pick up on goals that aren't about useful or quality software.

Fixed duration sprints can also force poor decisions, as your primary goal is time and not the other factors important to software. If you relax your use of sprints a bit, you can pick sprint times that fit the underlying problem better. The real goal with sprints is to focus on scoping the project, and improving the overall tempo of development. Sprints shouldn't be truncating quality for minimum viability, nor should they be pushing teams to burnout faster. Really, all we want from sprints is a framework to keep us focused on the right features and a strong speed of development.

Does 'sprint' equate to 'crunch time'?

No, it shouldn't.

This is a great question. My friend is coming from another domain, specifically small-scale custom manufacturing. He's not fully familiar with software development terminology (though he's vaguely familiar with how software is generally developed). Conversations like this are a great reminder that the language we've developed in this industry is insular, even though it is based on what we think are clear metaphors. When you think about if from an outsider's perspective, you can see how arbitrary the language is, and how the imagery doesn't fit perfectly.

This is especially true with terms like sprint. You can't literally sprint all of the time, as it suggests expending the maximum amount of energy over short distances. Software projects are anything but short, and the idea of sprinting through the entire thing is absurd. The PM concept of sprints doesn't match the real world use of the term very well.

A sprint is like a focused mini project. To some degree, it's like having a personal trainer guide your workouts and push you a bit. The idea is to use the mechanism to help you push to a clear goal.

More traditional PM structures projects as very long, boring wanderings through the desert. Waterfall projects, for example, don't produce workable software until much later in the process.

We might say, “we're authorizing blanket overtime” which tells the > guys we need as much effort as we can get right now. There are some > tasks on the shop floor which simply can't be done faster, so we > would add more hours. Or worst case a second shift. There's huge > inefficiency in adding a second team.

That is a common manufacturing practice for increasing bandwidth, but sprints aren't only about increasing output (though the term strongly suggests that). A sprint is more like setting up a smaller shop, with a special, focused project.

Ahhh. We would job things out to other manufacturers ...

That's not quite it either, though I do find using sprints for special ops is especially useful, as you push the focus of the tool to problems that need special focus.

The concept of project sprints was really to make things feel more focused and tactical. It was a backlash on the traditional waterfall planning (i.e., requirements, design, implementation, and testing).

But after using sprints a bunch, they can become empty words for traditional project management things. The trick is really to keep sprint projects focused on smaller, but useful deliveries.

Sprints really just become project phases like those in the waterfall model, except that you've organized the deliveries differently. You try to partition design into things you can deliver on shorter intervals, rather than aiming to deliver one giant thing at the end. There is a huge advantage to delivering these small and useful pieces early on, as you learn a lot about what works (and doesn't) well before the end of it all.

I was saying earlier that the concept of sprinting suggests running fast, which if you think about it is so expensive in terms of long-term motivation and energy. It's an odd side-effect of the name itself, as running fast isn't part of the definition of a software sprint, yet the term implies it so strongly.

There's this pattern of making special languages for our formal processes, based on the ways we approach building software. It's difficult to find ways to describe processes that don't eventually become ideology or warped in some way. The simple act of naming the parts can change how we think of them, if we lose focus on the real goals.

I can't think of an analogue in manufacturing ...

It eventually boils down to the same set of problems as with projects in any industry. I've seen special-ops style projects in manufacturing before:

One re-manufacturing mill I worked with started a finger-joint mill. They took their best staff and had them design and built the new plant in stages, apart from their traditional management techniques. They split the work into smaller chunks, and had something up and running early.

Ahh, I see!

They treated it like a special thing, which helped them escape their traditional planning techniques.

Of course sprints aren't really just special-ops anymore, but that's where the concept started from. We used to say “Let's push hard and get this out by next week”. We used this push mindset well before Agile methodologies were well-defined, and sprints were really an extension of that.

Yeah we did that too, but I guess we were so unfocused that it was > normal to work on random projects.

In the end it's just different words for the same things. The mindset of making projects smaller and focusing them on early and useful releases is the best part of sprints. The concept of isolating design to reduce scope is also great.

Really, sprints cut across the grain of the old waterfall style planning, repackaging a lot of the risk to earlier deliverables. It really only works if your sprints produce production quality deliverables though, which is why it's so important to avoid serving the process over the goal of building something incredible.

Sprints are a good example of allowing the process to become the focus, instead of obsessing over what you're making. This is what frustrates me the most about software development methodologies: when great concepts become ideology and the underlying intent is lost. You start spewing the bullshit terms without even thinking. They become bullshit when the term is the focus, rather than the outcome.

e.g., Let's spike this feature for the next sprint.

Or, Our sprints are only 1 week, so that story is too much for us right now ...

It's disingenuous to use these words all the time. It allows a lot of passive-aggressive behaviour too, as you can use the noble act of following the process as a way to beat down legitimate concerns about what customers or the business needs.

What I see happen is that smart people can control production and planning by using the process language in clever ways. Arguing things for the sake of process can be a sign of that problem, i.e., elevating the process above the intent.

The problem isn't in the process, however, it's just a reminder that it's easy to focus on the wrong goal. Sticking to your process is often a much clearer path than figuring out what your users really need, or how to design that thing in such a way that it's a viable business. And because the process can be easier to succeed at, it becomes an easy target for people on a career path.

As software leaders we just need to remind our people to apply the process and tools for the outcome. We want to build useful things for our fans and users, and we want these projects to serve the needs of the business. The processes and methodologies are just tools to this end, and not the end itself.

#SoftwareEngineering #Weblog

I review a lot of code these days. It's an incredible way to nudge a team of bright people toward greatness. It allows me to look at problems from the outside with a perspective of experience and distance from the low level design. The perspective is important too, and I see things that I often miss in my own code. You see, we're easily blinded when we're too close to the problems we're trying to solve.

A change in perspective is worth 80 IQ points. —– Alan Kay

Think of it like this: walk up to the front of a large building. You won't be able to see around it anymore: the roads leading away from it are obscured, and the metropolis surrounding it is dwarfed by its stature. Your perspective limits you.

Rule 0: no one has immunity

I require review for most code that comes from my teams, including my own. Github pull requests are a great tool for this, as you package up a change, relate it to a bunch of tracked issues, and open it up for peer review. It allows you and your team to comment, iterate, and accept or deny each patch. There are lots of other ways to do this, of course, but the mix of branching and discourse is brilliant.

Rule 1: reviewed code is always better code

You don't have to review all of the code produced by your teams, but it is incredible value for the effort. It's far easy to bump design and learning along in small steps, and far cheaper too. There are fewer surprises, fewer rewrites, and a much easier learning curve for everyone.

I can review code for a few teams of developers, given a mix of skills, in a few hours a day. That includes time to talk over issues with individuals and group whiteboard sessions. The code review leads to design review, and a solid understanding of where your teams stands on a daily basis. Getting the team productive gets you to shipping.

Rule 2: develop simple ways to sample code

Understanding every line of code is difficult, especially given the chaos that creeps into the daily life of a team lead. Even after taming the chaos you still need to find ways to focus on code in ways that finds the important holes. How do I do this? Simple heuristics, or in other words: rules for finding crap.

For example, when reviewing anything in Python, Ruby, Perl, or PHP, I look for programming conventions from other languages. Parsing strings using strpos? Is there a better way in the language? Is there a faster way? Is there a cleaner way? I call this: alien syndrome. It's easy to spot, and even though it's sometimes the best solution, it's more often a sign of a developer who has failed to really understand the language. A sign that they're limiting their thinking to what they know and not in the potential of the tool at hand.

Rule 3: write down your rules and philosophies

Write your rules and style guidelines down and share them with the team. Make them into manifestos or a top-$n list. Have fun with it.

Here's my 8-bit list for reviewing code:

1. Alien syndrome – the code uses a convention or mechanism from a different style of language. C-isms in Python? That's weird!
2. Exception litter – a function spreads its error cases throughout larger algorithms. Is the data unreliable? Failing early seems more humane than dragging it out.
3. In-and-out buggers – the code mixes generation with logic. Can the logic be split into something more easily tested? Can the generation be split into some sort of adapter or template?
4. Elbow-itis – the design or code has several elbows. Does the complexity of the code or design fit the complexity of the feature? Not!
5. Inconceivable – language or library functions used in odd ways, or where much better mechanisms are available with a bit of searching. Do the functions do what the developer thinks they mean to do? Various uses of regex are especially subject, as they represent incredibly dense logic.
6. Are we there yet? – algorithms or classes that drone on well past their welcome. What makes the code so lengthy? The rhythm of the code is much like prose: it should feel good, not boring.
7. How DRY I am – repeated code is stupid. I may be lazy, but testing something more than once is crazy talk. It's also expensive and prone to errors on several levels. Bad mojo!
8. Preflight sanity – code fails the basic mustard: neat, tidy, edible, digestible, and food-safe. Loose lips sink ships. Sloppy code is crap. Or, if code can't meet basic minimum limits, then it's likely flawed in other ways.

I add heuristics regularly, based on simple misconceptions, new techniques, or especially creative developers (often including myself). I focus on different types of problems with different developers, and with different types of designs.

Rule 4: review should use what people do best

Let the computers worry about syntax checking, enforcing persnickety grammar rules, and testing for large and complex things. Code review does not replace the compiler, the lint tools, the unit tests, the system tests, or the load tests. How could it?

I focus my time on things that I can use to teach better techniques, on patterns I can use to prevent entire classes of problems in the future. In the rare case where there are grievous basic issues, I call everyone to the bat cave.

Rule 5: to the bat cave!

Some problems deserve more than one-on-one time or group training. You know the type of problem I'm talking about: that API that's warty and gross, or those SQL queries that are covered in slop. Picking these things apart piece by piece would be painful and cruel. Didn't someone tell you not to pick at the scabby things? No?!

These loathsome problems require a graft and not just a simple nudge. Solve the problem properly, cleanly, and with some tact. No sense in making it painful or tedious. Write a standard or a guideline, and plan some real time to scour the grit out of the gunk. Fix it up right, and take the fix out of band to the mess it solves. No sense in dragging developers through the mud. Pick them up, set them straight, and show them how to solve the problem properly.

Rule 6: there is no rule 6

There are always problems that don't fit your experience. Roll with it and remember your team is awesome. Support them, look for the long game, and base your recommendations on solid first principles and fact. Keep it clean. Keep it fun. And remember: code review is a form of programming itself. You're encoding your team's success using a simple series of feedback loops.

#SoftwareEngineering #TheArtOf

It's not that these tools and techniques are bad in themselves, but our use of each should be fit into a well tuned approach to building software. An entire project delivery should be tidy, professional, and complete.

There are a few causes to the problem of organizational buildup. Our software is limited, our methods need improvement, and the unseen pieces of our project pile up and are left a mess once we ship. categories: rants

TODO lists, project management software, and GTD all have the same basic problem. They push you to produce things that aren't your end goal, to help you improve your focus, thinking, and (in the end) produce better quality software in less time. These artifacts clutter your archives; the very things that help you create more effectively get in the way. They're a form of noise to your project's signal.

It's not that these tools and techniques are bad, but our use of each should be fit into a well tuned approach to building software. An entire project delivery should be useful, professional, and complete.

There are a few causes to the problem of organizational buildup. Our software is limited, our methods need improvement, and the unseen pieces of our project pile up and are left a mess once we ship.

Everything is a file

Our tools treat each of our artifacts as equal. They're generic, as there are many approaches to managing projects, so very few assumptions are made about the importance of each piece of the process. Requirements, specifications, prototypes, estimates, and development plans are strongly interrelated, but few tools are able to capture this.

Even the application structure of project management and software development tools re-enforce the everything-is-equal by placing features in tabs, sections, and tree nodes similarly. While it's easy to organize product features into tabs and tree nodes, it keeps them at the same level of importance by default and we have to work to tame the mess.

A polished specification is much more important than the research and prototypes that fed into it. A completed user interface design is superior to the sketches and feedback that improved it. Our tools, however, rarely help us focus on these differences.

Rushed and unfinished

Our own methodology—our process of analysis, design, and development—contributes to the chaos of project bits. We don't like to fund (or spend time) on things that aren't part of the output, so prototypes, design documents, and specifications remain unfinished and unpolished.

We also produce, and subsequently discard, various artifacts as we're learning the process of building better software. These forays into improvement leave a wake of debris that is unlikely to be used again. This is a failing of team management and individual process: we just don't put enough emphasis on picking successful tools or polishing and curating the prerequisites.

Projects need teams that have a winning approach. We should not be exploring process on project time (side quests are better for this), and hence the exploration and failure of various processes should be insulated from active work. Why practice at game time?

Cooks who cannot clean

Our tools should serve us better. Our own methodology should avoid processes that fail us. In the end, though, we shouldn't leave the mess behind. Each member should take pride in what we deliver, including the parts unseen: the designs, specifications, drawings, models, and all the supporting pieces.

Our methods and techniques are not unfortunate side effects in building software. They are what's involved in building software. We sketch, we design, we carefully consider each piece. These supporting activities are not lesser to writing code, they make writing code better.

Less is more

There are many great reasons to use better tools, to understand the process of building software, and to curate the entire effort. We know that it improves communication, reduces bug rates, and reduced project costs. We know that it makes for better software. And yet we end projects late, incomplete, insufficient, and in a mess.

Let go of the tools that do not work. Ignore the processes that are obvious losers. Obsess over ways of documenting design that take less time, and produce something useful. Learn to write better, make your sketches count, and don't leave a mess at any time. The thinking that goes into your software is the software.

Know how to build software. Sketch, document, and plan. Be proud of the method of building software, and curate the in-between steps. Spend time between projects improving your process; finding ways to do less and better. And above all, curate your thinking and the things you record your thinking with. What you produce will be better for it.

#SoftwareEngineering #TheArtOf