Pressure To Produce

When programmers complain about the "pressure to produce more," I usually describe this as a positive thing. Programmers, managers, software companies, and consumers all want more and better software. It is the basic axiom of our field.

When I say "if I could find a way to produce 10 times as much quality software in a day, I would do it in a heartbeat," all the programmers' heads nod in violent agreement.

Most of us came to this field because we really like writing programs, solving problems, and making things work.  Learning algorithms and data structures was a small part of the answer, as was learning programming languages and idioms. But the underlying drive is still to make things that work using logic and flow and structure and all the IQ we can muster. We build because we love to build.

There is more than productivity to consider in software, such as building the right thing (product management), and building things well (craftsmanship), but productivity matters to everyone in this field.

This is the healthy pressure to produce.

There is an unhealthy pressure to produce as well. It comes from people making too many large promises. Again, they were doing the best they knew how to do, and there are usually significant opportunities for revenue or positioning.

When those who make promises aren't informed by a feedback loop that lets them know what is possible and how much reserve capacity the teams have (you do have slack, don't you?) then no rate of production will be sufficient. Too much will never be enough.

Worse, if those who make promises are in authority over those that have to deliver, and are not responsible to those who have to deliver (a common large corporation problem) then the problem is always seen as a failure of the development team to deliver.

Either way, the most positive result possible is to revise the system of production to its maximum productivity.  This involves having a more evolved theory of productivity, greater information to support decision-making, better teamwork, and more explicit management of code quality. It's all hard stuff that takes time.

The least positive way is to abandon all discipline and intelligent approaches and just write bad code as quickly as possible (usually with cut-paste-edit cycles) so that the resulting products will destroy your ability to move forward in the near future. The appeal of this approach is that you can begin it immediately (and may already have).

Either way, once you see the pressure to deliver you have to respond. You can refuse to change, take the low road, or take the high road. Only one of these is really interesting and has any promise for a brighter future but your team must make a choice.

Wallas' Four Stages of Creative Thought

If we want to see a greater theory of productivity, we have to recognize first that software development is thinking.  Once we get there, we need to understand how we can think better. One aid is to have more information so that we know what to do.

In the 1926 book The Art of Thought, Graham Wallas explained that having an idea (a creative solution) requires four distinct phases or steps:

1. Preparation (gathering of theory and data)
2. Incubation (letting the idea "cook" by doing something unrelated)
3. Enlightenment (the emerging of an idea, or "connection")
4. Verification (determining the validity of the idea)

Wallas noted that stages 1 and 4 may take minutes, hours, days, years, or decades. We are primarily interested in those that work on the sub-week scale when programming, but the fact remains that we need to allow the brain to work by feeding it information, giving it some time to process, and then verifying the ideas as they come.

Wallas refers to the mental state of approaching enlightenment as "intimation", which is a very positive and exciting mental state. You know this one: "Oh! Wait! I think I know the answer! It's....." accompanied by bouncing around the room in positive excitement.

Wallas' process is easily recognized by people who work in ideas (software, art, communication, etc).  Many of us have worked hard on one idea for hours or even days only to have it elude us as long as we concentrated on it. We finally surrender, only to have the answer come to us on waking from sleep (sometimes in the middle of the night) or else in the shower, while mowing the grass or exercising, while visiting with family, or while working on something entirely unrelated.

The second step (incubation) is one of the reasons why programming shops need to have ping-pong tables, foosball tables, video games, or a walking course. Without incubation it is hard to reach enlightenment.

Frustratingly, of the four steps, only the final one (verification) looks like "working."  An exceedingly naive manager would make sure that programmers are always typing, not "wasting" their time by reading, researching, or getting "distracted" by games or activities. A foolish manager would make sure developers are spending all their time and overtime concentrating on a single task, thus preventing the crucial incubation stage from occurring.

In a highly collaborative environment, knowledge is shared (preparation) and alternative solutions examined (verification) much more quickly than in solo work. We don't know of any situation where collaboration aids in enlightenment, nor can we say much about incubation other than the usefulness of having someone take a walk with us or play a quick game to distract us from a problem for a few minutes.

If we are wise, we should respect and not disrupt Wallas' four stages of thought.

Invisibility of Process, Visibility of Results

There are some special challenges with dealing with productivity of knowledge workers. Most of them have to do with the invisibility of the work and the difficulty in managing invisible work.

Programmers and testers don't assemble machinery or bend paperclips or mold parts from molten goo. They don't stack boxes or bricks, or swing hammers. The work they do has no physical manifestation, which makes it both hard to observe and hard to understand.

I don't blame managers in the 70s and 80s who counted lines of code. It was one of the few visible manifestations of the work programmers do. It was entirely misguided of course, and several of us have experienced net-negative lines of code in consecutive weeks of work (I've even had awkward and unpleasant meetings with managers for "messing up the metrics," ending in an admonition to stop it).

Other attempts to make the work visible count data fields on screens and in databases and on reports. This is a bit better, because it deals with the visible aspects of the system, not the bulk of the implementation. My one experience with function points was on a multi-year embedded control system which, based purely on data elements and screens, should have taken a few weeks to produce -- the function point measuring system at that time did not count decisions, constraints, or the number of operating-system workarounds involved in getting performance out of the system.

Management

Let's not overlook the other knowledge workers we see on a daily basis.  Management is also largely invisible and depends so heavily on intangibles that it's often hard to separate out the qualities that make managers effective compared to the ones that drive teams to create messy piles of junk.

Our managers much work in ideas, processes, directives, initiatives, relationships, impressions, signaling and messaging, systems, and concepts. 

Management has a lot of the same problem as programmers. It's hard to see the full range of effects that decisions might have.

Even a sigh, a snort, or an ill-timed change of body posture can have unfortunate unanticipated consequences for a manager. Consider the Moz story "He's not going to get very far" as an example of how developers interpret small clues as major directives.

Programmers and managers are in the same kind of pickle.

Our work is intellectual and invisible on both sides of the gap, and we rely on a relationship informative feedback cycles to improve our performance.

If "stuff that works" is our goal, then we need to return to the ideas that make it possible to build stuff that really works and really solves problems for people in the real world.

Metrics

So if software is not best measured in lines of code or data elements, how can we understand the software process better? How can we create a better theory of productivity for a largely invisible system?


The gold standard of XP teams is "running, tested features (RTF)."  It considers the value-producing part of the system (features), rather than the cost-inducing part (code). Even though features are not of uniform size and scale, we know that more value-producing parts is better than fewer.

Sadly, a lot of shops are still working on a system of pre-made promises that stretch well into the future. If we are producing a lot of running tested features but we are not on track to make all the promises that have been made by other people, then we start to deal with the pressure to produce in a less healthy way.

The ratio of things that the company says it will do compared to the things it can actually do is a crucial point of reputation. Are we dealing with under-delivery or over-promise? What is the best result we can expect when the say:do ratio of the company is out of proportion?

The wrong answer is to crank up more metrics to show how busy we are. Sadly, the common "answer" in many companies is to do exactly this, and to fall back to the naive theory of productivity. Why? Because it's the best we know how to do in the circumstances.

The Best Job They Know How To Do

Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.

-- "The Retrospective Prime Directive" by Norm Kerth

Software, as Dr. Ralph Johnson informed us, is distilled experience.

Writing software, Ray Scheufler reminds us, is a process of making decisions.

To make decisions, we have to understand the system we're working in, and the consequences of our decisions. That means that most of the work of a programmer is learning, and very little of it actually involves typing.

Understanding any existing body of software is involves understanding

• the domain, 
• the user being served, 
• the specific problem being solved, 
• the solution chosen, 
• the techniques of safe software development, 
• the organization producing the project, and 
• the technology (language, operating system, network architecture, etc) in which the program is written. 

When I mention the organization producing the product, I'm implying that it is different to work on a project with your three closest friends than it is to work with 300 other programmers divided into 30 teams with three management groups (each with its own budget and agenda). The more vast and complex the organization is, the more simple and clear the code must be in order to avoid the errors that arise from misunderstanding of the code. People moving from small teams to large companies are often surprised at the difference context makes.

Business people appreciate the need for domain and technical knowledge. Programmers used to be expected to operate on nothing more than a specific design and their raw technical skills; these days they are routinely given more information and context.

It's hard to measure understanding, but if we increase the knowledge and understanding in a team, we have a mechanism for increasing the output in terms of running, tested features.

Note that managers are in the same pickle. They have to make decisions based on the information that is available to them. It would by hypocritical to expect managers to make decisions without feedback and with no opportunity to vet ideas, and then expect them to respect developers needs for more information.

If we are all part of a system of development, then our information value to those around us is a major asset. It's long past the time when it was enough for people to separate and do their own work quietly in the corner. There has to be a system of shared learning and communication if we are to improve as a team, a company, or a field.

"It is not enough to do your best; you must know what to do, and then do your best."

-- W.E.Deming

Agile Documentation

Agile is not against documentation. It is merely lean, in that we don't want to maintain piles of documentation that don't actually help us product value for customers.

We maintain the least non-test, non-code documentation we can afford.

To be minimalist, we recognize that a conversation is better than a whiteboard, a whiteboard a poster is better than a white paper, a white paper is better than a tome -- provided it's enough for us to be able to produce good, working code and collaborate.


Here are all the rules I know about agile documentation:

1. If the document is contractually required, of course we do it.
2. If the need is immediate and significant we create a document (UncleBob's law). In other words, we don't build documents in case someone needs them in the future, and we don't draw a document when a conversation or whiteboard will do. 

There might be other rules. To date, I don't know them.

Hoarding Knowledge: sharing enhances productivity

Queuing

 If only one person on the team understands the database, then any work done by other programmers that affects the design of the database must wait up for the one competent individual.  Sometimes that queuing becomes significant and impedes the group. If the expert is not available, the work must wait, or else be done in an inexpert way.

Loss

Experts can go away.

Teams euphemistically refer to this phenomenon as the "bus number" of the team -- the number of people who, if they were hit and killed by a bus, would doom the entire project to failure.  A "bus number" of one is an unacceptable risk. A bus number of 10 represents a well-mitigated loss-of-knowledge risk.

Thankfully, few developers are hit by buses in the street. Instead, experts tend to be hired away by competitors or companies working in entirely unrelated industries. When this happens, teams do their best to muddle along, sometimes making poor decisions along the way and damaging the performance and/or accuracy of their software.

Ransom for Experts

In pathological cases, this reliance on a few experts allows the silo-ed expert to demand a very high wage and a high level of autonomy at the threat of leaving the company. The alternative would leave the company with without his specific expertise. This strategy is aspiration of some programmers we find today, who avoid teamwork for fear of losing their advantage over their employer.

Retaining your best experts makes sense, and they should be protected, but when it degrades to a kind of technical blackmail, companies are advised to mitigate their risk.

Efficiency of Scale

Companies have a tendency to assign work to the single worker who can complete it fastest because of his or her knowledge in a domain. This reinforces the bus number phenomenon and increases the risk of knowledge loss.

It also limits the ability of the rest of the team to acquire and develop necessary skills.

There is no efficiency of scale in software development. It is thinking work, not labor. It is produced through decisions, not machinery.  Batching up work does not make it more efficient, but less so. Specialization creates queuing and a reliance on individual effort instead of creating flow and accountability at the team level.

There is a dichotomy here. One one hand, a person who knows the domain and the technology best will be able to produce a given piece of work faster.  On the other hand, batching and queuing is waste (inventory and wait, in case you're keeping score) that makes the team slower and less predictable.

So now what?

The work system, in order to be efficient and productive, must be based on the idea of gathering and spreading information. It is not necessary that it preserves and exalts specialization among the members.

There will always be experts, and expertise is a real advantage. You should have the most expert developers you can possibly afford. You should give them rewarding and interesting (ie "hard") work and the ability to complete it well.

However, you don't want that expertise to be held exclusively in the head of one person.

Use techniques such as pair programming to spread knowledge, so that more "workaday" decisions can be competently made by many of the individuals on a team.

Having complimentary expertise and good knowledge transfer makes for better throughput for the team, even if some work must still be handled by experts.

Increasing Effort is Unsafe in Too Many Ways

The part of the effort-over-time we know from studies is that illness and fatality are clearly influenced by effort over time.

We find that overtime tends to reduce true productivity, though it tends to increase raw metrics of output. According to the Scheduled Overtime and Labor Productivity: Quantitative Analysis,  the most commonly reported factor is worker fatigue, both mental and physical.

Ron Jeffries also noted in his article on sustainable pace that:

A common effect of putting teams under pressure is that they will reduce their concentration on quality and focus instead on “just banging out code”. They’ll hunker down, stop helping each other so much, reduce testing, reduce refactoring, and generally revert to just coding. The impact of this is completely predictable: defect injection goes way up, code quality goes way down, progress measured in terms of net working features drops substantially.

From my own article, An Agile Pace:

Constant overtime is harmful. According to author and famed software developer Robert Martin, “After the first day or two, every hour of overtime requires another hour of straight time to clean up after.” This is what Evan Robinson's IGDA paper told us six years ago (and Yourdon’s Death March well before that), but the naive appeal of overtime continues unabated. 

This retired Despair.com poster is sadly right.

The problem isn't with effort, but with knowing what to do better. One of my clients told me "if you don't know what else is possible, you tend to keep on working the way you know."

It might help for people in the software field to become aware of the Yerkes-Dodson Law, which shows that some stress/arousal/pressure/need is necessary to reach peak performance, but that excessive stress degrades performance quite sharply.

Yerkes-Dodson Law

We need to get past acting like productivity is a linear function of effort and move on to something that we can work with.

Are you up for an experiment?

We've noticed that at some level our raw effort over time will degrade productivity, but how great would it be to have the data to identify your "sweet spot?"

I find that I can't function on continuous 10 hour days, and that a few weeks of longer hours or more travel will leave me less sharp and less decisive for several days. I start to get that "zombie expression" or "glaze" when looking at difficult code or tests. I can feel that I'm running at 80%, 70%, or even 60% productivity. I'm not keen on admitting that. But can I track it for myself?

Some people claim that they're more productive with a 50 hour week than 35 or 40, but those observations tend to be based on how they feel about their productivity and not on actual measures, even subjective ones.


Measuring your time-on-task and your feeling of productivity as well as actual output for a week would create a standard against which you could experiment.  I'm starting Monday. I will try to make it visible and transparent.


Collecting data takes the issue past the argument that "I don't like it" or "it shows dedication" to something that relates to actually getting work done.


Would you publicly track your productivity?

Misunderstanding Quality

Dr. Tsuda is saying that Western industry is satisfied to improve quality to a level where visible figures may shed doubt about the economic benefit of further improvement. As someone enquired, “How low may we go in quality without losing customers?” This question packs a mountain of misunderstanding into a few choice words. It is typical of management’s misunderstanding in America. In contrast, the Japanese go right ahead and improve the process without regard to figures. They thus improve productivity, decrease costs, and capture the market.

Deming, W. Edwards (2011-11-09). Out of the Crisis (p. 2). MIT Press. Kindle Edition.