Filter Signal from Noise - Cause and Effect

Filtering signal from noise is a constant requirement for all. Failure to do so can lead to significant learning at similar costs :-). One of the most common errors in filtering is incorrect identification of cause and effect. Often we assign causal relationships where none exist or entirely different relationship exists. This was illustrated beautifully by Levitt and Dubner in Freakonomics.

Concepts of filtering signal from noise and assigning right causal relationships are equally relevant in software development world, especially as you rise up the ladder - stakes get higher, source of information increase, and messages are more conflicting.

The most common trap we need to be aware of while assigning causal relationships is that we would often assign relationships that suit our needs and feelings at that point of time. Ask a manager why release has been delayed and you would hear reasons ranging from understaffed team to low productivity or limited capability of existing team. Interestingly if you pose the same question to the team you would hear reasons ranging from unclear requirements and poor estimations to poor management and low motivation levels - typical huh?

Same situation being analyzed by (hopefully) rational and smart people and yet such radically different reasons. If you noticed, reasons cited suited the person assigning the cause - point a finger on something/someone else and keep myself and my work above suspicion - convenient, isn't it? If this sounds familiar, BEWARE, this is a trap. You might unconsciously being denying yourself a chance at improving your delivery and at the same time building great relationships with people around you. If your reasoning is built to suit your convenience or to help you choose the path of least resistance, it should ring an alarm in your mind - this may not be true. Better to find out truth sooner than later when damage has been done.

Ummm ... but how to find out truth? Ask yourself a simple question - How do I know that X caused Y? If answer to that question does not contain concrete facts and data, you are falling in the trap. More often than not, you would discover that the reasoning is composed entirely of opinions and assumptions. To illustrate the point, when manager was asked why release was delayed he knowingly or unknowingly reasoned: I did a good job with planning and estimation, time lines were fair, an average team should have been able to deliver so this team must be limited in its capabilities. Notice - all assumptions and opinions. Likewise when a team member responded, she might have reasoned : I worked hard, put in extra hours, it couldn't have been done better, time lines were so off from the real effort and there were so many unplanned changes. Had we had a better manager, we would have delivered - see the common thread?

Another prominent fact in managing software development is uncertainty - too many things change along the way and we often have to make some choices before we have all the information for making the best choice. There is nothing wrong with making a best choice based on available information. If you were to wait until you had all the information, chances are your decision would have either been made for you or would be inconsequential. However, a side effect is that it attaches us to the decisions we made. Problem with this phenomenon is that now we are likely to come up with justifications to stick to the choices rather than consider the new facts and correct our course. This is caused by cognitive dissonance.

In simple terms, once we have made a decision, we are likely to create reasons to justify the decision rather than basing a decision on good reasons. If you are not consciously checking yourself, this can lead to costly mistakes which were otherwise avoidable.

This phenomenon is more visible at higher levels of management
- channelized meager resource for developing iPhone app - but none of our customers use an iPhone ... but they would some day, it is the thing of tomorrow so lets keep adding features so that we are ready. It might never occur that thing of tomorrow can be done tomorrow.
- made a hiring mistake at senior level - you need to give them some time to show results ... ignore feedback from others, people always have issues with change. It might never occur that by the time free run is over, irreversible damage might have been done.

Fortunately, same question and introspection can save us from this trap as well. All reasoning is based in opinions and assumptions - they would use iPhone tomorrow, people have problem with change. There is no data and facts to support that this is indeed the case.

However, there is a catch - we all have a different perspective which is shaped by our past experiences. May be giving senior executive a free run worked for them in the past. May be customers moved to new gadgets as anticipated in the past organization. Such experiences are likely to cloud the judgment and sway our decisions. How to protect ourselves from such errors? First and foremost, we need to recognize and acknowledge that this is a hunch or experience not data or facts. Whenever we are going with the experience, we need to evaluate each case on its merits. When experience was positive, what were the things that contributed to success? We need to distill the data from the successful experience and then figure out if we can find similarities in our current situation.

E.g. Giving a senior exec a free run worked last time even though people complained. Were the complaints composed of opinions or facts? Chances are for bad hires complains would be composed of facts and data while for good hires they would be made of opinions.

Myth about multitasking

True masterpiece from Scott Adams that shows one of the ills plaguing all levels of contemporary software development. Numerous researches have shown that human brain is not effective at multitasking. In fact multitasking slows us down since multitasking makes us deal with irrelevant information as well.

Yet, all of us have been seduced by the lure of multitasking - overestimating our capabilities and trying to do two(pun intended) much at the same time. Responding to emails while attending meetings, preparing plans while sitting through architectural discussions, watching tv while taking a call from home, we have all tried our hands at it in many forms.

Reflecting on my experiences, I do not remember much of meetings/conversations during which I checked emails and my email responses at that time weren't great either. Unless we stop thinking of the "other things" we are bound to miss the finer details of our current task. That would also explain why reading in the bathroom is not multitasking unless you want to concentrate on finer details of using the bathroom ;-)


So, is multitasking overrated? How does one manage high pressure day(and night) job with new information flowing in constantly?

The answer to those question lies in how you adopt multitasking. Effective human multitasking is not very different from computer multitasking. A processor gives "appearance" of handling multiple tasks at once. However, at any given moment only one task is being executed - not a single cycle is spared for another task. Only a higher priority task can preempt the current task. If current task is interrupted (waiting for input/resource etc) then next higher priority task is picked up for execution.

In a very similar fashion, instead of attempting multiple tasks at the same time, we should prioritize and pick up highest priority task first and then give it our complete focus. Periodically, new information reaches us (new tasks, events) which should be used to adjust our priority and we should continue with execution of highest priority task. Undivided attention to a task is the only way of getting a task done with good quality. This is especially true of high priority tasks where there is little room for iterations for refinement and delays are just as bad as poor quality.

In short, only way of multitasking effectively is to deal with multiple tasks at the same time but not in the same moment.

As a manager, it is extremely important to understand this reality. It is all too easy to put a lot of tasks on a team member's plate. This might render them ineffective or affect the quality of their delivery. If a team member reports little progress on multiple tasks or cites one task as reason for lag in another, it is a strong indication that you need to sit with them and help them prioritize and execute. It might take a few iterations/weeks before team gets into the groove of prioritizing and then executing with focus but it is worth the effort, after all this is indeed manager's highest priority task - building a highly productive team.

Another one: