Filter Signal from Noise - Cause and Effect

Filtering signal from noise is a constant requirement for all. Failure to do so can lead to significant learning at similar costs :-). One of the most common errors in filtering is incorrect identification of cause and effect. Often we assign causal relationships where none exist or entirely different relationship exists. This was illustrated beautifully by Levitt and Dubner in Freakonomics.

Concepts of filtering signal from noise and assigning right causal relationships are equally relevant in software development world, especially as you rise up the ladder - stakes get higher, source of information increase, and messages are more conflicting.

The most common trap we need to be aware of while assigning causal relationships is that we would often assign relationships that suit our needs and feelings at that point of time. Ask a manager why release has been delayed and you would hear reasons ranging from understaffed team to low productivity or limited capability of existing team. Interestingly if you pose the same question to the team you would hear reasons ranging from unclear requirements and poor estimations to poor management and low motivation levels - typical huh?

Same situation being analyzed by (hopefully) rational and smart people and yet such radically different reasons. If you noticed, reasons cited suited the person assigning the cause - point a finger on something/someone else and keep myself and my work above suspicion - convenient, isn't it? If this sounds familiar, BEWARE, this is a trap. You might unconsciously being denying yourself a chance at improving your delivery and at the same time building great relationships with people around you. If your reasoning is built to suit your convenience or to help you choose the path of least resistance, it should ring an alarm in your mind - this may not be true. Better to find out truth sooner than later when damage has been done.

Ummm ... but how to find out truth? Ask yourself a simple question - How do I know that X caused Y? If answer to that question does not contain concrete facts and data, you are falling in the trap. More often than not, you would discover that the reasoning is composed entirely of opinions and assumptions. To illustrate the point, when manager was asked why release was delayed he knowingly or unknowingly reasoned: I did a good job with planning and estimation, time lines were fair, an average team should have been able to deliver so this team must be limited in its capabilities. Notice - all assumptions and opinions. Likewise when a team member responded, she might have reasoned : I worked hard, put in extra hours, it couldn't have been done better, time lines were so off from the real effort and there were so many unplanned changes. Had we had a better manager, we would have delivered - see the common thread?

Another prominent fact in managing software development is uncertainty - too many things change along the way and we often have to make some choices before we have all the information for making the best choice. There is nothing wrong with making a best choice based on available information. If you were to wait until you had all the information, chances are your decision would have either been made for you or would be inconsequential. However, a side effect is that it attaches us to the decisions we made. Problem with this phenomenon is that now we are likely to come up with justifications to stick to the choices rather than consider the new facts and correct our course. This is caused by cognitive dissonance.

In simple terms, once we have made a decision, we are likely to create reasons to justify the decision rather than basing a decision on good reasons. If you are not consciously checking yourself, this can lead to costly mistakes which were otherwise avoidable.

This phenomenon is more visible at higher levels of management
- channelized meager resource for developing iPhone app - but none of our customers use an iPhone ... but they would some day, it is the thing of tomorrow so lets keep adding features so that we are ready. It might never occur that thing of tomorrow can be done tomorrow.
- made a hiring mistake at senior level - you need to give them some time to show results ... ignore feedback from others, people always have issues with change. It might never occur that by the time free run is over, irreversible damage might have been done.

Fortunately, same question and introspection can save us from this trap as well. All reasoning is based in opinions and assumptions - they would use iPhone tomorrow, people have problem with change. There is no data and facts to support that this is indeed the case.

However, there is a catch - we all have a different perspective which is shaped by our past experiences. May be giving senior executive a free run worked for them in the past. May be customers moved to new gadgets as anticipated in the past organization. Such experiences are likely to cloud the judgment and sway our decisions. How to protect ourselves from such errors? First and foremost, we need to recognize and acknowledge that this is a hunch or experience not data or facts. Whenever we are going with the experience, we need to evaluate each case on its merits. When experience was positive, what were the things that contributed to success? We need to distill the data from the successful experience and then figure out if we can find similarities in our current situation.

E.g. Giving a senior exec a free run worked last time even though people complained. Were the complaints composed of opinions or facts? Chances are for bad hires complains would be composed of facts and data while for good hires they would be made of opinions.

Where did Agile get it wrong?

Dont't get me wrong, I am not anti-agile. I love agile and a lot of good things that it brings to the table. However, it also has its share of shortcomings and problems. I am going to focus on one specific shortcoming for this post - effect of daily standup meetings on the team.

Focus of agile is to maintain a sustainable pace of work. For all agile practices "in the ideal world" it has come to mean a sustained constant pace of work "all the time". Humans, unlike computers, do not produce same number of processing cycles every day. There are days when one feels out of element and does not want to code and then there are days when one can accomplish far more than what would be expected of a daily average. Problem with agile is that what used to be hidden behind a weekly status update now comes to fore - non uniform productivity.

Even the most brilliant and passionate software developers would have off days when they are not in the groove. And these are the days when it is hardest to own up to doing nothing or lie. It is a lose-lose situation for the employee as well as the organization.

What that means is either the team member needs to stand up in front of his/her entire team and say "Ummm, I was feeling kind of offish yesterday and could not finish the API implementation and there was nothing blocking my work" or they need to lie outright claiming progress on work or a fake obstacle while deep inside they know that they would be working harder during remaining week to make it up.


If the process drives you to a corner and makes you feel stupid or makes you lie, it has lost its purpose and sanctity. Once you start tooling it, the fine line between what is right and wrong starts to blur and when the need arises to tool it a little further, one has precedents and some reasonable explanation to convince self to do it.

Some organizations say that they encourage people to be honest and it is okay if they did not accomplish anything on a given day. They can stand up and say so and it is fine. This, as most discover at their own expense, is fantasy land. Why? Imagine yourself in the shoes of the manager when the delivery gets delayed. What does his boss tell him - "You knew that your team was not working, people said in daily meetings that nothing was done and yet you took no action. Whole purpose of agile to raise flags quickly."

There is a conflict - manager needs to make sure everything is on track everyday and raise flag quickly. It would take amazing rapport between manager and team member for latter to say that they accomplished nothing and it to be fine. And I have not yet touched upon the effect on other team members and bad example it would set for younger team members.

Manager would have tough time explaining why it was ok for Tim to not accomplish anything today but same is not acceptable for Tom. Even though it might be true that Tim would be able to stretch and make up while Tom's delivery would almost certainly get delayed (you know it if you have been working with the team). However, when you are talking at day level, there is no room for adjustment and no justification for differentiation.

Long and short - expectation of same level of productivity every day is unrealistic and since software development is all about people, this is a fundamental shortcoming of agile.

Alignment - Part 2

Let us look at the first challenge in getting organizational alignment - Communicating vision and objectives.

Organization's vision statement should come from the very top and be incorporated in every employees function.

The first step of course is to have a vision and come out with a vision statement. Sounds very simple, and yet so many organizations get this wrong. They formulate superlative and unrealistic vision statements which do not reflect in the work done by their staff and only place their vision statement is visible is walls of meeting rooms. I'd probably have another post dedicated to vision statements, but for this post lets go back to communicating vision and objectives.

It is extremely important for top leadership to adopt and evangelize the organization's vision statement. Quarterly and annual all hands and other platforms of interaction with the rest of the employees are good opportunities to communicate the vision. Other communication platforms such as email, recorded videos on intranet website, brown bags are also very helpful and should be used skilfully.

An important aspect is consistently adopting the vision statement in their own work life. Nothing is more impressive than a leader who lives by the vision statement and nothing can drive home a point better than narrating an anecdote which exemplifies adoption of vision statement. We as humans would have more profound experience if our leader went the extra mile or took some pains to live by vision statement as compared to when someone in a suit talked about exceeding customer expectations by building high quality products.

Another extremely important platform for communicating the vision is new hire trainings. It is strategically important for employees to understand the vision right from the beginning. Deep understanding of guiding principles and vision should be built right from the start. Building this understanding should start from induction program itself. This becomes very critical for organizations that are growing rapidly. If you induct a lot of employees who do not understand organization's vision, you would end up spending a lot of time and effort cleaning up the mess created by culturally confused and unaligned work force.

Equally relevant is aligned leadership team - at all levels. Every leader in the organization should exemplify and evangelize the vision. This is at least as important if not more than communication from top leaders. Why? I'm glad you asked.

Employees spend far more time working with these leaders. They probably get to see/hear from top level management only once a quarter but they talk to their supervisors and managers every day. The way their managers perform their duties has most profound impact on the work style employees will adopt. In fact it would be great to have adoption of vision in work as a promotion and performance evaluation criteria for all managers. All manager should work to reinforce the organization's vision.



Lastly, further reinforcement should be done at grass root level by recognizing exemplary work done in alignment to the vision. If lack of alignment to the vision is demonstrated, it should be dealt with immediately and corrected as necessary. Too often I notice that this recognition and correction is left out for want of time/budget or deferred till performance review cycle. Do not defer it. What we do becomes our style and habit and habits are hard to break.

All employees should feel personally responsible for realizing the vision and at the same time understand that alignment to the vision is not optional. If this message is successfully imbibed by all, you'd have a very powerful organization moving with great speed in its predetermined direction.

Alignment - Part 1

Organizational alignment towards a common goal is one of the most important fundamental requirements for success. This is especially true for organizations that are staffed with highly talented and productive people and for startups. Not to imply that other businesses can get away without alignment, but problem surfaces quickly and is more pronounced in the former.

What is alignment? Alignment is every member of staff understands the vision and objective and these are the guiding principles of his or her actions.

What are the challenges in the way of getting aligned? To answer that question, let us examine what would it take to get a team member aligned.

1. Communicating vision and objectives - Team members need to understand what are they building - is it a website or a website that sells computers or a website that sells low end computers or a website that sells low end computers to primarily housewives and elderly who are not tech savvy - noticed the difference?

2. Right work environment - It is extremely important to maintain a work environment where it is ok to fail as long as we are learning from mistakes and improving continuously. A CYA environment changes the motivations and aligns everyone to do just that - CYA, instead of working towards a common goal. I am almost tempted to include motivation as a separate point but that is dependent on work environment to a large extent and its implications should be fairly obvious.

3. Alignment with personal goals - Different people work for different things. Some are working for cool development work opportunity, some are working to fulfill their duties on personal front, some are working for money/stocks, and so on - the list is almost as big as the team. With such varied motivational forces, getting the entire team to work towards a single goal of building a product that delights its customers is not trivial.

How do you make sure that person working for sexy scratch development work does not cut corners with testing or builds the required features and not cool features?

These are the three most important factors to build a great company that moves like a well oiled machine towards its common goals. If people understand what they are doing and why, if they are motivated and not afraid to act, and they believe that actions are aligned with their personal goals, you would have a well aligned organization.

The Art of Firefighting - Part 2

Localization of reference is a concept that holds true throughout the software development life cycle. There is a clear pattern of few chunks of code evolving significantly at any given time. These could be most bug infested code segments or segments most impacted by a change request or impact of a feature addition - irrespective of the nature and phase of software development, there would invariably be few high churn code chunks (I like to call them fire pockets).

Exactly what chunks are churning may vary with the development life cycle stage, e.g. towards the beginning of development, framework and API code might see significant changes while other modules might just be static stubs while later frameworks would have stabilized but other modules would see high activity. Still later in the development cycle some bug infested fire pockets might see a lot of code change.

How is this information relevant for firefighting? :-)

Let us examine some typical characteristics of our situation.

- There isn't enough time. Not the routine "not enough time" but a severe time constraint.
- Lack of time often leads to less than usual unit testing and QA testing.
- Code changes are likely to introduce more bugs.
- More the code changes, more likely that new bugs would be introduced and this relationship is not linear.

Add to these the fact that bulk of code changes are happening in a few fire pockets and voila next mantra to fire fighting is right there - "do not mix tasks". Never ever try to fix multiple issues in one go. It would take far less time to fix an issue, test it, check in the fix, and then proceed to tackle the next issue rather than to implement two or more fixes at one go and then proceed with testing them together.

While you are in the moment, it might appear that you have a handle on the issues and you can easily fix more than one issues at one go but this is similar to running hard and fast in a forest. Every new bug introduced would take so much more time to find and fix simply because lot more code has churned and consequently has low confidence value.

I have witnessed way too many delays and heart burns originating from the temptation of clubbing fixes and yet I have to concede that temptation to add just one more fix to the mix is strong. It is a fire that fuels itself.

To carry the jungle analogy further, if you try to build a shelter and cook food at the same time, you are likely to send an open invitation for dinner to some cohabitants and guess who's on the menu?

Moreover, if you pause and think, how much time difference are we talking about in the two approaches: when you fix one, test it, check it in and then pick up next Vs when you fix both, test them both, and check them in together. Even if we assume best case scenario, i.e. no new bugs were introduced, the only effort saved is time for an extra code commit and one round of set of tests that were common to both the fixes. Given that you are in a fire pocket, racing against time, with reduced testing effort, I'd leave it up to you to calculate the probability of such a best case scenario and its RoI.

This article is second in the series, you can find the first one here - The Art of Firefighting - Part 1.

PS: Pointers to relevant Dilbert cartoons are welcome.

The Art of Firefighting - Part 1

Firefighting is one of the most common experiences that everyone involved with software development goes through. Irrespective of your domain, country, organization, or experience level you are at, if you are involved with writing software, you have to go through this trial by fire.

Scary as it sounds, these are also the moments of highest achievements that bring out the best in you and form part of your most cherished work related memories. This is the stuff that your brags at friends evening out and anecdotes narrated at job interviews are made of.

While everyone shares the burden, tremendous onus lies on the manager to turn these testing times into pleasant memories. A few crucial decisions made during these stressful times are what differentiate the cherished memories from bad experiences that taught everyone one more way of not doing things.

So what is it that takes to lead a successful firefight?

First and foremost, it takes calmness and composure. Easier said than done, I am afraid. It is all too easy to get hyper in the thick of things and act on impulse. However, it takes a lot of self control and effort to stay calm and focused. This is extremely important in order to retain the capability of making rational decisions when it matters the most.

If you are in a fire fighting mode, chances are you are going to run into one problem after another, some foreseen and some unforeseen. Your composure would let you select the right response to foreseen problems and devise smart workarounds for unforeseen ones.

Think of it as crossing a forest. If you are going to start running hard and fast, chances are that it is going to become a very very harsh ordeal with unpleasant ending. However, if you have the composure to stop and listen to jungle sounds, invest your time in climbing a tree to get some sense of surroundings, take into account time of day and need for shelter, chances are you would cross the jungle and live to boast of it during your evenings out with friends ;-)

Here is a more relevant example. You are end of the integration testing phase (which is a hotbed for many a crisis) and your backend API team has just pulled an all night miracle fix enabling rest of the teams to continue with the IT. Its early in the morning and they are winding up, hoping to get some well deserved sleep. What is wrong with this scenario?

Ans: Late in the IT phase, you would be proceeding with IT without a core team in office. Imagine what happens when a bug pops up (as it invariably does) and initial findings point a finger of suspicion towards the backend API implementation. Imagine how much it can slow down other teams when API team is missing from discussions and clarifications get delayed. Worse, how much combined productive and precious time is lost. While it might feel right that entire team is working hard to keep the deadlines and they are all motivated and pumped up about the delivery, these feelings will quickly evaporate when the pumped up team is blamed for delays and lack of support.

If an all night was absolutely necessary, some backend API team members should have been sent home last evening so that they can carry on IT with the rest of the team the next day. Some form of hand off also needs to be planned so that the day time fighters know what the night warriors did in the battlefield. These small foresights on the part of the manager are crucial to success of the delivery. However, it would take some composure to not give in to impulsive action of having everyone working to solve the issue asap.

Climb a tree, get a sense of what may lie ahead and you would survive to brag.

PS: If you find a relevant Dilbert for this article, please let me know.