Project Pulse – the shape of the distribution matters
TeamFit is adding a new tool to its platform in June, project pulse. This is a simple way to ask people on a team how a project is going. Later on we will extend this so that people can let each other know how well a project did.
We are asking this so that our users can close the loop on project performance. We will use project pulse to help us better predict success, and to tease out which skills and patterns are contributing to success.
Most good ideas are simple to test and we have been testing project pulse internally using paper cards and a spreadsheet. We have already found a simple but very interesting result.
The shape of the distribution is just as important as the actual project pulse number, and changes in the shape of the distribution suggest interesting things about team dynamics.
In a short time we have already seen five different distributions.
Normal This is the Bell curve that we all learned about in statistics. Most value cluster around the mean (the average) and the mean and the median are close together. At this point we are not ascribing any particular meaning to scores with a normal distribution.
Even: This one shows up as a straight line in the data, there are roughly the same number of values for each value. This has actually been more common than data sets resembling normal distributions, probably because the small number of data points in each data set.
Skew Right: This distribution occurs when more people give a positive rating than negative. The median score will be higher than the average score.
Skew Left: This is the inverse of the Skew Right distribution. The media score will be lower than the average score. We have seen cases where a team has changed from Skew Right to Skew Left without the average score changing!
Bimodal: This may be the scariest distribution. The team is divided between optimists and pessimists. When you see a bimodal distribution you will want to talk to the team to understand what is happening.
At this point, we believe that the Skew Right distribution is a positive signal and that the Skew Left and Bimodal signals are negative.
Just as interesting as the distribution of any one data set are changes in the distribution. We have already seen a data series go from Even to Bimodal! This means that some people on the team became more sceptical of the projects future success while others became more optimistic. The average value remained the same, but the team dynamics had changed in an important way.
We have also seen a dataset go from skewed left to skewed right. Again, the mean value did not change but the median went from left of the average to right of the average. Is this a good sign? In general, more team members are now optimistic about the project’s outcome, but some people have become less optimistic (otherwise the mean would have also gone up).
Both patterns, the emergence of a bimodal distribution or a shift in skew are the sort of things project leaders should explore. In the case of the bimodal distribution, one might want to ask the following questions.
Are the optimists and pessimists performing different functions?
It means one thing if all of the engineers have one view and all of the business people have another than if the optimists and pessimists are mixed across functional roles.
Do the optimists and pessimists have different levels of experience?
It is worth knowing if all of the experienced people fall into one group (optimists or pessimists) while the inexperienced people fall into another group.
Do customer-facing people have a different view of the project than people who are internally focussed?
Sometimes the people managing the client have insights into the project’s likely success that people who are internally focussed do not have. In this case, better team communication may change the distribution.
In our work on skill ranking, we have had to account for individual bias in scoring. Some people tend to make rank other people relatively high or relatively low on a skill. There is the person whose default score for another person is low. When this person gives a low score it has comparatively little information, but when they give a high score it is worth noticing.
The same is true when it comes to predicting project success. Some people are by nature optimists, I am one, if I was not I would not start companies or make early stage investments. So when I say something I engage in is likely to succeed that carries less predictive value than if I say something I am committed to is likely to fail. There are pessimists as well for whom the opposite is true.
In our SkillRank™ algorithm, we factor in the SkillRank™ of the person making the judgement in calculating confidence. A person with a great deal of experience in big data is likely to be a better judge of another person’s skill in the area than a person who does not have relevant skills or who only has them at a low level.
In the same way, some people are going to be better at predicting project outcomes than others. Other people will perform the role of leading indicators, when they become optimistic or pessimistic they team pulse is likely to move in their direction over time.
As TeamFit builds its database of team pulse scores for different teams with different combinations of skills we will be able to see these patterns and weave them into our data model. It is going to be an interesting journey and we hope you will join us!
