Improve Project Evaluation & Selection

In PPM you learn the results of funded proposals, but you never know what would have happened with the unfunded proposals. Furthermore, the funded proposals are your best prospects; they constitute a nonrandom sample of your proposals.

Any statistical analysis of your PPM results must address this situation. Otherwise, the analysis is flawed and its reslts can harm your decision-making. Star DS is developing and testing metrics that properly analyze PPM results. These metrics will estimate the following qualities of your PPM:

• The quality of your project prioritization
  If prioritization is the basis of your project selection, you must evaluate it. Star DS is developing the first and only metric that evaluates project prioritization.
• The success rate of your projects
  Star DS is developing a metric that estimates the project success rates that result from all cutoff values, hurdle rates or amounts of funding down a ranking of projects.
• A project's probability of success
  Star DS is testing a model that translates a project's evaluation into an unbiased, objective estimate of its probability of success. The analysis may even estimate success as a function of each project attribute. Of course, you can compare the objective estimates to your subjective ones.
• Correlation of projects' scores and attributes with success
  When correlating project scores with results, standard statistical techniques underestimate the correlation. In extreme cases, scoring models look useless and project selection appears to be random. Star DS is developing a metric that correctly estimates the correlation between project scores and results. If you have data from enough projects, we will try to correlate project attributes with results as well.
• Accuracy and precision of project attributes
  Most PPM models -- whether heuristic or optimization -- rely on estimates of project attributes. You need a metric that evaluates your estimates, or as the saying goes, "Garbage in, garbage out!" Star DS is developing statistics that measure the accuracy and precision of your estimates. By identifying inaccurate or imprecise attributes, you will know how to improve your project evaluation model.
• Better project evaluations
  Historical data may improve your estimates of project attributes. However, using only the data from funded proposals can impart an optimistic bias to your estimates. You must use historical data from all previous proposals: funded and unfunded. Star DS is developing statistical techniques for using all of your historical data.