How to keep score

Scoring systems provide a shortcut to evaluating a product or service by assigning a numeric value to the item.

For example, wine critics employ a variety of scoring systems to evaluate wine. One system, created by the famous wine critic Robert Parker, uses a scale that ranges from 50 to 100 points. Of the 50 points up for grabs, 5 points are available for rating color or appearance, aroma and bouquet may be awarded 15 points, flavor and finish merit 20 points, and the potential for improvement—or aging—assigns up to 10 points.

The scores try to assign an overall value to the wine generated from the individual scores given to those four criteria. For someone interested in choosing a wine for drinking tonight, a wine with a 90-point score that reflects its higher aging potential may be an inferior choice when contrasted with a 88 point wine that is not as age worthy. If the consumer intends to drink the wine immediately, its age worthiness does not matter, yet the wine is rated higher because of it.

Scoring Presents Bias

Purchasing health information technology is obviously a risky and more difficult decision than choosing a bottle of red wine for dinner.

To help with this purchasing decision, organizations often use product ratings provided by self-appointed industry organizations to evaluate enterprise applications. Although often valuable in guiding a purchasing decision, these ratings suffer from the same problems consumers face in purchasing wine, choosing a hotel, or buying a product from Amazon. The rules that define scientifically valid statistics apply equally to determine the value and usefulness of product reviews.

For example, sample size represents one criteria that helps determine the validity and reliability of reviews. We all trust a product review that includes hundreds of individual opinions much more than one that includes only ten responses. This same intuitive rationale applies to the use of statistics in evaluating product reviews.

Beware Those Under 30

One rule of thumb is that a sample size of less than 30 indicates a questionable result. Therefore, the next time you read an enterprise software product review, examine the number of respondents. If it is less than 30, ignore the overall product review and focus on the individual responses.

Many of these product reviews include overall product scores. These scores, subject to similar statistical flaws as wine scoring systems, often deliver even more misleading information than the product reviews themselves. These scores, frequently "calculated out" to three or more digits, ignore the basic mathematical significant digits concept taught in every freshman chemistry class. (Here is a link to refresh your understanding of significant digits - https://en.wikipedia.org/wiki/Significant_figures.)

Without doing the calculations here, it is easy to logically conclude that a product review with a sample size of less than 30 should not include a product score with three significant digits. Making such calculations by introducing digits to imply a level pf precision that does not exist misleads those relying upon the product review to make important product decisions.

How to Choose

Here are some principles that apply to the purchase of any health information technology product:

  • Start any evaluation with an honest assessment of the organization's strategic vision and link that vision to reason for purchasing the software
  • Broadly explore available options without allowing review scores to bias this initiative
  • Seek a strong vendor partner who you trust
  • Avoid software demonstrations to evaluate features and functionality, an impossible task for these complex software systems
  • Establish pre-implementation metrics to evaluate the purchasing decision

Until rating organizations agree to embrace more scientific and defensible methods in their building of enterprise software reviews, provider organizations must resist the temptation to bias their decision making on these flawed evaluations. These organizations are better served by defining their decision making process, appropriately utilizing product reviews, and executing on their decision-making plan to evaluate all viable vendor organizations.

The views, opinions and positions expressed within these guest posts are those of the author alone and do not represent those of Becker's Hospital Review/Becker's Healthcare. The accuracy, completeness and validity of any statements made within this article are not guaranteed. We accept no liability for any errors, omissions or representations. The copyright of this content belongs to the author and any liability with regards to infringement of intellectual property rights remains with them.

© Copyright ASC COMMUNICATIONS 2018. Interested in LINKING to or REPRINTING this content? View our policies by clicking here.

 

Top 40 Articles from the Past 6 Months