Selecting The Proper Test Statistics to Ensure Your Electronic Algorithms Suits Your Business Needs
By John Shepard
A number of organizations in a variety of fields are utilizing electronic algorithms to assist in decision support. These electronic algorithms use a variety of different methods to collect data and monitor for user specified events. Healthcare providers and analytics organizations are beginning to utilize electronic algorithms for both predictive analytics and clinical decision support (CDS). Utilizing software like ShepardHealth’s CDS software Fixxer™, healthcare facilities can identify trends and then set real-time notifications that will alert staff when their business rules are met. The alert recipients can then address the issue to ensure no negative outcomes occur.
The goal for any algorithm or diagnostic test is to be 100% in all categories, but generally, algorithm and analytics developers must be realistic and keep in mind what the end goal is. In medical diagnostics, this is a well-known idea. When creating different tests to diagnosis different diseases, a number of different statistics are utilized to demonstrate how well the test performs. However, the desired usage of an algorithm or diagnostic test needs to be in context when examining these test statistics, as the intended purpose of one test may be different from another.
Let’s assume there is an algorithm utilized by a healthcare facility to identify sepsis[i]. Identifying patients with sepsis in a timely manner is important to ensure the best outcomes for patients[ii] [iii]. With any algorithm or test there are four possibilities:
Algorithms, like diagnostic tests, should be chosen based on the results of trials. Utilizing a CDS software like Fixxer™, healthcare facilities can utilize a number of different algorithms depending on the business need and the intended action that the algorithm alert will enable. To figure out what algorithm design is best for your healthcare facility, you must ask yourself a number of questions. For example:
1. Does your organization have regulatory requirements or must-do tasks? Do you want to ensure you identify a certain event every time? Is the cost of an FP alert minimal?
- If you answer yes, then the goal for this algorithm would be to minimize FN which can be measured using the negative predictive value [NPV=TN/(TN+FN)]. If an FP alert costs the healthcare facility very little, then maximizing the NPV alone will help ensure you never miss a regulatory requirement.
- If the cost of an FP alert is high or if workers will begin to complain of alert fatigue, then minimizing FN and FP will be key. This is identified using the accuracy [ACC=(TP+TN)/(TP+TN+FP+FN)]. Achieving an algorithm with a high level of accuracy is very difficult and rare to find, but a gold mind if you find them. These algorithms are “pie in the sky” projects.
2. Are you okay with missing a few cases to ensure when the alert does trigger it is a TP case? Is the financial cost or risk of alert fatigue very high? Does your algorithm need to be “On-the-money” every-time?
- The goal for an algorithm, in this case, would be to minimize FP cases, which is identified using the positive predictive value [PPV=TP/(TP+FP)]. Maximizing the PPV will help to ensure workers are never wrongly alerted. Quality improvement initiatives require buy-in from workers on the ground, and this is difficult at times if the alert or algorithm is wrong
Everyone wants the “Pie In The Sky” algorithm, but in the majority of the time, developers need to identify their true need utilizing business context and test-statics that are relevant to your business need. ShepardHealth can work with healthcare facilities to select an algorithm that produces their desired outcomes for sepsis and many other disease states.