Written by Corinne Stroum
It’s that time of year: as I drive by my local farm stands, a unique cherry stands out among the rest – the Rainier cherry. Pink and red, sweet, it’s a cultivar specific to the Pacific Northwest. This cherry didn’t exist for me growing up in New Jersey. I don’t know whether they weren’t available in my locale or my family didn’t purchase them. I have, however, come to see that they epitomize some of the best work in healthcare machine learning (ML). My reasons are:
- Locality. Rainier cherries, named for our local Mt. Rainier, are unique to this region. As I will explain below, solutions that rely upon healthcare ML models best achieve their goals when local nuances are part of the model.
- Time-sensitivity. These cherries are available for only a few weeks each year. Similarly, healthcare ML models may reflect the time the data scientists trained the model. Models are also subject to drift, a scenario in which the input variables or predictive endpoint shift over time.
- Availability. Rainier cherries aren’t just at the supermarket or even the farmer’s market but are sold out of the backs of vans that pop up on the side of the road. This speaks to solutions that deploy model outputs where the consumers are. Most everyone in healthcare IT know that it is near-impossible to pull a user from a primary working tool, such as the electronic medical record (EMR) or care management software.
Over the last few years, we have seen many machine learning vendors entering the healthcare space. I’ve been asked: “How should I vet these vendors? Who is reliable, and how do I know if they’ll deliver a quality model?” I think about context. It would keep costs low and installation straightforward to deploy a pre-trained model.
Pre-trained models can ship and even test with very high-performance metrics. Many vendors will even tout that they have trained these well-performing models on comprehensive, longitudinal datasets. However, they are missing the Rainier cherries: the unique fingerprint specific to your data. The services you may offer your patients or members in collaboration with a community organization that does not fit quite right into your EMR’s structured footprint. The supplemental benefits your health plan offers - a value-add that has become increasingly common among Medicare Advantage plans in the last few years. Local details change the nature of events in your data, and your model should benefit from this information.
The rebuttal is natural: wouldn’t an off-the-shelf solution and pre-trained model be less expensive? Wouldn’t it be faster to implement and adopt? Absolutely. For some scenarios, these models are suitable. There is little downside to the pint of cherries you can find at the supermarket, to continue my analogy.
However, you miss the upside of discovering a sweet local varietal. Instead of asking your prospective vendor, “how well does your model perform in the lab?” try “how will your model suit my data, my unique needs, and my business patterns?” or “How will you ensure this model performs well on my data?” Seek a vendor who welcomes additional sources of data rather than charges you for it. This additional information is how your model will exceed expectations beyond model performance metrics. Ensure your vendor will seek out that local fit: to your data, your workflow, and, most of all, your users.
