Building a scalable and accurate medical data labeling pipeline
Examine the unique challenges with medical data labeling, the relative lack of accuracy produced by traditional data labeling methods, and a more accurate and scalable alternative based on collective intelligence.
The Medical AI Transformation
Healthcare is undergoing a massive transformation through the adoption of medical AI. This is a trend that we expect to accelerate as more AI companies receive approval for reimbursements. (for more on that, check out this article from Luke Oakden-Rayner)
AI is, of course, only as good as the data used to train the model. It takes a tremendous amount of highly accurate and meticulously labeled data to properly train the latest deep learning models. As such, the data labeling has ballooned into a multi-billion dollar industry, primarily by the explosive growth of data labeling for the autonomous vehicle, AR/VR, and retail industries
Medical data labeling requires skill
However, within healthcare, there are unique challenges with acquiring highly accurate medical training data. First and foremost, it is a task that requires skill. This means that traditional data labeling methods are unable to produce labels accurately enough to be useful for a medical AI application.
Many practitioners have attempted to compensate for the skills gap by creating extensive training programs for their labelers or hiring teams of board certified physicians. These measures drive cost up and are still not sufficient in delivering accurate results since they rely solely on the credentials and education of the labelers rather than evaluating their recent performance on each specific data labeling task.
To put this in a different perspective, if you were to ask a radiologist how good they are, they will say they are ‘good’. Digging deeper, if you ask them if they are better at finding calcifications or masses in breast x-rays, they might be able to give a subjective answer but they have no quantifiable way of indicating their performance relative to their peers. What’s more, it is becoming well known that medical experts disagree at alarming rates. In fact, a recent study by Cheng et al (2013) found that radiologists disagreed on 16% of CT scans at a level 1 trauma center. This isn’t a knock on physicians. Rather, it is the result of not having a common way to evaluate the labeling performance of experts on a given task.
"Radiologists disagreed on 16% of CT scans at a level 1 trauma center" - Cheng et al., (2013)
Challenges with medical data labeling
In addition to the inability to access labeler performance and reconcile disagreements, there are many other challenges with medical data labeling as show in the table below:
In our effort to better understand the challenges in medical data labeling, we interviewed dozens of experts in the medical AI and annotation space. We captured their insights into what we hope is a descriptive guide for anyone looking to enhance the accuracy and performance of their medical data labeling efforts.
From this, we created a free guide for anyone who is looking to improve their medical data labeling.
You'll learn the following:
Why medical data labeling is different
Learn the unique challenges of working with medical data including the high skill needed for labeling and managing privacy concerns of PHI
How to collect medical data
Explore ways to acquire medical data including open-source, in-house and through licensing and partnerships
How to clean and enrich medical data
Understand ways to clean, classify and segment medical data and when to employ each labeling method
Options for medical data labeling
Review data labeling vendor models including in-house experts, hiring medical students, hybrid teams and crowdsourced options
How to evaluate accuracy of medical data labels
Grasp how to evaluate the accuracy of your medical data labels and understand where traditional methods fall short
The benefits of collective intelligence
Discover a new method for data labeling that offers superior accuracy vs other methods by aggregating multiple opinions