Pros and cons of using an AI-based diagnosis for diabetic retinopathy

The U.S. Food and Drug Administration (FDA) in April 2018 gave its first approval of an artificial intelligence (AI) algorithm for the detection of diabetic retinopathy (DR) in the offices of non-ophthalmic healthcare practitioners.¹

IDx-DR (IDx, LLC) is paired with a non-mydriatic retinal camera (TRC-NW400, Topcon) and captures images that are sent to a cloud-based server. That server then utilizes IDx-DR software and a “deep-learning” algorithm to detect retinal findings consistent with DR based on autonomous comparison with a large dataset of representative fundus images.

The FDA statement says that if captured images “are of sufficient quality,” the software provides the doctor with one of two results:
• If more than mild DR detected, refer to an eyecare professional (ECP)
• If the results are negative for more than mild DR, rescreen in 12 months

Approval data
FDA approval was based on submission of a 900-subject study using IDx-DR in a primary-care setting (10 sites) with automated image analysis of two 45-degree digital images per eye (one centered on the macula, one centered on the optic nerve).

These images were compared against stereo, widefield fundus imaging interpreted by the Wisconsin Fundus Photograph Reading Center (FPRC). The FPRC based its results on the Early Treatment Diabetic Retinopathy Study Severity Scale (ETDRS) and widefield stereo photographs combined with macular optical coherence tomography (OCT) for detection of diabetic macular edema (DME).²

More than minimal DR was defined as the presence of ETDRS levels 35 or higher (microaneurysms plus hard exudates, cotton wool spots, and/or mild retinal hemorrhages) and/or DME in at least one eye.³ Some 96 percent of images acquired in primary care were deemed of sufficient quality for algorithmic assessment. This was after human operators received four hours of training with the system. 

The technology was 87 percent sensitive and 90 percent specific for detecting more than mild DR. Of note, the algorithm correctly identified 100 percent of subjects with ETDRS levels of 43 or higher DR (moderate non-proliferative disease or worse, defined as microaneurysms plus mild intraretinal microvascular abnormality in up to three quadrants or either moderate retinal hemorrhage in four quadrants or severe hemorrhage in one quadrant).³

Public health aspect
The public health argument for widespread, autonomous detection of diabetes-related eye disease centers on the fact that many patients do not receive dilated retinal exams by ECPs at recommended intervals.⁴

Detecting referable or treatable disease in patients who would not otherwise receive eye exams may save both vision and money. In addition to helping patients, identifying these diseases early will also allow primary-care practitioners (PCPs) to boost their quality measures, protect their income, and may even represent a new revenue stream depending on how third-party payers address reimbursement for AI services.

The marginal costs of operating AI is almost nil once the new technology has been acquired by the PCP.⁵ Correspondence from the PCP to the ECP has been shown to be more important for compliance with dilated eye examinations.⁶

Downsides of AI
However, ECPs must educate PCPs and endocrinologists about the downsides of the current iteration of AI for diabetes eye care.

Diabetes patients frequently have ocular disease other than DR (glaucoma, age-related macular degeneration, cataract, dry eye, to name the most prevalent). These diseases require a comprehensive (dilated) eye examination for proper diagnosis and management.

A negative AI-based finding may give PCPs and patients a false sense of security about the totality of their ocular status. The relatively high false positive rate means some patients may have or will soon have a sight-threatening eye disease.

Individual patient risk of disease severity and vision loss is predicated on disease duration, degree and consistency of metabolic control, and accurate staging of DR.

Emerging evidence from the Joslin Diabetes Center shows that DR lesions in the peripheral retina are highly predictive of which patients will develop proliferative DR, the blinding form of the disease.⁷

The leading cause of vision loss in patients with diabetes is diabetic macular edema (DME). The gold standard for DME detection is stereoscopic macular examination coupled with spectral-domain optical coherence tomography (SD-OCT). Though all subjects with ETDRS level 43 or higher DR were detected via IDx-DR, how many cases of subtle DME were missed because this data was not reported by investigators?

I have concern that AI algorithms designed to pass or fail individual patients may hinder appropriate education and targeted intervention for patients with milder DR.

Working together
In the U.S., I estimate about 58,000 ECPs (optometrists, ophthalmologists, retinal specialists) who are trained to diagnose and manage the spectrum of diabetes-related eye disease. The U.S. has committed substantial resources in education and instrumentation to help ECPs identify these patients earlier and educate patients appropriately based on individualized risk factors and current stage of disease.

It makes economic sense for non-ophthalmic providers to work collaboratively with ECPs in their communities to ensure that patients receive appropriate diagnoses and care.

References:

1. U.S. Food & Drug Administration. FDA permits marketing of artificial intelligence-based device to detect certain diabetes-related eye problems. Available at: https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm604357.htm. Accessed 6/21/18.
2. Abramoff M. Artificial intelligence for automated detection of diabetic retinopathy in primary care. Paper presented at: Macular Society; February 22, 2018; Beverly Hills, CA. Available at: http://webeye.ophth.uiowa.edu/abramoff/MDA-MacSocAbst-2018-02-22.pdf.
3. Early Treatment Diabetic Retinopathy Study Research Group. Grading diabetic retinopathy from stereoscopic color fundus photographs--an extension of the modified Airlie House classification. ETDRS report number 10. Ophthalmology. 1991 May;98(5 Suppl):786-806.
4. Paksin-Hall A, Dent ML, Dong F, Ablah E. Factors contributing to diabetes patients not receiving annual dilated eye examinations. Ophthalmic Epidemiol. 2013 Oct;20(5):281-7.
5. Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, Wang Y, Dong Q, Shen H, Wang Y. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2017;2(4):230-243. doi:10.1136/svn-2017-000101.
6. Storey PP, Murchison AP, Pizzi LT, Hark LA, Dai Y, Leiby BE, Haller JA.  Impact of physician communication on diabetic eye examination adherence: Results From a Retrospective Cohort Analysis. Retina. 2016 Jan;36(1):20-7.
7. Silva PS, Cavallerano JD, Haddad NM, Kwak H, Dyer KH, Omar AF, Shikari H, Aiello LM, Sun JK, Aiello LP. Peripheral lesions identified on ultrawide field imaging predict increased risk of diabetic retinopathy progression over 4 years. Ophthalmology. 2015 May;122(5):949-56.