Joseph Lo

Overview:

My research uses computer vision and machine learning to improve medical imaging, focusing on breast and CT imaging. There are three specific projects:

(1) We design deep learning models to diagnose breast cancer from mammograms. We perform single-shot lesion detection, multi-task segmentation/classification, and image synthesis. Our goal is to improve radiologist diagnostic performance and empower patients to make personalized treatment decisions. This work is funded by NIH, Dept of Defense, Cancer Research UK, and other agencies.

(2) We create "digital twin" anatomical models that are based on actual patient data and thus contain highly realistic anatomy. With customized 3D printing, these virtual phantoms can also be rendered into physical form to be scanned on actual imaging devices, which allows us to assess image quality in new ways that are clinically relevant.

(3) We are building a computer-aided triage platform to classify multiple diseases across multiple organs in chest-abdomen-pelvis CT scans. Our hospital-scale data sets have hundreds of thousands of patients. This work includes natural language processing to analyze radiology reports as well as deep learning models for organ segmentation and disease classification.

Positions:

Professor in Radiology

Radiology
School of Medicine

Professor in the Department of Electrical and Computer Engineering

Electrical and Computer Engineering
Pratt School of Engineering

Member of the Duke Cancer Institute

Duke Cancer Institute
School of Medicine

Education:

B.S.E.E. 1988

Duke University

Ph.D. 1993

Duke University

Research Associate, Radiology

Duke University

Grants:

Predicting Breast Cancer With Ultrasound and Mammography

Administered By
Radiology
Awarded By
National Institutes of Health
Role
Principal Investigator
Start Date
End Date

Improved Diagnosis of Breast Microcalcification Clusters

Administered By
Radiology
Awarded By
National Institutes of Health
Role
Principal Investigator
Start Date
End Date

Accurate Models for Predicting Radiation-Induced Injury

Administered By
Radiation Oncology
Awarded By
National Institutes of Health
Role
Investigator
Start Date
End Date

Computer Aid for the Decision to Biopsy Breast Lesions

Administered By
Radiology
Awarded By
US Army Medical Research
Role
Co Investigator
Start Date
End Date

Computer Aid for the Decision to Biopsy Breast Lesions

Administered By
Radiology
Awarded By
National Institutes of Health
Role
Investigator
Start Date
End Date

Publications:

Utility of a Rule-Based Algorithm in the Assessment of Standardized Reporting in PI-RADS.

RATIONALE AND OBJECTIVES: Adoption of the Prostate Imaging Reporting & Data System (PI-RADS) has been shown to increase detection of clinically significant prostate cancer on prostate mpMRI. We propose that a rule-based algorithm based on Regular Expression (RegEx) matching can be used to automatically categorize prostate mpMRI reports into categories as a means by which to assess for opportunities for quality improvement. MATERIALS AND METHODS: All prostate mpMRIs performed in the Duke University Health System from January 2, 2015, to January 29, 2021, were analyzed. Exclusion criteria were applied, for a total of 5343 male patients and 6264 prostate mpMRI reports. These reports were then analyzed by our RegEx algorithm to be categorized as PI-RADS 1 through PI-RADS 5, Recurrent Disease, or "No Information Available." A stratified, random sample of 502 mpMRI reports was reviewed by a blinded clinical team to assess performance of the RegEx algorithm. RESULTS: Compared to manual review, the RegEx algorithm achieved overall accuracy of 92.6%, average precision of 88.8%, average recall of 85.6%, and F1 score of 0.871. The clinical team also reviewed 344 cases that were classified as "No Information Available," and found that in 150 instances, no numerical PI-RADS score for any lesion was included in the impression section of the mpMRI report. CONCLUSION: Rule-based processing is an accurate method for the large-scale, automated extraction of PI-RADS scores from the text of radiology reports. These natural language processing approaches can be used for future initiatives in quality improvement in prostate mpMRI reporting with PI-RADS.
Authors
Zhang, D; Neely, B; Lo, JY; Patel, BN; Hyslop, T; Gupta, RT
MLA Citation
Zhang, Dylan, et al. “Utility of a Rule-Based Algorithm in the Assessment of Standardized Reporting in PI-RADS.Acad Radiol, July 2022. Pubmed, doi:10.1016/j.acra.2022.06.024.
URI
https://scholars.duke.edu/individual/pub1532265
PMID
35909050
Source
pubmed
Published In
Acad Radiol
Published Date
DOI
10.1016/j.acra.2022.06.024

Anomaly Detection of Calcifications in Mammography Based on 11,000 Negative Cases.

In mammography, calcifications are one of the most common signs of breast cancer. Detection of such lesions is an active area of research for computer-aided diagnosis and machine learning algorithms. Due to limited numbers of positive cases, many supervised detection models suffer from overfitting and fail to generalize. We present a one-class, semi-supervised framework using a deep convolutional autoencoder trained with over 50,000 images from 11,000 negative-only cases. Since the model learned from only normal breast parenchymal features, calcifications produced large signals when comparing the residuals between input and reconstruction output images. As a key advancement, a structural dissimilarity index was used to suppress non-structural noises. Our selected model achieved pixel-based AUROC of 0.959 and AUPRC of 0.676 during validation, where calcification masks were defined in a semi-automated process. Although not trained directly on any cancers, detection performance of calcification lesions on 1,883 testing images (645 malignant and 1238 negative) achieved 75% sensitivity at 2.5 false positives per image. Performance plateaued early when trained with only a fraction of the cases, and greater model complexity or a larger dataset did not improve performance. This study demonstrates the potential of this anomaly detection approach to detect mammographic calcifications in a semi-supervised manner with efficient use of a small number of labeled images, and may facilitate new clinical applications such as computer-aided triage and quality improvement.
Authors
Hou, R; Peng, Y; Grimm, LJ; Ren, Y; Mazurowski, MA; Marks, JR; King, LM; Maley, CC; Hwang, ES; Lo, JY
MLA Citation
Hou, Rui, et al. “Anomaly Detection of Calcifications in Mammography Based on 11,000 Negative Cases.Ieee Trans Biomed Eng, vol. 69, no. 5, May 2022, pp. 1639–50. Pubmed, doi:10.1109/TBME.2021.3126281.
URI
https://scholars.duke.edu/individual/pub1502472
PMID
34788216
Source
pubmed
Published In
Ieee Trans Biomed Eng
Volume
69
Published Date
Start Page
1639
End Page
1650
DOI
10.1109/TBME.2021.3126281

Multi-label annotation of text reports from computed tomography of the chest, abdomen, and pelvis using deep learning.

BACKGROUND: There is progress to be made in building artificially intelligent systems to detect abnormalities that are not only accurate but can handle the true breadth of findings that radiologists encounter in body (chest, abdomen, and pelvis) computed tomography (CT). Currently, the major bottleneck for developing multi-disease classifiers is a lack of manually annotated data. The purpose of this work was to develop high throughput multi-label annotators for body CT reports that can be applied across a variety of abnormalities, organs, and disease states thereby mitigating the need for human annotation. METHODS: We used a dictionary approach to develop rule-based algorithms (RBA) for extraction of disease labels from radiology text reports. We targeted three organ systems (lungs/pleura, liver/gallbladder, kidneys/ureters) with four diseases per system based on their prevalence in our dataset. To expand the algorithms beyond pre-defined keywords, attention-guided recurrent neural networks (RNN) were trained using the RBA-extracted labels to classify reports as being positive for one or more diseases or normal for each organ system. Alternative effects on disease classification performance were evaluated using random initialization or pre-trained embedding as well as different sizes of training datasets. The RBA was tested on a subset of 2158 manually labeled reports and performance was reported as accuracy and F-score. The RNN was tested against a test set of 48,758 reports labeled by RBA and performance was reported as area under the receiver operating characteristic curve (AUC), with 95% CIs calculated using the DeLong method. RESULTS: Manual validation of the RBA confirmed 91-99% accuracy across the 15 different labels. Our models extracted disease labels from 261,229 radiology reports of 112,501 unique subjects. Pre-trained models outperformed random initialization across all diseases. As the training dataset size was reduced, performance was robust except for a few diseases with a relatively small number of cases. Pre-trained classification AUCs reached > 0.95 for all four disease outcomes and normality across all three organ systems. CONCLUSIONS: Our label-extracting pipeline was able to encompass a variety of cases and diseases in body CT reports by generalizing beyond strict rules with exceptional accuracy. The method described can be easily adapted to enable automated labeling of hospital-scale medical data sets for training image-based disease classifiers.
Authors
D'Anniballe, VM; Tushar, FI; Faryna, K; Han, S; Mazurowski, MA; Rubin, GD; Lo, JY
MLA Citation
D’Anniballe, Vincent M., et al. “Multi-label annotation of text reports from computed tomography of the chest, abdomen, and pelvis using deep learning.Bmc Med Inform Decis Mak, vol. 22, no. 1, Apr. 2022, p. 102. Pubmed, doi:10.1186/s12911-022-01843-4.
URI
https://scholars.duke.edu/individual/pub1517986
PMID
35428335
Source
pubmed
Published In
Bmc Medical Informatics and Decision Making
Volume
22
Published Date
Start Page
102
DOI
10.1186/s12911-022-01843-4

Prediction of Upstaging in Ductal Carcinoma in Situ Based on Mammographic Radiomic Features.

Background Improving diagnosis of ductal carcinoma in situ (DCIS) before surgery is important in choosing optimal patient management strategies. However, patients may harbor occult invasive disease not detected until definitive surgery. Purpose To assess the performance and clinical utility of mammographic radiomic features in the prediction of occult invasive cancer among women diagnosed with DCIS on the basis of core biopsy findings. Materials and Methods In this Health Insurance Portability and Accountability Act-compliant retrospective study, digital magnification mammographic images were collected from women who underwent breast core-needle biopsy for calcifications that was performed at a single institution between September 2008 and April 2017 and yielded a diagnosis of DCIS. The database query was directed at asymptomatic women with calcifications without a mass, architectural distortion, asymmetric density, or palpable disease. Logistic regression with regularization was used. Differences across training and internal test set by upstaging rate, age, lesion size, and estrogen and progesterone receptor status were assessed by using the Kruskal-Wallis or χ2 test. Results The study consisted of 700 women with DCIS (age range, 40-89 years; mean age, 59 years ± 10 [standard deviation]), including 114 with lesions (16.3%) upstaged to invasive cancer at subsequent surgery. The sample was split randomly into 400 women for the training set and 300 for the testing set (mean ages: training set, 59 years ± 10; test set, 59 years ± 10; P = .85). A total of 109 radiomic and four clinical features were extracted. The best model on the test set by using all radiomic and clinical features helped predict upstaging with an area under the receiver operating characteristic curve of 0.71 (95% CI: 0.62, 0.79). For a fixed high sensitivity (90%), the model yielded a specificity of 22%, a negative predictive value of 92%, and an odds ratio of 2.4 (95% CI: 1.8, 3.2). High specificity (90%) corresponded to a sensitivity of 37%, positive predictive value of 41%, and odds ratio of 5.0 (95% CI: 2.8, 9.0). Conclusion Machine learning models that use radiomic features applied to mammographic calcifications may help predict upstaging of ductal carcinoma in situ, which can refine clinical decision making and treatment planning. © RSNA, 2022.
Authors
Hou, R; Grimm, LJ; Mazurowski, MA; Marks, JR; King, LM; Maley, CC; Lynch, T; van Oirsouw, M; Rogers, K; Stone, N; Wallis, M; Teuwen, J; Wesseling, J; Hwang, ES; Lo, JY
MLA Citation
Hou, Rui, et al. “Prediction of Upstaging in Ductal Carcinoma in Situ Based on Mammographic Radiomic Features.Radiology, vol. 303, no. 1, Apr. 2022, pp. 54–62. Pubmed, doi:10.1148/radiol.210407.
URI
https://scholars.duke.edu/individual/pub1505281
PMID
34981975
Source
pubmed
Published In
Radiology
Volume
303
Published Date
Start Page
54
End Page
62
DOI
10.1148/radiol.210407

Technical note: Controlling the attenuation of 3D-printed physical phantoms for computed tomography with a single material.

PURPOSE: The purpose of this work was to characterize and improve the ability of fused filament fabrication to create anthropomorphic physical phantoms for CT research. Specifically, we sought to develop the ability to create multiple levels of X-ray attenuation with a single material. METHODS: CT images of 3D printed cylinders with different infill angles and printing patterns were assessed by comparing their 2D noise power spectra to determine the conditions that produced minimal and uniform noise. A backfilling approach in which additional polymer was extruded into an existing 3D printed background layer was developed to create multiple levels of image contrast. RESULTS: A print with nine infill angles and a rectilinear infill pattern was found to have the best uniformity, but the printed objects were not as uniform as a commercial phantom. An HU dynamic range of 600 was achieved by changing the infill percentage from 40% to 100%. The backfilling technique enabled control of up to eight levels of contrast within one object across a range of 200 HU, similar to the range of soft tissue. A contrast detail phantom with six levels of contrast and an anthropomorphic liver phantom with four levels of contrast were printed with a single material. CONCLUSION: This work improves the uniformity and levels of contrast that can be achieved with fused filament fabrication, thereby enabling researchers to easily create more detailed physical phantoms, including realistic, anthropomorphic textures.
Authors
Tong, H; Pegues, H; Samei, E; Lo, JY; Wiley, BJ
MLA Citation
Tong, Huayu, et al. “Technical note: Controlling the attenuation of 3D-printed physical phantoms for computed tomography with a single material.Med Phys, vol. 49, no. 4, Apr. 2022, pp. 2582–89. Pubmed, doi:10.1002/mp.15494.
URI
https://scholars.duke.edu/individual/pub1510786
PMID
35191035
Source
pubmed
Published In
Med Phys
Volume
49
Published Date
Start Page
2582
End Page
2589
DOI
10.1002/mp.15494

Research Areas:

Breast Neoplasms
Clinical Trials as Topic
Computer Simulation
Decision Making, Computer-Assisted
Decision Support Systems, Clinical
Decision Support Techniques
Image Processing, Computer-Assisted
Imaging, Three-Dimensional
Machine learning
Mammography
Models, Structural
Pattern Recognition, Automated
Radiographic Image Interpretation, Computer-Assisted
Radiology
Technology Assessment, Biomedical
Tomosynthesis