Joseph Lo

Overview:

My research uses computer vision and machine learning to improve medical imaging, focusing on breast and CT imaging. There are three specific projects:

(1) We design deep learning models to diagnose breast cancer from mammograms. We perform single-shot lesion detection, multi-task segmentation/classification, and image synthesis. Our goal is to improve radiologist diagnostic performance and empower patients to make personalized treatment decisions. This work is funded by NIH, Dept of Defense, Cancer Research UK, and other agencies.

(2) We create "digital twin" anatomical models that are based on actual patient data and thus contain highly realistic anatomy. With customized 3D printing, these virtual phantoms can also be rendered into physical form to be scanned on actual imaging devices, which allows us to assess image quality in new ways that are clinically relevant.

(3) We are building a computer-aided triage platform to classify multiple diseases across multiple organs in chest-abdomen-pelvis CT scans. Our hospital-scale data sets have hundreds of thousands of patients. This work includes natural language processing to analyze radiology reports as well as deep learning models for organ segmentation and disease classification.

Positions:

Professor in Radiology

Radiology
School of Medicine

Professor of Biomedical Engineering

Biomedical Engineering
Pratt School of Engineering

Professor in the Department of Electrical and Computer Engineering

Electrical and Computer Engineering
Pratt School of Engineering

Member of the Duke Cancer Institute

Duke Cancer Institute
School of Medicine

Education:

B.S.E.E. 1988

Duke University

Ph.D. 1993

Duke University

Research Associate, Radiology

Duke University

Grants:

Predicting Breast Cancer With Ultrasound and Mammography

Administered By
Radiology
Awarded By
National Institutes of Health
Role
Principal Investigator
Start Date
End Date

Improved Diagnosis of Breast Microcalcification Clusters

Administered By
Radiology
Awarded By
National Institutes of Health
Role
Principal Investigator
Start Date
End Date

Accurate Models for Predicting Radiation-Induced Injury

Administered By
Radiation Oncology
Awarded By
National Institutes of Health
Role
Investigator
Start Date
End Date

Computer Aid for the Decision to Biopsy Breast Lesions

Administered By
Radiology
Awarded By
US Army Medical Research
Role
Co Investigator
Start Date
End Date

Computer Aid for the Decision to Biopsy Breast Lesions

Administered By
Radiology
Awarded By
National Institutes of Health
Role
Investigator
Start Date
End Date

Publications:

A new method to accurately identify single nucleotide variants using small FFPE breast samples.

Most tissue collections of neoplasms are composed of formalin-fixed and paraffin-embedded (FFPE) excised tumor samples used for routine diagnostics. DNA sequencing is becoming increasingly important in cancer research and clinical management; however it is difficult to accurately sequence DNA from FFPE samples. We developed and validated a new bioinformatic pipeline to use existing variant-calling strategies to robustly identify somatic single nucleotide variants (SNVs) from whole exome sequencing using small amounts of DNA extracted from archival FFPE samples of breast cancers. We optimized this strategy using 28 pairs of technical replicates. After optimization, the mean similarity between replicates increased 5-fold, reaching 88% (range 0-100%), with a mean of 21.4 SNVs (range 1-68) per sample, representing a markedly superior performance to existing tools. We found that the SNV-identification accuracy declined when there was less than 40 ng of DNA available and that insertion-deletion variant calls are less reliable than single base substitutions. As the first application of the new algorithm, we compared samples of ductal carcinoma in situ of the breast to their adjacent invasive ductal carcinoma samples. We observed an increased number of mutations (paired-samples sign test, P < 0.05), and a higher genetic divergence in the invasive samples (paired-samples sign test, P < 0.01). Our method provides a significant improvement in detecting SNVs in FFPE samples over previous approaches.
Authors
Fortunato, A; Mallo, D; Rupp, SM; King, LM; Hardman, T; Lo, JY; Hall, A; Marks, JR; Hwang, ES; Maley, CC
MLA Citation
Fortunato, Angelo, et al. “A new method to accurately identify single nucleotide variants using small FFPE breast samples.Brief Bioinform, vol. 22, no. 6, Nov. 2021. Pubmed, doi:10.1093/bib/bbab221.
URI
https://scholars.duke.edu/individual/pub1485970
PMID
34117742
Source
pubmed
Published In
Brief Bioinform
Volume
22
Published Date
DOI
10.1093/bib/bbab221

A Data Set and Deep Learning Algorithm for the Detection of Masses and Architectural Distortions in Digital Breast Tomosynthesis Images.

Importance: Breast cancer screening is among the most common radiological tasks, with more than 39 million examinations performed each year. While it has been among the most studied medical imaging applications of artificial intelligence, the development and evaluation of algorithms are hindered by the lack of well-annotated, large-scale publicly available data sets. Objectives: To curate, annotate, and make publicly available a large-scale data set of digital breast tomosynthesis (DBT) images to facilitate the development and evaluation of artificial intelligence algorithms for breast cancer screening; to develop a baseline deep learning model for breast cancer detection; and to test this model using the data set to serve as a baseline for future research. Design, Setting, and Participants: In this diagnostic study, 16 802 DBT examinations with at least 1 reconstruction view available, performed between August 26, 2014, and January 29, 2018, were obtained from Duke Health System and analyzed. From the initial cohort, examinations were divided into 4 groups and split into training and test sets for the development and evaluation of a deep learning model. Images with foreign objects or spot compression views were excluded. Data analysis was conducted from January 2018 to October 2020. Exposures: Screening DBT. Main Outcomes and Measures: The detection algorithm was evaluated with breast-based free-response receiver operating characteristic curve and sensitivity at 2 false positives per volume. Results: The curated data set contained 22 032 reconstructed DBT volumes that belonged to 5610 studies from 5060 patients with a mean (SD) age of 55 (11) years and 5059 (100.0%) women. This included 4 groups of studies: (1) 5129 (91.4%) normal studies; (2) 280 (5.0%) actionable studies, for which where additional imaging was needed but no biopsy was performed; (3) 112 (2.0%) benign biopsied studies; and (4) 89 studies (1.6%) with cancer. Our data set included masses and architectural distortions that were annotated by 2 experienced radiologists. Our deep learning model reached breast-based sensitivity of 65% (39 of 60; 95% CI, 56%-74%) at 2 false positives per DBT volume on a test set of 460 examinations from 418 patients. Conclusions and Relevance: The large, diverse, and curated data set presented in this study could facilitate the development and evaluation of artificial intelligence algorithms for breast cancer screening by providing data for training as well as a common set of cases for model validation. The performance of the model developed in this study showed that the task remains challenging; its performance could serve as a baseline for future model development.
Authors
Buda, M; Saha, A; Walsh, R; Ghate, S; Li, N; Swiecicki, A; Lo, JY; Mazurowski, MA
MLA Citation
Buda, Mateusz, et al. “A Data Set and Deep Learning Algorithm for the Detection of Masses and Architectural Distortions in Digital Breast Tomosynthesis Images.Jama Netw Open, vol. 4, no. 8, Aug. 2021, p. e2119100. Pubmed, doi:10.1001/jamanetworkopen.2021.19100.
URI
https://scholars.duke.edu/individual/pub1494586
PMID
34398205
Source
pubmed
Published In
Jama Network Open
Volume
4
Published Date
Start Page
e2119100
DOI
10.1001/jamanetworkopen.2021.19100

Multimodal Patient-Specific Registration for Breast Imaging Using Biomechanical Modeling with Reference to AI Evaluation of Breast Tumor Change.

<h4>Background</h4>The strategy to combat the problem associated with large deformations in the breast due to the difference in the medical imaging of patient posture plays a vital role in multimodal medical image registration with artificial intelligence (AI) initiatives. How to build a breast biomechanical model simulating the large-scale deformation of soft tissue remains a challenge but is highly desirable.<h4>Methods</h4>This study proposed a hybrid individual-specific registration model of the breast combining finite element analysis, property optimization, and affine transformation to register breast images. During the registration process, the mechanical properties of the breast tissues were individually assigned using an optimization process, which allowed the model to become patient specific. Evaluation and results:&nbsp;The proposed method has been extensively tested on two datasets collected from two independent institutions, one from America and another from Hong Kong.<h4>Conclusions</h4>Our method can accurately predict the deformation of breasts from the supine to prone position for both the Hong Kong and American samples, with a small target registration error of lesions.
Authors
Xue, C; Tang, F-H; Lai, CWK; Grimm, LJ; Lo, JY
MLA Citation
Xue, Cheng, et al. “Multimodal Patient-Specific Registration for Breast Imaging Using Biomechanical Modeling with Reference to AI Evaluation of Breast Tumor Change.Life (Basel, Switzerland), vol. 11, no. 8, July 2021. Epmc, doi:10.3390/life11080747.
URI
https://scholars.duke.edu/individual/pub1489690
PMID
34440490
Source
epmc
Published In
Life (Basel)
Volume
11
Published Date
DOI
10.3390/life11080747

Mask Embedding in conditional GAN for Guided Synthesis of High
Resolution Images

Recent advancements in conditional Generative Adversarial Networks (cGANs) have shown promises in label guided image synthesis. Semantic masks, such as sketches and label maps, are another intuitive and effective form of guidance in image synthesis. Directly incorporating the semantic masks as constraints dramatically reduces the variability and quality of the synthesized results. We observe this is caused by the incompatibility of features from different inputs (such as mask image and latent vector) of the generator. To use semantic masks as guidance whilst providing realistic synthesized results with fine details, we propose to use mask embedding mechanism to allow for a more efficient initial feature projection in the generator. We validate the effectiveness of our approach by training a mask guided face generator using CELEBA-HQ dataset. We can generate realistic and high resolution facial images up to the resolution of 512*512 with a mask guidance. Our code is publicly available.
Authors
Ren, Y; Zhu, Z; Li, Y; Lo, J
URI
https://scholars.duke.edu/individual/pub1395859
Source
arxiv

Retina-Match: Ipsilateral Mammography Lesion Matching in a Single Shot Detection Pipeline

In mammography and tomosynthesis, radiologists use the geometric relationship of the four standard screening views to detect breast abnormalities. To date, computer aided detection methods focus on formulations based only on a single view. Recent multi-view methods are either black box approaches using methods such as relation blocks, or perform extensive, case-level feature aggregation requiring large data redundancy. In this study, we propose Retina-Match, an end-to-end trainable pipeline for detection, matching, and refinement that can effectively perform ipsilateral lesion matching in paired screening mammography images. We demonstrate effectiveness on a private, digital mammography data set with 1,016 biopsied lesions and 2,000 negative cases.
Authors
Ren, Y; Lu, J; Liang, Z; Grimm, LJ; Kim, C; Taylor-Cho, M; Yoon, S; Marks, JR; Lo, JY
MLA Citation
Ren, Y., et al. “Retina-Match: Ipsilateral Mammography Lesion Matching in a Single Shot Detection Pipeline.” Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12905 LNCS, 2021, pp. 345–54. Scopus, doi:10.1007/978-3-030-87240-3_33.
URI
https://scholars.duke.edu/individual/pub1499563
Source
scopus
Published In
Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume
12905 LNCS
Published Date
Start Page
345
End Page
354
DOI
10.1007/978-3-030-87240-3_33

Research Areas:

Breast Neoplasms
Clinical Trials as Topic
Computer Simulation
Decision Making, Computer-Assisted
Decision Support Systems, Clinical
Decision Support Techniques
Image Processing, Computer-Assisted
Imaging, Three-Dimensional
Machine learning
Mammography
Models, Structural
Pattern Recognition, Automated
Radiographic Image Interpretation, Computer-Assisted
Radiology
Technology Assessment, Biomedical
Tomosynthesis