Using Large Text to Image Models with Structured Prompts for Skin Disease Identification: A Case Study

Sajith Rajapaksa, Jean Marie Uwabeza Vianney, Renell Castro, Farzad Khalvati, Shubhra Aich

ICCVW 2023 pp. 2686-2693

doi:10.1109/ICCVW60793.2023.00284 /iccvw/2023/rajapaksa2023iccvw-using/

Abstract

This paper investigates the potential usage of large text-to-image (LTI) models for the automated diagnosis of a few skin conditions with rarity or a severe lack of annotated datasets. As the input to the LTI model, we provide the targeted instantiation of a generic but succinct prompt structure designed upon careful observations of the conditional narratives from the standard medical textbooks. In this regard, we pave the path to utilizing accessible textbook descriptions for automated diagnosis of conditions with data scarcity through the lens of LTI models. Experiments show the efficacy of the proposed framework, including much better localization of the infected regions. Moreover, it has the immense possibility for generalization across the medical sub-domains to mitigate the data scarcity issue, and debias automated diagnostics from the all-pervasive racial biases.

PDF ICCVW Semantic Scholar

Cite

Text

Rajapaksa et al. "Using Large Text to Image Models with Structured Prompts for Skin Disease Identification: A Case Study." IEEE/CVF International Conference on Computer Vision Workshops, 2023. doi:10.1109/ICCVW60793.2023.00284

Markdown

[Rajapaksa et al. "Using Large Text to Image Models with Structured Prompts for Skin Disease Identification: A Case Study." IEEE/CVF International Conference on Computer Vision Workshops, 2023.](https://mlanthology.org/iccvw/2023/rajapaksa2023iccvw-using/) doi:10.1109/ICCVW60793.2023.00284

BibTeX

@inproceedings{rajapaksa2023iccvw-using,
  title     = {{Using Large Text to Image Models with Structured Prompts for Skin Disease Identification: A Case Study}},
  author    = {Rajapaksa, Sajith and Vianney, Jean Marie Uwabeza and Castro, Renell and Khalvati, Farzad and Aich, Shubhra},
  booktitle = {IEEE/CVF International Conference on Computer Vision Workshops},
  year      = {2023},
  pages     = {2686-2693},
  doi       = {10.1109/ICCVW60793.2023.00284},
  url       = {https://mlanthology.org/iccvw/2023/rajapaksa2023iccvw-using/}
}