TMRGM: A Template-Based Multi-Attention Model for X-Ray Imaging Report Generation

Authors

  • Xuwen Wang Institute of Medical Information and Library, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China https://orcid.org/0000-0003-3022-6513
  • Yu Zhang Institute of Medical Information and Library, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China
  • Zhen Guo Institute of Medical Information and Library, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China https://orcid.org/0000-0002-7454-0750
  • Jiao Li Institute of Medical Information and Library, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, China https://orcid.org/0000-0001-6391-8343

DOI:

https://doi.org/10.2991/jaims.d.210428.002

Keywords:

Chest X-ray, Deep learning, Thoracic abnormality recognition, Medical imaging report generation, Attention mechanism, Medical imaging report template

Abstract

The rapid growth of medical imaging data brings heavy pressure to radiologists for imaging diagnosis and report writing. This paper aims to extract valuable information automatically from medical images to assist doctors in chest X-ray image interpretation. Considering the different linguistic and visual characteristics in reports of different crowds, we proposed a template-based multi-attention report generation model (TMRGM) for the healthy individuals and abnormal ones respectively. In this study, we developed an experimental dataset based on the IU X-ray collection to validate the effectiveness of TMRGM model. Specifically, our method achieves the BLEU-1 of 0.419, the METEOR of 0.183, the ROUGE score of 0.280, and the CIDEr of 0.359, which are comparable with the SOTA models. The experimental results indicate that the proposed TMRGM model is able to simulate the reporting process, and there is still much room for improvement in clinical application.

References

Interagency Working Group on Medical Imaging Committee on Science, National Science and Technology Council, Roadmap for Medical Imaging Research and Development, Washington D.C., USA, 2017, pp. 1-19. https://trumpwhitehouse.archives.gov/wp-content/uploads/2017/12/Roadmap-for-Medical-Imaging-Research-and Development-2017.pdf

Y. LeCun and Y. Bengio, Convolutional networks for images, speech, and time series, M.A. Arbib (editor), The Handbook of Brain Theory and Neural Networks, MIT Press, Cambridge, MA, USA, Vol. 3361, 1995. http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.32.9297

G.S. Lodwick, Computer-aided diagnosis in radiology. A research plan, Invest. Radiol., Vol. 1, 1966, pp. 72-80.

Z.C. Lipton, J. Berkowitz, and C. Elkan, A critical review of recurrent neural networks for sequence learning, in Computer Science, 2015. arXiv:1506.00019v4

Published

2021-05-05

How to Cite

1.
Wang X, Zhang Y, Guo Z, Li J. TMRGM: A Template-Based Multi-Attention Model for X-Ray Imaging Report Generation. JAIMS [Internet]. 2021 May 5 [cited 2023 Feb. 4];2(1-2):21-32. Available from: http://oapublishing-jaims.com/jaims/article/view/60