Cross-attention-based late fusion network for medical visual question answering from radiology images | |
| Author | Lameesa, Aiman |
| Call Number | AIT Thesis no.DSAI-23-13 |
| Subject(s) | Question-answering systems Computer vision Information visualization |
| Note | A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Data Science and Artificial Intelligence |
| Publisher | Asian Institute of Technology |
| Abstract | Image and question matching is greatly important in Medical Visual Question Answering (MVQA) in order to accurately assess the visual-semantic correspondence between a radiology image and a question. However, the recent state-of-the-art methods focus solely on the contrastive learning between entire image and question words. Though contrastive learning successfully model the global relationship between an image and a question, it cannot capture the fine-grained alignments conveyed between them image re gions and question words. To address this limitation, we propose a novel Cross-Attention based Late Fusion (CALF) network in MVQA tasks by combining image and question features in a unified deep model. In our proposed approach, we use self-attention to ef fectively leverage intra-modal relationships within each modality and implement cross attention to emphasize the inter-modal associations between image regions and ques tion words. By simultaneously considering intra-modal and inter-modal relationships, our proposed method significantly improves the performance of MVQA. Experimental results on benchmark datasets, such as, SLAKE and VQA-RAD demonstrate that our proposed approach outperforms the existing methods. |
| Year | 2023 |
| Type | Thesis |
| School | School of Engineering and Technology |
| Department | Department of Information and Communications Technologies (DICT) |
| Academic Program/FoS | Data Science and Artificial Intelligence (DSAI) |
| Chairperson(s) | Chaklam Silpasuwanchai; |
| Examination Committee(s) | Mongkol Ekpanyapong; |
| Scholarship Donor(s) | AIT Scholarships; |
| Degree | Thesis (M. Sc.) - Asian Institute of Technology, 2023 |