Employing UMLS as an ontology to achieve robust reasoning in an intelligent tutoring system for medical problem-based learning | |
| Author | Kazi, Hameedullah |
| Call Number | AIT Diss. no.CS-10-04 |
| Subject(s) | Intelligent tutoring systems Medical care--Study and teaching |
| Note | A dissertation submitted in partial fulfillment of the requirements towards the degree of Doctor of Philosophy in Computer Science, School of Engineering and Technology |
| Publisher | Asian Institute of Technology |
| Series Statement | Dissertation ; no. CS-10-04 |
| Abstract | As knowledge based systems, the problem solving in intelligent tutoring systems comprises two tasks: evaluating student solutions and returning feedback. Both tasks are susceptible to brittleness. Tutoring systems typically contain a set of approved solution for a given problem scenario. Student solutions are evaluated by comparing them against the set of approved solutions. Plausible solutions, that don't match the approved set, but are otherwise acceptable of close to acceptable, are rejected by the system as being incorrect. hints generated by the system are also typically tailored in such a way that they are effective only within the knowledge confines of the approved solution. Student hypotheses that don't match the approved solution but are partially correct, receive little acknowledgment from the system as feedback. This confines the reasoning of the student to the narrow set of hypotheses instructed by the system, stifling a broader reasoning that may very well be applicable to the problem presented. Additionally, the hint generation mechanism relies on having the student model, which requires extensive effort to build, leading to the problem of knowledge acquisition bottleneck. A robust tutoring system should allow students a diverse range of domain concepts, assess their solution in the context of broad knowledge and steer them towards a correct solution if they deviate, The system should relate the student solution to a correct solution and provide feedback thai is relevant to the context of the problem solving activity. Yet at the same time, the development of such an system should not place extensive burden on knowledge acquisition. This dissertation addresses the issue of robustness in tutoring systems and describes a system implementation in the domain of medical problem based learning. We describe the design of the system METEOR (Medical Tutor Employing Ontology for Robustness), inspired by its predecessor COMET, which is a medical tutoring system for collaborative problem based learning, developed and representing a subset of an existing and widely available medical knowledge source, UMLS(Unified Medical Language System) as the domain ontology in the METEOR tutoring system. Solutions to problem scenarios are collected from domain experts and are combined with tables in the UMLS to form the domain mode. The concept hierarchy and relationships among concepts in the UMLS are exploited for inference purposes. Through the inference mechanism, the system is able to expand the solution space and accept a greater variety of plausible student solutions beyond the scope of the explicitly encoded ones. The inference mechanism also enables the system to assess the partial correctness of student solutions and return acknowledgment as feedback, The concept hierarchy in the domain ontology is leveraged off to generate hints relevant to the context of the student problem solving activity, without recourse to an explicit student model. The use of an existing knowledge source to facilitate assessment and generating feedback also eases the knowledge acquisition bottleneck. Evaluation of the system accuracy in accepting inferred plausible solutions indicates accuracy close to that of human experts, who agreed among themselves with Pearson Correlation Coefficient of 0.48 and P<0.05. As a result of expanding the plausible solution space through inference, the system precision in accepting correct solutions dropped by 32% while the recall increased by a factor of five, compared to the system that only accepted explicitly encoded solutions. Furthermore, the geometric mean of sensitivity and specificity was increased by 0.33. Evaluation of expert agreement with system generated hints on a 5-point likert scale resulted in an average score of 4.44 (r=0.9018, p < 0.05). Hints containing partial correctness feedback scored significantly higher than those without it (Mann-Whitney, p < 0.001). Evaluation of student learning outcomes led to highly significant learning gains (Mann-Whitney, p < 0.001), which outperformed those obtained through the predecessor COMET system. |
| Year | 2010 |
| Corresponding Series Added Entry | Asian Institute of Technology. Dissertation ; no. CS-10-04 |
| Type | Dissertation |
| School | School of Engineering and Technology (SET) |
| Department | Department of Information and Communications Technologies (DICT) |
| Academic Program/FoS | Computer Science (CS) |
| Chairperson(s) | Haddawy, Peter; |
| Examination Committee(s) | Janecek, Paul;Siriwan Suebnukarn;Manukid Parnichkun; |
| Degree | Thesis (Ph.D.) - Asian Institute of Technology, 2010 |