International Journal of Medical Informatics 64 (2001) 341 – 354 Validation of a knowledge based reminder system for diagnostic test ordering in general practice Rianne Bindels a, *, Ron A.G. Winkens b,c , Peter Pop b , Jan W.J. van Wersch d , Jan Talmon a , Arie Hasman a a Department of Medical Informatics, Maastricht Uniersity, P.O. Box 616, 6200 MD Maastricht, The Netherlands b Transmural and Diagnostic Center, Uniersity Hospital Maastricht, P.O. Box 5800, 6202 AZ Maastricht, The Netherlands c Department of General Practice, Maastricht Uniersity, P.O. Box 616, 6200 MD Maastricht, The Netherlands d Atrium Medical Center Heerlen, Heerlen, P.O. Box 4446, 6401 CX Heerlen, The Netherlands Abstract We describe the validation of a real-time automated reminder system that assists General Practitioners (GP) in appropriate test ordering. We compared the comments of human experts with the comments of the reminder system using a retrospective random selection of 253 request forms. A panel of three expert physicians judged the requested tests independently based on their interpretations of the practice guidelines. The majority assessment of the physicians was compared with the assessment of the reminder system. In case the system’s output differed from the majority assessment the written practice guidelines were consulted. On average 1.75 reminders were produced per form. In total 32 of the 442 given reminders (7%) were given incorrectly. The amount of information and the level of detail (the specificity of the terms) in which the GP describes the patients’ medical status are crucial for the reminder system to react correctly. © 2001 Elsevier Science Ireland Ltd. All rights reserved. Keywords: Clinical decision support; Primary health care; Practice guidelines; Test ordering; Reminder systems; Validation; Knowledge base www.elsevier.com/locate/ijmedinf 1. Introduction The validity and reliability of peer review of medical practice is known to be question- able due to the variability in assessment of the peer reviewers [1,2]. Usually there is one reviewer. Peer review can be expanded for example, by using multiple reviewers in the process, or by guidelines and standards or computerized decision support [3]. The use of knowledge based decision sup- port systems is becoming more and more popular in medicine [4–8]. These systems are developed to assist physicians in making medical decisions to provide better care. To manage the quest for diagnostic certainty [9] and to improve appropriateness of diagnostic test requests the GRIF automated reminder * Corresponding author. Tel.: +31-43-3882295; fax: +31- 43-3884170. E-mail address: r.bindels@mi.unimaas.nl (R. Bindels). 1386-5056/01/$ - see front matter © 2001 Elsevier Science Ireland Ltd. All rights reserved. PII:S1386-5056(01)00207-6