The National Center for Biomedical Ontology is developing a system for automated, ontology-based access to online biomedical resources. The system’s indexing workflow processes the text metadata of diverse resources such as datasets from GEO and ArrayExpress to annotate and index them with concepts from appropriate ontologies. This indexing requires the use of a concept-recognition system to identify the presence of ontology concepts in the resource metadata. In this paper, we present a comprehensive comparison of two concept recognizers – NIH’s MetaMap and the University of Michigan’s MGREP. We utilize a number of data sources and dictionaries to evaluate the concept recognizers in terms of precision, recall, speed of execution, scalability and customizability. Our evaluations demonstrate that MGREP has a clear edge over MetaMap for large-scale applications. Based on our analysis we also suggest areas of potential improvements for MGREP.
