System Evaluation

The effectiveness of the proposed system is measured by precision and recall [28]. Here, precision is the ratio of correctly extracted medical concepts out of all extracted medical concepts. Recall is the ratio of correctly extracted medical concepts out of all medical concepts that should have been extracted. Precision and recall for the proposed system is calculated as:

Results and Discussion

Based on the concept of precision and recall given in Equations 8.1 and 8.2, the matrix-based proposed system generates approximately a 60% recall and a 30% precision for all concepts collectively (test, problem, treatment). The precision of the system is reduced because it uses the concept of parallel pattern matching between two matrices and generates many non-medical entities also. To improve the precision of the system, some post-processing rules are introduced, after which precision is increased to 70% approximately. A comparison of precision between before and after post-processing is given for Beth and Partners data in Figure 8.2. The performance of the system is measured for every concept (problem, test, and treatment) separately and compared with the Partners and Beth results (see Figure 8.3). It is observed that the precision of the Beth data is greater than the Partners data for the problem and treatment concepts, but for the test concept it is equal. It is found that the recall of the Partners dataset is higher than for the Beth dataset for every concept. The system is compared with different methods, such as rule based [13], dictionary look-up [3], the chunker module of MetaMap 2010 [4], and MetaMap 2013 version [13]. The precision and recall of the proposed system is better than these methods (see Figures 8.4 and 8.5).

FIGURE 8.2 Comparison of precision before and after post-processing rules.

FIGURE 8.3 Comparison of performance of proposed method for treatment, problem, and test concepts based on precision and recall.

FIGURE 8.4 Comparison between precision of different text classification methods.

After conducting error analysis on the results, it is observed that improvement in recall is possible to some extent. A few concepts are missing because of the inaccurate boundary identification of sequences of entities. For frequent pattern generation, the minimum threshold value is 2 and patterns those are having frequency 2 or more than 2, they have considered. But frequency 1 patterns are missed and the corresponding words were not extracted. In this system, only the POS feature was focused. In future work, the same system could be developed by using more features, like word, n-gram, semantics of words, lemma, prefix, and suffix.

Boundary identification of multiword clinical concepts is the big issue [29] which is principally found in the problem and treatment concepts because the gold-standard

FIGURE 8.5 Comparison between recall of different text classification methods.

dataset contains a large sequence of entities in problem concepts like “minimal right internal carotid artery disease” or “thrombosis of the right renal artery.” A large sequence of entities is also found in treatment concepts like “a left carotid endarterectomy,” “oxycodone-acetaminophen,” “saphaneous vein graft —► posterior descending artery” or “a permanent dual chamber rate responsive pacemaker." The system performed strict matching; in future, partial matching could be performed which might increase the recall of the system. The pruning of non-medical concepts w'as done using a UMLS database, but large sequences of entities are not correctly matched in this database to medical words. This issue could be determined using relaxed matching of clinical concepts.

Text pre-processing included regular expressions for the identification of special characters, w hich are treated as word splitters, though only some words are identified in the gold-standard data which included these characters in between. Examples of composite words such as “severe 3 vessel disease,” “saphaneous vein graft —► obtuse marginal,” “the patient’s pacing wires,” “heel/shin,” and “leg, emg & apos” are inaccurately predicted as concepts using the anticipated method.

Conclusions and Future Work

In the proposed solution, medical concepts, such as signs and symptoms of diseases, tests, and treatments, are identified using parallel pattern matching of two matrices followed by a rule-based method. The pruning of non-medical concepts increased the performance of the system by using concept mapping with semantic types of UMLS. It is noted that the efficiency of the system is improved over MetaMap 2010, MetaMap 2013v2, and dictionary look-up methods. In future, recall could be further improved by adding new' features like prefix and suffix, stemming, word, semantics of w'ords, or n-gram in the same method. Machine learning concepts are not used, which is why the performance of the system is not dependent on the size of the dataset. The proposed system could be implemented on small and large datasets in an identical way.

but the processing time for large datasets would increase. In future work, for the fast processing of large text, the proposed method could be implemented in a distributed environment.

Acknowledgments

We would like to thank the organizers of the 2010 i2b2 challenge for designing, training, and evaluating corpora. We also wish to thank the U.S. National Library of Medicine for allowing us to access clinical work through UMLS.

References

• 1. Jiang, M., Chen. Y., Liu, M., Rosenbloom. S.T., Mani, S.. Denny, J.C.. Xu. H.: A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries. Journal of the American Medical Informatics Association: JAMIA 18, 601-606 (2011).
• 2. Kim, Y.. Riloff. E.: A stacked ensemble for medical concept extraction from clinical notes. AMIA Jt Summits on Translational Science Proceedings 2015, 69-73 (2015).
• 3. Torii. M., Wagholikar, K., Liu, H.: Using machine learning for concept extraction on clinical documents from multiple data sources. Journal of the American Medical Informatics Association: JAMIA 18, 580-587 (2011).
• 4. Minard, A.-L.. Ligozat, A.-L., Ben Abacha, A.. Bernhard. D., Cartoni. B., Deleger. L., Grau, B., Rosset, S., Zweigenbaum, P.. Grouin. C.: Hybrid methods for improving information access in clinical documents: Concept, assertion, and relation identification. Journal of the American Medical Informatics Association 18, 588 (2011).
• 5. Dehghan. A.: Boundary identification of events in clinical named entity recognition. arXiv preprint arXiv 1308, 1004 (2013).
• 6. Kang, N.. Afzal, Z., Singh, B.. van Mulligen, E.M.. Kors. J.A.: Using an ensemble system to improve concept extraction from clinical records. Journal of Biomedical Informatics 45, 423-428 (2012).
• 7. Abacha, A.B.. Zweigenbaum, P: Medical entity recognition: A comparison of semantic and statistical methods. In: Proceedings of BioNLP 2011 Workshop, pp. 56-64. Association for Computational Linguistics, Portland, OR (2011).
• 8. Uzuner, O., South. B.R., Shen. S.. DuVall, S.L.: 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. Journal of the American Medical Informatics Association: JAMIA 18. 552-556 (2011).
• 9. Suominen. H.. Salantera, S., Velupillai, S., Chapman, W.W., Savova, G., Elhadad, N.. Pradhan, S.. South, B.R.. Mowery, D.L.. Jones, G.J.F., Leveling, J., Kelly. L., Goeuriot, L.. Martinez, D.. Zuccon. G.: Overview of the ShARe/CLEF eHealth evaluation lab 2013. In: Forner. P. Muller. H.. Paredes. R., Rosso, P, Stein, B. (eds.), Information Access Evaluation. Multilinguality, Multimodality, and Visualization: 4th International Conference of the CLEF Initiative, CLEF 2013, Valencia, Spain, September 23-26, 2013. Proceedings, pp. 212-231. Springer Berlin Heidelberg. Berlin (2013).
• 10. Kim, J.-D., Ohta, T.. Tsuruoka. Y.. Tateisi. Y., Collier, N.: Introduction to the bio-entity recognition task at JNLPBA. In: Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications Geneva. Switzerland (2004).
• 11. Hirschman, L., Yeh. A., Blaschke, C., Valencia, A.: Overview of BioCreAtlvE: Critical assessment of information extraction for biology. BMC Bioinformatics 6, SI (2005).
• 12. Yeh. A., Morgan, A.. Colosimo, M., Hirschman, L.: BioCreAtlvE task 1A: Gene mention finding evaluation. BMC Bioinformatics 6, S2 (2005).
• 13. Kim, Y., Riloff, E., Hurdle. J.F.: A study of concept extraction across different types of clinical notes. AMIA Annual Symposium Proceedings 2015, 737-746 (2015).