Named Entity Recognition in Hindi using Maximum Entropy and Transliteration

Authors: Sujan Kumar Saha, Partha Sarathi Ghosh, Sudeshna Sarkar, Pabitra Mitra

Polibits, 38, pp. 33-42, 2008.

Abstract: Named entities are perhaps the most important indexing element in text for most of the information extraction and mining tasks. Construction of a Named Entity Recognition (NER) system becomes challenging if proper resources are not available. Gazetteer lists are often used for the development of NER systems. In many resource-poor languages gazetteer lists of proper size are not available, but sometimes relevant lists are available in English. Proper transliteration makes the English lists useful in the NER tasks for such languages. In this paper, we have described a Maximum Entropy based NER system for Hindi. We have explored different features applicable for the Hindi NER task. We have incorporated some gazetteer lists in the system to increase the performance of the system. These lists are collected from the web and are in English. To make these English lists useful in the Hindi NER task, we have proposed a two-phase transliteration methodology. A considerable amount of performance improvement is observed after using the transliteration based gazetteer lists in the system. The proposed transliteration based gazetteer preparation methodology is also applicable for other languages. Apart from Hindi, we have applied the transliteration approach in Bengali NER task and also achieved performance improvement.

Keywords: Gazetteer list preparation; named entity recognition; natural language processing; transliteration

PDF: Named Entity Recognition in Hindi using Maximum Entropy and Transliteration, Alternative link