Polibits, Vol. 48, pp. 79-83, 2013.
Abstract: The ever-increasing web is an important source for building large-scale corpora. However, dynamically generated web pages often contain much irrelevant and duplicated text, which impairs the quality of the corpus. To ensure the high quality of web-based corpora, a good boilerplate removal algorithm is needed to extract only the relevant content from web pages. In this article, we present an automatic text extraction procedure, GoldMiner, which by enhancing a previously published boilerplate removal algorithm, minimizes the occurrence of irrelevant duplicated content in corpora, and keeps the text more coherent than previous tools. The algorithm exploits similarities in the HTML structure of pages coming from the same domain. A new evaluation document set (CleanPortalEval) is also presented, which can demonstrate the power of boilerplate removal algorithms for web portal pages.
Keywords: Corpus building, boilerplate removal, the web as corpus
PDF: More Effective Boilerplate Removal - the GoldMiner Algorithm
PDF: More Effective Boilerplate Removal - the GoldMiner Algorithm