Saved in:
| Main Author: | |
|---|---|
| Format: | Artículo científico |
| Language: | en |
| Published: |
Instituto Politécnico Nacional
2013
|
| Subjects: | |
| Online Access: | https://www.redalyc.org/articulo.oa?id=402640462011 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Table of Contents:
- More Effective Boilerplate Removal—the GoldMiner Algorithm István Endrédy Attila Novák Computación Corpus building the web as corpus boilerplate removal The ever-increasing web is an important source forbuilding large-scale corpora. However, dynamically generatedweb pages often contain much irrelevant and duplicated text,which impairs the quality of the corpus. To ensure the highquality of web-based corpora, a good boilerplate removalalgorithm is needed to extract only the relevant contentfrom web pages. In this article, we present an automatictext extraction procedure, GoldMiner, which by enhancing apreviously published boilerplate removal algorithm, minimizesthe occurrence of irrelevant duplicated content in corpora,and keeps the text more coherent than previous tools. Thealgorithm exploits similarities in the HTML structure of pagescoming from the same domain. A new evaluation document set(CleanPortalEval) is also presented, which can demonstrate thepower of boilerplate removal algorithms for web portal pages. 2013 artículo científico 1870-9044 https://www.redalyc.org/articulo.oa?id=402640462011 en http://www.redalyc.org/revista.oa?id=4026 Polibits application/pdf Instituto Politécnico Nacional Polibits (México) Vol.48