An Automated Algorithm for Extracting Website Skeleton
Conference Proceeding Article
The huge amount of information available on the Web has attracted many research efforts into developing wrappers that extract data from webpages. However, as most of the systems for generating wrappers focus on extracting data at page-level, data extraction at site-level remains a manual or semi-automatic process. In this paper, we study the problem of extracting website skeleton, i.e. extracting the underlying hyperlink structure that is used to organize the content pages in a given website. We propose an automated algorithm, called the Sew algorithm, to discover the skeleton of a website. Given a page, the algorithm examines hyperlinks in groups and identifies the navigation links that point to pages in the next level in the website structure. The entire skeleton is then constructed by recursively fetching pages pointed by the discovered links and analyzing these pages using the same process. Our experiments on real life websites show that the algorithm achieves a high recall with moderate precision.
Databases and Information Systems | Numerical Analysis and Scientific Computing
Data Management and Analytics
Database Systems for Advanced Applications: 9th International Conference, DASFAA 2004, Jeju Island, Korea, March 17-19, 2003: Proceedings
City or Country
LIU, Zehua; NG, Wee-Keong; and LIM, Ee Peng.
An Automated Algorithm for Extracting Website Skeleton. (2004). Database Systems for Advanced Applications: 9th International Conference, DASFAA 2004, Jeju Island, Korea, March 17-19, 2003: Proceedings. 2973, 799-811. Research Collection School Of Information Systems.
Available at: http://ink.library.smu.edu.sg/sis_research/1040