saradoc

Different URLs with Similar Text. Such duplicate URLs are prevalent in Web sites, as Web server software often uses aliases and redirections, and dynamically generates the same page from various different URL requests. The Algorithm for discovering rules that transform a given URL to others that are likely to have similar content. DUST Buster mines DUST effectively from previous crawl logs or Web server logs Verifying these rules via sampling requires fetching few actual Web pages. Search engines can benefit from information about DUST to increase the effectiveness of crawling, reduce indexing overhead, and improve the quality of popularity statistics such as Page Rank. My Responsibilities § Responsible for this project right from requirements to implementation. § Code Reviews, Unit & Integration testing. Duration From: JAN/10 To: JUN/10 Software Dreamweaver, JAVA 2.0, MySQL, My Eclipse, Apache Tomcat