An Efficient Clustering System for the Measure of Page (Document) Authoritativeness

Abstract

A collection of documents D1 of a search result R1 is a cluster if all the documents in D1 are similar in a way and dissimilar to another collection say D2 for a given query Q1. Implying that, given a new query Q2, the search result R2 may pose an intersection or a union of documents from D1 and D2 or more to form D3. However within these collections say D1, D2, D3 etc, one or two pages certainly would be better in relevance to the query that invokes them. Such a page is regarded being ‘authoritative’ than others. Therefore in a query context, a given search result has pages of authority. The most important measure of a search engine’s efficiency is the quality of its search results. This work seeks to cluster search results to ease the matching of searched documents with user’s need by attaching a page authority value (pav). We developed a classifier that falls in the margin of supervised and unsupervised learning which would be computationally feasible and producing most authoritative pages. A novel searching and clustering engine was developed using several measure-factors such as anchor text, proximity, page rank, and features of neighbors to rate the pages so searched. Documents or corpora of known measures from the Text Retrieval Conference (TREC), the Initiative for the Evaluation of XML Retrieval (INEX) and Reuter’s Collection, were fed into our work and evaluated comparatively with existing search engines (Google, VIVISIMO and Wikipedia). We got very impressive results based on our evaluation. Additionally, our system could add a value – pav to every searched and classified page to indicate a page’s relevance over the other. A document is a good match to a query if the document model is likely to generate the query, which will in turn happen if the document contains the query words often. This approach thus provides a different realization of some of the basic ideas for document ranking which could be applied through some acceptable rules: number of occurrence, document zone and relevance measures. The biggest problem facing users of web search engines today is the quality of the results they get back. While the results are often amusing and expand users' horizons, they are often frustrating and consume precious time. We have made available a better page ranker that do not depend heavily on the page developer’s inflicted weights but considers the actual factors within and without the target page. Though very experimental on research collections, the user can within the collection of the first ten search results listing, extract his or her relevant pages with ease. Keywords: page Authoritativeness, page Rank, search results, clustering algorithm, web crawling.