Information retrieval (IR) has changed considerably in recent years with the expansion of the
World Wide Web and the advent of modern and inexpensive graphical user interfaces and …

We report our experience in implementing UbiCrawler, a scalable distributed Web crawler,
using the Java programming language. The main features of UbiCrawler are platform …

The key factors for the success of the World Wide Web are its large size and the lack of a
centralized control over its contents. Both issues are also the most important source of …

This article compares several page ordering strategies for Web crawling under several
metrics. The objective of these strategies is to download the most" important" pages" early" …

Continuous growth of the Web and user bases forces web search engine companies to
make costly investments on very large compute infrastructures. The scalability of these …

In this paper, we present a characterization study of search-engine crawlers. For the
purposes of our work, we use Web-server access logs from five academic sites in three …

Web crawling has become an important aspect of web search, as the WWW keeps getting
bigger and search engines strive to index the most important and up to date content. Many …

Search Engine for South-East Europe (SE4SEE) is a socio-cultural search engine running
on the grid infrastructure. It offers a personalized, on-demand, country-specific, category …

Centralized crawlers are not adequate to spider meaningful and relevant portions of the
Web. A crawler with good scalability and load balancing can bring growth to performance …

Intermediaries are software entities, deployed on hosts of the wireline and wireless network,
that mediate the interaction between clients and servers of the World Wide Web. In this …