A major factor in the recent success of large language models is the use of enormous and
ever-growing text datasets for unsupervised pre-training. However, naively training a model …

Information about pretraining corpora used to train the current best-performing language
models is seldom discussed: commercial models rarely detail their data, and even open …

This is a survey of the science and practice of web crawling. While at first glance web
crawling may appear to be merely an application of breadth-first-search, the truth is that …

The amount of near-duplicates in web crawls like the ClueWeb or Common Crawl demands
from their users either to develop a preprocessing pipeline for deduplication, which is costly …

Presence of duplicate documents in the World Wide Web adversely affects crawling,
indexing and relevance, which are the core building blocks of web search. In this paper, we …

Duplicate URLs have brought serious troubles to the whole pipeline of a search engine,
from crawling, indexing, to result serving. URL normalization is to transform duplicate URLs …

Recently, Large Language Models (LLMs) have undergone a significant transformation,
marked by a rapid rise in both their popularity and capabilities. Leading this evolution are …

The Internet has witnessed the proliferation of applications and services that rely on HTTP
as application protocol. Users play games, read emails, watch videos, chat and access web …

With the push for transparency and open data, many datasets and data repositories are
becoming available on the Web. This opens new opportunities for data-driven exploration …

The Semantic Web is constantly gaining momentum, as more and more Web sites and
content providers adopt its principles. At the core of these principles lies the Linked Data …