A website can contain hundreds of pages without any of them being visible from the main menu. Between orphan URLs, mistakenly blocked content, and pages buried several clicks deep, the actual mapping of a site rarely matches what its structure suggests.
Accessing all the pages of a website requires understanding how search engines, browsers, and configuration files interact to render (or hide) content.
Inaccessible page or simply unfound page: a technical distinction
When a page does not display, two very different situations coexist. The first: the page exists on the server, responds with an HTTP 200 code, but no internal link points to it. This is an orphan page, technically accessible if you know its URL, but invisible to crawlers and visitors.
The second: the page is blocked by a directive in the robots.txt file, protected by authentication, or returns a 403 or 404 code. In this case, even knowing the exact address, access is restricted or impossible.
The confusion between these two cases is common. A URL absent from Google search results is not necessarily deleted. It may simply be excluded from the index because a robots.txt file asks crawlers not to explore it. The available data does not always allow for a clear distinction without manual verification: you need to test the URL directly in a browser and then check the server logs to confirm the actual status of the page.
To illustrate this logic applied to a specific site, you can access all the pages of Le Site de Julia via its sitemap, which allows you to check which URLs are declared and which are missing from this list.
Robots.txt and RFC 9309: what these files really control
The robots.txt file is often misunderstood. Many consider it a security mechanism. In reality, robots.txt does not constitute an access control. It merely indicates to cooperative bots the paths they can or cannot explore.
A non-cooperative bot, or a human user, can ignore these directives and freely access the content. A URL forbidden in robots.txt can even appear in search results if discovered through external links. Sensitive content should be protected by server-side authentication.
Since the publication of RFC 9309 in September 2022, the Robots Exclusion Protocol has a formal framework. Key points to remember include:
- The rules apply to a specific host, protocol, and port, meaning that a robots.txt on port 443 does not cover port 80 of the same domain.
- In case of conflicting directives, the most specific rule prevails, with priority given to the Allow directive in case of equal specificity.
- A temporary outage or unavailability of the robots.txt file can produce opposite effects depending on the bots: some will suspend crawling, while others will consider everything allowed.
This last point is often overlooked. A server that returns a 500 error instead of the robots.txt file does not block access: it makes it unpredictable.
XML Sitemap: mapping a site without a guarantee of completeness
The XML sitemap is the most direct tool for listing the pages of a website. It is usually found at /sitemap.xml, and it can also be referenced in the robots.txt file via the Sitemap directive.
The sitemap facilitates the work of search engines by explicitly declaring the URLs to be indexed. However, a sitemap is never proof of completeness. Several reasons for this include:
- Pages may exist on the server without being included in the sitemap, either by oversight or because they were created after the last generation of the file.
- The Sitemap directive is not part of the core protocol defined by RFC 9309. A bot can therefore ignore it without violating the standard.
- Some CMS generate incomplete sitemaps that exclude types of content (taxonomy pages, archives, PDF documents).
To verify the actual coverage of a site, the only reliable method is to cross-reference the sitemap data with server logs and the HTTP responses obtained during a full crawl. SEO crawling tools traverse the structure of internal links and identify discrepancies between declared pages and those that are actually accessible.
Anti-bot controls and filtering by agent type: fragmented access to pages
The discovery of all the pages of a site is increasingly affected by anti-bot protection systems. Services like Cloudflare now allow administrators to distinguish search bots, automated agents, and AI training bots.
This segmentation creates a new situation: a site can remain perfectly visible in Google search results while blocking access to other types of crawlers. For an SEO professional trying to map a competitor’s site, or for a site owner checking their own coverage, the results of a crawl vary depending on the user-agent used by the crawling tool.
In practice, this means that a classic SEO crawling tool may be denied access to pages that Googlebot explores without difficulty. Field reports vary on this point: some crawlers bypass these restrictions by mimicking a standard browser user-agent, while others fail silently without indicating the blockage.
Checking the server access logs remains the most reliable method to confirm which bots actually access which pages. Without this verification, any list of pages remains partial.
Accessing all the pages of a website is not a single technical issue, but an overlay of checks: HTTP status, presence in the sitemap, robots.txt directives, user-agent filtering. Each layer can hide or reveal different URLs, and no isolated method provides a complete picture.
