A 404 error tells the visitor, and the crawler, that the address requested does not exist on the server. On a small business site they appear quietly: a product page taken down, a URL changed during a redesign, a link to a third-party site that has reorganised its content. Screaming Frog SEO Spider remains the reference tool for finding them — provided you configure it properly.
This article started with a question from someone I had helped audit a company website. Weeks later he could no longer find a 404 we had looked at together, and the webmaster insisted nothing had changed. His case sums up the traps of this diagnosis: a crawl that does not "see" the same thing twice, and response codes that are easy to confuse.
When a 404 error appears to vanish
Two broken links had been repaired. A third 404, spotted in an earlier crawl, was nowhere in the new report; all that turned up were three URLs answering with a 301. Re-running the crawl produced three explanations, none of them mysterious.
- A corrected page stops being a 404. A crawler photographs the site as it passes through. It has no memory. Once the link is fixed, the error leaves the report — the expected outcome, not a bug.
- A redirect moves the URL to another filter. The second address now answered with a 301: someone, or a plugin, had redirected it. It had moved from "Client Error (4xx)" to "Redirection (3XX)".
- The configuration was not the same. The free version stops at 500 URLs, and JavaScript rendering had not been enabled. Where the menu is injected by a script, a share of the internal links is never followed.
Before concluding that an error has gone, compare three things: the crawl configuration, the date of the crawl, and the exact response code the URL returns.
Why 404 errors are worth handling
One assumption needs dismantling. Google does not penalise a site for containing 404 pages. Its documentation is explicit: all 4xx errors except 429 are handled the same way. The URL is dropped from the index if it was in it, new URLs returning 404 are not processed, and crawling of that address gradually slows (Google Search Central, HTTP status codes, checked September 2026). A 404 on a page that no longer has any reason to exist is a normal, correct answer. The real problems sit elsewhere.
- Broken internal links. Every link pointing at a 404 is an interrupted journey and a visible sign of neglected maintenance. If the missing page had earned external links, that value now passes to nothing.
- Soft 404s. An empty page, or a "not found" message served with a 200 status. Google notes that such pages keep being crawled for no return, and points to the page indexing report for finding them (Google, managing crawl budget). The same document reserves the crawl budget question for sites above one million pages, or above 10,000 pages updated daily. For a smaller site, the stakes are the visitor and the internal linking.
- Dead outbound links. A link to a resource that no longer exists undermines the credibility of content that is otherwise perfectly current.
Configuring Screaming Frog for a 404 audit
Screaming Frog SEO Spider is free up to 500 URLs per crawl. The annual licence is listed at £199 on the publisher's pricing page (Screaming Frog pricing, version 24.3, checked September 2026). Beyond a few hundred pages, the licence becomes necessary.
The settings that change the result
- Configuration > Spider > Crawl. Check that the crawl boxes are ticked for images, CSS, JavaScript and external links. Without the external links box, no broken outbound link is ever tested.
- Configuration > Spider > Rendering. In "Text Only" mode, only the raw HTML is analysed. Switch to "JavaScript" if your menu, filters or content blocks are generated in the browser. This is the most common cause of a gap between two crawls of the same site.
- Limits and speed. A crawl depth or URL limit set too low cuts the crawl short before it reaches deep pages. Crawling too fast against shared hosting produces "No Response" rows that do not exist in reality.
Reading the Response Codes tab
The "Response Codes" tab filters by response family. The useful ones here are "Client Error (4xx)", "Redirection (3XX)", "Redirect Chain", "Redirect Loop" and "No Response". For each failing URL, the "Inlinks" panel lists the source pages, the anchor text and the link type.
That list is what you work from: you do not fix the 404 itself, you fix the pages pointing at it. "Bulk Export" writes every inlink to 4xx URLs into a spreadsheet — the simplest way to hand the work to a developer. To confirm a single URL outside the crawler:
curl -sI https://example.com/old-page/ | head -n 1
> HTTP/2 404Fixing: one decision per type of case
Not every 404 is handled the same way. The table below sums up the choices I put to clients.
| Situation | Recommended action | Watch out for |
|---|---|---|
| Typo in an internal link | Correct the link in the source page | Do not add a redirect to paper over a typing mistake |
| Page moved or renamed | 301 redirect to the new URL, then update the internal links | Redirect chains building up after successive redesigns |
| Page deleted, close equivalent exists | 301 redirect to the equivalent page | Redirecting to the home page, which ignores the visitor's intent |
| Page deleted, no equivalent | Leave the 404 (or 410) and remove the internal links | Check the error page really returns a 404 status |
| Outbound link to a vanished resource | Find the new address, otherwise remove the link | Search the Wayback Machine or the organisation's own site |
On redirects, Google is clear that a permanent redirect (301 or 308) is a signal that the target should become canonical, whereas a temporary redirect (302 or 307) is not (Google, redirects and Google Search). For a page that has moved for good, use the 301. On WordPress, the Redirection plugin (version 5.10 in August 2026) handles these rules without editing .htaccess. If the errors come from a wider migration, the website redesign guide covers mapping old URLs to new ones before launch rather than after.
Outbound links: tracing the missing resource
The old "cache:" reflex no longer works. Google removed the "Cached" link from its results in early February 2024 and retired the operator shortly afterwards (Search Engine Land, 1 February 2024). To recover a page that has gone, use the Wayback Machine to see what was there, then look for the equivalent on the organisation's current site. If nothing equivalent exists, delete the link rather than leave it pointing into the void.
Cross-checking with Search Console
Screaming Frog shows what a crawler finds by following your links. Search Console shows what Google has actually encountered — not the same set: old URLs, addresses linked from other sites, entries still in a stale sitemap.
The relevant report is now called "Page indexing", replacing the former "Coverage". It separates out reasons including "Not found (404)" and "Soft 404". Google notes that Googlebot will probably keep trying the URL for some time, and recommends a 301 if the page has moved. Once the corrections are made, the "Validate fix" button starts a check that can take around two weeks (Search Console Help, page indexing report). If these reports are unfamiliar territory, our page on Google indexing explains how to read them without over-interpreting them.
Keeping the errors from coming back
A 404 audit is only useful if it is repeated. The rhythm I suggest:
- a crawl after every significant release — a new section, a batch of products withdrawn, a change of theme;
- a monthly look at the "Page indexing" report in Search Console;
- one team rule: no page is deleted without a decision on its redirect taken at the same time;
- on WordPress, the Broken Link Checker plugin (WPMU DEV, version 2.4.14 in August 2026) monitors links continuously. On modest hosting, prefer its cloud mode to the local scan, which leans on the database.
Finally, check that your own 404 page returns a 404 status code and offers a search box or links to the main sections. It is the only part of this audit a visitor ever sees. If the crawl returns several hundred errors, or redirect chains inherited from successive rebuilds, the problem is structural: that belongs in a full technical SEO audit, and it helps to scope the audit before anyone starts crawling.
Common questions
Why can't Screaming Frog find a 404 error I saw somewhere else?
A crawl reflects the state of the site at the moment it runs, with the configuration you chose. A URL limit reached, JavaScript rendering left off, external links not tested, or a redirect added in between account for most discrepancies. Compare the configurations first.
Do 404 errors damage search rankings?
Google treats a 404 as a page that no longer exists: it is dropped from the index and crawled less often, with no penalty to the rest of the site. What does cause harm is internal links pointing at those pages, the loss of any external links they had earned, and soft 404s.
Should every 404 page be redirected?
No. A page that has moved, or been replaced by a close equivalent, deserves a 301. A page deleted with no equivalent should stay a 404 or a 410. Redirecting it to the home page gives the visitor nothing and hides the problem.
What is the difference between a 404 and a 410?
A 410 states that the resource was removed for good; a 404 says only that it cannot be found. Google's documentation says it handles all 4xx errors the same way apart from 429, so the choice makes little difference to indexing.
Is the free version of Screaming Frog enough for a small business site?
It crawls up to 500 URLs, which covers a typical brochure site. Beyond that, or to use JavaScript rendering and bulk exports, you need the paid annual licence, listed by the publisher at £199 in September 2026.
How do I recover the content behind an outbound link that now returns 404?
Google withdrew the "Cached" link and the cache: operator in 2024. Use the Wayback Machine to see the old page, then look for the equivalent on the publisher's current site. If there is none, remove the link.