Automatic link checker

Broken links are a common problem, particularly for a website which is already old and of medium size, which is an annoyance for visitors, for search engine optimisation, and for webmasters. While there are good tools available to check this point, it would be better to have something entirely automatic, so a webmaster does not even have to think about this point.

To handle this issue, I configured a GitHub action which checks for broken links 1/ when pushing a new version of the website and 2/ from time to time to avoid link rot.

Choice of the link checker tool

A first version of the link checker system used the lychee link checker because it was an easy pick with a ready-made GitHub action. However, it handled some edge cases poorly, like delayed responses, so I had to choose another tool.

After some benchmarks, I settled on LinkChecker, a mature tool written in Python.

Wrapper script

In addition to the mere LinkChecker program, I needed some convenience scripts to:

• Handle the different configuration between my computer (WSL) and the GitHub CI.

• Have a shortcut to avoid repeating the flags of LinkChecker.

• Generate a summary with statistics, redirects, and a list of ignored links to manually check from time to time.

• Avoid putting too much stuff into the action files.

At the beginning, there were several bash scripts using grep/sed/awk. Then for CSV parsing the bash scripts used inline Python. Then I reworked these to use only a single Python script, available as linkchecker.py in the repository.

GitHub action file

The GitHub action file allows checking for broken links 1/ when changes are made and 2/ monthly, on the first day of each month, to avoid link rot. Additionally, GitHub sends me an email when this action fails so I’m warned that I have to update links.

Ignored links

According to the latest statistics, out of 655 links checked, 25 links are ignored, falling into the following cases:

• Cloudflare-protected sites, which prevent bots from checking them.

• LinkedIn, which also has bot protection.

• Wikimedia, which has a rate-limitation. Note that currently LinkChecker has no per-site rate-limitation.

• archive.org, which for unclear reasons can be checked from my computer but has issues when checked by the GitHub CI.

• doi.org, redirecting towards Cloudflare sites.

Cache

During development, it is common to run the test script many times. Checking all URLs each time would be inconvenient and risk triggering various anti-bot protections. A cache mechanism is therefore implemented to check external links only when needed.

Post-checking

LinkChecker triggers some anti-bot protections on certain sites. A post-checking phase is performed using primp. A previous version used curl_cffi, but following some doubts about curl_cffi, I preferred to switch to primp.

Cloudflare detection

In previous versions, I had a manual list of Cloudflare URLs to ignore due to Cloudflare’s bot block. Now, URLs blocked by Cloudflare are automatically flagged as such and cached so they are not checked again for 1 month, to avoid tickling the dragon too much.

Summary page

The script also produces a very convenient summary page for display on GitHub actions pages, like the one shown below (slightly older, with 26 ignored links instead of the current 25):