Automatic link checker

Broken links are a common problem, particularly for a website which is already old and of medium size, which is an annoyance for visitors, for search engine optimisation, and for webmasters. While there are good tools available to check this point, it would be better to have something entirely automatic, so a webmaster does not even have to think about this point.

To handle this issue, I configured a GitHub action which checks for broken links 1/ when pushing a new version of the website and 2/ from time to time to avoid link rot.

Choice of the link checker tool

A first version of the link checker system used the lychee link checker because it was an easy pick with a ready-made GitHub action. However, it handled some edge cases poorly, like delayed responses, so I had to choose another tool.

After some benchmarks, I settled on LinkChecker, a mature tool written in Python.

Wrapper script

In addition to the mere LinkChecker program, I needed some convenience scripts to:

• Handle the different configuration between my computer (WSL) and the GitHub CI.

• Have a shortcut to avoid repeating the flags of LinkChecker.

• Generate a summary with statistics, redirects, and a list of ignored links to manually check from time to time.

• Avoid putting too much stuff into the action files.

At the beginning, there were several bash scripts using grep/sed/awk. Then for CSV parsing the bash scripts used inline Python. Then I reworked these to use only a single Python script, available as linkchecker.py in the repository.

GitHub action file

The GitHub action file allows checking for broken links 1/ when changes are made and 2/ monthly, on the first day of each month, to avoid link rot. Additionally, GitHub sends me an email when this action fails so I’m warned that I have to update links.

Ignored links

According to the latest statistics, out of 655 links checked, 25 links are ignored, falling into the following cases:

• Cloudflare-protected sites, which prevent bots from checking them.

• LinkedIn, which also has bot protection.

• Wikimedia, which has a rate-limitation. Note that currently LinkChecker has no per-site rate-limitation.

• archive.org, which for unclear reasons can be checked from my computer but has issues when checked by the GitHub CI.

• doi.org, redirecting towards Cloudflare sites.

Summary page

The script also produces a very convenient summary page for display on GitHub actions pages, like the one shown below (slightly older, with 26 ignored links instead of the current 25):