LinkChecker is a program that checks web page links to make sure that they work. When writing web pages it is easy to make a small typing error that results in a link that doesn’t work. Checking all the links on a web site is time consuming, and it is hard to be sure that they have all been checked. This program checks each link on each page one by one, to make sure that they all work.
Usage
The program can be run from the command line like this:
java -jar /path/to/LinkChecker.jar http://www.mysite.com/index.html -d2000This will load the index page of www.mysite.com and will check that every link works. Each page that it finds a link to that is also on he same server will be checked in turn. Thus the whole site should eventually be checked, as long as there are links between all the pages. The -d option specifies a delay time in milliseconds. -d2000 specifies a delay of 2 seconds between each page access. This is avoid overloading the server. Each valid address is cached, so that it only needs to be looked up once. If the computer is hosting a copy of a website, and is not connected to the Internet then the -l will cause it to only check local addresses.
The program can only check web pages that are well-formed, which means that they comply with the w3c XML specification. This probably means that it is limited to XHTML web pages. The DTD’s for XHTML are included in the .jar file. If the web pages use different DTD’s then the program will only work if the computer is connected to the Internet.
The program can also be called directly from an Ant build script. First of all the task must be defined:
<taskdef name="checklinks"
classname="uk.co.matthewwest.LinkChecker.LinkCheckerTask"
classpath="/path/to/LinkChecker.jar" />Then the task can be called like an other Ant task:
<checklinks href="http://www.mysite.com/index.html"
delay="150" local="no" />Version 0.2
Version 0.2 corrects a couple of problems caused by upgrading of other parts. Apache now gives longer content headers which include the charset while version 0.1 was looking for just “text/html”. Modern versions of Java include an XML parser which gives a SAX warning seemingly every time it comes accross an entity, so version 0.2 suppresses warnings unless ‘debug’ is set to true.
Libraries
Version 0.1 of the program is supplied with Xerces, an XML parser developed by the Apache Foundation. Version 0.2 assumes a more modern version of Java, which comes with an XML library built in. Ant is used to build the project. There are also some DTD’s included which were downloaded from the w3c website.
Download
The program is available for download as a zip file , or as a tar file . My code is released under the terms of the GPL.
How It Works
Each page is parsed by a SAX parser. Every time the parser encounters a link(<a>, <link>, <img>, <form>) it calls checkUrl(). This checks a HashSet of previously validated addresses, and if the URL is not among them it attempts to connect to a server to retrieve it. If it can then the URL is added to the HashSet. If the URL is on the same server as the first page and hasn’t been checked it is added to a LinkedList of pages pending checking. If it cannot retrieve the page then it prints a message. It then moves on to the next page in the list.