Using Xenu to crawl your own sites

I’ve used Xenu for years – probably as long as I’ve been coding websites – and it’s been around longer than that. You’ll see from the website that it’s incredibly old school, but sometimes the tried and tested tools are the ones you should stick with.

I’ve just helped a friend merge two sites together and we’ve used mysql to change some of the internal links but there’s still a chance that some might have been missed. Xenu will let him know what the state of play is.

Start off with giving the site you want to index. I always check external links because it doesn’t hurt to know if any of the sites you’ve linked to have vanished.

xenu1

As Xenu starts running you’ll see the links it has discovered and there’s a progress bar at the bottom that reflects it’s current status. With every new page it checks there is the potential to discover a treasure trove of new links and change the % done dramatically.

xenu2

 

 

Once upon a time I used to use Xenu’s reports but as sites have grown and become more complex the report itself has become too cumbersome. It doesn’t hurt to take a look if you’ve checked a small site, just so  you know what is there.

xenu3

Instead I sort the results by status and check out each line of errors

xenu4

 

By right mouse clicking on a line and selecting URL Properties from the context menu I can see which pages in my site link to the broken link. If the list is really long then it’s likely to be a sitewide link, fix it once and it fixes for all.

xenu5

 

For the links that are ok, double click on them and you’ll be taken to the page. You can still check out the URL properties which helps you find links out to sites you’d rather not be linking to.

Categories

Recent Comments

Tags

Be First to Comment

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.