application icon

Scrutiny 7 Troubleshooting and FAQs

What could be wrong?


If your scan or results didn't turn out as expected, see whether the answer is here.

Pages time out / the web server stops responding / 509 server error

This isn't uncommon. Some servers will respond to many simultaneous requests, but some will have trouble coping, or may deliberately stop responding if being bombarded from the same IP.

There are a couple of things you can do. First of all, the 'threads' slider sets the number of requests that Scrutiny/Integrity can make at once. If you move this to the extreme left, then Scrutiny/Integrity will send one request at a time, and process the result before sending the next. This alone may work. If not, then there's a box above that slider which allows you to set a delay (in seconds). You can set this to what you like, but a fraction of a second may be enough.

Note that introducing a delay will only be effective with a very low number of threads.

If your server is simply being slow to respond, you can increase the timeout.

If you have to allow querystrings because there's a page id in there, but a session id or some other parameter is causing the crawl to go on for ever, then Scrutiny now has the option to ignore only the session id (or another single parameter). See the 'Advanced' tab

404s are reported for links that are fine

This can happen with certain servers where both http:// links and https:// links appear on the site. It appears that some servers don't like rapid requests for both http and https urls. Try starting at a https:// url and blacklisting http:// links (make a rule 'do not check urls containing http://') and see whether the https:// links then return the correct code.

It's also worth changing the user-agent string in Preferences, servers can sometimes respond differently to a UA string which isn't a recognised browser

Also see the question below about certain social networking sites

A link to [a social networking site ie Youtube, Facebook] is reported as a bad link or an error in Scrutiny, but the link works fine in my browser?

In your browser, log out of the site in question, then visit the link. You'll then be seeing the same page that Scrutiny sees because, by default, it doesn't attempt to authenticate.

If you see a page that says something like 'you need to be logged in to see this content' then this is the answer. It's debatable whether a site should return a 404 if the page is asking you to log in, but that should be taken up with the site in question.

You have several options. You could switch on authentication & cookies in Scrutiny (and log in using the button to the right of those checkboxes). You could set up a rule so that Scrutiny does not check these links, or you could change your profile on the social networking site so that the content is visible to everyone.

If the problem is LinkedIn links giving status 999, then this is a different problem, LinkedIn is detecting the automated requests and is sending the 999 codes in protest. The only way to avoid this (as far as I know) is to seriously throttle Scrutiny by turning down the threads to a minimum and introducing a long delay, but this will seriously slow your scan, and so it may be preferable to set up a rule to ignore the LinkedIn links

Limitations

If your site is a larger site then the memory use and demand on the processor and HD (virtual memory) will increase as the lists of pages crawled and links checked get longer.

Scrutiny has become much more efficient over the last couple of versions, but if the site is large enough (millions of links) then the app will eventually run out of resources and obviously can't continue.

- Make sure Integrity isn't going into a loop or crawling the same page multiple times because of a session id or date in a querystring - you can switch off querystrings in the settings, but make sure that content that you want to crawl isn't controlled by information in the querystring (eg a page id)

- See if you're crawling unnecessary pages, such as a messageboard. To Integrity and Scrutiny, a well-used messageboard can look like tens of thousands of unique pages and it will try to list and check all of those pages. Again, you can exclude these pages by blacklisting part of the url or querystring or ignoring querystrings.

- You can crawl the site in parts. You can do this by scanning by subdomain, scanning by directory or using blacklisting or whitelisting.

If you start within a subdomain (eg engineering.mysite.com ) the scan will be limited to that subdomain if the setting 'consider subdomains of the root domain internal' is switched off
If you start within a 'directory' (eg mysite.com/engineering ) the scan will be limited to that directory
if you create a whitelist rule which says 'only follow links containing /manual/ the scan will be limited to urls which contain that fragment.


I use Google advertising on my pages and don't want hits on these ads from my IP address

The Google Adsense code on your page is just a snippet of javascript and doesn't contain the adverts or the links. When a browser loads the page, it runs the javascript and the ads are then pulled in. Integrity and Scrutiny don't run javascript (make sure the Render page (run javascript) option is turned off in Scrutiny) so it won't see any ads or find the links within them.

A link that's shown as "www.mysite.com/../page.html" is reported as an error but when I click it in the browser it works perfectly well

Sometimes a link is written in the html as '../mypage.html'. The ../ means that the page is to be found in the directory above, which is fine as long as the link is deep in the site. If it appears on a top-level page in that form, then it's technically incorrect because no-one should have access to the directory above your domain. Browsers tend to tolerate this and assume the link is supposed to point to the root of your site. By default, Scrutiny does not make this assumption and reports the error. Since v6.8.1 there is a preference to "tolerate ../ that travels above the domain" (General tab)

A link that uses non-ascii or unicode characters is reported as an error but when I click it in the browser it works perfectly well

Integrity and Scrutiny now handle non-ascii characters in urls but not if within the domain name.

Internationalized Domain Names (IDN) have only recently entered web standards and at present Integrity and Scrutiny do not support them.

What do the red and orange colours mean in the list?

To check a link, Scrutiny sends a request and receives a status code back from your server (200, 404, whatever).

The 'status' column tells you the code. 200 codes means that the link is good, 300 means there's something not quite right (usually a redirection) but the link still works, 400 codes mean that the link is bad and the page can't be accessed and 500 codes mean some kind of error with the server. So the higher the number, the worse the error and Scrutiny colours these (by default) white, orange and red.

If you don't consider a redirection a problem, then you can now switch the orange colour off in Preferences (Links tab). You can also choose different colours or even turn off this colouring completely in Preferences (General tab)

(There's a full list of all the possible status codes here: http://en.wikipedia.org/wiki/List_of_HTTP_status_codes) but Scrutiny does helpfully give you a description of the status as well as the code number.

a 200 is shown for a link where the server doesn't exist

Your provider may be recognising the fact and inserting a page of their own (possibly with a search box and some ads which benefit them financially) and returning a 200 code. They call this a helpful service, but it's unhelpful to us when we're trying to find bad links.

You may be able to ask your service provider to turn this behaviour off (either via a page on their website or by contacting them). Failing that you can use the 'soft 404' feature to raise a problem for such urls. There is a longer explanation of this problem and solution here.

It crashes

This is rare, as far as we're aware, and when it does happen we really would like to know. Please use this form to send some details to help us.

If your site is an extremely large site (millions of pages) then you may be hitting a limit. See limitations above for more details and suggestions.

Disc space is eaten while Scrutiny runs

This should only happen for very large sites, and since version 6, Integrity and Scrutiny will be much less resource-hungry. Here are some measures to make Scrutiny more efficient.

Go to the settings for your site, Options tab, there are four checkboxes which are labeled 'these options may have a serious impact on resources' - uncheck them if you can, particularly grammar checking and keyword analysis.

Make sure that the javascript option is switched off. This should only be used in very rare circumstances where page content including links are generated by javascript. It's in your site's settings on the Advanced tab ('Render page (run javascript)')

Also uncheck Settings > Options > Archive pages while crawling and Preferences > SEO > Count occurrences of keywords in content. If either of these boxes are checked, Scrutiny necessarily caches the page content. Depending on the size and number of your pages this can mean a significant amount of space. Unless you save the archive after the scan, this cache will be deleted when you quit or failing that, when you start the next scan.