Website crawlers / spiders

A curated list of web crawling software for Mac OSX

A number of apps, free and paid which will scan your website and perform various functions such as create sitemaps, check markup, make SEO checks, archive the site or convert the format.

A1 Website Analyzer

A1 Website Analyzer

from: Microsys
price: $69
demo: free and unrestricted for 30 days, after that, free version has 500-url limit
platform(s): Windows / Mac OSX (10.10+)

Crawls a website, reports broken links, analyse SEO parameters, on-page keyword analysis, spell-check, html and css validation, site search, file sizes and download times.

More information

BLT logo

BLT (Better Link Tester)

from: Before Dawn Solutions
price: $19.99
demo: free and unrestricted for 14 days
platform(s): native Mac OSX

Crawls a website, reports broken links, extensible.

More information

CheckSite app icon

CheckSite

from: Colorize
price: $24.99
demo: trial for 30 days
platform(s): Java (Windows, OS X, Linux, Android)

Finds links to other pages (in testing I've found that it tests pages linked from the starting page, couldn't persuade it to crawl further than that).

Performs a number of checks, browser compatibility, html validation, presence of robots.txt, missing alt text etc.

More information

HTMLtoMD logo

HTMLtoMD

from: PeacockMedia
price: Free
platform(s): native Mac OSX

Crawls a website, sucks / archives the content after converting to Markdown

More information

Integrity from PeacockMedia

Integrity

from: PeacockMedia
price: Free
platform(s): native Mac OSX

Crawls a website, reports broken links, broken images. Integrity Plus adds xml sitemap, search boxes, manage multiple sites ($15).

More information

Screaming Frog SEO Spider logo

Screaming Frog SEO Spider

from: Screaming Frog
price: £149 per year
demo: free restricted to 500 link crawl limit
platform(s): Java (windows/mac/ubuntu)

Crawls a website, reports broken links, broken images, broken CSS, script. Lists pages and provides xml sitemap, reports seo information.

More information

Scrutiny logo

Scrutiny

from: PeacockMedia
price: $115
demo: free and unrestricted for 30 days
platform(s): native Mac OSX

Crawls a website, reports broken links, broken images, lists pages and provides xml sitemap, reports seo information, spell-check, site search.

More information

Sitemap Automator logo

Sitemap Automator

from: Rage software
price: $21.95
platform(s): native Mac OSX, native Windows

Crawls a website, reports broken links, creates xml sitemap.

More information

SiteSucker logo

SiteSucker

from: Richard Cranisky
price: £4.99
platform(s): native Mac OSX

Crawls a website, sucks / archives the content.

More information

Webbee web spider

from: eebew.com
price: subscription, ranging from $15 per month to $129 per year
platform(s): Java (Windows, OS X, Linux)

Crawls a website, reports response codes, descriptions, titles, headings etc. Some SEO factors such as images without alt tag, duplicate titles, descriptions.

More information

Weblight logo

Weblight

from: Illumit
price: $33
demo: free and unrestricted for 30 days
platform(s): Java (Mac OSX, Windows, Unix)

Crawls a website, reports broken links, broken images, lists pages and provides xml sitemap, reports seo information, reports html validity.

Website not currently responding