Website crawlers / spiders

A curated list of web crawling software for Mac OSX

A number of apps, free and paid which will scan your website and perform various functions such as create sitemaps, check markup, make SEO checks, archive the site or convert the format.

BLT logo

BLT (Better Link Tester)

from: Before Dawn Solutions
price: $19.99
demo: free and unrestricted for 14 days
platform(s): native Mac OSX

Crawls a website, reports broken links, extensible.

More information

CheckSite app icon

CheckSite

from: Colorize
price: $24.99
demo: trial for 30 days
platform(s): Java (Windows, OS X, Linux, Android)

Finds links to other pages (in testing I've found that it tests pages linked from the starting page, couldn't persuade it to crawl further than that).

Performs a number of checks, browser compatibility, html validation, presence of robots.txt, missing alt text etc.

More information

HTMLtoMD logo

HTMLtoMD

from: PeacockMedia
price: Free beta
platform(s): native Mac OSX

Crawls a website, sucks / archives the content after converting to Markdown

More information

Integrity from PeacockMedia

Integrity

from: PeacockMedia
price: Free
platform(s): native Mac OSX

Crawls a website, reports broken links, broken images. Integrity Plus adds xml sitemap, search boxes, manage multiple sites.

More information

Screaming Frog SEO Spider logo

Screaming Frog SEO Spider

from: Screaming Frog
price: £145 per year
demo: free restricted to 500 link crawl limit
platform(s): Java (windows/mac/ubuntu)

Crawls a website, reports broken links, broken images, broken CSS, script. Lists pages and provides xml sitemap, reports seo information.

More information

Scrutiny logo

Scrutiny

from: PeacockMedia
price: $95
demo: free and unrestricted for 30 days
platform(s): native Mac OSX

Crawls a website, reports broken links, broken images, lists pages and provides xml sitemap, reports seo information, reports html validity.

More information

Sitemap Automator logo

Sitemap Automator

from: Rage software
price: $29.95
platform(s): native Mac OSX, native Windows

Crawls a website, reports broken links, creates xml sitemap.

More information

SiteOrbiter logo

SiteOrbiter

from: Alex Nygren
price: $9.99
platform(s): native Mac OSX

Crawls a website, reports broken links, lists pages and creates visualisation

More information

SiteSucker logo

SiteSucker

from: Richard Cranisky
price: £4.99
platform(s): native Mac OSX

Crawls a website, sucks / archives the content.

More information

Weblight logo

Weblight

from: Illumit
price: $33
demo: free and unrestricted for 30 days
platform(s): Java (Mac OSX, Windows, Unix)

Crawls a website, reports broken links, broken images, lists pages and provides xml sitemap, reports seo information, reports html validity.

More information