Crawl tools are used to deeply analyse a website from a technical and on-page SEO perspective. They work similarly to Google’s robots — they crawl your site’s pages, follow links, and collect data about each page. They also allow you to analyse the server logs left by crawl bots.
A crawl tool (or crawler) is software that automatically visits the pages of your site, following links from page to page, to collect technical and SEO information. It is the same principle as search engine bots like Googlebot, but with an interface that lets you visualise and analyse the collected data. Crawl tools are indispensable for any serious SEO professional, whether for auditing a new site, detecting technical issues, or monitoring the health of an existing site.
What Is Log Analysis?
When a robot visits a web page, it leaves a trace on the website’s server. This trace is simply called a “log”.
In SEO, we analyse the logs left by robots — notably Googlebot (there are many types of Googlebot, such as those dedicated to images, videos, etc.).
However, this is complex to analyse and takes a phenomenal amount of time. Different crawl tools allow you to aggregate them and make them more human-readable.
What Is Log Analysis?
In SEO, log analysis is very important for all sites primarily wanting to optimise their crawl budget.
Indeed, Google generally does not have the time to crawl an entire website, so it is strategic to optimise accordingly.
Worth knowing: according to BDM, 20 billion sites are visited (crawled) by Google every day. In 2020, Google published its Webspam report revealing no fewer than 40 billion spam pages found per day. And this costs a lot of money — which is why managing crawl budget is increasingly widespread in the SEO sphere. A site optimised for Google and its crawl is a site that ranks better.
Log analysis also allows you to know precisely what Google knows about your site and what it has understood. It lets you know whether the content and products you present to it are known to Google and will therefore have the potential to rank in its results. Since Google only partially reveals this information through Search Console, log analysis allows you to obtain precise data.
The data from server logs allows you to analyse and understand, day by day, the pages Google sees on a site, those it ignores, those that generate traffic, those that should generate traffic but do not, and those that should not generate traffic but do anyway. Etc…
Log file analysis will ultimately allow you to rework a site’s structure, its internal linking, the distribution of its PageRank, and a number of technical criteria.
1. ScreamingFrog, the Ultimate Crawl Tool
ScreamingFrog is an SEO tool dedicated to crawling and therefore to deep site analysis. It is moreover the third most-used tool by French SEO professionals according to BDM (2022). It is used by 60% of respondents, just behind Google Search Console and SEMrush.
And for good reason. Indeed, this more than comprehensive tool will allow you to analyse your site — whether you are a beginner SEO or one of the top SEO experts.
Unlike the other tools presented here, ScreamingFrog is installed on a computer and is therefore not accessible online.
Here is what the interface looks like:
ScreamingFrog allows you to analyse your site, both from the on-site SEO and technical SEO perspectives, including notably:
- Identifying the SEO fundamentals of each page (H(n), Titles, Meta Description)
- Checking the HTTP status codes of pages (1xx, 2xx, 3xx, 4xx, 5xx)
- Viewing redirections / canonical tags / sitemaps
- Pagination
- Everything related to Hreflang
- The Meta-Robots-Tag directives of pages
- Structured data & rich snippets
- Page speed / Core Web Vitals (improvement suggestions such as indicating unused CSS, minifying CSS, etc.)
- Checking whether pages are indexable
- Checking AMP
- Checking a site’s internal linking (treemap visualisation, based on PageRank, etc.)
- Detecting content duplication
- … (and a great many other things).
Available for free, paid version from £149/year
2. Botify
Botify is a French SEO software publisher, specialised in the technical SEO optimisation of websites. The company was founded in 2012 by Stéphane Chauvin, Thomas Grange and Adrien Ménard. Its headquarters are based in Paris but it is only available in the English language. The company has offices in Europe and the United States.
Botify offers three main software modules:
- Botify Analytics (BA) — a very powerful solution for SEO crawling of websites.
- Botify Log Analyzer (BLA) — a market-leading solution for log analysis in SEO.
- Botify RealKeywords (BK) — which allows you to cross-reference all the data returned by your Search Console (GSC) with the technical structure of the site.
The three software blocks can (and generally should) work simultaneously to carry out a comprehensive and relevant analysis of a site’s technical aspects. Botify is a very powerful and complex software that allows you to strongly optimise your crawl budget usage, your internal PageRank, and the fine-tuned optimisation of each URL on the site.
Only available on request. Pricing based on your site.
3. OnCrawl
OnCrawl is a heavyweight among crawl tools and serves to do even more advanced things in addition to log analysis.
- SEO Crawler: roughly the same things as those mentioned for ScreamingFrog
- Log Analyzer: bot hits (mobile, desktop, SEA, news, images and other verticals), explored resources, active pages and many other indicators
- OnCrawl Data: Reveal the impact of SEO metrics — Analyse SEO traffic in detail — Control your backlinks: all by linking to a site’s real data
- OnCrawl BI: Evolution of KPIs over time – Track trends – Demonstrate SEO return on investment – Implementing SEO reporting at scale
- OnCrawl Labs: ready-to-use machine learning algorithms for prediction, creation and SEO automation (real-time indexing – text generation – anomaly detection – long-tail performance prediction)
- OnCrawl Developer / Integrations: independent, Big Data export, custom reports, automated tasks and actions, documented — can be integrated with GSC, GA, Majestic, AT Internet, Adobe Analytics.
From €49/month, available without prior request.
4. DeepCrawl
Focused on crawling and automation.
1. Deep Crawl Detection
World-class technical SEO analyses and insights that help you improve your site’s technical health, increase your performance in search engine results pages, and generate greater revenue.
Main features
- Technical SEO crawler
- Segment data to identify priorities
- Flexible API to use data where you need it
2. DeepCrawl Protect
SEO Automation Centre
The world’s first suite of automated SEO testing that helps ensure code is SEO-optimised and error-free before being published to your site, enabling you to publish code without risking generating problems.
Main features
- Direct integration into your CI/CD pipeline
- 207 automated SEO QA tests
- Set thresholds corresponding to your SEO priorities
Only available on request. Pricing based on your site.
5. Seolyzer
Seolyzer is a technical SEO-oriented crawl tool, French, written for French speakers, and available for free.
It allows among other things:
- Log analysis
- Crawler
- Crawl evolution
- Cross-analysis
- Monitoring
For those who want to go further in technical SEO easily, it is the ideal tool.
It is simple, fast, accessible at all levels and not very expensive if you want to step up by taking a paid plan.
As with the many tools mentioned previously, Seolyzer will be able to do the same “basic” things on the On-Page & Technical problem detection side.
In any case, the best thing is to test it!