ReadableBot: The Readable.io Indexer

Indexing

Our engine is built to index at a friendly rate - we would rather score URLs and websites slowly than risk affecting their performance for real customers. We typically won't retrieve more than one URL per second, although for larger sites our system will attempt to increase that rate tentatively and monitor for any signs of impact on performance.

User-Agent

Our user agent is "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Readable/1.1.4" (please note that the version number following "Readable/" will change periodically and the version number shown here may not be current). Many websites will server different versions of text to search engine spiders, and some will block access to anything they don't recognise. Since our job is to measure the readability of the page a typical user sees, we do our best to make sure that's what we are seeing.

JavaScript

Our engine does not support pages rendered with JavaScript at the moment, though that is something we are working on and hope to support soon.

Robots.txt

We respect instructions from the robots.txt file. If you want to specifically exclude anything from being indexed by the Readable.io engine, you can use the user-agent "ReadableBot". We have included an example of this on our own robots.txt file.

Try Our Text Scoring Tool

See what you can learn about your content today.

Try It Now »

Dave Child

Dave is the founder of Readable.io, and has been building websites since the early 90s. He's one of those fortunate people who gets to do what he loves for a living.

       


Comments


Readable.io Premium!

Premium Subscribers get access to a whole host of specialist readability tools, including text, URL and file scoring, from as little as $5 per month!

Go Premium Today

  • Unlimited Text Readability
  • File (Word Docs, PDFs, etc) Scoring
  • Bulk Text Readability
  • Bulk URL Readability
  • Readability API

"A really great tool for writers who care more about clarity than flourish."

Patrick O'Shaughnessy, Investor

Premium Popup Trigger