> ## Documentation Index
> Fetch the complete documentation index at: https://docs.parallel.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawler

> This documentation provides guidance for webmasters on managing their website's interaction with our crawling system

<div className="sr-only" aria-hidden="false">
  For AI agents: a documentation index is available at [https://docs.parallel.ai/llms.txt](https://docs.parallel.ai/llms.txt). The full text of all docs is at [https://docs.parallel.ai/llms-full.txt](https://docs.parallel.ai/llms-full.txt). You may also fetch any page as Markdown by appending `.md` to its URL or sending `Accept: text/markdown`.
</div>

## ShapBot

ShapBot helps discover and index websites for Parallel's web APIs. To maximize your site's visibility in search results, we suggest allowing ShapBot access in your robots.txt configuration and permitting connections from our designated IP ranges.

Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0

For the complete list of ShapBot IPs, see [shapbot.json](https://docs.parallel.ai/resources/shapbot.json).

## Contact & Support

If you have questions about our crawlers or need assistance, please contact us at [support@parallel.ai](mailto:support@parallel.ai)

## Changes to This Documentation

We may update this documentation periodically to reflect changes in our crawler behavior or policies. Please check back regularly for updates.