Crawl & data access
Crawling at a fair pace is free. If you need more, request higher limits or a bulk feed.
Need the whole catalog? Download the daily snapshot instead of crawling page by page.
Free for everyone
- Fair-pace crawling: robots.txt
- Public API: OpenAPI
- Sitemaps: sitemap.xml
- Catalog dump: catalog.jsonl
- MCP tools: MCP manifest
Need more?
Higher limits: get a free API key, or ask here for a higher plan.
Bulk feed: request a regularly updated catalog export, so you don't need to crawl.
Higher limits and bulk feeds are free for research, open-source and partner projects. Commercial high-volume access is available. Tell us what you need.
Data feed for AI companies
Daily full catalog with checksums, licenses and where models run, change feed, higher limits, uptime promise, a direct contact. Free for research and open source.
How crawlers identify themselves
- Send one stable User-Agent with your crawler name and a page about it, for example ExampleBot/1.0 (+https://example.com/bot).
- Use the same User-Agent from every IP address. We count named crawlers by User-Agent, not by IP.
- Follow robots.txt, including Crawl-delay.
- On the data feed, send your key as Authorization: Bearer <key> or X-Api-Key: <key>. Keyed requests get the key's limits, not the shared crawler budget.
A crawler that sends a lot of traffic without a key may get a 429 with Retry-After and a link to this page. That is a pause, not a block: research tools and agents are always welcome at a fair pace.
Request access
For agents and scripts
One call. Only contact_email and intended_use are required. JSON or form posts both work.
curl -sS -X POST 'https://huggingbay.xyz/api/v1/crawl-access/requests' -H 'content-type: application/json' -d '{"contact_email":"you@example.org","intended_use":"Describe what you will do with the data"}'MCP: tool request_crawl_access at https://huggingbay.xyz/api/mcp?profile=crawl. Machine-readable summary: crawl-access.json.