{"schemaVersion":"hugging-bay.crawl-access.v1","summary":"Fair-pace crawling is free. Need faster or bulk access? Request access in one call.","policy":"Higher limits and bulk feeds are free for research, open-source and partner projects. Commercial high-volume access is available. Tell us what you need.","page":"https://huggingbay.xyz/crawl-access","free":[{"name":"Fair-pace crawling","url":"https://huggingbay.xyz/robots.txt","note":"Follow robots.txt, including any Crawl-delay."},{"name":"Public API","url":"https://huggingbay.xyz/openapi.json"},{"name":"Sitemaps","url":"https://huggingbay.xyz/sitemap.xml"},{"name":"MCP tools","url":"https://huggingbay.xyz/.well-known/mcp.json"},{"name":"Catalog dump","url":"https://huggingbay.xyz/api/dumps/catalog.jsonl?limit=1000"}],"options":[{"id":"higher_limits","title":"Higher limits","description":"Request an access key with a higher rate limit."},{"id":"bulk_feed","title":"Bulk feed","description":"Request a regularly updated catalog export so you do not need to crawl."},{"id":"data_partnership","title":"Data partnership","description":"For AI companies: the data feed (daily full catalog, change feed, a data feed key and terms that fit your use)."},{"id":"other","title":"Other","description":"Tell us what you need."}],"request":{"method":"POST","url":"https://huggingbay.xyz/api/v1/crawl-access/requests","contentTypes":["application/json","application/x-www-form-urlencoded"],"required":["contact_email","intended_use"],"schema":{"type":"object","required":["contact_email","intended_use"],"properties":{"contact_email":{"type":"string","format":"email","maxLength":254,"description":"Where we reply. Kept private."},"intended_use":{"type":"string","minLength":3,"maxLength":1000,"description":"What you will do with the data."},"name":{"type":"string","maxLength":120,"description":"Name or organization."},"user_agent":{"type":"string","maxLength":300,"description":"Your crawler's User-Agent."},"volume_per_day":{"type":"integer","minimum":0,"maximum":1000000000,"description":"Expected requests per day."},"option":{"type":"string","enum":["higher_limits","bulk_feed","data_partnership","other"],"default":"other","description":"higher_limits, bulk_feed, data_partnership or other."},"use_type":{"type":"string","enum":["research","open-source","partner","commercial"],"description":"research, open-source, partner or commercial."}},"additionalProperties":false},"example":{"contact_email":"you@example.org","intended_use":"Index model metadata for a research search engine","user_agent":"ExampleBot/1.0 (+https://example.org/bot)","volume_per_day":50000,"option":"bulk_feed","use_type":"research"},"curl":"curl -sS -X POST 'https://huggingbay.xyz/api/v1/crawl-access/requests' -H 'content-type: application/json' -d '{\"contact_email\":\"you@example.org\",\"intended_use\":\"Describe what you will do with the data\"}'","response":"201 with id and statusUrl (200 with duplicate true for a repeat). We reply by email."},"status":{"method":"GET","urlTemplate":"https://huggingbay.xyz/api/v1/crawl-access/requests/{id}","auth":"none","note":"Shows status only; never your contact details."},"mcp":{"endpoint":"https://huggingbay.xyz/api/mcp?profile=crawl","profile":"crawl","tool":"request_crawl_access"},"dataFeed":{"title":"Data feed for AI companies","summary":"Daily full catalog with checksums, licenses and where models run, change feed, higher limits, uptime promise, a direct contact.","free":"Free for research and open source.","includes":["The daily full catalog: every public model with file lists, sizes, SHA-256 checksums and license labels.","Where models run: download readiness and the files you can fetch today.","A change feed, so you pull only what changed since your last copy.","Higher limits for live lookups with your own data feed key.","An uptime promise for the feed.","A direct contact for questions and fixes."],"freeForEveryone":["Fair-pace crawling that follows robots.txt.","The public API, MCP tools, sitemaps and the bounded catalog dump.","A free API key with a higher limit of its own."],"howToGet":"Ask us. We set up a data feed key and reply by email with terms.","contact":"https://huggingbay.xyz/contact?topic=data-feed","identify":["Send one stable User-Agent with your crawler name and a page about it, for example ExampleBot/1.0 (+https://example.com/bot).","Use the same User-Agent from every IP address. We count named crawlers by User-Agent, not by IP.","Follow robots.txt, including Crawl-delay.","On the data feed, send your key as Authorization: Bearer <key> or X-Api-Key: <key>. Keyed requests get the key's limits, not the shared crawler budget."]},"overLimit":"Crawlers over the free limit without a key get a 429 with Retry-After and links to this offer. It is a pause, not a block.","privacy":"We use these details only to reply to your request."}