Bulk data and partnerships
Need the whole catalog? Ask for a bulk feed instead of crawling.
Data partnership
If you build search, training or agent products and need the whole catalog, crawling page by page is slow for you and costly for us. Ask for a bulk feed instead.
- A regularly updated export of public model facts: sources, license labels, file lists and SHA-256 hashes.
- A partner key with a high rate limit for live lookups.
- Terms that fit your use: attribution, research or a commercial license.
Tell us who you are and what you need. We reply by email.
Free for everyone
- Fair-pace crawling: robots.txt
- Public API: OpenAPI
- Sitemaps: sitemap.xml
- Catalog dump: catalog.jsonl
- MCP tools: MCP manifest
Need more?
Higher limits: get a free API key, or ask here for a higher plan.
Bulk feed: request a regularly updated catalog export, so you don't need to crawl.
Higher limits and bulk feeds are free for research, open-source and partner projects. Commercial high-volume access is available. Tell us what you need.
Data feed for AI companies
Daily full catalog with checksums, licenses and where models run, change feed, higher limits, uptime promise, a direct contact. Free for research and open source.
How crawlers identify themselves
- Send one stable User-Agent with your crawler name and a page about it, for example ExampleBot/1.0 (+https://example.com/bot).
- Use the same User-Agent from every IP address. We count named crawlers by User-Agent, not by IP.
- Follow robots.txt, including Crawl-delay.
- On the data feed, send your key as Authorization: Bearer <key> or X-Api-Key: <key>. Keyed requests get the key's limits, not the shared crawler budget.
A crawler that sends a lot of traffic without a key may get a 429 with Retry-After and a link to this page. That is a pause, not a block: research tools and agents are always welcome at a fair pace.
Request access
For agents and scripts
One call. Only contact_email and intended_use are required. JSON or form posts both work.
curl -sS -X POST 'https://huggingbay.xyz/api/v1/crawl-access/requests' -H 'content-type: application/json' -d '{"contact_email":"you@example.org","intended_use":"Describe what you will do with the data"}'MCP: tool request_crawl_access at https://huggingbay.xyz/api/mcp?profile=crawl. Machine-readable summary: crawl-access.json.