Aller au contenu principal
The Hugging BayTrouver un modèle

AI models that run on this device.

Two small chat models run in a computer’s browser here; every other model is a file you download.

How do I run open LLMs on a consumer GPU with GGUF quantization?

Choose a GGUF file supported by your app, then check its size and your available GPU memory. Quantization and file size alone do not establish quality, speed or fit at your chosen context length.

Answer records

Check model files and sizes

License terms and download availability are separate. Read the exact upstream license before using a model.

Answer response assembled as of . This dates the returned rows, not an upstream model update or a successful model run. Check each linked file record for current availability.

Models or files
12
Citation links
6
Data links
3

  1. legraphista/glm-4-9b-chat-IMat-GGUF

    Model from Hugging Face for text generation: legraphista/glm-4-9b-chat-IMat-GGUF · gguf

    License
    unknown. License: unknown License details
    Hosting
    1 hosted file. Inspect the file record for current availability. 30 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Recorded task: text generation. Listed formats: gguf. This listing does not measure quality or prove runtime support.

  2. AnkitAI/Parable-Qwen3-8B-Claude-Fable-5-GGUF

    Model from Hugging Face for text generation: AnkitAI/Parable-Qwen3-8B-Claude-Fable-5-GGUF · gguf

    License
    apache-2.0. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    1 hosted file. Inspect the file record for current availability. 7 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Recorded task: text generation. Listed formats: gguf. Recorded license: apache-2.0. This listing does not measure quality or prove runtime support.

  3. LiquidAI/LFM2.5-230M-GGUF

    Model from Hugging Face for text generation: LiquidAI/LFM2.5-230M-GGUF · gguf

    License
    unknown. License: unknown License details
    Hosting
    1 hosted file. Inspect the file record for current availability. 15 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Recorded task: text generation. Listed formats: gguf. This listing does not measure quality or prove runtime support.

  4. LiquidAI/LFM2.5-2.6B-GGUF

    Model from Hugging Face for text generation: LiquidAI/LFM2.5-2.6B-GGUF · gguf

    License
    unknown. License: unknown License details
    Hosting
    1 hosted file. Inspect the file record for current availability. 15 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Recorded task: text generation. Listed formats: gguf. This listing does not measure quality or prove runtime support.

  5. mudler/locate-anything.cpp-gguf

    Model from Hugging Face for object detection: mudler/locate-anything.cpp-gguf · gguf

    License
    unknown. License: unknown License details
    Hosting
    1 hosted file. Inspect the file record for current availability. 9 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Recorded task: object detection. Listed formats: gguf. This listing does not measure quality or prove runtime support.

  6. ornith-ai/Ornith-1.0-35B-GGUF

    Model from Hugging Face for text generation: ornith-ai/Ornith-1.0-35B-GGUF · transformers

    License
    mit. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    7 hosted files. Inspect the file record for current availability. 9 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Recorded task: text generation. Listed formats: gguf. Recorded license: mit. This listing does not measure quality or prove runtime support.

  7. rippertnt/HyperCLOVAX-SEED-Text-Instruct-1.5B-Q4_K_M-GGUF

    Model from Hugging Face for llm: rippertnt/HyperCLOVAX-SEED-Text-Instruct-1.5B-Q4_K_M-GGUF

    License
    unknown. License: unknown License details
    Hosting
    1 hosted file. Inspect the file record for current availability. 7 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Listed formats: gguf. This listing does not measure quality or prove runtime support.

  8. unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF

    Model from Hugging Face for llm: unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF · transformers

    License
    apache-2.0. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    32 hosted files. Inspect the file record for current availability. 33 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Listed formats: gguf. Recorded license: apache-2.0. This listing does not measure quality or prove runtime support.

  9. prism-ml/Bonsai-27B-gguf

    Model from Hugging Face for text generation: prism-ml/Bonsai-27B-gguf · llama.cpp

    License
    apache-2.0. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    9 hosted files. Inspect the file record for current availability. 14 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Recorded task: text generation. Listed formats: gguf. Recorded license: apache-2.0. This listing does not measure quality or prove runtime support.

  10. handy-computer/parakeet-unified-en-0.6b-gguf

    Model from Hugging Face for automatic speech recognition: handy-computer/parakeet-unified-en-0.6b-gguf · transcribe.cpp

    License
    cc-by-4.0. cc-by-4.0: commercial use, modification, and redistribution are permitted with attribution. License details
    Hosting
    1 hosted file. Inspect the file record for current availability. 7 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Recorded task: automatic speech recognition. Listed formats: gguf. Recorded license: cc-by-4.0. This listing does not measure quality or prove runtime support.

  11. handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf

    Model from Hugging Face for automatic speech recognition: handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf · transcribe.cpp

    License
    unknown. License: unknown License details
    Hosting
    1 hosted file. Inspect the file record for current availability. 8 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Recorded task: automatic speech recognition. Listed formats: gguf. This listing does not measure quality or prove runtime support.

  12. prism-ml/Ternary-Bonsai-27B-gguf

    Model from Hugging Face for text generation: prism-ml/Ternary-Bonsai-27B-gguf · llama.cpp

    License
    apache-2.0. Permissive open license: commercial use, modification, and redistribution are permitted. License details
    Hosting
    15 hosted files. Inspect the file record for current availability. 16 files listed upstream.
    File evidence
    No exact file name or SHA-256 is recorded in this answer row. Open the file record for current file-level evidence; unknown availability is unconfirmed.
    Why listed
    The catalog returned it for the search "gguf". Recorded task: text generation. Listed formats: gguf. Recorded license: apache-2.0. This listing does not measure quality or prove runtime support.

Page generated 2026-10-24. Catalog records may be cached or incomplete.

Cite this page

https://huggingbay.xyz/answers/run-llms-on-consumer-gpu

How do I run open LLMs on a consumer GPU with GGUF quantization?. Hugging Bay. https://huggingbay.xyz/answers/run-llms-on-consumer-gpu

Share this page

Share: Reddit Hacker News 微博

机器可读引用包 (machine-readable citations): citations.json · 中文指南: llms-zh.txt

Quick answer: compatibility and sources

How do I run open LLMs on a consumer GPU with GGUF quantization?

It depends on your task, available memory, file format and license. The records below match this guide’s filters. Check the linked file evidence before choosing: a catalog match does not prove runtime support, safety or answer quality.

An answer pack is a question-specific filter over catalog records. It is not a benchmark, safety review or guarantee that a model will run.

As of .

The machine-readable answer record and its citation list link to the source data behind this guide. Catalog matches are not a benchmark or model test.

Updated · Page-copy source revision 8871bab84e1cf80aea1686a944ec335078a6c036 · Source files: apps/web/server.mjs, apps/web/lib/answers-copy.mjs, apps/web/lib/answers-landing-html.mjs, apps/web/lib/answer-pack-html.mjs, apps/web/src/App.tsx, apps/web/lib/aeo-page.mjs · Full revision record

Key facts

Answer pack
Filters catalog rows for one reader question; order is not a quality ranking.
Citation API
Machine-readable citations link each row to source endpoints.
Hosting line
Describes delivery status separately from license permission.
Unmeasured runs
No row implies a successful model test unless run evidence says so.

Evidence you can inspect (plain text)

  • GET /api/answers indexes public answer packs.
  • GET /api/answers/run-llms-on-consumer-gpu exposes machine-readable rows when the slug exists.
  • Open /artifact/hf-model-qwen-qwen3-8b/files for model exact filenames, availability and any recorded SHA-256.
  • Open /evidence/artifacts/hf-model-qwen-qwen3-8b for run-readiness evidence tied to that record; absent receipts stay unmeasured.
  • Open /api/artifacts/hf-model-qwen-qwen3-8b/card.md for machine-readable source facts (not a runtime test).
  • Guides do not assert model tests unless linked run evidence includes a receipt.

Unmeasured states are stated explicitly; this page does not invent run scores or compatibility guarantees.

Which date applies to which fact

  • Answer API rows assembled:
  • Committed page-copy sources:
  • Upstream model metadata: Dates on artifact records when shown; not implied by page-copy revision
  • Run receipts: Only on /evidence pages when recorded; otherwise unmeasured

Page-copy revision (Updated line above) identifies committed text sources, not upstream model edits or test outcomes.

We use an optional cookie to count visits.

Privacy