Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
AI and LLM integration

SafetensorError: Error while deserializing header: HeaderTooLarge

Errorsafetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

Short answer

A safetensors file starts with 8 bytes that give the size of a JSON header, and the library caps that header at 100MB. HeaderTooLarge means the first bytes are not a valid size, so the file is not real weights. It is usually a small Git LFS pointer text file or a damaged download. Look at the file, then download it again.

  • Applies to Python 3.12, safetensors 0.x, huggingface_hub 1.x
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Look at the size and the first bytes of the file. A real weights file is many megabytes; a pointer is a few lines of text.

    ls -l model.safetensors
    head -c 200 model.safetensors
  2. 2Match what you see, then jump to its fix

    1. 1The file is under 1 KB and head prints text starting with version https://git-lfs.github.com/spec/v1The file is a Git LFS pointer, not the weightsFix
    2. 2The file has a normal size but head shows HTML or garbage, or the transfer was interruptedThe download is damaged or is not a safetensors fileFix

What does the error look like?

  • safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

Cause 1 of 2

The file is a Git LFS pointer, not the weights

1Check

The Git LFS spec says a pointer is a UTF-8 text file under 1024 bytes with the keys version, oid and size. The safetensors format reads the first 8 bytes as the header size, so text there gives an impossible size, far above the 100MB header cap.

head -n 3 model.safetensors

2Fix

Download the real file with the Hugging Face CLI. It prints the local path of the file.

hf download your-user/your-model model.safetensors

3Verify

The file is now large and opens.

# use the path that hf download printed
ls -lh PATH_FROM_HF_DOWNLOAD
python -c "from safetensors import safe_open; f = safe_open('PATH_FROM_HF_DOWNLOAD', framework='pt'); print(len(list(f.keys())))"

Confirmed by raw.githubusercontent.com

Did this fix it?

Cause 2 of 2

The download is damaged or is not a safetensors file

1Check

The safetensors README states a 100MB limit on the header, kept to stop parsing of extremely large JSON. A cut-off or wrong file fails this check the same way.

file model.safetensors

2Fix

Force a fresh download so the cached copy is not reused.

from huggingface_hub import hf_hub_download
path = hf_hub_download(repo_id="your-user/your-model", filename="model.safetensors", force_download=True)
print(path)

3Verify

Open the file with safe_open and list its tensors.

python -c "from safetensors import safe_open; print(len(list(safe_open('PATH_FROM_ABOVE', framework='pt').keys())))"

Confirmed by huggingface.co

Did this fix it?

How do I stop it coming back?

  • Download model files with the hf command or huggingface_hub, which handle large files.
  • After copying a model between machines, compare file sizes with the Hub page.
  • If you use git for models, install the large-file extension the Hub names before cloning.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

AI

LangChain says it could not import the chromadb package

ImportError: Could not import chromadb python package. Please install it with `pip install chromadb`.

The message comes from LangChain's old Chroma class, which cannot import chromadb in the Python that runs your code.

2 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.