Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
AI and LLM integration

TextLoader fails on a text file with an encoding error

ErrorRuntimeError: Error loading notes.txt

Short answer

TextLoader opens the file with the encoding you give it, or the system default if you give none. When the bytes do not match, it raises RuntimeError: Error loading <path>, chained from a UnicodeDecodeError. Pass the file's real encoding, or set autodetect_encoding=True. Check the result, because autodetect can return an empty document.

  • Applies to langchain-community 0.3 and later, Python 3.10-3.14, Windows 11, Ubuntu 24.04
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Scroll up in the traceback to the exception the RuntimeError was chained from. It names the codec that failed and the byte it failed on.

    python - <<'PY'
    import locale
    print(locale.getencoding())
    PY
  2. 2Match what you see, then jump to its fix

    1. 1RuntimeError: Error loading <path> with a UnicodeDecodeError above it, and no encoding passed to TextLoaderThe file is not in the encoding Python used to open itFix
    2. 2Some files load and others raise the error, even with encoding='utf-8' setA folder of files in mixed encodingsFix
    3. 3UnicodeDecodeError only on Windows, for files that are valid UTF-8Windows default code page instead of UTF-8Fix

What does the error look like?

  • RuntimeError: Error loading notes.txt
  • UnicodeDecodeError: codec can't decode byte

Cause 1 of 3

The file is not in the encoding Python used to open it

1Check

With encoding=None, TextLoader uses the system default. Python's open() docs say that default is platform dependent unless UTF-8 mode is on, so the same file can load on one machine and fail on another.

python -c "import locale; print(locale.getencoding())"
file -i notes.txt

2Fix

Tell the loader which encoding the file really uses. UTF-8 is the usual answer; use the codec name that file -i reports if it is something else.

from langchain_community.document_loaders import TextLoader

docs = TextLoader("notes.txt", encoding="utf-8").load()

3Verify

load() returns one Document with the text in page_content.

print(len(docs), docs[0].page_content[:80])

Confirmed by raw.githubusercontent.com

Did this fix it?

Cause 2 of 3

A folder of files in mixed encodings

1Check

One fixed encoding cannot fit every file. TextLoader has a flag for this: if the first read fails with a UnicodeDecodeError and autodetect_encoding is True, it tries candidate encodings in order and keeps the first that decodes.

file -i data/*.txt

2Fix

Turn on autodetect_encoding. If no candidate decodes, the loader does not raise: it yields a Document with empty text. So test for empty content.

from langchain_community.document_loaders import TextLoader

loader = TextLoader("data/old-export.txt", autodetect_encoding=True)
docs = loader.load()
if not docs[0].page_content.strip():
    raise ValueError("could not decode data/old-export.txt")

3Verify

The file loads and the text is not empty. Re-save such files as UTF-8 so you can drop the flag.

print(docs[0].page_content[:80])

Confirmed by raw.githubusercontent.com

Did this fix it?

Cause 3 of 3

Windows default code page instead of UTF-8

1Check

Without UTF-8 mode, Python on Windows uses the ANSI code page as the default text encoding. UTF-8 mode became the default only in Python 3.15.

python -c "import sys; print(sys.version_info[:2], sys.flags.utf8_mode)"

2Fix

Pass encoding explicitly in code, or switch UTF-8 mode on for the run. The Python docs warn that setting PYTHONUTF8 system-wide affects every Python 3.7+ program, so prefer the per-run flag.

py -X utf8 app.py

3Verify

The same script loads the file with no encoding argument.

py -X utf8 -c "import sys; print(sys.flags.utf8_mode)"

Confirmed by docs.python.org

Did this fix it?

How do I stop it coming back?

  • Always pass encoding to TextLoader instead of relying on the machine default.
  • Convert source files to UTF-8 once, when you ingest them.
  • Treat an empty page_content after autodetect as a failed load, not a success.
  • Run the same loader in CI on the OS you deploy to, so a Windows-only default shows up early.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

AI

LangChain says it could not import the chromadb package

ImportError: Could not import chromadb python package. Please install it with `pip install chromadb`.

The message comes from LangChain's old Chroma class, which cannot import chromadb in the Python that runs your code.

2 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.