TextLoader fails on a text file with an encoding error
RuntimeError: Error loading notes.txtShort answer
TextLoader opens the file with the encoding you give it, or the system default if you give none. When the bytes do not match, it raises RuntimeError: Error loading <path>, chained from a UnicodeDecodeError. Pass the file's real encoding, or set autodetect_encoding=True. Check the result, because autodetect can return an empty document.
Fix it fast
1Find out which cause you have
Scroll up in the traceback to the exception the RuntimeError was chained from. It names the codec that failed and the byte it failed on.
python - <<'PY' import locale print(locale.getencoding()) PY2Match what you see, then jump to its fix
- 1
RuntimeError: Error loading <path> with a UnicodeDecodeError above it, and no encoding passed to TextLoaderThe file is not in the encoding Python used to open itFix - 2
Some files load and others raise the error, even with encoding='utf-8' setA folder of files in mixed encodingsFix - 3
UnicodeDecodeError only on Windows, for files that are valid UTF-8Windows default code page instead of UTF-8Fix
- 1
What does the error look like?
RuntimeError: Error loading notes.txtUnicodeDecodeError: codec can't decode byte
Cause 1 of 3
The file is not in the encoding Python used to open it
1Check
With encoding=None, TextLoader uses the system default. Python's open() docs say that default is platform dependent unless UTF-8 mode is on, so the same file can load on one machine and fail on another.
python -c "import locale; print(locale.getencoding())"
file -i notes.txt2Fix
Tell the loader which encoding the file really uses. UTF-8 is the usual answer; use the codec name that file -i reports if it is something else.
from langchain_community.document_loaders import TextLoader
docs = TextLoader("notes.txt", encoding="utf-8").load()3Verify
load() returns one Document with the text in page_content.
print(len(docs), docs[0].page_content[:80])Confirmed by raw.githubusercontent.com
Cause 2 of 3
A folder of files in mixed encodings
1Check
One fixed encoding cannot fit every file. TextLoader has a flag for this: if the first read fails with a UnicodeDecodeError and autodetect_encoding is True, it tries candidate encodings in order and keeps the first that decodes.
file -i data/*.txt2Fix
Turn on autodetect_encoding. If no candidate decodes, the loader does not raise: it yields a Document with empty text. So test for empty content.
from langchain_community.document_loaders import TextLoader
loader = TextLoader("data/old-export.txt", autodetect_encoding=True)
docs = loader.load()
if not docs[0].page_content.strip():
raise ValueError("could not decode data/old-export.txt")3Verify
The file loads and the text is not empty. Re-save such files as UTF-8 so you can drop the flag.
print(docs[0].page_content[:80])Confirmed by raw.githubusercontent.com
Cause 3 of 3
Windows default code page instead of UTF-8
1Check
Without UTF-8 mode, Python on Windows uses the ANSI code page as the default text encoding. UTF-8 mode became the default only in Python 3.15.
python -c "import sys; print(sys.version_info[:2], sys.flags.utf8_mode)"2Fix
Pass encoding explicitly in code, or switch UTF-8 mode on for the run. The Python docs warn that setting PYTHONUTF8 system-wide affects every Python 3.7+ program, so prefer the per-run flag.
py -X utf8 app.py3Verify
The same script loads the file with no encoding argument.
py -X utf8 -c "import sys; print(sys.flags.utf8_mode)"Confirmed by docs.python.org
How do I stop it coming back?
- Always pass encoding to TextLoader instead of relying on the machine default.
- Convert source files to UTF-8 once, when you ingest them.
- Treat an empty page_content after autodetect as a failed load, not a success.
- Run the same loader in CI on the OS you deploy to, so a Windows-only default shows up early.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
- stegnor.comMy company site: 804 pages from a Python and Jinja generator, with 15 free browser tools.
- MenuloA restaurant discovery platform in Germany, kept current by a Python ingestion pipeline.
- ShopGoAn e-commerce aggregator with a Python scraping engine behind its catalog, past 10K Google Play downloads.
Related fixes
AI
ollama Python library fails with httpx.ConnectError
httpx.ConnectError
httpx.ConnectError means the client could not open a connection to the address it used.
3 causes, each with a fixAI
Azure OpenAI says the API deployment for this resource does not exist
The API deployment for this resource does not exist
Azure OpenAI does not take a model name in the model field.
2 causes, each with a fixAI
Azure OpenAI fails with an SSL certificate verification error
httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED]
The client checks the endpoint's certificate against its CA bundle, which is certifi by default.
2 causes, each with a fixAI
LangChain says it could not import the chromadb package
ImportError: Could not import chromadb python package. Please install it with `pip install chromadb`.
The message comes from LangChain's old Chroma class, which cannot import chromadb in the Python that runs your code.
2 causes, each with a fixAI
Gemini API returns 429 RESOURCE_EXHAUSTED on the free tier
429 RESOURCE_EXHAUSTED
Gemini limits are applied per Google Cloud project, not per API key, so a new key in the same project does not reset anything.
3 causes, each with a fixAI
ChatPromptTemplate is missing the variable agent_scratchpad
Expected: ['agent_scratchpad', 'input']
A tool-calling agent keeps its tool calls and tool results in a prompt slot named agent_scratchpad.
2 causes, each with a fix