Python stops with UnicodeDecodeError: 'charmap' codec can't decode byte
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 3852: character maps to <undefined>Short answer
Python decoded the file with the wrong encoding. Before Python 3.15, open() without an encoding argument uses the platform locale, and a legacy locale codec can raise this error. Pass the file's real encoding, usually utf-8. If it is not UTF-8, name the right codec or choose an error handler on purpose.
Fix it fast
1Find out which cause you have
Print which default encoding your Python uses for open(). If it is not UTF-8, that is where the error comes from.
python -c "import locale; print(locale.getencoding())"2Match what you see, then jump to its fix
- 1
The traceback ends in the line that calls open(...).read() or iterates a text file, and the file is UTF-8open() uses the locale encoding, not UTF-8Fix - 2
encoding='utf-8' now raises a UnicodeDecodeError with 'utf-8' codec insteadThe file itself is not UTF-8Fix - 3
Only a handful of lines fail and losing those characters is acceptableA few bad bytes in an otherwise good fileFix
- 1
What does the error look like?
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 3852: character maps to <undefined>Unhandled exception: 'charmap' codec can't decode byte
Cause 1 of 3
open() uses the locale encoding, not UTF-8
1Check
In text mode without encoding, Python uses UTF-8 only when UTF-8 Mode is on; otherwise it calls locale.getencoding(). Python 3.15 changes the default to UTF-8.
python -c "import locale; print(locale.getencoding())"2Fix
Pass encoding explicitly every time you open a text file.
with open("dump.sql", encoding="utf-8") as f:
sql = f.read()3Verify
Run the script again.
python import_dump.pyConfirmed by docs.python.org
Cause 2 of 3
The file itself is not UTF-8
1Check
The file may be in another encoding, for example Windows-1252 or Latin-1. Python lists cp1252 and latin_1 among its standard codecs.
python -c "open('dump.sql', encoding='utf-8').read()"2Fix
Open it with the encoding it really has. Do not guess blindly: choose the codec that matches the tool that produced the file.
with open("dump.sql", encoding="cp1252") as f:
sql = f.read()3Verify
Read the file and print a line that has accented text.
python -c "print(open('dump.sql', encoding='cp1252').read()[:200])"Confirmed by docs.python.org
Cause 3 of 3
A few bad bytes in an otherwise good file
1Check
The default error handler, strict, raises on every undecodable byte. Other handlers are available.
python -c "print(open('dump.sql', encoding='utf-8', errors='replace').read().count(chr(0xFFFD)))"2Fix
Use errors='replace' (puts U+FFFD where bytes could not be decoded) or errors='backslashreplace' (keeps the byte as a visible escape). Use this only when you accept the loss.
with open("dump.sql", encoding="utf-8", errors="backslashreplace") as f:
sql = f.read()3Verify
The script runs to the end; search the output for the markers you chose.
python import_dump.pyConfirmed by docs.python.org
How do I stop it coming back?
- Always pass encoding= when opening text files, so the code behaves the same on every machine.
- Export SQL and CSV files as UTF-8 and write that down next to the file.
- Read binary (mode 'rb') when you only copy a file and never need to interpret its text.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
- stegnor.comMy company site: 804 pages from a Python and Jinja generator, with 15 free browser tools.
- MenuloA restaurant discovery platform in Germany, kept current by a Python ingestion pipeline.
- ShopGoAn e-commerce aggregator with a Python scraping engine behind its catalog, past 10K Google Play downloads.
Related fixes
Databases
Installing the Airflow MySQL provider fails on pkg-config or mysqlclient
Building the mysqlclient wheel fails during the install
The Airflow MySQL provider depends on mysqlclient, which pip compiles against the MySQL C client library.
2 causes, each with a fixDatabases
asyncpg InterfaceError: another operation is in progress with SQLAlchemy
InterfaceError raised from a FastAPI endpoint that runs several queries at once
A single database connection can run one operation at a time.
4 causes, each with a fixDatabases
Django cannot load MySQLdb and asks whether mysqlclient is installed
Error loading MySQLdb module: Did you install mysqlclient or MySQL-python?
Django's MySQL backend imports MySQLdb, which is provided by the mysqlclient package.
3 causes, each with a fixDatabases
python manage.py migrate finishes but the tables for my app are missing
relation "myapp_book" does not exist
migrate only applies migration files that exist.
3 causes, each with a fixDatabases
Django save fails on PostgreSQL: null value in a column violates a not-null constraint
django.db.utils.IntegrityError: null value in column "price" of relation "shop_product" violates not-null constraint
A model field is NOT NULL in the database unless it has null=True.
3 causes, each with a fixDatabases
Django migration on PostgreSQL stops with 'pending trigger events'
django.db.utils.OperationalError: cannot ALTER TABLE "mytable" because it has pending trigger events
PostgreSQL runs a Django migration in one transaction.
2 causes, each with a fix