Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Databases and Redis

Python stops with UnicodeDecodeError: 'charmap' codec can't decode byte

ErrorUnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 3852: character maps to <undefined>

Short answer

Python decoded the file with the wrong encoding. Before Python 3.15, open() without an encoding argument uses the platform locale, and a legacy locale codec can raise this error. Pass the file's real encoding, usually utf-8. If it is not UTF-8, name the right codec or choose an error handler on purpose.

  • Applies to Python 3.12, Python 3.13, Python 3.14
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Print which default encoding your Python uses for open(). If it is not UTF-8, that is where the error comes from.

    python -c "import locale; print(locale.getencoding())"
  2. 2Match what you see, then jump to its fix

    1. 1The traceback ends in the line that calls open(...).read() or iterates a text file, and the file is UTF-8open() uses the locale encoding, not UTF-8Fix
    2. 2encoding='utf-8' now raises a UnicodeDecodeError with 'utf-8' codec insteadThe file itself is not UTF-8Fix
    3. 3Only a handful of lines fail and losing those characters is acceptableA few bad bytes in an otherwise good fileFix

What does the error look like?

  • UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 3852: character maps to <undefined>
  • Unhandled exception: 'charmap' codec can't decode byte

Cause 1 of 3

open() uses the locale encoding, not UTF-8

1Check

In text mode without encoding, Python uses UTF-8 only when UTF-8 Mode is on; otherwise it calls locale.getencoding(). Python 3.15 changes the default to UTF-8.

python -c "import locale; print(locale.getencoding())"

2Fix

Pass encoding explicitly every time you open a text file.

with open("dump.sql", encoding="utf-8") as f:
    sql = f.read()

3Verify

Run the script again.

python import_dump.py

Confirmed by docs.python.org

Did this fix it?

Cause 2 of 3

The file itself is not UTF-8

1Check

The file may be in another encoding, for example Windows-1252 or Latin-1. Python lists cp1252 and latin_1 among its standard codecs.

python -c "open('dump.sql', encoding='utf-8').read()"

2Fix

Open it with the encoding it really has. Do not guess blindly: choose the codec that matches the tool that produced the file.

with open("dump.sql", encoding="cp1252") as f:
    sql = f.read()

3Verify

Read the file and print a line that has accented text.

python -c "print(open('dump.sql', encoding='cp1252').read()[:200])"

Confirmed by docs.python.org

Did this fix it?

Cause 3 of 3

A few bad bytes in an otherwise good file

1Check

The default error handler, strict, raises on every undecodable byte. Other handlers are available.

python -c "print(open('dump.sql', encoding='utf-8', errors='replace').read().count(chr(0xFFFD)))"

2Fix

Use errors='replace' (puts U+FFFD where bytes could not be decoded) or errors='backslashreplace' (keeps the byte as a visible escape). Use this only when you accept the loss.

with open("dump.sql", encoding="utf-8", errors="backslashreplace") as f:
    sql = f.read()

3Verify

The script runs to the end; search the output for the markers you chose.

python import_dump.py

Confirmed by docs.python.org

Did this fix it?

How do I stop it coming back?

  • Always pass encoding= when opening text files, so the code behaves the same on every machine.
  • Export SQL and CSV files as UTF-8 and write that down next to the file.
  • Read binary (mode 'rb') when you only copy a file and never need to interpret its text.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.