Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Node.js, PHP and APIs

json_encode returns false or an empty result because of bad UTF-8

ErrorMalformed UTF-8 characters, possibly incorrectly encoded

Short answer

json_encode only accepts UTF-8. When a string holds other bytes (Latin-1 from an old table, a wrongly set connection charset, or a string cut in the middle of a character) it returns false and json_last_error_msg() says Malformed UTF-8. Fix the encoding where the data enters PHP. Use the substitute flag only as a safety net.

  • Applies to PHP 8.3, PHP 8.4
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Ask PHP why the encode failed, and which column holds the bad text. Throwing makes the failure impossible to miss.

    $json = json_encode($rows);
    if ($json === false) {
        echo json_last_error_msg();
    }
    // or throw instead of returning false:
    $json = json_encode($rows, JSON_THROW_ON_ERROR);
  2. 2Match what you see, then jump to its fix

    1. 1Rows that contain accents, Arabic, Urdu or emoji fail, plain English rows encode fineThe database connection is not talking UTF-8Fix
    2. 2The connection is already utf8mb4, yet one specific row or column always breaksThe stored text is in another encoding such as Windows-1252Fix
    3. 3The failing value is a shortened preview or excerpt, often 100 or 200 characters longA string was cut in the middle of a characterFix
    4. 4The bad bytes come from outside data you do not control, and one bad value must not break the whole responseYou need output now and can accept a replacement characterFix

What does the error look like?

  • Malformed UTF-8 characters, possibly incorrectly encoded
  • json_encode() returns false
  • JsonException: Malformed UTF-8 characters, possibly incorrectly encoded

Cause 1 of 4

The database connection is not talking UTF-8

1Check

mysqli starts with the server's default connection charset, which can be latin1. The text then reaches PHP in that charset and json_encode rejects it. Print the charset the connection really uses.

$mysqli = new mysqli('localhost', 'my_user', 'my_password', 'my_db');
echo $mysqli->character_set_name();

2Fix

Set the connection charset to utf8mb4 right after connecting. The manual calls set_charset() the preferred way, not a SET NAMES query.

$mysqli->set_charset('utf8mb4');

3Verify

Run the same query and encode it again. json_last_error_msg() should report No error.

json_encode($rows);
echo json_last_error_msg();

Confirmed by php.net

Did this fix it?

Cause 2 of 4

The stored text is in another encoding such as Windows-1252

1Check

Test each value with mb_check_encoding to find the rows that are not valid UTF-8.

foreach ($rows as $i => $row) {
    foreach ($row as $col => $val) {
        if (is_string($val) && !mb_check_encoding($val, 'UTF-8')) {
            echo "row $i column $col\n";
        }
    }
}

2Fix

Convert the string to UTF-8 and say which encoding it really is. Detection guesses and can be wrong, so name the source encoding when you know it. Best of all, repair the stored data once.

$clean = mb_convert_encoding($val, 'UTF-8', 'Windows-1252');

3Verify

Encode the converted value and compare the output with what you expect.

echo json_encode($clean, JSON_UNESCAPED_UNICODE);

Confirmed by php.net

Did this fix it?

Cause 3 of 4

A string was cut in the middle of a character

1Check

substr counts bytes. In UTF-8 a character can take up to four bytes, so cutting at a byte offset can leave half a character at the end.

grep -rn "substr(" src/ app/

2Fix

Use mb_substr, which counts characters, and pass the encoding explicitly.

$preview = mb_substr($text, 0, 200, 'UTF-8');

3Verify

Encode the new preview. A value that failed before should now pass.

var_dump(json_encode($preview));

Confirmed by php.net

Did this fix it?

Cause 4 of 4

You need output now and can accept a replacement character

1Check

PHP 7.2 and later has two flags for this. Ignore drops the invalid bytes. Substitute replaces each one with U+FFFD, so you can see where data was lost.

php -r 'echo json_encode("abc\xB1", JSON_INVALID_UTF8_SUBSTITUTE);'

2Fix

Add the substitute flag as a safety net. It hides the problem, so keep fixing the source too.

$json = json_encode($rows, JSON_INVALID_UTF8_SUBSTITUTE | JSON_THROW_ON_ERROR);

3Verify

The command above prints the string with \ufffd in place of the bad byte.

php -r 'echo json_encode("abc\xB1", JSON_INVALID_UTF8_IGNORE);'

Confirmed by php.net

Did this fix it?

How do I stop it coming back?

  • Create tables and connections as utf8mb4 and set the charset in the connection code, not per query.
  • Use the mb_* functions for any string length, cut or search on user text.
  • Pass JSON_THROW_ON_ERROR so a failed encode stops with a message instead of sending false.
  • Convert imported CSV and legacy data to UTF-8 once, on the way in.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Back end

Apache returns 403 when a URL contains %3F

a request like /page%3Fname=test is refused while /page?name=test works

Recent Apache httpd blocks a rewrite when the request contains an encoded question mark (%3F) and the rewritten target has a literal ?.

2 causes, each with a fix

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.