Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Databases and Redis

Replication stopped: error 1032, coordinator stopped because of worker errors

ErrorLast_SQL_Errno: 1032

Short answer

Error 1032 on a replica means a change from the source could not be applied because the replica's data no longer matches the source. Find the failing transaction in performance_schema, make the replica row match the source, and only skip the transaction (empty GTID transaction or skip counter) when you know it is safe.

  • Applies to MySQL 8.0.22+, MySQL 8.4
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    With a multi-threaded replica the SHOW REPLICA STATUS error is only a summary. The worker row holds the real error and, with GTIDs, the failing transaction.

    SELECT CHANNEL_NAME, WORKER_ID, APPLYING_TRANSACTION,
           LAST_ERROR_NUMBER, LAST_ERROR_MESSAGE
    FROM performance_schema.replication_applier_status_by_worker;
  2. 2Match what you see, then jump to its fix

    1. 1Error 1032 in the worker row, and the data it names differs between source and replicaThe replica's table has drifted from the sourceFix
    2. 2APPLYING_TRANSACTION shows a GTID in the form uuid:number, and the change is not needed on the replicaGTID replication: the one failing transaction can be skippedFix
    3. 3gtid_mode is OFF and the failing statement is not needed on the replicaFile and position replication: skip with the counterFix

What does the error look like?

  • Last_SQL_Errno: 1032
  • Can't find record in 'tablename'
  • Coordinator stopped because there were error(s) in the worker(s)

Cause 1 of 3

The replica's table has drifted from the source

1Check

Look at the transaction's events, then look for the row on both servers. The manual's advice is to find out whether the replica's table differs from the source's, and why.

-- on the replica
SHOW RELAYLOG EVENTS IN 'replica-relay-bin.000012' LIMIT 20;
-- on the source
SHOW BINLOG EVENTS IN 'binlog.000145' LIMIT 20;

2Fix

Make the replica's table identical to the source's (insert the missing row, or copy the table from the source), then restart the applier. If many tables differ, resynchronise the replica from a new snapshot of the source instead of patching row by row.

START REPLICA;

3Verify

Both threads report Yes and Last_SQL_Errno is 0.

SHOW REPLICA STATUS\G

Confirmed by dev.mysql.com

Did this fix it?

Cause 2 of 3

GTID replication: the one failing transaction can be skipped

1Check

Confirm that the transaction really should not run on the replica: it may be from an unknown source, or it may need to be applied by hand first.

SELECT APPLYING_TRANSACTION FROM performance_schema.replication_applier_status_by_worker;

2Fix

Stop the replica, commit an empty transaction that carries the failing GTID, set gtid_next back to automatic and start again. The replica sees the GTID as already applied and passes over it. The empty transaction lands in the binary log, so the manual suggests flushing and purging if this replica may become a source.

STOP REPLICA;
SET GTID_NEXT='aaa-bbb-ccc-ddd:N';
BEGIN;
COMMIT;
SET GTID_NEXT='AUTOMATIC';
START REPLICA;

3Verify

The worker row is clean and the replica moves on to the next transaction.

SELECT LAST_ERROR_NUMBER, LAST_ERROR_MESSAGE
FROM performance_schema.replication_applier_status_by_worker;

Confirmed by dev.mysql.com

Did this fix it?

Cause 3 of 3

File and position replication: skip with the counter

1Check

Check gtid_mode, and that the failing event is a single statement. The skip counter counts events, not transactions, and a statement that uses AUTO_INCREMENT or LAST_INSERT_ID() counts as two.

SHOW VARIABLES LIKE 'gtid_mode';

2Fix

Stop the replica, set the counter, start again. Use this only after you have decided that skipping is safe; the manual describes it as a last resort when a full resynchronisation is not practical.

STOP REPLICA;
SET GLOBAL sql_replica_skip_counter = 1;
START REPLICA;

3Verify

The counter resets to 0 on start and the SQL thread runs.

SHOW REPLICA STATUS\G

Confirmed by dev.mysql.com

Did this fix it?

How do I stop it coming back?

  • Never change data on a replica outside the replication threads; the manual names this as the cause of this kind of divergence.
  • Seed a new replica from a proper snapshot of the source with a recorded position.
  • Write down every skip you do, with the GTID or position, so the difference can be reconciled later.
  • Alert on replication errors from performance_schema so a stop is found in minutes.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.