How to Recover Crashed InnoDB Tables Safely

Recovering crashed InnoDB tables is not the same as repairing a MyISAM table. InnoDB protects transactions through a connected set of data pages, redo information, undo records and internal metadata. When that structure is damaged, REPAIR TABLE and mysqlcheck --repair do not reconstruct the missing InnoDB data.

The recovery objective is therefore very specific: preserve the original evidence, restore from a verified backup when possible, or start a protected copy in the lowest viable recovery mode long enough to export readable data. The exported schemas and rows are then loaded into a clean database instance.

This guide covers a severe case in which MySQL repeatedly fails during InnoDB initialization or crashes while reading an affected table. It includes a complete workflow for diagnosis, offline preservation, innodb_force_recovery, logical dumps, damaged-table extraction and validation. Commands are examples for a Linux server and must be adjusted for the installed MySQL or MariaDB version, service name, paths and authentication method.

If the data is valuable and no tested backup exists, stop working on the only copy. Take a storage snapshot or create an offline copy before changing recovery settings. Values of innodb_force_recovery at 4 or above can permanently damage data files.

Understand what an InnoDB crash can mean

An unclean shutdown does not automatically mean corruption. InnoDB normally replays redo information during startup and rolls incomplete transactions back. On a healthy storage system, that crash recovery may simply need time.

A genuine recovery incident becomes more likely when MySQL repeatedly exits at the same point, reports page checksum failures, cannot open a tablespace, raises an internal assertion, or receives short reads from the operating system. The cause may be confined to one table, or it may involve a shared component such as the system tablespace, redo files, undo tablespaces or data dictionary.

Common causes include:

• an interrupted filesystem or storage operation
• a failing disk, controller or virtual storage layer
• a full filesystem or exhausted inode allocation
• a truncated or missing tablespace file
• an unsafe copy of a live MySQL data directory
• a version or package change performed during an incomplete shutdown
• memory or hardware faults that produced damaged pages
• an application query reaching a corrupt page that had not been read recently

The correct response depends on the failure. Replacing configuration files will not repair a storage fault, and deleting log files will not repair a corrupt table. Diagnose the layer before choosing the recovery method.

Read the complete startup timeline

Start with the service manager and database error log. On a systemd server, the unit may be called mysqld, mysql or mariadb:

systemctl status mysqld --no-pager -l
journalctl -u mysqld --since "30 minutes ago" --no-pager

Check the configured error log as well. Typical locations include /var/log/mysql/error.log, /var/log/mysqld.log, /var/log/mariadb/mariadb.log and the database data directory. Do not rely on one line copied from a control panel. Save the complete sequence from process start to process exit.

The following is an original illustrative example. It is not copied from a customer server:

2026-08-23T02:14:51.204Z [Note] [InnoDB] Starting crash recovery from checkpoint 4871932501
2026-08-23T02:14:52.071Z [ERROR] [InnoDB] Page checksum mismatch in tablespace appdata/orders, page 18432
2026-08-23T02:14:52.071Z [ERROR] [InnoDB] Expected checksum 0x62a91c40, calculated 0x17bd03e8
2026-08-23T02:14:52.074Z [ERROR] [InnoDB] Unable to continue redo application for the affected page
2026-08-23T02:14:52.075Z [ERROR] [InnoDB] Plugin initialization stopped because a data page could not be validated
2026-08-23T02:14:52.076Z [ERROR] [Server] Data dictionary storage engine initialization failed
2026-08-23T02:14:52.077Z [ERROR] [Server] Aborting

This pattern suggests that normal crash recovery is reaching a damaged page. A different incident might show a short file read, permission error, missing file or lack of disk space. Those are not interchangeable diagnoses.

Check storage and operating-system evidence

Before enabling forced recovery, confirm that the server can reliably read and write storage:

df -hT
df -ih
journalctl -k --since "2 hours ago" --no-pager
dmesg -T | tail -n 200

Look for filesystem errors, I/O timeouts, device resets, read-only remounts and out-of-memory kills. If a physical disk is involved and SMART tools are available, preserve its health report. If the kernel is still reporting storage errors, continuing database recovery on the same device can damage both the source and the output files.

Also record the installed server build. The client and server packages can differ, so use more than one source when available:

mysqld --version
mysql --version
rpm -qa | grep -Ei 'mysql|mariadb' | sort

On Debian or Ubuntu, use dpkg -l instead of rpm -qa. Save the active MySQL configuration and note whether the server uses MySQL, Percona Server or MariaDB. Do not switch products or perform an upgrade during the extraction attempt.

Confirm the storage engine when MySQL still runs

An application may describe any unavailable table as crashed. If MySQL remains accessible, confirm the engine before applying an InnoDB procedure:

SELECT TABLE_SCHEMA, TABLE_NAME, ENGINE
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = 'appdata'
  AND TABLE_NAME = 'orders';

REPAIR TABLE applies to engines such as MyISAM, ARCHIVE and CSV, not InnoDB. For an InnoDB table that can still be read, export it before running intensive checks. A full scan can touch the damaged page and terminate the process.

If the problem appeared after a package update, confirm that the old server had fully stopped before the new binary started. If the log reports a missing file or permission error, verify the configured data directory, ownership and security policy instead of assuming page corruption.

Choose the least destructive recovery path

Use the first viable path in this order:

• restore the affected database from a recent backup that has passed a test restore
• export the affected database immediately if normal MySQL startup remains stable
• recover from a healthy replica if it contains the required transactions
• create an offline copy and test forced recovery on that copy
• extract unaffected databases and tables before attempting damaged objects
• rebuild a clean instance from the logical data that could be recovered

Do not make a forced-recovery instance the new production server. Its purpose is extraction, not continued service.

Freeze the incident and preserve an offline copy

First stop automated restart loops. A process supervisor, control panel or orchestration system may keep starting MySQL after every failure. Disable that behavior temporarily so each attempt is deliberate and documented.

Stop the database service and verify that no server process remains:

systemctl stop mysqld
systemctl is-active mysqld
pgrep -a mysqld

Use the correct unit name for the server. Do not copy the data directory while mysqld is still writing to it unless the snapshot system provides application-consistent database snapshots.

Record the important paths before copying:

mysqld --verbose --help 2>/dev/null | grep -A 1 "Default options"
grep -R "^[[:space:]]*datadir" /etc/my.cnf /etc/my.cnf.d /etc/mysql 2>/dev/null

Create a storage snapshot when the platform supports it. Otherwise copy the complete data directory to storage with enough free space. The following example assumes the database has stopped and the actual data directory is /var/lib/mysql:

mkdir -p /srv/mysql-recovery/source-copy
rsync -aHAX --numeric-ids /var/lib/mysql/ /srv/mysql-recovery/source-copy/

Preserve the full unit, not only a suspected .ibd file. Depending on the version and configuration, recovery may require the system tablespace, per-table tablespaces, redo information, undo tablespaces, general tablespaces, data dictionary and system schema.

Save the configuration and logs beside the copy:

mkdir -p /srv/mysql-recovery/evidence
cp -a /etc/my.cnf /srv/mysql-recovery/evidence/ 2>/dev/null || true
cp -a /etc/my.cnf.d /srv/mysql-recovery/evidence/ 2>/dev/null || true
cp -a /etc/mysql /srv/mysql-recovery/evidence/ 2>/dev/null || true
journalctl -u mysqld --since "24 hours ago" --no-pager > /srv/mysql-recovery/evidence/mysqld-journal.txt

Confirm that the copy has a plausible size and create a file manifest:

du -sh /var/lib/mysql /srv/mysql-recovery/source-copy
find /srv/mysql-recovery/source-copy -type f -print0 | sort -z | xargs -0 sha256sum > /srv/mysql-recovery/evidence/source-copy.sha256

Keep this preserved source copy unchanged. Create a second working copy for recovery tests. If a high recovery level makes the working files worse, you can return to the preserved state.

Export immediately when normal startup is possible

If MySQL starts normally and remains stable, export critical business data before experimenting with configuration. Start with the databases that would be hardest to recreate.

A typical logical export from a healthy InnoDB server is:

mysqldump --databases appdata billingdata \
  --routines --events --triggers \
  --single-transaction --quick --hex-blob \
  --set-gtid-purged=OFF \
  > /srv/mysql-recovery/critical-databases.sql \
  2> /srv/mysql-recovery/critical-databases.err

Authentication is intentionally omitted from these examples. Use socket authentication, a protected client option file or another approved credential method. Do not place a database password directly in a shared shell history or process list.

--single-transaction is useful only while the server can maintain a consistent transaction. If startup is unstable or forced recovery is required, dump objects in smaller groups and avoid assuming that one long transaction will finish.

MySQL documents mysqldump as a logical backup tool and requires explicit --routines and --events options when those objects must be included. Review the options supported by the installed client in the official mysqldump documentation and with mysqldump --help.

Use innodb_force_recovery only to extract data

When normal InnoDB startup fails on the protected working copy, innodb_force_recovery may disable enough background work to make readable data accessible. It does not repair damaged pages.

Add the option under the [mysqld] section of the configuration file actually read by the server:

[mysqld]
innodb_force_recovery=1

Start at 1. Start the copied instance, capture its complete log and test only basic reads. If it cannot start, stop it cleanly, confirm that no database process remains, restore a fresh working copy if necessary, and increase the value by one.

Do not jump directly to 6. MySQL defines progressively more invasive levels:

Values 4 through 6 must be treated as last-resort extraction modes on a separate copy. MySQL specifically warns that values of 4 or greater can permanently corrupt data files. The complete version-specific behavior is documented in Forcing InnoDB Recovery.

Older articles sometimes recommend unrelated purge-thread settings alongside forced recovery. Do not add legacy options automatically. Recovery-level behavior and configuration support vary by version, especially between MySQL and MariaDB. Use only the setting needed for the installed server and the failure shown in its log.

What to check after the forced start

Do not judge success only by a green service status. Review the new error log, then run low-impact checks:

SELECT VERSION();
SELECT @@innodb_force_recovery;
SHOW DATABASES;
SELECT TABLE_SCHEMA, COUNT(*) AS table_count
FROM information_schema.TABLES
WHERE TABLE_SCHEMA NOT IN ('information_schema','performance_schema','sys')
GROUP BY TABLE_SCHEMA;

Confirm that the intended recovery setting is active. Check whether critical schemas and tables are visible. Avoid application traffic, scheduled jobs and monitoring actions that execute complex queries. Even when the server accepts connections, individual tables may still crash a session or the entire process.

Build a recoverable database inventory

Create an inventory before starting a long dump. Exclude virtual schemas and record each database separately:

mkdir -p /srv/mysql-recovery/dumps

mysql --batch --skip-column-names -e "
SELECT SCHEMA_NAME
FROM information_schema.SCHEMATA
WHERE SCHEMA_NAME NOT IN ('information_schema','performance_schema','sys')
ORDER BY SCHEMA_NAME;" > /srv/mysql-recovery/dumps/database-list.txt

Review the file manually. The mysql schema contains accounts, privileges and server metadata, but copying or dropping it casually is unsafe. On a clean replacement instance, users should normally be recreated with supported account-management statements rather than importing an incompatible system schema from another server version.

Inventory the user tables as well:

mysql --batch --skip-column-names -e "
SELECT TABLE_SCHEMA, TABLE_NAME, ENGINE
FROM information_schema.TABLES
WHERE TABLE_TYPE='BASE TABLE'
  AND TABLE_SCHEMA NOT IN ('mysql','information_schema','performance_schema','sys')
ORDER BY TABLE_SCHEMA, TABLE_NAME;" \
> /srv/mysql-recovery/dumps/table-list.txt

This list becomes the recovery ledger. Mark databases and tables as complete, partial or failed instead of assuming that an all-databases command captured everything.

Dump databases individually and record every result

Exporting one database at a time keeps a failure in one schema from hiding successful work elsewhere. The following Bash example records the exit code and error output for every database:

while IFS= read -r db; do
  safe_name=$(printf '%s' "$db" | tr -c 'A-Za-z0-9_.-' '_')
  output="/srv/mysql-recovery/dumps/${safe_name}.sql"
  errors="/srv/mysql-recovery/dumps/${safe_name}.err"

  mysqldump --databases "$db" \
    --routines --events --triggers \
    --quick --skip-lock-tables --hex-blob \
    --no-tablespaces --set-gtid-purged=OFF \
    > "$output" 2> "$errors"

  status=$?
  printf '%s\t%s\t%s\n' "$db" "$status" "$(wc -c < "$output")" \
    >> /srv/mysql-recovery/dumps/results.tsv
done < /srv/mysql-recovery/dumps/database-list.txt

Option support differs across releases. Check the installed client before using this script. For MariaDB, use its matching dump utility and options rather than assuming full command-line compatibility with MySQL.

Export the most important schemas first. A corrupt low-value archive table should not prevent recovery of current orders, accounts or configuration data.

Verify logical dumps before trusting them

A file with an .sql extension is not automatically a successful backup. For every dump:

• check the command exit status
• read the corresponding error file
• confirm that the file size is plausible
• inspect the beginning for the expected database and table definitions
• inspect the end for an abrupt stop or incomplete statement
• store a checksum of the completed file
• perform a test import into a separate clean instance

Create a dump manifest after the extraction pass:

cd /srv/mysql-recovery/dumps
sha256sum *.sql > SHA256SUMS
column -t -s $'\t' results.tsv

Do not erase a source database because a dump command returned zero. Validation happens on a different instance, and the original protected copy remains unchanged until the business confirms the restored data.

Recover a table that fails during a full scan

A damaged page can stop a table dump at the same point on every attempt. Recover other tables first, then treat the failed table as a separate extraction project.

Start by saving its definition without reading row data:

mysqldump appdata orders \
  --no-data --triggers \
  > /srv/mysql-recovery/dumps/appdata-orders-schema.sql

If the table has a numeric primary key, find its readable bounds:

SELECT MIN(order_id), MAX(order_id) FROM appdata.orders;

Export rows in controlled ranges. Use non-overlapping boundaries and save each range independently:

mysqldump appdata orders \
  --no-create-info --skip-triggers --quick --skip-lock-tables \
  --where='order_id >= 1 AND order_id < 250000' \
  > /srv/mysql-recovery/dumps/orders-000001-249999.sql \
  2> /srv/mysql-recovery/dumps/orders-000001-249999.err

Continue with the next range. If one range fails, divide only that range into smaller sections until the unreadable area is isolated. Keep an explicit list of skipped keys or intervals. That is a partial recovery and must be reported as such.

If an ascending scan reaches a damaged region, a descending primary-key query may recover rows located after that region. Complex predicates and ordering can also touch damaged indexes, especially at high force-recovery levels, so start with the simplest query that can return useful rows.

For tables without a suitable key, SELECT ... INTO OUTFILE may help extract readable query results, subject to the server's secure_file_priv setting and FILE privilege. Exported delimited data also requires a separately preserved table definition and careful handling of character sets, NULL values and binary columns.

When one column triggers the failure

A large BLOB, TEXT value or secondary index can expose corruption that a narrower query avoids. Test selected columns on the working copy:

SELECT order_id, customer_id, created_at, status
FROM appdata.orders
WHERE order_id >= 500000 AND order_id < 510000
ORDER BY order_id;

Recovering core columns without an unreadable attachment may be more valuable than losing the entire row. Document every omitted column and range so application owners understand what was recovered.

Do not use --force and then call the resulting dump complete. Continuing after SQL errors can be useful for salvage, but the error log and recovery ledger must identify every object that failed.

Understand file-per-table recovery limits

A visible .ibd file is not a self-describing backup in every situation. Importing a file-per-table tablespace normally requires a compatible server, the exact table definition and matching tablespace metadata. Encryption, compression, partitions, discarded tablespaces, data-dictionary state and version differences can prevent import.

Transportable tablespace procedures can be valuable when logical reads are impossible, but they are not a universal shortcut. Perform this work on a separate instance and preserve the original file. Never overwrite the only .ibd copy during an import experiment.

Rebuild on a clean database instance

Once the readable data has been exported, provision a clean instance using the same database family and a compatible version. Avoid combining data recovery with a major-version migration. First recover and validate; then plan upgrades separately.

The clean instance must not contain innodb_force_recovery. Confirm its value before importing:

SELECT @@innodb_force_recovery;

The expected value is 0.

Load the schema and data dumps in a controlled order. For a complete multi-database dump:

mysql < /srv/mysql-recovery/dumps/critical-databases.sql \
  2> /srv/mysql-recovery/dumps/critical-databases-import.err

For a damaged table recovered in ranges, import its schema once, then load each verified data segment in primary-key order. Record every import exit code and review duplicate-key, foreign-key and character-set errors rather than suppressing them.

Recreate database accounts and grants through supported CREATE USER and GRANT statements appropriate to the target version. Reconfigure replication, scheduled events, backup jobs and monitoring only after the restored data has passed validation.

Do not move the damaged ibdata1, redo files or whole data directory into the clean instance. The logical rebuild is what separates recovered rows from the corrupted storage state.

Validate the recovered databases

Validation should answer both technical and business questions. A successful import proves that SQL statements were accepted, not that every required record survived.

Compare:

• database and table inventories
• row counts for critical tables and recovered key ranges
• minimum and maximum timestamps or identifiers
• recent orders, account records or other business-critical samples
• stored routines, triggers, views and scheduled events
• users and effective privileges
• application reads and writes in a controlled test environment
• foreign-key relationships and expected uniqueness rules
• replication state and binary-log configuration where relevant
• backup execution and a fresh test restore

Example inventory queries include:

SELECT TABLE_SCHEMA, COUNT(*) AS tables
FROM information_schema.TABLES
WHERE TABLE_TYPE='BASE TABLE'
  AND TABLE_SCHEMA NOT IN ('mysql','information_schema','performance_schema','sys')
GROUP BY TABLE_SCHEMA;

SELECT COUNT(*) AS orders,
       MIN(order_id) AS first_id,
       MAX(order_id) AS last_id,
       MIN(created_at) AS first_created,
       MAX(created_at) AS last_created
FROM appdata.orders;

Compare these values with previous reports, replicas, application exports or other independent records. For tables recovered in ranges, verify that the documented gaps match the actual restored keys.

Keep the protected source copy until the application owner has accepted the recovery and the new backup set has passed a restore test.

Remove recovery mode from every startup path

After extraction, remove the temporary option from the copied instance and any configuration-management source that could reintroduce it. Search common configuration paths:

grep -R "innodb_force_recovery" /etc/my.cnf /etc/my.cnf.d /etc/mysql 2>/dev/null

Do not return the damaged instance to production simply because it starts without the option. A clean logical rebuild and validation are required. Preserve the incident logs and document the exact level used, which objects failed, which ranges were skipped and how the replacement instance was tested.

Actions that commonly make recovery worse

Avoid these actions on the original data:

• running mysqlcheck --repair and assuming it repairs InnoDB pages
• increasing innodb_force_recovery directly to 6
• enabling production traffic while the server is in recovery mode
• deleting or replacing ibdata1, redo logs or undo files before preserving a copy
• dropping damaged databases before a verified export exists
• copying only selected .ibd files and discarding the rest of the data directory
• initializing MySQL over the existing data directory
• changing database products or major versions during the recovery attempt
• performing recursive ownership or security-label changes without evidence of a permission fault
• using a dump's file size as the only proof of success
• hiding failed tables or key ranges from the recovery report
• repeating uncontrolled restarts without saving each new log sequence

The old shortcut of dropping databases, moving every ib* file away and allowing MySQL to create new files can erase the only recoverable transaction and metadata state. A clean rebuild belongs on a separate instance after logical data has been exported and verified.

Prevent another InnoDB recovery emergency

Backups are useful only when they can be restored. Maintain automated backups with retention outside the database server and schedule test restores. Large or high-change databases may need suitable physical backups plus logical exports of critical schemas.

Monitor:

• backup completion and restore-test results
• filesystem capacity and inode use
• disk and virtual-storage health
• database crash loops and assertion errors
• replica health and replication lag
• unusual growth in redo, undo or temporary space
• package and major-version changes
• memory pressure and out-of-memory kills
• storage latency that can turn shutdowns into forced restarts

Plan enough free capacity for a complete offline copy, logical dumps and a clean replacement instance. During an incident, insufficient recovery storage often forces risky decisions.

InnoDB recovery questions

Can REPAIR TABLE fix an InnoDB table?

No. MySQL documents REPAIR TABLE for MyISAM, ARCHIVE and CSV tables. When an InnoDB table is corrupt, the supported direction is backup restoration or a controlled dump-and-reload workflow, using forced recovery only when necessary to make data readable.

Should I delete ibdata1 or the redo logs when MySQL will not start?

Not on the only copy. Those files can contain information required to interpret or recover the remaining data. Preserve the complete stopped data directory first. Rebuilding fresh InnoDB files happens on a clean instance after logical extraction, not by deleting evidence in place.

Is innodb_force_recovery=6 a repair method?

No. Level 6 skips redo roll-forward and can leave database pages inconsistent. It is a drastic extraction option for a separate working copy when lower levels cannot start the server. It does not make the data healthy.

Can the website remain online during forced recovery?

It should not be treated as a normal production database. Writes are restricted, reads can still crash on damaged pages, and high recovery levels expose inconsistent state. Route application traffic away and use the instance only for controlled extraction.

Does the same procedure apply to MariaDB?

The overall safety principles are the same: preserve an offline copy, start with the lowest recovery level, extract data and rebuild cleanly. Exact recovery behavior, configuration support and dump options vary by product and version, so check the matching MariaDB documentation and installed client help before running commands.

When should an administrator stop and escalate?

Escalate when the only copy is on failing storage, normal and low-level recovery startup both fail, critical tables crash every extraction attempt, encryption keys or tablespace metadata are missing, or the business cannot accept an undocumented partial recovery. Continuing blind experiments can reduce the remaining recovery options.

If this is an active data-loss incident, emergency server support can assess the surviving data before high-risk changes are made. Routine database monitoring, backup oversight and recovery planning are available through ongoing Linux server management.

Emergency server support

Is this problem affecting a live server now?

Bring in a server engineer for urgent diagnosis, recovery and a clearly scoped response to the active incident.

Get emergency server supportTalk to an engineer