fork database head (17411000) is inconsistent with state (12367000) Nodeos Error

Question

I have a private multi-node blockchain. Because of Nodeos failure, it stops with the dirty flag error. I restart it with the --hard-replay-blockchain, the blocks start replaying. Here, by mistake, I run --hard-replay-blockchain again. Two replay operations were ongoing. So, I stop the script and start again. Now I am getting an error,

error 2021-05-18T03:13:16.399 nodeos    main.cpp:125                  main                 ] 3020000 fork_database_exception: Fork database exception
fork database head (17411000) is inconsistent with state (12367000)
    {"db":12367000,"head":17411000}
    nodeos  controller.cpp:666 init

    {}
    nodeos  chain_plugin.cpp:1112 plugin_startup

This is a genesis node producing blocks. I have other p2p nodes (with the same genesis.json) receiving blocks from genesis. Please guide, how would I solve this? I don't want to delete the blocks and transactions. Or if I clear blocks files or data from the genesis node, how can I sync / restore them with other peers.

Answer 1

In your case it sounds like you have 3 servers with the same set of blocks. You could manually shutdown one of the 2 working servers and copy all the blocks and state to the non-working server.

Best way to copy is to create a tar archive, copy the archive, then restore it.

tar --sparse -cf mybackup.tar [insert location of nodeos blocks and state]

Also suggest to make regular backups in a similar fashion. Always stop nodeos before running the backup.

Answer 2

Since you have 2 servers in sync, you could remove all the nodeos state from the server that is corrupted and sync the blocks back from the properly working nodes.

Remove all the nodeos blocks and state, and start nodeos with just the genesis.

If in doubt make a backup of your blocks and state first before erasing so you can recover if something goes wrong.