Consul Mismatch Key

Posted on | Edited on | In

I haven’t dealt with infra for a while after the team shift. Today, I was got a ticket and its problem solving process refreshed my memory about some Consul operations, so I would like to document it here.

The initial issue was the DNS lookup service from our stack malfunction and it did not return update-to-date IP of some VMs. As we use Consul as discovery service in our distributed system, so the question came to it.

Check if the VM or service was registered to Consul:

1
2
3
4
5
6
# List the members of a Consul cluster
# Examine the Status(alive), Type(server/client), etc 
consul members

# List all services
consul catalog services

Neither VMs not service was there, went to check if the Consul daemon was good:

1
sudo systemctl status consul

Consul was running, but I observed some short error messages, to see full:

1
sudo journalctl -ex -u consul

The error was about Consul encrypt key mismatch:

1
2
3
4
5
error=
| 3 errors occurred:
|         * Failed to join 172.16.4.137:8301: No installed keys could decrypt the message
|         * Failed to join 172.16.4.153:8301: No installed keys could decrypt the message
|         * Failed to join 172.16.4.139:8301: No installed keys could decrypt the message

Checked the Consul /etc/consul/config.json(check your corresponding setup) file and compared with the Consul server config, spotted that the enctypt field did not match, to fix it:

  1. Replaced the wrong key with the correct one.
  2. Deleted the cached key file /.../serf/local.keyring, the root path please check Consul config JSON file’s data_dir field.

Then restarted the Consul service and problem got solved:

1
sudo systemctl restart consul

You can check the DNS again by dig with Consul as DNS server:

1
dig +short <consul service name>.service.consul @localhost -p 8600

Or if the VM /etc/resolv.conf was configured with Consul already:

1
nslookup <consul service name>
0%