All we need is an easy explanation of the problem, so here it is.
Adding 5th node getting an error
"Our replica set configuration is invalid or does not include us
Presently all 4 nodes are healthy.
when I try to add a2 both the arbiters will go unhealhty immediately whereas my arbiter is healhty on itself but not healthy in replica set.
Error for arbiter in replica set –
"lastHeartbeatMessage" : "Our replica set configuration is invalid or does not include us"
"lastHeartbeatMessage" : "Couldn't get a connection within the time limit"
m- data nodes
How to solve :
I know you bored from this bug, So we are here to help you! Take a deep breath and look at the explanation of your problem. We have many solutions to this problem, But we recommend you to use the first method because it is tested & true method that will 100% work for you.
Replica set should have ONLY one arbiter and ONLY when there is an even number of data-bearing nodes.
Your replica set has already an odd number of data-bearing nodes (m1, m2, and m3), so there should not be any arbiters.
Those error messages means:
a1 - rs.conf() don't have that a1 node in the list of that replica sets' config, it may have wrong IP or port-number a2 - there is firewall (or similar) what prevents connection with other RS nodes.
Note: Use and implement method 1 because this method fully tested our system.
Thank you 🙂