MongoDB OplogStart Missing

All we need is an easy explanation of the problem, so here it is.

I understand that some times MongoDB secondary data becomes stale at which point, we need to perform an init sync from the Primary instance.

But sometimes we see error logs that look like this:

2022-07-22T21:24:20.065+0000 I  REPL  [rsBackgroundSync] Starting rollback due to OplogStartMissing: Our last optime fetched: { ts: Timestamp(1658512549, 30), t: 57 }. source's GTE: { ts: Timestamp(1658513748, 2), t: 88 }

How is the above error message different from:

2022-08-01T02:30:21.915+0000 I  REPL     [replication-0] We are too stale to use xyz as a sync source. Blacklisting this sync source because our last fetched timestamp: Timestamp(1659314292, 1) is before their earliest timestamp: Timestamp(1659320029, 301) for 1min until: 2022-08-01T02:31:21.915+0000

I am trying to understand the difference between both of the above.

How to solve :

I know you bored from this bug, So we are here to help you! Take a deep breath and look at the explanation of your problem. We have many solutions to this problem, But we recommend you to use the first method because it is tested & true method that will 100% work for you.

Method 1

The first one is rolling back writes from one of the nodes.

When a secondary lost connectivity it claimed to be master and accepted some writes. The log entry says that 20 min before that the real primary accepted other writes, so the reunited secondary steps down and rolls back own writes.

The second one says the oplog size is not sufficient for such long downtime. Oplog is a capped collection. Newer writes overwrite older records. The last oplog entry on the node is Monday, 1 August 2022 00:38:12, the first oplog entry on at source is Monday, 1 August 2022 02:13:49. There is more than 1 and a half hours gap in the records, so mongo cannot reliably replay oplog.

Although you can work it around by increasing size of oplog in the config, it would make sense to address the connectivity issues. Mongo is designed to cover short network partitions, but is not as efficient with longer downtimes.

Note: Use and implement method 1 because this method fully tested our system.
Thank you 🙂

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply