Postgres Database flagged as STORAGE FULL whereas there's 45 Gb available

All we need is an easy explanation of the problem, so here it is.

I will start this post by stating that I am not very experienced in database administration, please accept my apologies for any unclear explanation you may find below.

We have a replica postgresql hosted on AWS RDS, which stopped replicating last week. The instance was flagged as Storage-full.

However, while looking at the free storage space on CloudWatch, we realized there were still roughly 46Gb available on the 50Gb-allocated instance. We increased the allocated space to 60Gb and everything went back to normal, but we knew the issue would come back and it did.

The main instance on which this one is replicated is auto vacuumed. I think any writes resulting on the main will be written on the replica, so this is probably not a vacuum issue.

I could not find any indication of problem in CloudWatch metrics.

It appears that the logs could be the issue here but I don’t know where to look to investigate this option.

I will edit the question with any relevant information you might suggest in the comments.
Thanks for your help.

How to solve :

I know you bored from this bug, So we are here to help you! Take a deep breath and look at the explanation of your problem. We have many solutions to this problem, But we recommend you to use the first method because it is tested & true method that will 100% work for you.

Method 1

Got an answer back from AWS support.

It appears the problem came from not having enough space left for inodes. We cannot control allocated space for inodes on RDS so the only solution here is to get a bigger instance.

The OOM Killer allowed to get our instance running again.

Note: Use and implement method 1 because this method fully tested our system.
Thank you 🙂

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0

Leave a Reply