Re: memory leak under heavy load?

> hi
> i think i've encountered a bug in postgresql 8.1.
> yet - i'm not reallty info submitting it to -bugs, as i have no way to
> successfully redo it again.
>
> basically
> i have server, with dual opteron, 4g of memory, 2gb of swap.
> everything working under centos 4.2.
> ...
> what i say is that postmaster user started to "eat" memory.
> it allocated *all* memory (both ram and swap), and then died.
> load on the machine jumped to something around 20.

I noticed a similar occurrence. We have a high-load PostgreSQL database
-- not a ridiculous amount of inserts or updates, but a huge variety of
diverse queries on some 200 tables.

We had noticed load averages of 3-4 on our database for the past couple
days. Then, this morning, Postgres got killed twice by the Linux
out-of-memory process killer. (Also on a dual Opteron, 4GB of memory.)
We were showing 3.5 GB of memory allocated to *something*, but stopping
Postgres completely for a few seconds didn't lower the number. It wasn't
taken by any process, which leads me to believe that it's a kernel bug.
One reboot later, everything is rosy -- load hovers around 1.2, there's
enough free memory to have a 2.5 GB buffer cache, and swap is untouched.

PostgreSQL 7.4 had run on this box flawlessly for six months -- bad RAM
forced us to take it down -- then again for another month until we
upgraded to 8.1 last week. Like the original poster, we're set up for
~500 MB of shared memory; certainly not enough to make the kernel kill
-9 postmaster. Kernel is 2.6.11-gentoo-r6, same as before the upgrade.

Also, this didn't happen in our test environment, which uses a similar
but x86 server. Perhaps this is AMD64 related?

--Will Glynn
Freedom Healthcare

Will Glynn <wglynn(at)freedomhealthcare(dot)org> wrote:
> Postgres completely for a few seconds didn't lower the number. It wasn't
> taken by any process, which leads me to believe that it's a kernel bug.

If it was a shared memory segment allocated a particular way (I
*think* it's "shm_open", I'm not 100% sure), it's not erronious for the
kernel to leave it behind after all processes are gone... see
http://lists.debian.org/debian-apache/2004/06/msg00188.html .

If postgres needs this much shared memory and wants it to go away on
a crash, I think (again, I'm a neophyte at this still, I havent even fixed
mod_bt fo rthis yet) that an mmap()ed file is the way to go... but then
don't you need enough harddrive space to support your shared memory?

I don't know, this whole things confusing...

- Tyler

On Fri, Dec 02, 2005 at 03:53:20PM -0800, Tyler MacDonald wrote:
> Will Glynn <wglynn(at)freedomhealthcare(dot)org> wrote:
> > Postgres completely for a few seconds didn't lower the number. It wasn't
> > taken by any process, which leads me to believe that it's a kernel bug.
>
> If it was a shared memory segment allocated a particular way (I
> *think* it's "shm_open", I'm not 100% sure), it's not erronious for the
> kernel to leave it behind after all processes are gone... see
> http://lists.debian.org/debian-apache/2004/06/msg00188.html .

But this shouldn't be an issue here. If you set the IPC_RMID flag then
the kernel should remove the segment when all users go away. This is
standard IPC behaviour and is documentated in the manpage...

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

> But this shouldn't be an issue here. If you set the IPC_RMID flag then
> the kernel should remove the segment when all users go away. This is
> standard IPC behaviour and is documentated in the manpage...
>

Would you please tell me where to find the manpage and how to set IPC_RMID flag?

--
Kathy Lo

On Thu, Dec 08, 2005 at 12:29:11PM +0800, Kathy Lo wrote:
> > But this shouldn't be an issue here. If you set the IPC_RMID flag then
> > the kernel should remove the segment when all users go away. This is
> > standard IPC behaviour and is documentated in the manpage...
> >
>
> Would you please tell me where to find the manpage and how to set IPC_RMID flag?

See the shmctl() manpage:

int shmctl(int shmid, int cmd, struct shmid_ds *buf);

One of the command ids is IPC_RMID

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

On 12/8/05, Martijn van Oosterhout <kleptog(at)svana(dot)org> wrote:
> On Thu, Dec 08, 2005 at 12:29:11PM +0800, Kathy Lo wrote:
> > > But this shouldn't be an issue here. If you set the IPC_RMID flag then
> > > the kernel should remove the segment when all users go away. This is
> > > standard IPC behaviour and is documentated in the manpage...
> > >
> >
> > Would you please tell me where to find the manpage and how to set IPC_RMID
> flag?
>
> See the shmctl() manpage:
>
> int shmctl(int shmid, int cmd, struct shmid_ds *buf);
>
> One of the command ids is IPC_RMID
>
Do I need to change the source code of postgresql if I want to set
IPC_RMID flag to solve this problem?

--
Kathy Lo

On Fri, Dec 09, 2005 at 12:15:44PM +0800, Kathy Lo wrote:
> > See the shmctl() manpage:
> >
> > int shmctl(int shmid, int cmd, struct shmid_ds *buf);
> >
> > One of the command ids is IPC_RMID
> >
> Do I need to change the source code of postgresql if I want to set
> IPC_RMID flag to solve this problem?

No because it's completely unrelated. The amount of shared memory
doesn't vary while the system is running and it gets removed once you
restart postgres. So it can't cause you to run out of memory (unless
you allocated a truly huge amount of memory that way, but that's bad
for other reasons). At worst it's gets stuffed into swap until you next
start postgres.

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.