Re: [Pgbuildfarm-members] guppie: 64MB RAM too small?

Christian Mair wrote:
> Hi,

Hi Christian!

>
> I've recently added guppie to the buildfarm:
> http://www.pgbuildfarm.org/cgi-bin/show_history.pl?nm=guppy&br=HEAD
>
> I see random failures on make check which appearantly are
> guppie's fault. Instead of the expected outputs one can see
> lines such as:
> "could not send startup packet: Broken pipe"
> or
> "psql: could not fork new process for connection: Resource temporarily
> unavailable"
>
> To me, these look like messages you get from overloading a machine
> (no more RAM, no more processes, ...)

not really "overloading"

>
> Now the question is: is guppie to small for make check?
> It is Open BSD 3.8 on a Pentium 1 with 64MB RAM and 256MB SWAP.
> It runs just the buildfarm, sshd and apache (totally idle), no gui.

I operate the other OpenBSD buildfarm member (spoonbill), if you are
running the buildfarm script under a user that falls into the "default"
login-class (/etc/login.conf) it will get a limit of 64 maxfiles and 64
processes.
This limit is too small for make check to complete - so either up the
limits for the default login class or move the buildfarm user to another
class with higher limits.

beside that one of my other buildfarm boxes (lionfish) happily works
with 48MB of RAM - so 64MB is plenty enough ;-)

Stefan

Hi,

I've recently added guppie to the buildfarm:
http://www.pgbuildfarm.org/cgi-bin/show_history.pl?nm=guppy&br=HEAD

I see random failures on make check which appearantly are
guppie's fault. Instead of the expected outputs one can see
lines such as:
"could not send startup packet: Broken pipe"
or
"psql: could not fork new process for connection: Resource temporarily
unavailable"

To me, these look like messages you get from overloading a machine
(no more RAM, no more processes, ...)

Now the question is: is guppie to small for make check?
It is Open BSD 3.8 on a Pentium 1 with 64MB RAM and 256MB SWAP.
It runs just the buildfarm, sshd and apache (totally idle), no gui.

Bye, Chris.

PS: I had the impression I passed make check at least once
when I made the dry runs, but I might have looked in the wrong place.

Stefan Kaltenbrunner wrote:

>Christian Mair wrote:
>
>
>>Hi,
>>
>>
>
>Hi Christian!
>
>
>
>>I've recently added guppie to the buildfarm:
>>http://www.pgbuildfarm.org/cgi-bin/show_history.pl?nm=guppy&br=HEAD
>>
>>I see random failures on make check which appearantly are
>>guppie's fault. Instead of the expected outputs one can see
>>lines such as:
>>"could not send startup packet: Broken pipe"
>>or
>>"psql: could not fork new process for connection: Resource temporarily
>>unavailable"
>>
>>To me, these look like messages you get from overloading a machine
>>(no more RAM, no more processes, ...)
>>
>>
>
>not really "overloading"
>
>
>
>>Now the question is: is guppie to small for make check?
>>It is Open BSD 3.8 on a Pentium 1 with 64MB RAM and 256MB SWAP.
>>It runs just the buildfarm, sshd and apache (totally idle), no gui.
>>
>>
>
>I operate the other OpenBSD buildfarm member (spoonbill), if you are
>running the buildfarm script under a user that falls into the "default"
>login-class (/etc/login.conf) it will get a limit of 64 maxfiles and 64
>processes.
>This limit is too small for make check to complete - so either up the
>limits for the default login class or move the buildfarm user to another
>class with higher limits.
>
>beside that one of my other buildfarm boxes (lionfish) happily works
>with 48MB of RAM - so 64MB is plenty enough ;-)
>
>
>

You can also reduce the amount of parallelism by setting
MAX_CONNECTIONS. There is an example in the standard config file,
commented out. Not sure if this will help though.

Stefan, what maxfiles/maxprocesses should be set for OpenBSD? Are the
initial settings hard limits or can we increase them in the config file?

cheers

andrew

>> Now the question is: is guppie to small for make check?
>> It is Open BSD 3.8 on a Pentium 1 with 64MB RAM and 256MB SWAP.
>> It runs just the buildfarm, sshd and apache (totally idle), no gui.
>
> I operate the other OpenBSD buildfarm member (spoonbill), if you are
> running the buildfarm script under a user that falls into the "default"
> login-class (/etc/login.conf) it will get a limit of 64 maxfiles and 64
> processes.
> This limit is too small for make check to complete - so either up the
> limits for the default login class or move the buildfarm user to another
> class with higher limits.

Ok!
Thanks a lot - that explains everything :)
I'll do that.

OpenBSD: too secure for mere mortals...

> beside that one of my other buildfarm boxes (lionfish) happily works
> with 48MB of RAM - so 64MB is plenty enough ;-)

Ok, good to know :)

Bye, Chris.

Andrew Dunstan wrote:
> Stefan Kaltenbrunner wrote:
>
>> Christian Mair wrote:
>>
>>
>>> Hi,
>>>
>>
>>
>> Hi Christian!
>>
>>
>>
>>> I've recently added guppie to the buildfarm:
>>> http://www.pgbuildfarm.org/cgi-bin/show_history.pl?nm=guppy&br=HEAD
>>>
>>> I see random failures on make check which appearantly are
>>> guppie's fault. Instead of the expected outputs one can see
>>> lines such as:
>>> "could not send startup packet: Broken pipe"
>>> or
>>> "psql: could not fork new process for connection: Resource temporarily
>>> unavailable"
>>>
>>> To me, these look like messages you get from overloading a machine
>>> (no more RAM, no more processes, ...)
>>>
>>
>>
>> not really "overloading"
>>
>>
>>
>>> Now the question is: is guppie to small for make check?
>>> It is Open BSD 3.8 on a Pentium 1 with 64MB RAM and 256MB SWAP.
>>> It runs just the buildfarm, sshd and apache (totally idle), no gui.
>>>
>>
>>
>> I operate the other OpenBSD buildfarm member (spoonbill), if you are
>> running the buildfarm script under a user that falls into the
>> "default" login-class (/etc/login.conf) it will get a limit of 64
>> maxfiles and 64 processes.
>> This limit is too small for make check to complete - so either up the
>> limits for the default login class or move the buildfarm user to
>> another class with higher limits.
>>
>> beside that one of my other buildfarm boxes (lionfish) happily works
>> with 48MB of RAM - so 64MB is plenty enough ;-)
>>
>>
>>
>
> You can also reduce the amount of parallelism by setting
> MAX_CONNECTIONS. There is an example in the standard config file,
> commented out. Not sure if this will help though.
>
> Stefan, what maxfiles/maxprocesses should be set for OpenBSD? Are the
> initial settings hard limits or can we increase them in the config file?

not sure what the minimal requirements are for a make check to complete
without lowering MAX_CONNECTIONS actually . I have mine set to 512/512
for the pgbuild-farm user.
I guess a lot less than that might be enough though ...

While at least the maxproc limit can be raised up to the hardlimit of
128 I don't think that is worth spending any development time on with
trying to raise it in the build-farm script ...

Stefan

On Mon, Mar 20, 2006 at 08:18:46PM +0100, Stefan Kaltenbrunner wrote:
> > You can also reduce the amount of parallelism by setting
> > MAX_CONNECTIONS. There is an example in the standard config file,
> > commented out. Not sure if this will help though.
> >
> > Stefan, what maxfiles/maxprocesses should be set for OpenBSD? Are the
> > initial settings hard limits or can we increase them in the config file?
>
> not sure what the minimal requirements are for a make check to complete
> without lowering MAX_CONNECTIONS actually . I have mine set to 512/512
> for the pgbuild-farm user.
> I guess a lot less than that might be enough though ...
>
> While at least the maxproc limit can be raised up to the hardlimit of
> 128 I don't think that is worth spending any development time on with
> trying to raise it in the build-farm script ...

Isn't MAX_CONNECTIONS one of those things that should be scaled
automagically by initdb? Or is the problem just that make check needs a
certain minimum number of connections...
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

On Tue, Mar 21, 2006 at 06:39:58AM -0600, Andrew Dunstan wrote:
> There seems to be some confusion here. The MAX_CONNECTIONS setting for "make
> check" is a limit on the parallelism of the test. It's a gadget we put in a
> couple of years ago to allow cygwin to run "make check" as it couldn't run
> 20 concurrent connections. It is not connected to the server setting that is
> indeed set up by initdb. Buildfarm very deliberately always runs with the
> default settings that initdb sets up.

Ok, so the problem was just that the database ran out of connections
because initdb had to set max_connections really low. Thanks for
clarifying.

It's probably worth adding a note about this somewhere in
http://pgfoundry.org/docman/view.php/1000040/4/PGBuildFarm-HOWTO.txt
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Jim C. Nasby said:
> On Mon, Mar 20, 2006 at 08:18:46PM +0100, Stefan Kaltenbrunner wrote:
>> > You can also reduce the amount of parallelism by setting
>> > MAX_CONNECTIONS. There is an example in the standard config file,
>> > commented out. Not sure if this will help though.
>> >
>> > Stefan, what maxfiles/maxprocesses should be set for OpenBSD? Are
>> > the initial settings hard limits or can we increase them in the
>> > config file?
>>
>> not sure what the minimal requirements are for a make check to
>> complete without lowering MAX_CONNECTIONS actually . I have mine set
>> to 512/512 for the pgbuild-farm user.
>> I guess a lot less than that might be enough though ...
>>
>> While at least the maxproc limit can be raised up to the hardlimit of
>> 128 I don't think that is worth spending any development time on with
>> trying to raise it in the build-farm script ...
>
> Isn't MAX_CONNECTIONS one of those things that should be scaled
> automagically by initdb? Or is the problem just that make check needs a
> certain minimum number of connections...

There seems to be some confusion here. The MAX_CONNECTIONS setting for "make
check" is a limit on the parallelism of the test. It's a gadget we put in a
couple of years ago to allow cygwin to run "make check" as it couldn't run
20 concurrent connections. It is not connected to the server setting that is
indeed set up by initdb. Buildfarm very deliberately always runs with the
default settings that initdb sets up.

cheers

andrew

Jim C. Nasby said:
> On Tue, Mar 21, 2006 at 06:39:58AM -0600, Andrew Dunstan wrote:
>> There seems to be some confusion here. The MAX_CONNECTIONS setting for
>> "make check" is a limit on the parallelism of the test. It's a gadget
>> we put in a couple of years ago to allow cygwin to run "make check" as
>> it couldn't run 20 concurrent connections. It is not connected to the
>> server setting that is indeed set up by initdb. Buildfarm very
>> deliberately always runs with the default settings that initdb sets
>> up.
>
> Ok, so the problem was just that the database ran out of connections
> because initdb had to set max_connections really low. Thanks for
> clarifying.
>
> It's probably worth adding a note about this somewhere in
> http://pgfoundry.org/docman/view.php/1000040/4/PGBuildFarm-HOWTO.txt --
>

No, let's start again.

The user's machine ran out of resources. That can't be because inbitdb set
max_connections too low - if anything it has probably set them too high. I
suggested that he could possibly limit resource use by limiting the number
of concurrent connections "make check" would use, by using the *UNRELATED*
MAX_CONNECTIONS=n make flag. This flag is not part of buildfarm - it
predates buildfarm in fact. It's part of the postgres build system - look in
pg_regress.sh and the associated make file. Buildfarm has support for it as
shown in the sample config file.

cheers

andrew

> No, let's start again.
>
> The user's machine ran out of resources. That can't be because inbitdb set
> max_connections too low - if anything it has probably set them too high. I
> suggested that he could possibly limit resource use by limiting the number
> of concurrent connections "make check" would use, by using the *UNRELATED*
> MAX_CONNECTIONS=n make flag. This flag is not part of buildfarm - it
> predates buildfarm in fact. It's part of the postgres build system - look in
> pg_regress.sh and the associated make file. Buildfarm has support for it as
> shown in the sample config file.

Yes,
Neither PostgreSQL nor the buildfarm code is to blame.
The problem was OpenBSD's restrictive default limit of max 64
user processes. I raised that on OpenBSD's side and as you can see
guppie turned green :)

Bye, Chris.

Adding -hackers.

On Tue, Mar 21, 2006 at 02:20:09PM +0100, Christian Mair wrote:
>
> > No, let's start again.
> >
> > The user's machine ran out of resources. That can't be because inbitdb set
> > max_connections too low - if anything it has probably set them too high. I
> > suggested that he could possibly limit resource use by limiting the number
> > of concurrent connections "make check" would use, by using the *UNRELATED*
> > MAX_CONNECTIONS=n make flag. This flag is not part of buildfarm - it
> > predates buildfarm in fact. It's part of the postgres build system - look in
> > pg_regress.sh and the associated make file. Buildfarm has support for it as
> > shown in the sample config file.
>
> Yes,
> Neither PostgreSQL nor the buildfarm code is to blame.
> The problem was OpenBSD's restrictive default limit of max 64
> user processes. I raised that on OpenBSD's side and as you can see
> guppie turned green :)

Ok, lets go back to my original point then: initdb should be made to
check that you can actually open as many connections as it's trying to
set max_connections to. If it can't, it should drop max_connections down
(and possibly add a note to postgresql.conf indicating that it did so).
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Jim C. Nasby wrote:

>Adding -hackers.
>
>
[removing -buildfarm :-) ]

>Ok, lets go back to my original point then: initdb should be made to
>check that you can actually open as many connections as it's trying to
>set max_connections to. If it can't, it should drop max_connections down
>(and possibly add a note to postgresql.conf indicating that it did so).
>
>

At the time it sets max_connections there is no server to test against.
initdb in fact never uses a standard client connection at all, and never
starts postmaster. To do a check on max_connections you would have to
start postmaster and then try to start that many client connections.
That's a lot of extra lifting to put into initdb for what is arguably at
worst a rare problem.

cheers

andrew

"Jim C. Nasby" <jnasby(at)pervasive(dot)com> writes:
> Ok, lets go back to my original point then: initdb should be made to
> check that you can actually open as many connections as it's trying to
> set max_connections to.

Apparently you are unaware that it's done that for a long time.

regards, tom lane

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> At the time it sets max_connections there is no server to test against.
> initdb in fact never uses a standard client connection at all, and never
> starts postmaster. To do a check on max_connections you would have to
> start postmaster and then try to start that many client connections.

max_connections *is* checked by initdb ... although only to the extent
of verifying we can make that many semaphores.

The parallel regression tests are not a particularly great reference
point for this anyway, because for each parallel test case you have not
only a server process, but a psql process, and in most shells a parent
shell process for the psql, ie 3x the nominal level of parallelism,
all running under the postgres userid. This isn't the normal usage
scenario, so it would not be reasonable to restrict max_connections to
1/3 the number of user processes per userid.

regards, tom lane

On Tue, Mar 21, 2006 at 09:59:40AM -0500, Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > At the time it sets max_connections there is no server to test against.
> > initdb in fact never uses a standard client connection at all, and never
> > starts postmaster. To do a check on max_connections you would have to
> > start postmaster and then try to start that many client connections.
>
> max_connections *is* checked by initdb ... although only to the extent
> of verifying we can make that many semaphores.

Ok, I thought there was at least some kind of check in there. Maybe it
should try and determine how many processes the postmaster user is
allowed as well; presumably something like ulimit would show this.

> The parallel regression tests are not a particularly great reference
> point for this anyway, because for each parallel test case you have not
> only a server process, but a psql process, and in most shells a parent
> shell process for the psql, ie 3x the nominal level of parallelism,
> all running under the postgres userid. This isn't the normal usage
> scenario, so it would not be reasonable to restrict max_connections to
> 1/3 the number of user processes per userid.

Certainly; it doesn't make sense to be mucking around just for the sake
of make check.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461