Re: count with high allocation

Lists: pgsql-general
From: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
To: GENERAL <pgsql-general(at)postgresql(dot)org>
Subject: count with high allocation
Date: 2010-04-07 14:11:41
Message-ID: 497357.69364.qm@web52502.mail.re2.yahoo.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Hi all,
Monitoring "top" in database server , i could noticed an query with reserved 8GB on physical memory.

select count(field) from big_table 1 inner join big_table2...

There is the possibility of using another function with less memory allocation?
Is there a way to limit the memory usage of the count?

Paul

____________________________________________________________________________________
Veja quais são os assuntos do momento no Yahoo! +Buscados
http://br.maisbuscados.yahoo.com


From: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
To: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Cc: GENERAL <pgsql-general(at)postgresql(dot)org>
Subject: Re: count with high allocation
Date: 2010-04-07 14:26:13
Message-ID: j2v162867791004070726g8735509s1af1eb0a31d21af2@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

Hello

2010/4/7 paulo matadr <saddoness(at)yahoo(dot)com(dot)br>:
> Hi all,
> Monitoring "top" in database server  , i could  noticed an query with
> reserved  8GB on physical memory.
>
> select count(field) from big_table  1 inner join big_table2...
>
> There is the possibility of using another function with less memory
> allocation?
> Is there a way to limit the memory usage of the count?

probably you have unactualised statistic. What is result of EXPLAIN
ANALYZE select count(... ?

I think, so planner uses hash arrays

try to

set enable_hashagg to off;
and again SELECT ...

Regards
Pavel Stehule

>
>
>
> Paul
>
>
> ________________________________
> Veja quais são os assuntos do momento no Yahoo! + Buscados: Top 10 -
> Celebridades - Música - Esportes


From: Scott Marlowe <scott(dot)marlowe(at)gmail(dot)com>
To: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Cc: GENERAL <pgsql-general(at)postgresql(dot)org>
Subject: Re: count with high allocation
Date: 2010-04-07 14:27:35
Message-ID: w2idcc563d11004070727s221d674dw722cac521ec1e87f@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Wed, Apr 7, 2010 at 8:11 AM, paulo matadr <saddoness(at)yahoo(dot)com(dot)br> wrote:
> Hi all,
> Monitoring "top" in database server  , i could  noticed an query with
> reserved  8GB on physical memory.

You are likely seeing the SHR and VIRT columns saying that. Lemme
guess, you've got 8G of shared memory allocated to pgsql?

If not, please show the lines from top that have you worried.


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Cc: GENERAL <pgsql-general(at)postgresql(dot)org>
Subject: Re: count with high allocation
Date: 2010-04-07 14:31:27
Message-ID: 14229.1270650687@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

paulo matadr <saddoness(at)yahoo(dot)com(dot)br> writes:
> Monitoring "top" in database server , i could noticed an query with reserved 8GB on physical memory.

> select count(field) from big_table 1 inner join big_table2...

> There is the possibility of using another function with less memory allocation?
> Is there a way to limit the memory usage of the count?

It seems quite likely that what top is telling you just reflects the
process touching all shared buffers, and has nothing to do with any
real "memory consumption". What do you have shared_buffers set to?

regards, tom lane


From: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: GENERAL <pgsql-general(at)postgresql(dot)org>, Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
Subject: Res: count with high allocation
Date: 2010-04-07 14:53:37
Message-ID: 933228.89946.qm@web52501.mail.re2.yahoo.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

EXPLAIN/TOP

EXPLAIN:
Aggregate (cost=3710076.58..3710076.59 rows=1 width=4)
-> Hash Join (cost=2668820.36..3692200.16 rows=7150570 width=4)
Hash Cond: (osunidade.attp_id = art.attp_id)
-> Hash Join (cost=2668819.29..3593878.75 rows=7150570 width=8)
Hash Cond: (osunidade.orse_id = os.orse_id)
-> Seq Scan on ordem_servico_unidade osunidade (cost=0.00..429514.00 rows=23418900 width=8)
-> Hash (cost=2598702.48..2598702.48 rows=4033665 width=4)
-> Hash Left Join (cost=1372486.83..2598702.48 rows=4033665 width=4)
Hash Cond: (os.cbdo_id = cobra.cbdo_id)
-> Merge Join (cost=0.00..880392.67 rows=4033665 width=8)
Merge Cond: (os.rgat_id = ra.rgat_id)
-> Index Scan using xfk1_ordem_servico on ordem_servico os (cost=0.00..879051.89 rows=13210693 width=12)
-> Index Scan using registro_atendimento_pkey on registro_atendimento ra (cost=0.00..548171.12 rows=5369913 width=4)
Filter: (rgat_id IS NOT NULL)
-> Hash (cost=897238.26..897238.26 rows=27340126 width=4)
-> Seq Scan on cobranca_documento cobra (cost=0.00..897238.26 rows=27340126 width=4)
-> Hash (cost=1.03..1.03 rows=3 width=4)
-> Seq Scan on atendimento_relacao_tipo art (cost=0.00..1.03 rows=3 width=4)

"TOP"
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

26809 postgres 16 0 16.5g 8.6g 8.6g S 1 27.5 1:28.84 postgres: user database 10.1.1.7(54033) SELECT

________________________________
De: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Cc: GENERAL <pgsql-general(at)postgresql(dot)org>
Enviadas: Quarta-feira, 7 de Abril de 2010 11:31:27
Assunto: Re: [GENERAL] count with high allocation

paulo matadr <saddoness(at)yahoo(dot)com(dot)br> writes:
> Monitoring "top" in database server , i could noticed an query with reserved 8GB on physical memory.

> select count(field) from big_table 1 inner join big_table2...

> There is the possibility of using another function with less memory allocation?
> Is there a way to limit the memory usage of the count?

It seems quite likely that what top is telling you just reflects the
process touching all shared buffers, and has nothing to do with any
real "memory consumption". What do you have shared_buffers set to?

regards, tom lane

--
Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

____________________________________________________________________________________
Veja quais são os assuntos do momento no Yahoo! +Buscados
http://br.maisbuscados.yahoo.com


From: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
To: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, GENERAL <pgsql-general(at)postgresql(dot)org>
Subject: Re: count with high allocation
Date: 2010-04-07 15:10:23
Message-ID: g2t162867791004070810w9e874a34xb68e75f8e511f90d@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

please, EXPLAIN ANALYZE

and try to execute

set enable_hashagg to off before as second variant. It have to take less memory

regards
Pavel Stehule

2010/4/7 paulo matadr <saddoness(at)yahoo(dot)com(dot)br>:
> EXPLAIN/TOP
>
> EXPLAIN:
> Aggregate  (cost=3710076.58..3710076.59 rows=1 width=4)
>   ->  Hash Join  (cost=2668820.36..3692200.16 rows=7150570 width=4)
>         Hash Cond: (osunidade.attp_id = art.attp_id)
>         ->  Hash Join  (cost=2668819.29..3593878.75 rows=7150570 width=8)
>               Hash Cond: (osunidade.orse_id = os.orse_id)
>               ->  Seq Scan on ordem_servico_unidade osunidade
> (cost=0.00..429514.00 rows=23418900 width=8)
>               ->  Hash  (cost=2598702.48..2598702.48 rows=4033665 width=4)
>                     ->  Hash Left Join  (cost=1372486.83..2598702.48
> rows=4033665 width=4)
>                           Hash Cond: (os.cbdo_id = cobra.cbdo_id)
>                           ->  Merge Join  (cost=0.00..880392.67 rows=4033665
> width=8)
>                                 Merge Cond: (os.rgat_id = ra.rgat_id)
>                                 ->  Index Scan using xfk1_ordem_servico on
> ordem_servico os  (cost=0.00..879051.89 rows=13210693 width=12)
>                                 ->  Index Scan using
> registro_atendimento_pkey on registro_atendimento ra  (cost=0.00..548171.12
> rows=5369913 width=4)
>                                       Filter: (rgat_id IS NOT NULL)
>                           ->  Hash  (cost=897238.26..897238.26 rows=27340126
> width=4)
>                                 ->  Seq Scan on cobranca_documento cobra
> (cost=0.00..897238.26 rows=27340126 width=4)
>         ->  Hash  (cost=1.03..1.03 rows=3 width=4)
>               ->  Seq Scan on atendimento_relacao_tipo art  (cost=0.00..1.03
> rows=3 width=4)
>
> "TOP"
> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>
> 26809 postgres  16   0 16.5g 8.6g 8.6g S    1 27.5   1:28.84 postgres: user
> database 10.1.1.7(54033) SELECT
> ________________________________
> De: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
> Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
> Cc: GENERAL <pgsql-general(at)postgresql(dot)org>
> Enviadas: Quarta-feira, 7 de Abril de 2010 11:31:27
> Assunto: Re: [GENERAL] count with high allocation
>
> paulo matadr <saddoness(at)yahoo(dot)com(dot)br> writes:
>> Monitoring "top" in database server  , i could  noticed an query with
>> reserved  8GB on physical memory.
>
>> select count(field) from big_table  1 inner join big_table2...
>
>> There is the possibility of using another function with less memory
>> allocation?
>> Is there a way to limit the memory usage of the count?
>
> It seems quite likely that what top is telling you just reflects the
> process touching all shared buffers, and has nothing to do with any
> real "memory consumption".  What do you have shared_buffers set to?
>
>             regards, tom lane
>
> --
> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>
> ________________________________
> Veja quais são os assuntos do momento no Yahoo! + Buscados: Top 10 -
> Celebridades - Música - Esportes


From: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
To: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Res: count with high allocation
Date: 2010-04-07 16:41:02
Message-ID: 708167.67310.qm@web52505.mail.re2.yahoo.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

with
set enable_hashagg to off , I give the same allocation.

________________________________
De: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>; GENERAL <pgsql-general(at)postgresql(dot)org>
Enviadas: Quarta-feira, 7 de Abril de 2010 12:10:23
Assunto: Re: [GENERAL] count with high allocation

please, EXPLAIN ANALYZE

and try to execute

set enable_hashagg to off before as second variant. It have to take less memory

regards
Pavel Stehule

2010/4/7 paulo matadr <saddoness(at)yahoo(dot)com(dot)br>:
> EXPLAIN/TOP
>
> EXPLAIN:
> Aggregate (cost=3710076.58..3710076.59 rows=1 width=4)
> -> Hash Join (cost=2668820.36..3692200.16 rows=7150570 width=4)
> Hash Cond: (osunidade.attp_id = art.attp_id)
> -> Hash Join (cost=2668819.29..3593878.75 rows=7150570 width=8)
> Hash Cond: (osunidade.orse_id = os.orse_id)
> -> Seq Scan on ordem_servico_unidade osunidade
> (cost=0.00..429514.00 rows=23418900 width=8)
> -> Hash (cost=2598702.48..2598702.48 rows=4033665 width=4)
> -> Hash Left Join (cost=1372486.83..2598702.48
> rows=4033665 width=4)
> Hash Cond: (os.cbdo_id = cobra.cbdo_id)
> -> Merge Join (cost=0.00..880392.67 rows=4033665
> width=8)
> Merge Cond: (os.rgat_id = ra.rgat_id)
> -> Index Scan using xfk1_ordem_servico on
> ordem_servico os (cost=0.00..879051.89 rows=13210693 width=12)
> -> Index Scan using
> registro_atendimento_pkey on registro_atendimento ra (cost=0.00..548171.12
> rows=5369913 width=4)
> Filter: (rgat_id IS NOT NULL)
> -> Hash (cost=897238.26..897238.26 rows=27340126
> width=4)
> -> Seq Scan on cobranca_documento cobra
> (cost=0.00..897238.26 rows=27340126 width=4)
> -> Hash (cost=1.03..1.03 rows=3 width=4)
> -> Seq Scan on atendimento_relacao_tipo art (cost=0.00..1.03
> rows=3 width=4)
>
> "TOP"
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 26809 postgres 16 0 16.5g 8.6g 8.6g S 1 27.5 1:28.84 postgres: user
> database 10.1.1.7(54033) SELECT
> ________________________________
> De: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
> Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
> Cc: GENERAL <pgsql-general(at)postgresql(dot)org>
> Enviadas: Quarta-feira, 7 de Abril de 2010 11:31:27
> Assunto: Re: [GENERAL] count with high allocation
>
> paulo matadr <saddoness(at)yahoo(dot)com(dot)br> writes:
>> Monitoring "top" in database server , i could noticed an query with
>> reserved 8GB on physical memory.
>
>> select count(field) from big_table 1 inner join big_table2...
>
>> There is the possibility of using another function with less memory
>> allocation?
>> Is there a way to limit the memory usage of the count?
>
> It seems quite likely that what top is telling you just reflects the
> process touching all shared buffers, and has nothing to do with any
> real "memory consumption". What do you have shared_buffers set to?
>
> regards, tom lane
>
> --
> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>
> ________________________________
> Veja quais são os assuntos do momento no Yahoo! + Buscados: Top 10 -
> Celebridades - Música - Esportes


From: Scott Marlowe <scott(dot)marlowe(at)gmail(dot)com>
To: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Cc: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>, pgsql-general(at)postgresql(dot)org
Subject: Re: count with high allocation
Date: 2010-04-07 16:44:40
Message-ID: w2idcc563d11004070944r2b7c9038y2beba66d65d203c6@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

On Wed, Apr 7, 2010 at 10:41 AM, paulo matadr <saddoness(at)yahoo(dot)com(dot)br> wrote:
> with
> set enable_hashagg to off , I give the same allocation.

So what is your setting for shared_buffers? Cause this looks pretty
normal to me.


From: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
To: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: count with high allocation
Date: 2010-04-07 16:51:24
Message-ID: u2u162867791004070951gf2634edepdd1c0978764f9757@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

2010/4/7 paulo matadr <saddoness(at)yahoo(dot)com(dot)br>:
> with
> set enable_hashagg to off , I give the same allocation.

ok, then problem will be other.

what is result of:

show shared_buffers;
show work_mem;

Regards
Pavel Stehule
>
> ________________________________
> De: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
> Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
> Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>; GENERAL <pgsql-general(at)postgresql(dot)org>
> Enviadas: Quarta-feira, 7 de Abril de 2010 12:10:23
> Assunto: Re: [GENERAL] count with high allocation
>
> please, EXPLAIN ANALYZE
>
> and try to execute
>
> set enable_hashagg to off before as second variant. It have to take less
> memory
>
> regards
> Pavel Stehule
>
> 2010/4/7 paulo matadr <saddoness(at)yahoo(dot)com(dot)br>:
>> EXPLAIN/TOP
>>
>> EXPLAIN:
>> Aggregate  (cost=3710076.58..3710076.59 rows=1 width=4)
>>   ->  Hash Join  (cost=2668820.36..3692200.16 rows=7150570 width=4)
>>         Hash Cond: (osunidade.attp_id = art.attp_id)
>>         ->  Hash Join  (cost=2668819.29..3593878.75 rows=7150570 width=8)
>>               Hash Cond: (osunidade.orse_id = os.orse_id)
>>               ->  Seq Scan on ordem_servico_unidade osunidade
>> (cost=0.00..429514.00 rows=23418900 width=8)
>>               ->  Hash  (cost=2598702.48..2598702.48 rows=4033665 width=4)
>>                     ->  Hash Left Join  (cost=1372486.83..2598702.48
>> rows=4033665 width=4)
>>                           Hash Cond: (os.cbdo_id = cobra.cbdo_id)
>>                           ->  Merge Join  (cost=0.00..880392.67
>> rows=4033665
>> width=8)
>>                                 Merge Cond: (os.rgat_id = ra.rgat_id)
>>                                 ->  Index Scan using xfk1_ordem_servico on
>> ordem_servico os  (cost=0.00..879051.89 rows=13210693 width=12)
>>                                 ->  Index Scan using
>> registro_atendimento_pkey on registro_atendimento ra
>> (cost=0.00..548171.12
>> rows=5369913 width=4)
>>                                       Filter: (rgat_id IS NOT NULL)
>>                           ->  Hash  (cost=897238.26..897238.26
>> rows=27340126
>> width=4)
>>                                 ->  Seq Scan on cobranca_documento cobra
>> (cost=0.00..897238.26 rows=27340126 width=4)
>>         ->  Hash  (cost=1.03..1.03 rows=3 width=4)
>>               ->  Seq Scan on atendimento_relacao_tipo art
>> (cost=0.00..1.03
>> rows=3 width=4)
>>
>> "TOP"
>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>
>> 26809 postgres  16   0 16.5g 8.6g 8.6g S    1 27.5   1:28.84 postgres:
>> user
>> database 10.1.1.7(54033) SELECT
>> ________________________________
>> De: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
>> Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
>> Cc: GENERAL <pgsql-general(at)postgresql(dot)org>
>> Enviadas: Quarta-feira, 7 de Abril de 2010 11:31:27
>> Assunto: Re: [GENERAL] count with high allocation
>>
>> paulo matadr <saddoness(at)yahoo(dot)com(dot)br> writes:
>>> Monitoring "top" in database server  , i could  noticed an query with
>>> reserved  8GB on physical memory.
>>
>>> select count(field) from big_table  1 inner join big_table2...
>>
>>> There is the possibility of using another function with less memory
>>> allocation?
>>> Is there a way to limit the memory usage of the count?
>>
>> It seems quite likely that what top is telling you just reflects the
>> process touching all shared buffers, and has nothing to do with any
>> real "memory consumption".  What do you have shared_buffers set to?
>>
>>             regards, tom lane
>>
>> --
>> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-general
>>
>> ________________________________
>> Veja quais são os assuntos do momento no Yahoo! + Buscados: Top 10 -
>> Celebridades - Música - Esportes
>
>


From: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
To: pgsql-general(at)postgresql(dot)org
Subject: Res: count with high allocation
Date: 2010-04-07 17:06:13
Message-ID: 310064.83193.qm@web52508.mail.re2.yahoo.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

postgres=# show shared_buffers;
shared_buffers
----------------
16GB
(1 row)

postgres=# show work_mem;
work_mem
----------
5MB
(1 row)

________________________________
De: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Cc: pgsql-general(at)postgresql(dot)org
Enviadas: Quarta-feira, 7 de Abril de 2010 13:51:24
Assunto: Re: [GENERAL] count with high allocation

2010/4/7 paulo matadr <saddoness(at)yahoo(dot)com(dot)br>:
> with
> set enable_hashagg to off , I give the same allocation.

ok, then problem will be other.

what is result of:

show shared_buffers;
show work_mem;

Regards
Pavel Stehule
>
> ________________________________
> De: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
> Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
> Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>; GENERAL <pgsql-general(at)postgresql(dot)org>
> Enviadas: Quarta-feira, 7 de Abril de 2010 12:10:23
> Assunto: Re: [GENERAL] count with high allocation
>
> please, EXPLAIN ANALYZE
>
> and try to execute
>
> set enable_hashagg to off before as second variant. It have to take less
> memory
>
> regards
> Pavel Stehule
>
> 2010/4/7 paulo matadr <saddoness(at)yahoo(dot)com(dot)br>:
>> EXPLAIN/TOP
>>
>> EXPLAIN:
>> Aggregate (cost=3710076.58..3710076.59 rows=1 width=4)
>> -> Hash Join (cost=2668820.36..3692200.16 rows=7150570 width=4)
>> Hash Cond: (osunidade.attp_id = art.attp_id)
>> -> Hash Join (cost=2668819.29..3593878.75 rows=7150570 width=8)
>> Hash Cond: (osunidade.orse_id = os.orse_id)
>> -> Seq Scan on ordem_servico_unidade osunidade
>> (cost=0.00..429514.00 rows=23418900 width=8)
>> -> Hash (cost=2598702.48..2598702.48 rows=4033665 width=4)
>> -> Hash Left Join (cost=1372486.83..2598702.48
>> rows=4033665 width=4)
>> Hash Cond: (os.cbdo_id = cobra.cbdo_id)
>> -> Merge Join (cost=0.00..880392.67
>> rows=4033665
>> width=8)
>> Merge Cond: (os.rgat_id = ra.rgat_id)
>> -> Index Scan using xfk1_ordem_servico on
>> ordem_servico os (cost=0.00..879051.89 rows=13210693 width=12)
>> -> Index Scan using
>> registro_atendimento_pkey on registro_atendimento ra
>> (cost=0.00..548171.12
>> rows=5369913 width=4)
>> Filter: (rgat_id IS NOT NULL)
>> -> Hash (cost=897238.26..897238.26
>> rows=27340126
>> width=4)
>> -> Seq Scan on cobranca_documento cobra
>> (cost=0.00..897238.26 rows=27340126 width=4)
>> -> Hash (cost=1.03..1.03 rows=3 width=4)
>> -> Seq Scan on atendimento_relacao_tipo art
>> (cost=0.00..1.03
>> rows=3 width=4)
>>
>> "TOP"
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>
>> 26809 postgres 16 0 16.5g 8.6g 8.6g S 1 27.5 1:28.84 postgres:
>> user
>> database 10.1.1.7(54033) SELECT
>> ________________________________
>> De: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
>> Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
>> Cc: GENERAL <pgsql-general(at)postgresql(dot)org>
>> Enviadas: Quarta-feira, 7 de Abril de 2010 11:31:27
>> Assunto: Re: [GENERAL] count with high allocation
>>
>> paulo matadr <saddoness(at)yahoo(dot)com(dot)br> writes:
>>> Monitoring "top" in database server , i could noticed an query with
>>> reserved 8GB on physical memory.
>>
>>> select count(field) from big_table 1 inner join big_table2...
>>
>>> There is the possibility of using another function with less memory
>>> allocation?
>>> Is there a way to limit the memory usage of the count?
>>
>> It seems quite likely that what top is telling you just reflects the
>> process touching all shared buffers, and has nothing to do with any
>> real "memory consumption". What do you have shared_buffers set to?
>>
>> regards, tom lane
>>
>> --
>> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-general
>>
>> ________________________________
>> Veja quais são os assuntos do momento no Yahoo! + Buscados: Top 10 -
>> Celebridades - Música - Esportes
>
>

--
Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

____________________________________________________________________________________
Veja quais são os assuntos do momento no Yahoo! +Buscados
http://br.maisbuscados.yahoo.com


From: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
To: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: count with high allocation
Date: 2010-04-07 17:12:18
Message-ID: j2r162867791004071012qddb6959ds5c39630d6569be8e@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-general

---------- Forwarded message ----------
From: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Date: 2010/4/7
Subject: Res: [GENERAL] count with high allocation
To: pgsql-general(at)postgresql(dot)org

shared_buffer is too large. It is good for server with 64GB RAM. It
can be about 1/2 RAM for dedicated server. PostgreSQL allocate shared
memory after start - before your query, and use it as memory cache.

postgres=# show shared_buffers;
 shared_buffers
----------------
 16GB
(1 row)

shared_buffers + work_mem * max_connection < 90% of dedicated memory

Regards
Pavel Stehule

postgres=# show work_mem;
 work_mem
----------
 5MB
(1 row)

________________________________
De: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
Cc: pgsql-general(at)postgresql(dot)org
Enviadas: Quarta-feira, 7 de Abril de 2010 13:51:24
Assunto: Re: [GENERAL] count with high allocation

2010/4/7 paulo matadr <saddoness(at)yahoo(dot)com(dot)br>:
> with
> set enable_hashagg to off , I give the same allocation.

ok, then problem will be other.

what is result of:

show shared_buffers;
show work_mem;

Regards
Pavel Stehule
>
> ________________________________
> De: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
> Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
> Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>; GENERAL <pgsql-general(at)postgresql(dot)org>
> Enviadas: Quarta-feira, 7 de Abril de 2010 12:10:23
> Assunto: Re: [GENERAL] count with high allocation
>
> please, EXPLAIN ANALYZE
>
> and try to execute
>
> set enable_hashagg to off before as second variant. It have to take less
> memory
>
> regards
> Pavel Stehule
>
> 2010/4/7 paulo matadr <saddoness(at)yahoo(dot)com(dot)br>:
>> EXPLAIN/TOP
>>
>> EXPLAIN:
>> Aggregate  (cost=3710076.58..3710076.59 rows=1 width=4)
>>   ->  Hash Join  (cost=2668820.36..3692200.16 rows=7150570 width=4)
>>         Hash Cond: (osunidade.attp_id = art.attp_id)
>>         ->  Hash Join  (cost=2668819.29..3593878.75 rows=7150570 width=8)
>>               Hash Cond: (osunidade.orse_id = os.orse_id)
>>               ->  Seq Scan on ordem_servico_unidade osunidade
>> (cost=0.00..429514.00 rows=23418900 width=8)
>>               ->  Hash  (cost=2598702.48..2598702.48 rows=4033665 width=4)
>>                     ->  Hash Left Join  (cost=1372486.83..2598702.48
>> rows=4033665 width=4)
>>                           Hash Cond: (os.cbdo_id = cobra.cbdo_id)
>>                           ->  Merge Join  (cost=0.00..880392.67
>> rows=4033665
>> width=8)
>>                                 Merge Cond: (os.rgat_id = ra.rgat_id)
>>                                 ->  Index Scan using xfk1_ordem_servico on
>> ordem_servico os  (cost=0.00..879051.89 rows=13210693 width=12)
>>                                 ->  Index Scan using
>> registro_atendimento_pkey on registro_atendimento ra
>> (cost=0.00..548171.12
>> rows=5369913 width=4)
>>                                       Filter: (rgat_id IS NOT NULL)
>>                           ->  Hash  (cost=897238.26..897238.26
>> rows=27340126
>> width=4)
>>                                 ->  Seq Scan on cobranca_documento cobra
>> (cost=0.00..897238.26 rows=27340126 width=4)
>>         ->  Hash  (cost=1.03..1.03 rows=3 width=4)
>>               ->  Seq Scan on atendimento_relacao_tipo art
>> (cost=0.00..1.03
>> rows=3 width=4)
>>
>> "TOP"
>> PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>
>> 26809 postgres  16   0 16.5g 8.6g 8.6g S    1 27.5   1:28.84 postgres:
>> user
>> database 10.1.1.7(54033) SELECT
>> ________________________________
>> De: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
>> Para: paulo matadr <saddoness(at)yahoo(dot)com(dot)br>
>> Cc: GENERAL <pgsql-general(at)postgresql(dot)org>
>> Enviadas: Quarta-feira, 7 de Abril de 2010 11:31:27
>> Assunto: Re: [GENERAL] count with high allocation
>>
>> paulo matadr <saddoness(at)yahoo(dot)com(dot)br> writes:
>>> Monitoring "top" in database server  , i could  noticed an query with
>>> reserved  8GB on physical memory.
>>
>>> select count(field) from big_table  1 inner join big_table2...
>>
>>> There is the possibility of using another function with less memory
>>> allocation?
>>> Is there a way to limit the memory usage of the count?
>>
>> It seems quite likely that what top is telling you just reflects the
>> process touching all shared buffers, and has nothing to do with any
>> real "memory consumption".  What do you have shared_buffers set to?
>>
>>             regards, tom lane
>>
>> --
>> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-general
>>
>> ________________________________
>> Veja quais são os assuntos do momento no Yahoo! + Buscados: Top 10 -
>> Celebridades - Música - Esportes
>
>

--
Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

________________________________
Veja quais são os assuntos do momento no Yahoo! + Buscados: Top 10 -
Celebridades - Música - Esportes