Re: pg_largeobject

Hi

pg_largeobject is creating performance issues as it grow due to single
point storage(for all tables)

is there any alternate apart from bytea ?

like configuration large-object-table at table-column level and oid
PK(primary key) stored at pg_largeobject

Thanks
Sridhar

On 3/29/2016 2:13 AM, Sridhar N Bamandlapally wrote:
> Hi
>
> pg_largeobject is creating performance issues as it grow due to single
> point storage(for all tables)
>
> is there any alternate apart from bytea ?
>
> like configuration large-object-table at table-column level and oid
> PK(primary key) stored at pg_largeobject
>

I would as soon use a NFS file store for larger files like images,
audio, videos, or whatever. use SQL for the relational metadata.

just sayin'....

--
john r pierce, recycling bits in santa cruz

all media files are stored in database with size varies from 1MB - 5GB

based on media file types and user-group we storing in different tables,
but PostgreSQL store OID/Large-object in single table (pg_largeobject), 90%
of database size is with table pg_largeobject

due to size limitation BYTEA was not considered

Thanks
Sridhar

On Tue, Mar 29, 2016 at 3:05 PM, John R Pierce <pierce(at)hogranch(dot)com> wrote:

> On 3/29/2016 2:13 AM, Sridhar N Bamandlapally wrote:
>
>> Hi
>>
>> pg_largeobject is creating performance issues as it grow due to single
>> point storage(for all tables)
>>
>> is there any alternate apart from bytea ?
>>
>> like configuration large-object-table at table-column level and oid
>> PK(primary key) stored at pg_largeobject
>>
>>
> I would as soon use a NFS file store for larger files like images, audio,
> videos, or whatever. use SQL for the relational metadata.
>
> just sayin'....
>
>
>
> --
> john r pierce, recycling bits in santa cruz
>
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

Some time ago I had to setup a replicated file system between multiple linux servers. I tried everything I could based on postgres, including large objects, but everything was significantly slower than a regular filesystem.

My conclussion: postgres is not suitable for storing large files efficiently.

Do you need that for replication, or just for file storage?

Alvaro Aguayo
Jefe de Operaciones
Open Comb Systems E.I.R.L.

Oficina: (+51-1) 3377813 | RPM: #034252 / (+51) 995540103  | RPC: (+51) 954183248
Website: www.ocs.pe

Sent from my Sony Xperia™ smartphone

---- Sridhar N Bamandlapally wrote ----

all media files are stored in database with size varies from 1MB - 5GB

based on media file types and user-group we storing in different tables,
but PostgreSQL store OID/Large-object in single table (pg_largeobject), 90%
of database size is with table pg_largeobject

due to size limitation BYTEA was not considered

Thanks
Sridhar

On Tue, Mar 29, 2016 at 3:05 PM, John R Pierce <pierce(at)hogranch(dot)com> wrote:

> On 3/29/2016 2:13 AM, Sridhar N Bamandlapally wrote:
>
>> Hi
>>
>> pg_largeobject is creating performance issues as it grow due to single
>> point storage(for all tables)
>>
>> is there any alternate apart from bytea ?
>>
>> like configuration large-object-table at table-column level and oid
>> PK(primary key) stored at pg_largeobject
>>
>>
> I would as soon use a NFS file store for larger files like images, audio,
> videos, or whatever. use SQL for the relational metadata.
>
> just sayin'....
>
>
>
> --
> john r pierce, recycling bits in santa cruz
>
>
>
> --
> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

We are doing application/database migration compatible with postgresql on
cloud, DR/replication also in plan

at present I feel need of configurable multi-table storage instead of
pg_largeobject only

Thanks
Sridhar

On Tue, Mar 29, 2016 at 6:08 PM, Alvaro Aguayo Garcia-Rada <
aaguayo(at)opensysperu(dot)com> wrote:

> Some time ago I had to setup a replicated file system between multiple
> linux servers. I tried everything I could based on postgres, including
> large objects, but everything was significantly slower than a regular
> filesystem.
>
> My conclussion: postgres is not suitable for storing large files
> efficiently.
>
> Do you need that for replication, or just for file storage?
>
> Alvaro Aguayo
> Jefe de Operaciones
> Open Comb Systems E.I.R.L.
>
> Oficina: (+51-1) 3377813 | RPM: #034252 / (+51) 995540103 | RPC:
> (+51) 954183248
> Website: www.ocs.pe
>
> Sent from my Sony Xperia™ smartphone
>
>
> ---- Sridhar N Bamandlapally wrote ----
>
>
> all media files are stored in database with size varies from 1MB - 5GB
>
> based on media file types and user-group we storing in different tables,
> but PostgreSQL store OID/Large-object in single table (pg_largeobject), 90%
> of database size is with table pg_largeobject
>
> due to size limitation BYTEA was not considered
>
> Thanks
> Sridhar
>
>
>
> On Tue, Mar 29, 2016 at 3:05 PM, John R Pierce <pierce(at)hogranch(dot)com>
> wrote:
>
>> On 3/29/2016 2:13 AM, Sridhar N Bamandlapally wrote:
>>
>>> Hi
>>>
>>> pg_largeobject is creating performance issues as it grow due to single
>>> point storage(for all tables)
>>>
>>> is there any alternate apart from bytea ?
>>>
>>> like configuration large-object-table at table-column level and oid
>>> PK(primary key) stored at pg_largeobject
>>>
>>>
>> I would as soon use a NFS file store for larger files like images, audio,
>> videos, or whatever. use SQL for the relational metadata.
>>
>> just sayin'....
>>
>>
>>
>> --
>> john r pierce, recycling bits in santa cruz
>>
>>
>>
>> --
>> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-general
>>
>
>

Amongst all my tries, I also tried that. I created two tables, one for basic file data, and another for file content(splitted in pages, as in large objects), but the performance was almost the same as with pg_largeobject; he great difference was that, with my own tables, I could replicate without problems with pgpool2, which was troublesome with large objects.

Based on my experience, I would seriously recommend to search for another solution, as postgres may not be suitable for large files storage. In my case, I ended up using MS DFS-R, but there are some other solutions like Ceph, GlusterFS, and many others. Also, I've recently heard about MongoDB, which has it's own database-backed filesystem optimized for large files; never tried it, but may be worth a try.

Regards,

Alvaro Aguayo
Jefe de Operaciones
Open Comb Systems E.I.R.L.

Oficina: (+51-1) 3377813 | RPM: #034252 / (+51) 995540103 | RPC: (+51) 954183248
Website: www.ocs.pe

----- Original Message -----
From: "Sridhar N Bamandlapally" <sridhar(dot)bn1(at)gmail(dot)com>
To: "Alvaro Aguayo Garcia-Rada" <aaguayo(at)opensysperu(dot)com>
Cc: "John R Pierce" <pierce(at)hogranch(dot)com>, "PostgreSql-general" <pgsql-general(at)postgresql(dot)org>
Sent: Tuesday, 29 March, 2016 10:09:10
Subject: Re: [GENERAL] pg_largeobject

We are doing application/database migration compatible with postgresql on
cloud, DR/replication also in plan

at present I feel need of configurable multi-table storage instead of
pg_largeobject only

Thanks
Sridhar

On Tue, Mar 29, 2016 at 6:08 PM, Alvaro Aguayo Garcia-Rada <
aaguayo(at)opensysperu(dot)com> wrote:

> Some time ago I had to setup a replicated file system between multiple
> linux servers. I tried everything I could based on postgres, including
> large objects, but everything was significantly slower than a regular
> filesystem.
>
> My conclussion: postgres is not suitable for storing large files
> efficiently.
>
> Do you need that for replication, or just for file storage?
>
> Alvaro Aguayo
> Jefe de Operaciones
> Open Comb Systems E.I.R.L.
>
> Oficina: (+51-1) 3377813 | RPM: #034252 / (+51) 995540103 | RPC:
> (+51) 954183248
> Website: www.ocs.pe
>
> Sent from my Sony Xperia™ smartphone
>
>
> ---- Sridhar N Bamandlapally wrote ----
>
>
> all media files are stored in database with size varies from 1MB - 5GB
>
> based on media file types and user-group we storing in different tables,
> but PostgreSQL store OID/Large-object in single table (pg_largeobject), 90%
> of database size is with table pg_largeobject
>
> due to size limitation BYTEA was not considered
>
> Thanks
> Sridhar
>
>
>
> On Tue, Mar 29, 2016 at 3:05 PM, John R Pierce <pierce(at)hogranch(dot)com>
> wrote:
>
>> On 3/29/2016 2:13 AM, Sridhar N Bamandlapally wrote:
>>
>>> Hi
>>>
>>> pg_largeobject is creating performance issues as it grow due to single
>>> point storage(for all tables)
>>>
>>> is there any alternate apart from bytea ?
>>>
>>> like configuration large-object-table at table-column level and oid
>>> PK(primary key) stored at pg_largeobject
>>>
>>>
>> I would as soon use a NFS file store for larger files like images, audio,
>> videos, or whatever. use SQL for the relational metadata.
>>
>> just sayin'....
>>
>>
>>
>> --
>> john r pierce, recycling bits in santa cruz
>>
>>
>>
>> --
>> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-general
>>
>
>

Sridhar N Bamandlapally wrote:

> due to size limitation BYTEA was not considered

You could adopt for a custom table the same kind of structure that
pg_largeobject has, that is an ordered series of BYTEA chunks.

# \d pg_largeobject
Table "pg_catalog.pg_largeobject"
Column | Type | Modifiers
--------+---------+-----------
loid | oid | not null
pageno | integer | not null
data | bytea |

Say you create a table looking like this:
(
object_id int
pageno integer
data bytea
)
with a unique index on (object_id,pageno),
and octet_length(data) never exceeding a reasonable max size,
such as 1MB, so if you want a 5GB object, that's just 5*1024 rows in
that table, one row per pageno.

It's really a good plan if your client code cooperates by streaming
contents ordered by pageno instead of handling the blob as
a monolith.

About the chunk size, by comparison, the large object facility limits
pg_largeobject.data to a quarter of a page, or 2048 bytes per row
(=LOBLKSIZE, see comments around
http://doxygen.postgresql.org/large__object_8h_source.html#l00072 )

Having your own table has several advantages:
- it contains much less rows for the same contents, if the choosen chunk
size is
much larger than 2048 bytes.
- TOAST storage is enabled so that the main relation is way smaller.
- it can be partitioned.
- it can have triggers (good for custom replication)

The drawback being that your application has to provide the equivalent
code to the lo_* client-side and server-side functions that it needs.
But that's a relatively easy work for a programmer, especially if the blobs
happen to be immutable, as is often the case.

Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite

I am not saying that this will solve your problem (I never tried id even
though I keep it in my radar), but this project seems to implement
something close to what Daniel is describing:

https://github.com/andreasbaumann/pgfuse

+ it gives you a FUSE wrapper so the client can use fs calls.

the proposed schema is here
https://github.com/andreasbaumann/pgfuse/blob/master/schema.sql

On Tue, Mar 29, 2016 at 5:09 PM, Sridhar N Bamandlapally <
sridhar(dot)bn1(at)gmail(dot)com> wrote:

> We are doing application/database migration compatible with postgresql on
> cloud, DR/replication also in plan
>
> at present I feel need of configurable multi-table storage instead of
> pg_largeobject only
>
> Thanks
> Sridhar
>
>
> On Tue, Mar 29, 2016 at 6:08 PM, Alvaro Aguayo Garcia-Rada <
> aaguayo(at)opensysperu(dot)com> wrote:
>
>> Some time ago I had to setup a replicated file system between multiple
>> linux servers. I tried everything I could based on postgres, including
>> large objects, but everything was significantly slower than a regular
>> filesystem.
>>
>> My conclussion: postgres is not suitable for storing large files
>> efficiently.
>>
>> Do you need that for replication, or just for file storage?
>>
>> Alvaro Aguayo
>> Jefe de Operaciones
>> Open Comb Systems E.I.R.L.
>>
>> Oficina: (+51-1) 3377813 | RPM: #034252 / (+51) 995540103 | RPC:
>> (+51) 954183248
>> Website: www.ocs.pe
>>
>> Sent from my Sony Xperia™ smartphone
>>
>>
>> ---- Sridhar N Bamandlapally wrote ----
>>
>>
>> all media files are stored in database with size varies from 1MB - 5GB
>>
>> based on media file types and user-group we storing in different tables,
>> but PostgreSQL store OID/Large-object in single table (pg_largeobject), 90%
>> of database size is with table pg_largeobject
>>
>> due to size limitation BYTEA was not considered
>>
>> Thanks
>> Sridhar
>>
>>
>>
>> On Tue, Mar 29, 2016 at 3:05 PM, John R Pierce <pierce(at)hogranch(dot)com>
>> wrote:
>>
>>> On 3/29/2016 2:13 AM, Sridhar N Bamandlapally wrote:
>>>
>>>> Hi
>>>>
>>>> pg_largeobject is creating performance issues as it grow due to single
>>>> point storage(for all tables)
>>>>
>>>> is there any alternate apart from bytea ?
>>>>
>>>> like configuration large-object-table at table-column level and oid
>>>> PK(primary key) stored at pg_largeobject
>>>>
>>>>
>>> I would as soon use a NFS file store for larger files like images,
>>> audio, videos, or whatever. use SQL for the relational metadata.
>>>
>>> just sayin'....
>>>
>>>
>>>
>>> --
>>> john r pierce, recycling bits in santa cruz
>>>
>>>
>>>
>>> --
>>> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
>>> To make changes to your subscription:
>>> http://www.postgresql.org/mailpref/pgsql-general
>>>
>>
>>
>

Is there any way we can change the segment file size,

I am trying to look into the possibility of segment file size Vs bytea size
limitation

PostgreSQL installation

step 1: ./configure --enable-largefile --with-segsize ( throwing error
"configure: error: Large file support is not enabled. Segment size cannot
be larger than 1GB" )

Thanks
Sridhar

On Tue, Mar 29, 2016 at 9:01 PM, Daniel Verite <daniel(at)manitou-mail(dot)org>
wrote:

> Sridhar N Bamandlapally wrote:
>
> > due to size limitation BYTEA was not considered
>
> You could adopt for a custom table the same kind of structure that
> pg_largeobject has, that is an ordered series of BYTEA chunks.
>
> # \d pg_largeobject
> Table "pg_catalog.pg_largeobject"
> Column | Type | Modifiers
> --------+---------+-----------
> loid | oid | not null
> pageno | integer | not null
> data | bytea |
>
> Say you create a table looking like this:
> (
> object_id int
> pageno integer
> data bytea
> )
> with a unique index on (object_id,pageno),
> and octet_length(data) never exceeding a reasonable max size,
> such as 1MB, so if you want a 5GB object, that's just 5*1024 rows in
> that table, one row per pageno.
>
> It's really a good plan if your client code cooperates by streaming
> contents ordered by pageno instead of handling the blob as
> a monolith.
>
> About the chunk size, by comparison, the large object facility limits
> pg_largeobject.data to a quarter of a page, or 2048 bytes per row
> (=LOBLKSIZE, see comments around
> http://doxygen.postgresql.org/large__object_8h_source.html#l00072 )
>
> Having your own table has several advantages:
> - it contains much less rows for the same contents, if the choosen chunk
> size is
> much larger than 2048 bytes.
> - TOAST storage is enabled so that the main relation is way smaller.
> - it can be partitioned.
> - it can have triggers (good for custom replication)
>
> The drawback being that your application has to provide the equivalent
> code to the lo_* client-side and server-side functions that it needs.
> But that's a relatively easy work for a programmer, especially if the blobs
> happen to be immutable, as is often the case.
>
> Best regards,
> --
> Daniel Vérité
> PostgreSQL-powered mailer: http://www.manitou-mail.org
> Twitter: @DanielVerite
>

Sridhar N Bamandlapally <sridhar(dot)bn1(at)gmail(dot)com> wrote:

> Is there any way we can change the segment file size, 
>
> I am trying to look into the possibility of segment file size Vs bytea size
> limitation
>
> PostgreSQL installation
>
> step 1:  ./configure --enable-largefile --with-segsize ( throwing error
> "configure: error: Large file support is not enabled. Segment size cannot be
> larger than 1GB" )

check if your filesystem supports large files:

getconf FILESIZEBITS /some/path

If the result is 64, LFS is supported.

Regards, Andreas Kretschmer
--
Andreas Kretschmer
http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services