error on online backup using pg_basebackup tool

Hi 4 all!!

As Ido not know if it is my mistake or actually a bug, sending the two
topics.

When trying to perform online backup using the pg_basebackup tool, I get
the message: The folder is not empty.

steps to reproduce the error:
1 - install postgres in the / usr / local / pgsql
2 - initialize the database: / usr / local / pgsql / bin / -D /data
3 - set up pg_hba.conf to allow replication (used by online backup with
pg_basebackup)
4 - create the directories /data/newtblspc and /data/pg_tblspc/newtblspc2
5 - start with postgres -D /data
6 - create table spaces:
CREATE TABLESPACE newtblspc LOCATION '/data/newtblspc';
CREATE TABLESPACE newtblspc2 LOCATION '/data/pg_tblspc/newtblspc2'
7 - mkdir /data2
8 - Now perform a backup using pg_basebackup -D /data2 and get the error
message.

--
T.'.A.'.F.'.,
*Gerdan Rezende dos Santos *
*Po*stgreSQL & EnterpriseDB Specialist, Support, Training & Services
+55 (61) 9645-1525

On Thu, Sep 3, 2015 at 10:28 AM, Gerdan Rezende dos Santos
<gerdan(at)gmail(dot)com> wrote:
> 6 - create table spaces:
> CREATE TABLESPACE newtblspc LOCATION '/data/newtblspc';
> CREATE TABLESPACE newtblspc2 LOCATION '/data/pg_tblspc/newtblspc2'
> 7 - mkdir /data2
> 8 - Now perform a backup using pg_basebackup -D /data2 and get the error
> message.

You are performing a base backup on a node that has tablespaces on the
same hosts, so tablespaces are going to overlap. Isn't that your
problem? Note that 9.4 you can use tablespace-mapping to address the
problem, for older versions you are going to need some manual
operations before being able to run the standby and the master on the
same server.
--
Michael

I use the tablespaces without problem but the backup online not
complete... but when I attempt to online backup using the pg_basebackup
displays the error message saying q pg_tblspc is not clean and the process
is terminated without sucess

Em quarta-feira, 2 de setembro de 2015, Michael Paquier <
michael(dot)paquier(at)gmail(dot)com> escreveu:

> On Thu, Sep 3, 2015 at 10:28 AM, Gerdan Rezende dos Santos
> <gerdan(at)gmail(dot)com <javascript:;>> wrote:
> > 6 - create table spaces:
> > CREATE TABLESPACE newtblspc LOCATION '/data/newtblspc';
> > CREATE TABLESPACE newtblspc2 LOCATION '/data/pg_tblspc/newtblspc2'
> > 7 - mkdir /data2
> > 8 - Now perform a backup using pg_basebackup -D /data2 and get the error
> > message.
>
> You are performing a base backup on a node that has tablespaces on the
> same hosts, so tablespaces are going to overlap. Isn't that your
> problem? Note that 9.4 you can use tablespace-mapping to address the
> problem, for older versions you are going to need some manual
> operations before being able to run the standby and the master on the
> same server.
> --
> Michael
>

--
T.'.A.'.F.'.,
*Gerdan Rezende dos Santos *
*Po*stgreSQL & EnterpriseDB Specialist, Support, Training & Services
+55 (61) 9645-1525

On Thu, Sep 3, 2015 at 12:20 PM, Gerdan Rezende dos Santos
<gerdan(at)gmail(dot)com> wrote:
>[...]

Note 1: Please don't top-post.
Note 2: Please don't post across multiple mailing lists.

> I use the tablespaces without problem but the backup online not complete...
> but when I attempt to online backup using the pg_basebackup displays the
> error message saying q pg_tblspc is not clean and the process is terminated
> without sucess.

That's what I am saying. You will need to redirect manually the soft
links of pg_tbspc into other paths if you want to run a standby on the
same host as the master.

That's not a bug, just an operation mistake.
--
Michael

Em quinta-feira, 3 de setembro de 2015, Michael Paquier <
michael(dot)paquier(at)gmail(dot)com> escreveu:

> On Thu, Sep 3, 2015 at 12:20 PM, Gerdan Rezende dos Santos
> <gerdan(at)gmail(dot)com <javascript:;>> wrote:
> >[...]
>
> Note 1: Please don't top-post.
> Note 2: Please don't post across multiple mailing lists.
>
> > I use the tablespaces without problem but the backup online not
> complete...
> > but when I attempt to online backup using the pg_basebackup displays the
> > error message saying q pg_tblspc is not clean and the process is
> terminated
> > without sucess.
>
> That's what I am saying. You will need to redirect manually the soft
> links of pg_tbspc into other paths if you want to run a standby on the
> same host as the master.
>
> That's not a bug, just an operation mistake.
> --
> Michael
>

In other server with same configuration the proccess not complete too!

--
T.'.A.'.F.'.,
*Gerdan Rezende dos Santos *
*Po*stgreSQL & EnterpriseDB Specialist, Support, Training & Services
+55 (61) 9645-1525

El 03/09/15 a las 00:54, Gerdan Rezende dos Santos escribió:
>
>
> In other server with same configuration the proccess not complete too!

Add --verbose to the pg_basebackup execution and send over the output
error message.

I find your procedure confusing, and I'd recommend you *not* to put
tablespaces in $PGDTA/pg_tblspc/. That directory is for postgres to put
the links to where the actual tablespace is (normally on another partition)

--
Martín Marqués http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Thu, Sep 3, 2015 at 7:20 AM, Martín Marqués <martin(at)2ndquadrant(dot)com>
wrote:

> El 03/09/15 a las 00:54, Gerdan Rezende dos Santos escribió:
> >
> >
> > In other server with same configuration the proccess not complete too!
>
> Add --verbose to the pg_basebackup execution and send over the output
> error message.
>
> I find your procedure confusing, and I'd recommend you *not* to put
> tablespaces in $PGDTA/pg_tblspc/. That directory is for postgres to put
> the links to where the actual tablespace is (normally on another partition)
>
> --
> Martín Marqués http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
>

This message on verbose mode: pg_basebackup: directory "/data/pg_tblspc/"
exists but is not empty

From machine A to machine B

pb_basebackup started in machine B, connecting on machine A

On machine B
pb_basebackup -D /data/pg_tblspc/ -h machineA

T.'.A.'.F.'.,
*Gerdan Rezende dos Santos *
*Po*stgreSQL & EnterpriseDB Specialist, Support, Training & Services
+55 (61) 9645-1525

2015-09-03 16:48 GMT-03:00 Gerdan Rezende dos Santos <gerdan(at)gmail(dot)com>:
> On Thu, Sep 3, 2015 at 7:20 AM, Martín Marqués <martin(at)2ndquadrant(dot)com>
> wrote:
>>
>> El 03/09/15 a las 00:54, Gerdan Rezende dos Santos escribió:
>> >
>> >
>> > In other server with same configuration the proccess not complete too!
>>
>> Add --verbose to the pg_basebackup execution and send over the output
>> error message.
>>
>> I find your procedure confusing, and I'd recommend you *not* to put
>> tablespaces in $PGDTA/pg_tblspc/. That directory is for postgres to put
>> the links to where the actual tablespace is (normally on another
>> partition)
>
> This message on verbose mode: pg_basebackup: directory "/data/pg_tblspc/"
> exists but is not empty

The error message is quite clear, it has to use /data/pg_tblspc/ to
put a tablespace, but that directory is not empty.

Why are you putting tablespaces inside PGDATA? It really makes no
sense. Just leave the objects you have there in the default
tablespace.

Regards,

--
Martín Marqués http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Thu, Sep 3, 2015 at 5:01 PM, Martín Marqués <martin(at)2ndquadrant(dot)com>
wrote:

> 2015-09-03 16:48 GMT-03:00 Gerdan Rezende dos Santos <gerdan(at)gmail(dot)com>:
> > On Thu, Sep 3, 2015 at 7:20 AM, Martín Marqués <martin(at)2ndquadrant(dot)com>
> > wrote:
> >>
> >> El 03/09/15 a las 00:54, Gerdan Rezende dos Santos escribió:
> >> >
> >> >
> >> > In other server with same configuration the proccess not complete too!
> >>
> >> Add --verbose to the pg_basebackup execution and send over the output
> >> error message.
> >>
> >> I find your procedure confusing, and I'd recommend you *not* to put
> >> tablespaces in $PGDTA/pg_tblspc/. That directory is for postgres to put
> >> the links to where the actual tablespace is (normally on another
> >> partition)
> >
> > This message on verbose mode: pg_basebackup: directory "/data/pg_tblspc/"
> > exists but is not empty
>
> The error message is quite clear, it has to use /data/pg_tblspc/ to
> put a tablespace, but that directory is not empty.
>
> Why are you putting tablespaces inside PGDATA? It really makes no
> sense. Just leave the objects you have there in the default
> tablespace.
>
> Regards,
>
> --
> Martín Marqués http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
>

This problem, I found in my client ...
Why are you putting in PGDATA table ? He can not say to me
The solution then is to continue using the start_backup , rsync and
stop_backup. ;(

Thanks!

Hi Gerdan,

2015-09-03 17:41 GMT-03:00 Gerdan Rezende dos Santos <gerdan(at)gmail(dot)com>:
>
> This problem, I found in my client ...
> Why are you putting in PGDATA table ? He can not say to me
> The solution then is to continue using the start_backup , rsync and
> stop_backup. ;(

Or use repmgr. ;)

If you intend to use repmgr3 (will only work with postgres 9.3 and
above) you'll need to pass it the --rsync-only option so it doesn't
use pg_basebackup to clone.

Regards,

--
Martín Marqués http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Thu, Sep 3, 2015 at 5:53 PM, Martín Marqués <martin(at)2ndquadrant(dot)com>
wrote:

> Hi Gerdan,
>
> 2015-09-03 17:41 GMT-03:00 Gerdan Rezende dos Santos <gerdan(at)gmail(dot)com>:
> >
> > This problem, I found in my client ...
> > Why are you putting in PGDATA table ? He can not say to me
> > The solution then is to continue using the start_backup , rsync and
> > stop_backup. ;(
>
> Or use repmgr. ;)
>
> If you intend to use repmgr3 (will only work with postgres 9.3 and
> above) you'll need to pass it the --rsync-only option so it doesn't
> use pg_basebackup to clone.
>
> Regards,
>
> --
> Martín Marqués http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
>

Matín, the client have a BART from EDB.
EDB BART only use pg_basebackup.
Its my problem.

On Thu, Sep 3, 2015 at 4:41 PM, Gerdan Rezende dos Santos <gerdan(at)gmail(dot)com>
wrote:

> On Thu, Sep 3, 2015 at 5:01 PM, Martín Marqués <martin(at)2ndquadrant(dot)com>
> wrote:
>
>> 2015-09-03 16:48 GMT-03:00 Gerdan Rezende dos Santos <gerdan(at)gmail(dot)com>:
>> > On Thu, Sep 3, 2015 at 7:20 AM, Martín Marqués <martin(at)2ndquadrant(dot)com>
>> > wrote:
>> >>
>> >> El 03/09/15 a las 00:54, Gerdan Rezende dos Santos escribió:
>> >> >
>> >> >
>> >> > In other server with same configuration the proccess not complete
>> too!
>> >>
>> >> Add --verbose to the pg_basebackup execution and send over the output
>> >> error message.
>> >>
>> >> I find your procedure confusing, and I'd recommend you *not* to put
>> >> tablespaces in $PGDTA/pg_tblspc/. That directory is for postgres to put
>> >> the links to where the actual tablespace is (normally on another
>> >> partition)
>> >
>> > This message on verbose mode: pg_basebackup: directory
>> "/data/pg_tblspc/"
>> > exists but is not empty
>>
>> The error message is quite clear, it has to use /data/pg_tblspc/ to
>> put a tablespace, but that directory is not empty.
>>
>> Why are you putting tablespaces inside PGDATA? It really makes no
>> sense. Just leave the objects you have there in the default
>> tablespace.
>>
>> Regards,
>>
>> --
>> Martín Marqués http://www.2ndQuadrant.com/
>> PostgreSQL Development, 24x7 Support, Training & Services
>>
>
>
> This problem, I found in my client ...
> Why are you putting in PGDATA table ? He can not say to me
> The solution then is to continue using the start_backup , rsync and
> stop_backup. ;(
>
>
You can always move the folder out of there. Depending on the version of
PostgreSQL different steps need to be take. If I remember correctly for
version 9.3 you can move the symlink to point to the new directory. Pre 9.3
you need to move the symlink and the pointer inside the pg tables.

-Joseph Kregloh

> Thanks!
>

Hi Gerdan,

2015-09-03 18:06 GMT-03:00 Gerdan Rezende dos Santos <gerdan(at)gmail(dot)com>:
>
> Matín, the client have a BART from EDB.
> EDB BART only use pg_basebackup.
> Its my problem.

AFAIK BART is a backup & recovery tool, and I thought you were trying
to spin new standbys, which is why I recommended repmgr. If you need
to do backups then repmgr is not the way to go. There are other
alternatives.

Regards,

--
Martín Marqués http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 9/3/15 5:07 PM, Joseph Kregloh wrote:
> You can always move the folder out of there. Depending on the version of
> PostgreSQL different steps need to be take. If I remember correctly for
> version 9.3 you can move the symlink to point to the new directory. Pre
> 9.3 you need to move the symlink and the pointer inside the pg tables.

You can move the directory and modify the symlink on 9.2 or later
without any additional complications.

However - and this is very important - PostgreSQL must be shut down
during this operation.

My recommendation would be to move both tablespaces out of $PGDATA.
There has been a lot of discussion about disallowing it in future
releases as it causes problems for various tools.

--
-David
david(at)pgmasters(dot)net