Re: pg_upgrade in 9.3.5 can not start cluster

Hi,

we've encountered possible bug in debian packages, both squeeze and wheezy seem to be affected.

We're using following repositories:
http://apt.postgresql.org/pub/repos/apt/ squeeze-pgdg/main amd64 Packages
http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg/main amd64 Packages

This bug makes it currently impossible to run pg_upgrade script on 9.3 and therefore successfully upgrade from 9.2 to 9.3. It seems to be a typo somewhere in the source code. Unfortunately, I don't have the logs anymore, but the problem was that pg_upgrade refused to start cluster with an error that went like 'parameter "full_page_writes" requires a Boolean value'. The problem was that the startup command had a typo in it and contained '-c full_page_writes=off-c anotherparameter...' (Notice no space after =off).

I think the pg_upgrade script worked when we were upgrading from 9.2 to 9.3.4, upgrade from 9.2 to 9.3.5 does not work anymore. As a workaround we have used pg_upgrade from 9.3.4 package and it worked.

Could you guys look at it? Unfortunately due to internal policy I can't test any patches in our environment, but this should be easily reproducible on a clean debian box.

Regards,
Jirka Chadima

Re: Chadima Jiří 2014-10-01 <D1FA90F570B9E3449D7EF1A4D43D1736B7203591(at)CZPAV00092(dot)pa(dot)jtfg(dot)com>
> Hi,
>
> we've encountered possible bug in debian packages, both squeeze and wheezy seem to be affected.
>
> We're using following repositories:
> http://apt.postgresql.org/pub/repos/apt/ squeeze-pgdg/main amd64 Packages
> http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg/main amd64 Packages
>
> This bug makes it currently impossible to run pg_upgrade script on 9.3 and therefore successfully upgrade from 9.2 to 9.3. It seems to be a typo somewhere in the source code. Unfortunately, I don't have the logs anymore, but the problem was that pg_upgrade refused to start cluster with an error that went like 'parameter "full_page_writes" requires a Boolean value'. The problem was that the startup command had a typo in it and contained '-c full_page_writes=off-c anotherparameter...' (Notice no space after =off).
>
> I think the pg_upgrade script worked when we were upgrading from 9.2 to 9.3.4, upgrade from 9.2 to 9.3.5 does not work anymore. As a workaround we have used pg_upgrade from 9.3.4 package and it worked.
>
> Could you guys look at it? Unfortunately due to internal policy I can't test any patches in our environment, but this should be easily reproducible on a clean debian box.

It's not reproducible on a clean Debian box.

Do you have full_page_writes mentioned in
/etc/postgresql/9.2/main/pg_ctl.conf or the like?

Christoph
--
cb(at)df7cb(dot)de | http://www.df7cb.de/

Actually, we don't... I have managed another clean environment, so we will try the same process with the same configuration again and I'll let you know.

Jirka

-----Original Message-----
From: Christoph Berg [mailto:cb(at)df7cb(dot)de]
Sent: Saturday, October 04, 2014 8:47 PM
To: Chadima Jiří
Cc: pgsql-pkg-debian(at)postgresql(dot)org
Subject: Re: [pgsql-pkg-debian] pg_upgrade in 9.3.5 can not start cluster

Re: Chadima Jiří 2014-10-01 <D1FA90F570B9E3449D7EF1A4D43D1736B7203591(at)CZPAV00092(dot)pa(dot)jtfg(dot)com>
> Hi,
>
> we've encountered possible bug in debian packages, both squeeze and wheezy seem to be affected.
>
> We're using following repositories:
> http://apt.postgresql.org/pub/repos/apt/ squeeze-pgdg/main amd64
> Packages http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg/main
> amd64 Packages
>
> This bug makes it currently impossible to run pg_upgrade script on 9.3 and therefore successfully upgrade from 9.2 to 9.3. It seems to be a typo somewhere in the source code. Unfortunately, I don't have the logs anymore, but the problem was that pg_upgrade refused to start cluster with an error that went like 'parameter "full_page_writes" requires a Boolean value'. The problem was that the startup command had a typo in it and contained '-c full_page_writes=off-c anotherparameter...' (Notice no space after =off).
>
> I think the pg_upgrade script worked when we were upgrading from 9.2 to 9.3.4, upgrade from 9.2 to 9.3.5 does not work anymore. As a workaround we have used pg_upgrade from 9.3.4 package and it worked.
>
> Could you guys look at it? Unfortunately due to internal policy I can't test any patches in our environment, but this should be easily reproducible on a clean debian box.

It's not reproducible on a clean Debian box.

Do you have full_page_writes mentioned in /etc/postgresql/9.2/main/pg_ctl.conf or the like?

Christoph
--
cb(at)df7cb(dot)de | http://www.df7cb.de/

We have finally found out why this happens, it's probably a slight issue in argument handling in pg_upgrade. When we run the upgrade command in this form:

/usr/lib/postgresql/9.3/bin/pg_upgrade -b /usr/lib/postgresql/9.2/bin -B /usr/lib/postgresql/9.3/bin -d /var/lib/postgresql/9.2/main/ -D /var/lib/postgresql/9.3/main/ -O "-c config_file=/etc/postgresql/9.3/main/postgresql.conf" -o "-c config_file=/etc/postgresql/9.2/main/postgresql.conf"

Cluster startup fails on

could not connect to new postmaster started with the command:
"/usr/lib/postgresql/9.3/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "/var/lib/postgresql/9.3/main/" -o "-p 50432 -b -c autovacuum_multixact_freeze_max_age=2000000000 -c synchronous_commit=off -c fsync=off -c full_page_writes=off-c config_file=/etc/postgresql/9.3/main/postgresql.conf -c listen_addresses='' -c unix_socket_permissions=0700 -c unix_socket_directories='/var/lib/postgresql'" start
Failure, exiting

When we issue the upgrade command with additional spaces before config_file options (both -O and -o), all works as expected and upgrade works flawlessly.

/usr/lib/postgresql/9.3/bin/pg_upgrade -b /usr/lib/postgresql/9.2/bin -B /usr/lib/postgresql/9.3/bin -d /var/lib/postgresql/9.2/main/ -D /var/lib/postgresql/9.3/main/ -O " -c config_file=/etc/postgresql/9.3/main/postgresql.conf" -o " -c config_file=/etc/postgresql/9.2/main/postgresql.conf"

I am pretty sure that the command without space worked before and now it doesn't, we haven't changed our upgrade guide for months.

Jirka

-----Original Message-----
From: Chadima Jiří
Sent: Monday, October 06, 2014 10:39 AM
To: 'Christoph Berg'
Cc: pgsql-pkg-debian(at)postgresql(dot)org
Subject: RE: [pgsql-pkg-debian] pg_upgrade in 9.3.5 can not start cluster

Actually, we don't... I have managed another clean environment, so we will try the same process with the same configuration again and I'll let you know.

Jirka

-----Original Message-----
From: Christoph Berg [mailto:cb(at)df7cb(dot)de]
Sent: Saturday, October 04, 2014 8:47 PM
To: Chadima Jiří
Cc: pgsql-pkg-debian(at)postgresql(dot)org
Subject: Re: [pgsql-pkg-debian] pg_upgrade in 9.3.5 can not start cluster

Re: Chadima Jiří 2014-10-01 <D1FA90F570B9E3449D7EF1A4D43D1736B7203591(at)CZPAV00092(dot)pa(dot)jtfg(dot)com>
> Hi,
>
> we've encountered possible bug in debian packages, both squeeze and wheezy seem to be affected.
>
> We're using following repositories:
> http://apt.postgresql.org/pub/repos/apt/ squeeze-pgdg/main amd64
> Packages http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg/main
> amd64 Packages
>
> This bug makes it currently impossible to run pg_upgrade script on 9.3 and therefore successfully upgrade from 9.2 to 9.3. It seems to be a typo somewhere in the source code. Unfortunately, I don't have the logs anymore, but the problem was that pg_upgrade refused to start cluster with an error that went like 'parameter "full_page_writes" requires a Boolean value'. The problem was that the startup command had a typo in it and contained '-c full_page_writes=off-c anotherparameter...' (Notice no space after =off).
>
> I think the pg_upgrade script worked when we were upgrading from 9.2 to 9.3.4, upgrade from 9.2 to 9.3.5 does not work anymore. As a workaround we have used pg_upgrade from 9.3.4 package and it worked.
>
> Could you guys look at it? Unfortunately due to internal policy I can't test any patches in our environment, but this should be easily reproducible on a clean debian box.

It's not reproducible on a clean Debian box.

Do you have full_page_writes mentioned in /etc/postgresql/9.2/main/pg_ctl.conf or the like?

Christoph
--
cb(at)df7cb(dot)de | http://www.df7cb.de/