Re: Re: [GENERAL] 7.0 vs. 7.1 (was: latest version?)

Hi,
What is the latest version of PostgreSQL?
Is there something like 7.1?

Pawel

Pawel Wegrzyn wrote:
>
> Hi,
> What is the latest version of PostgreSQL?
> Is there something like 7.1?

The most recent version 7.0.2. 7.1 is about to come - I am looking
forward to it as well.

Regards,
Mit freundlichem Gruß,
Holger Klawitter
--
Holger Klawitter +49 (0)251 484 0637
holger(at)klawitter(dot)de http://www.klawitter.de/

On Wed, 25 Oct 2000, Holger Klawitter wrote:

> Pawel Wegrzyn wrote:
> >
> > Hi,
> > What is the latest version of PostgreSQL?
> > Is there something like 7.1?
>
> The most recent version 7.0.2. 7.1 is about to come - I am looking
> forward to it as well.

7.0.3 is about to come out, 7.1 is about 2 months away yet :)

The Hermit Hacker <scrappy(at)hub(dot)org> writes:

> On Wed, 25 Oct 2000, Holger Klawitter wrote:
>
> > Pawel Wegrzyn wrote:
> > >
> > > Hi,
> > > What is the latest version of PostgreSQL?
> > > Is there something like 7.1?
> >
> > The most recent version 7.0.2. 7.1 is about to come - I am looking
> > forward to it as well.
>
> 7.0.3 is about to come out, 7.1 is about 2 months away yet :)

How compatible with 7.0 and 7.1 be from an application standpoint?
Will applications linked with libraries from 7.0 be able to talk to
the 7.1 database? Any changes in library major versions? The other
way?

The reason I'm asking is that Red Hat wants to maintain binary
compatibility in a for all x in y.x (that's what distribution
numbering means to us, other Linux distributions have other (and
sometimes rather weird) schemes), but I'm also interested in upgrading
the database component.

--
Trond Eivind Glomsrød
Red Hat, Inc.

[ Charset ISO-8859-1 unsupported, converting... ]
> The Hermit Hacker <scrappy(at)hub(dot)org> writes:
>
> > On Wed, 25 Oct 2000, Holger Klawitter wrote:
> >
> > > Pawel Wegrzyn wrote:
> > > >
> > > > Hi,
> > > > What is the latest version of PostgreSQL?
> > > > Is there something like 7.1?
> > >
> > > The most recent version 7.0.2. 7.1 is about to come - I am looking
> > > forward to it as well.
> >
> > 7.0.3 is about to come out, 7.1 is about 2 months away yet :)
>
> How compatible with 7.0 and 7.1 be from an application standpoint?
> Will applications linked with libraries from 7.0 be able to talk to
> the 7.1 database? Any changes in library major versions? The other
> way?

Historically, all applications have been able to talk to newer servers,
so a 6.4 client can talk to a 7.0 postmaster, and I believe 7.0 clients
can talk to 7.1 postmasters.

We usually do not go the other way, where 6.5 clients can not talk to
6.4 postmasters. I believe 7.0->7.1 will be able to talk in any
7.0.X/7.1 client and server combination.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian wrote:
> Trond Eivind Glomsrød wrote:
> > How compatible with 7.0 and 7.1 be from an application standpoint?
> > Will applications linked with libraries from 7.0 be able to talk to
> > the 7.1 database? Any changes in library major versions? The other
> > way?

> Historically, all applications have been able to talk to newer servers,
> so a 6.4 client can talk to a 7.0 postmaster, and I believe 7.0 clients
> can talk to 7.1 postmasters.

> We usually do not go the other way, where 6.5 clients can not talk to
> 6.4 postmasters. I believe 7.0->7.1 will be able to talk in any
> 7.0.X/7.1 client and server combination.

He's meaning the libpq version for dynamic link loading. Is the
libpq.so lib changing versions (like the change from 6.5.x to 7.0.x
changed from libpq.so.2.0 to libpq.so.2.1, which broke binary RPM
compatibility for other RPM's linked against libpq.so.2.0, which failed
when libpq.so.2.1 came on the scene). I think the answer is no, but I
haven't checked the details yet.

Not just libpq, though -- libpgtcl.so has also been problematic.

Of course, the file format on disk changes (again!), which is a whole
'nother issue for RPM's......
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> I believe 7.0->7.1 will be able to talk in any
> 7.0.X/7.1 client and server combination.

This should work as far as simple connectivity goes, but 7.0
applications that do queries against system catalogs might find
that their queries need to be updated. For example, psql's backslash
commands didn't recognize views for awhile due to pg_class.relkind
changes.

regards, tom lane

[ Charset ISO-8859-1 unsupported, converting... ]
> Bruce Momjian wrote:
> > Trond Eivind Glomsr?d wrote:
> > > How compatible with 7.0 and 7.1 be from an application standpoint?
> > > Will applications linked with libraries from 7.0 be able to talk to
> > > the 7.1 database? Any changes in library major versions? The other
> > > way?
>
> > Historically, all applications have been able to talk to newer servers,
> > so a 6.4 client can talk to a 7.0 postmaster, and I believe 7.0 clients
> > can talk to 7.1 postmasters.
>
> > We usually do not go the other way, where 6.5 clients can not talk to
> > 6.4 postmasters. I believe 7.0->7.1 will be able to talk in any
> > 7.0.X/7.1 client and server combination.
>
> He's meaning the libpq version for dynamic link loading. Is the
> libpq.so lib changing versions (like the change from 6.5.x to 7.0.x
> changed from libpq.so.2.0 to libpq.so.2.1, which broke binary RPM
> compatibility for other RPM's linked against libpq.so.2.0, which failed
> when libpq.so.2.1 came on the scene). I think the answer is no, but I
> haven't checked the details yet.

I usually up the .so version numbers before entering beta. That way,
they get marked as newer than older versions.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

[Taken off GENERAL, added HACKERS to cc:]

Bruce Momjian wrote:
> > He's meaning the libpq version for dynamic link loading. Is the
> > libpq.so lib changing versions (like the change from 6.5.x to 7.0.x
> > changed from libpq.so.2.0 to libpq.so.2.1, which broke binary RPM
> > compatibility for other RPM's linked against libpq.so.2.0, which failed
> > when libpq.so.2.1 came on the scene). I think the answer is no, but I
> > haven't checked the details yet.

> I usually up the .so version numbers before entering beta. That way,
> they get marked as newer than older versions.

May I ask: is it necessary? Have there been version-bumping changes to
libpq since 7.0.x? (With the rate that necessary improvement is
happening to PostgreSQL, probably).

Let me explain:
RPM's contain a plethora of dependency information, some of which is
added manually, but most of which is generated automatically. These
dependencies are based on which 'soname' is needed to satisfy dynamic
linking requirements, interpreter requirements, etc. With version
numbers as part of the name, a change in version numbers changes the
dependency.
Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be
fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0
might fail if 2.1 were to be loaded under it (hypothetically)).

Now, that doesn't directly effect the PostgreSQL RPM's. What it does
effect is the guy who wants to install PHP from with PostgreSQL support
enabled and cannot because of a failed dependency. Who gets blamed?
PostgreSQL.

Trond may correct me on this, but I don't know of a workaround for
this. And any workaround has to be applied to packages that depend upon
PostgreSQL, not to the PostgreSQL RPM's (which I would gladly modify) --
although I am going to try something -- I know that a symlink to the old
soname works, even though it is a kludge and, IMO, stinks like a
polecat.

But, enough rant. That _is_ I believe what Trond was asking about. We
have been bitten before with people installing the PHP from RedHat 6.2
after installing the PostgreSQL 7.0.x RPMset -- and dependency failures
wreaked havoc.

So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then?

Actually, Bruce, it would do me and Trond a great favor if a list of
what so's are getting bumped and to what version were to be posted. At
least we can plan for a transition at that point.

I just hate to pull a threepeat on RedHat customers. (RH 5.0 shipped PG
6.2.1. RH 5.1 shipped PG 6.3.2. BONG!) (RH 6.0 shipped 6.4.2 (bong!) RH
6.1 shipped 6.5.2 (double BONG!)). RH 7 shipped 7.0.x (small bong) --
RH 7.1 ships 7.1.x (ouch bong).

Whew. Trond, you ready for this?

[Note: I have been ill, so this message may be more incoherent than my
normal scattered self]
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

[ Blind CC to general added for comment below.]

> [Taken off GENERAL, added HACKERS to cc:]
>
> Bruce Momjian wrote:
> > > He's meaning the libpq version for dynamic link loading. Is the
> > > libpq.so lib changing versions (like the change from 6.5.x to 7.0.x
> > > changed from libpq.so.2.0 to libpq.so.2.1, which broke binary RPM
> > > compatibility for other RPM's linked against libpq.so.2.0, which failed
> > > when libpq.so.2.1 came on the scene). I think the answer is no, but I
> > > haven't checked the details yet.
>
> > I usually up the .so version numbers before entering beta. That way,
> > they get marked as newer than older versions.
>
> May I ask: is it necessary? Have there been version-bumping changes to
> libpq since 7.0.x? (With the rate that necessary improvement is
> happening to PostgreSQL, probably).

No, only major releases have bumps.

>
> But, enough rant. That _is_ I believe what Trond was asking about. We
> have been bitten before with people installing the PHP from RedHat 6.2
> after installing the PostgreSQL 7.0.x RPMset -- and dependency failures
> wreaked havoc.
>
> So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then?
>
> Actually, Bruce, it would do me and Trond a great favor if a list of
> what so's are getting bumped and to what version were to be posted. At
> least we can plan for a transition at that point.

See pgsql/src/tools/RELEASE_CHANGES. I edit interfaces/*/Makefile and
increase the minor number for every interface by one.

Let me add one thing on this RPM issue. There has been a lot of talk
recently about RPM's, and what they should do, and what they don't do,
and who should be blamed. Unfortunately, much of the discussion has
been very unproductive and more like 'venting'.

I really don't appreciate people 'venting' on these lists, especially
since we have _nothing_ to do with RPM's. All we do is make the
PostgreSQL software.

If people want to discuss RPM's on the ports list, or want to create a
new list just about RPM's, that's OK, but venting is bad, and venting on
a list that has nothing to do with RPM's is even worse.

What would be good is for someone to constructively make a posting about
the known problems, and come up with acceptible solutions. Asking us to
fix it really isn't going to help because we don't deal with RPM's here,
and we don't have enough free time to make significant changes to meet
the needs of RPM's.

Also, remember we support many Unix platforms, and Linux is only one of
them.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be
> fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0
> might fail if 2.1 were to be loaded under it (hypothetically)).

If so, I claim RPM is broken.

The whole point of major/minor version numbering for .so's is that
a minor version bump is supposed to be binary-upward-compatible.
If the RPM stuff has arbitrarily decided that it won't honor that
definition, why do we bother with multiple numbers at all?

> So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then?

To answer your question, there are no pending changes in libpq that
would mandate a major version bump (ie, nothing binary-incompatible,
AFAIK). We could ship it with the exact same version number, but then
how are people to tell whether they have a 7.0 or 7.1 libpq?

regards, tom lane

> Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> > Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be
> > fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0
> > might fail if 2.1 were to be loaded under it (hypothetically)).
>
> If so, I claim RPM is broken.
>
> The whole point of major/minor version numbering for .so's is that
> a minor version bump is supposed to be binary-upward-compatible.
> If the RPM stuff has arbitrarily decided that it won't honor that
> definition, why do we bother with multiple numbers at all?
>
> > So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then?
>
> To answer your question, there are no pending changes in libpq that
> would mandate a major version bump (ie, nothing binary-incompatible,
> AFAIK). We could ship it with the exact same version number, but then
> how are people to tell whether they have a 7.0 or 7.1 libpq?

Yes, we need to have new numbers so binaries from different releases use
the proper .so files.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian wrote:
> Lamar Owen wrote:
> > May I ask: is it necessary? Have there been version-bumping changes to
> > libpq since 7.0.x? (With the rate that necessary improvement is
> > happening to PostgreSQL, probably).

> No, only major releases have bumps.

> See pgsql/src/tools/RELEASE_CHANGES. I edit interfaces/*/Makefile and
> increase the minor number for every interface by one.

Thanks for the pointer.

> Let me add one thing on this RPM issue. There has been a lot of talk
> recently about RPM's, and what they should do, and what they don't do,
> and who should be blamed. Unfortunately, much of the discussion has
> been very unproductive and more like 'venting'.

> I really don't appreciate people 'venting' on these lists, especially
> since we have _nothing_ to do with RPM's. All we do is make the
> PostgreSQL software.

> What would be good is for someone to constructively make a posting about
> the known problems, and come up with acceptible solutions. Asking us to
> fix it really isn't going to help because we don't deal with RPM's here,
> and we don't have enough free time to make significant changes to meet
> the needs of RPM's.

Which is why I stepped up to the plate last year to help with RPM's.

I apologize if you took my post (which I edited greatly) as 'venting' --
it was not my intention to 'vent', much less offend. I just want to
know what to expect from the 7.1 release. I feel that that is germane
to the Hackers list, as the knowledge necessary to answer the question
is to be found on the list. (and you answered the question above).

Like it or not, in the eyes of many people having solid RPM's is a core
issue. If there are gotchas, I want to document them so people don't
get blindsided. Or work around them. Or ask why the change is
necessary in the first place.

I appreciate the fact that we are not here to make it easy for
distributors to package our software. I also appreciate the fact that
if you don't at least make an effort to work with major distributors
(and RedHat, TurboLinux, Caldera, and SuSE together comprise a major
userbase) that you run the risk of not being distributed in favor of an
inferior product.

I also appreciate and applaud the cross-platform mentality of the
PostgreSQL developers. Linux is far from the only OS to be supported by
PostgreSQL, true. But Linux is also the most popular OS for PostgreSQL
deployment.

However, there are known problems that can bite people who are not using
RPM's and are not running Linux. Some of those problems are such that
it will take someone with more knowledge than I currently possess to
solve. One is the issue of upgrading/migrating tools. This is not an
RPM-specific issue. To me, that is the only big issue that I have
spoken about in a way that could even remotely be construed as
'venting'. And it is not a Linux-specific issue. It is a core issue.

I'll shut up now, as I have cross-distribution RPM problems to solve.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

> > What would be good is for someone to constructively make a posting about
> > the known problems, and come up with acceptable solutions. Asking us to
> > fix it really isn't going to help because we don't deal with RPM's here,
> > and we don't have enough free time to make significant changes to meet
> > the needs of RPM's.
>
> Which is why I stepped up to the plate last year to help with RPM's.
>
> I apologize if you took my post (which I edited greatly) as 'venting' --
> it was not my intention to 'vent', much less offend. I just want to
> know what to expect from the 7.1 release. I feel that that is germane
> to the Hackers list, as the knowledge necessary to answer the question
> is to be found on the list. (and you answered the question above).

No, I was not pointing to you when I mentioned venting. There have been
other RPM threads lately. I just want information on how to make things
better for RPM's, not vents.

> Like it or not, in the eyes of many people having solid RPM's is a core
> issue. If there are gotchas, I want to document them so people don't
> get blindsided. Or work around them. Or ask why the change is
> necessary in the first place.

Sure.

> I appreciate the fact that we are not here to make it easy for
> distributors to package our software. I also appreciate the fact that
> if you don't at least make an effort to work with major distributors
> (and RedHat, TurboLinux, Caldera, and SuSE together comprise a major
> userbase) that you run the risk of not being distributed in favor of an
> inferior product.

Let them. It is their decision. Frankly, I have seen this attitude
before, and I don't like it. Just the mention that "Gee, if you don't
cooperate, we may yank you," is really a veiled threat. Now, I know you
aren't saying that, but the "if you don't play nice, we will drop you"
argument sounds a lot more like MS that a Linux vendor should be acting,
especially since they are not telling us what they want or assisting in
the work.

The "We are big. Just fix it and let us know when it is ready" attitude
does not work here, and that is what I am hearing mostly from the RPM
people.

> I also appreciate and applaud the cross-platform mentality of the
> PostgreSQL developers. Linux is far from the only OS to be supported by
> PostgreSQL, true. But Linux is also the most popular OS for PostgreSQL
> deployment.

True, it is the most popular, but that doesn't make the others less
important.

This whole statement comes across as, "You run on Linux, and look, you
took the time to run on other OS's too. How quaint."

In the history of this project, Linux was an after-thought. None of our
platforms are inferior or superior, except to the extent that the
platform does not support Unix standard functions (like NT/Cygwin).

> However, there are known problems that can bite people who are not using
> RPM's and are not running Linux. Some of those problems are such that
> it will take someone with more knowledge than I currently possess to
> solve. One is the issue of upgrading/migrating tools. This is not an
> RPM-specific issue. To me, that is the only big issue that I have
> spoken about in a way that could even remotely be construed as
> 'venting'. And it is not a Linux-specific issue. It is a core issue.

Again, your comments where quite helpful. We need more of them. We
need to hear more about the problems people are having with RPM's, and
how to make them better.

There must be a list of known problems. Let's hear them, so we can try
to solve them as a group. However, in general, we do not make dramatic
change to work around OS bugs, and do not plan to make major changes to
work around the limitations of RPM's. My bet is that some middle layer
can be created that will fix that for us.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:

> Let them. It is their decision. Frankly, I have seen this attitude
> before, and I don't like it. Just the mention that "Gee, if you don't
> cooperate, we may yank you," is really a veiled threat. Now, I know you
> aren't saying that, but the "if you don't play nice, we will drop you"
> argument sounds a lot more like MS that a Linux vendor should be acting,
> especially since they are not telling us what they want or assisting in
> the work.

FWIW, I've never threatened to do so. If I wanted to, I would just do
it[1] - threats are bad and never cause anything but bad feelings.

That being said, my favorite wishes (in addition to as much SQL
compliance and performance as possible, of course) are:

* migration on upgrade
* old libraries being able to speak to newer databases, so old
binaries can continue working after database upgrades
* good sonames on libraries - if a library hasn't changed, bumping the
number to show it's part of a new version isn't necesarry. If it is
backwards compatible, just bump the minor version, if it isn't, bump
the major version. Or even better, use versioned symbols (I don't
know how many other OSes than Linux and Solaris supports this,
though).

As for assisting, at least Red Hat contributes to a lot of projects,
some of which are important to postgres on one or more platforms: gdb,
gcc, glibc and the linux kernel. There just isn't enough resources to
do everything, but I try to help out with the RPMs.

When we make patches for packages, we try to cooperate with the
author(s) to get them in - happily, we haven't had much of a need for
that with postgresql.

> The "We are big. Just fix it and let us know when it is ready" attitude
> does not work here, and that is what I am hearing mostly from the RPM
> people.

I haven't heard anyone say that.

> There must be a list of known problems. Let's hear them, so we can try
> to solve them as a group. However, in general, we do not make dramatic
> change to work around OS bugs, and do not plan to make major changes to
> work around the limitations of RPM's.

I don't think there are any apart from the upgrade issues - if library
versioning follows the standard, that certainly won't be a problem.

[1] which I'm not even close to doing - I've spent a bit of time lately
hunting down aliasing bugs in MySQL which causes wrong SQL query
results if compiled with "-O2". Ouch.
--
Trond Eivind Glomsrød
Red Hat, Inc.

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> > Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be
> > fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0
> > might fail if 2.1 were to be loaded under it (hypothetically)).

You link against libpq.so.2 , not libpq.so.2.1. This isn't a problem.

> If the RPM stuff has arbitrarily decided that it won't honor that
> definition, why do we bother with multiple numbers at all?

There is no such problem.

> > So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then?
>
> To answer your question, there are no pending changes in libpq that
> would mandate a major version bump (ie, nothing binary-incompatible,
> AFAIK). We could ship it with the exact same version number, but then
> how are people to tell whether they have a 7.0 or 7.1 libpq?

If there isn't any changes, why bump it?
--
Trond Eivind Glomsrød
Red Hat, Inc.

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:

> > How compatible with 7.0 and 7.1 be from an application standpoint?
> > Will applications linked with libraries from 7.0 be able to talk to
> > the 7.1 database? Any changes in library major versions? The other
> > way?
>
> Historically, all applications have been able to talk to newer servers,
> so a 6.4 client can talk to a 7.0 postmaster, and I believe 7.0 clients
> can talk to 7.1 postmasters.

Great - that was what I wanted to know.

> We usually do not go the other way, where 6.5 clients can not talk to
> 6.4 postmasters. I believe 7.0->7.1 will be able to talk in any
> 7.0.X/7.1 client and server combination.

Thanks!

--
Trond Eivind Glomsrød
Red Hat, Inc.

Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:

> Bruce Momjian wrote:
> > Trond Eivind Glomsrød wrote:
> > > How compatible with 7.0 and 7.1 be from an application standpoint?
> > > Will applications linked with libraries from 7.0 be able to talk to
> > > the 7.1 database? Any changes in library major versions? The other
> > > way?
>
> > Historically, all applications have been able to talk to newer servers,
> > so a 6.4 client can talk to a 7.0 postmaster, and I believe 7.0 clients
> > can talk to 7.1 postmasters.
>
> > We usually do not go the other way, where 6.5 clients can not talk to
> > 6.4 postmasters. I believe 7.0->7.1 will be able to talk in any
> > 7.0.X/7.1 client and server combination.
>
> He's meaning the libpq version for dynamic link loading.

Not only - I'm interested in both issues.

> Is the libpq.so lib changing versions (like the change from 6.5.x to
> 7.0.x changed from libpq.so.2.0 to libpq.so.2.1, which broke binary
> RPM compatibility for other RPM's linked against libpq.so.2.0, which
> failed when libpq.so.2.1 came on the scene

Huh? Shouldn't happen.

> Not just libpq, though -- libpgtcl.so has also been problematic.

I don't think we ship that as a dynamic library.

--
Trond Eivind Glomsrød
Red Hat, Inc.

Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:

> Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be
> fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0
> might fail if 2.1 were to be loaded under it (hypothetically)).
>
> Now, that doesn't directly effect the PostgreSQL RPM's. What it does
> effect is the guy who wants to install PHP from with PostgreSQL support
> enabled and cannot because of a failed dependency. Who gets blamed?
> PostgreSQL.
>
> Trond may correct me on this, but I don't know of a workaround for
> this.

There usually are no such problems, and I'm not aware of any specific
to postgresql either.

--
Trond Eivind Glomsrød
Red Hat, Inc.

[ Charset ISO-8859-1 unsupported, converting... ]
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
>
> > Let them. It is their decision. Frankly, I have seen this attitude
> > before, and I don't like it. Just the mention that "Gee, if you don't
> > cooperate, we may yank you," is really a veiled threat. Now, I know you
> > aren't saying that, but the "if you don't play nice, we will drop you"
> > argument sounds a lot more like MS that a Linux vendor should be acting,
> > especially since they are not telling us what they want or assisting in
> > the work.
>
> FWIW, I've never threatened to do so. If I wanted to, I would just do
> it[1] - threats are bad and never cause anything but bad feelings.

Sounds good.

>
> That being said, my favorite wishes (in addition to as much SQL
> compliance and performance as possible, of course) are:

OK, here is a nice list I can address constructively:

>
> * migration on upgrade

Yes, we have problems. 7.1 will have a more robust pg_dump, and I hope
that fixes most of those problems. As you see issues here, we need to
hear about it.

> * old libraries being able to speak to newer databases, so old
> binaries can continue working after database upgrades

We have always been able to do that. Old clients can talk to newer
databases, though new can't necessary talk to older. To the extent that
the client assumes a particular database structure, we have problems
there, especially psql. I most cases, those match the server, so I
think we are OK here, and most 3-rd party stuff doesn't touch the system
tables in areas that are changed frequently.

> * good sonames on libraries - if a library hasn't changed, bumping the
> number to show it's part of a new version isn't necesarry. If it is
> backwards compatible, just bump the minor version, if it isn't, bump
> the major version. Or even better, use versioned symbols (I don't
> know how many other OSes than Linux and Solaris supports this,
> though).

We only bump minor .so numbers, except for 6.5 I think where we had a
major overhaul of libpq and the major was bumped. I don't see another
major bump on the horizon. I don't think we have ever shipped a server
that could not talk to clients at least one major revison backwards.

The big question is how RPM's handle that. I have no idea.

> As for assisting, at least Red Hat contributes to a lot of projects,
> some of which are important to postgres on one or more platforms: gdb,
> gcc, glibc and the linux kernel. There just isn't enough resources to
> do everything, but I try to help out with the RPMs.

We only need help to the extent RPM people are asking for major feature
additions that affect only RPM/Linux, and frankly, the RPM/Linux users
should be supplying patches to us for that anyway. No need for the
company to get involved.

> When we make patches for packages, we try to cooperate with the
> author(s) to get them in - happily, we haven't had much of a need for
> that with postgresql.

Yes, I have never seen one.

> > The "We are big. Just fix it and let us know when it is ready" attitude
> > does not work here, and that is what I am hearing mostly from the RPM
> > people.
>
> I haven't heard anyone say that.

Some of the RPM users have made some demands that sound a little like
that. :-)

> > There must be a list of known problems. Let's hear them, so we can try
> > to solve them as a group. However, in general, we do not make dramatic
> > change to work around OS bugs, and do not plan to make major changes to
> > work around the limitations of RPM's.
>
> I don't think there are any apart from the upgrade issues - if library
> versioning follows the standard, that certainly won't be a problem.

I would love to get a detailed list of upgrade problems so we can be
sure 7.1 has them fixed. Certainly 7.1 is already a big improvement for
upgrades.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

> > To answer your question, there are no pending changes in libpq that
> > would mandate a major version bump (ie, nothing binary-incompatible,
> > AFAIK). We could ship it with the exact same version number, but then
> > how are people to tell whether they have a 7.0 or 7.1 libpq?
>
> If there isn't any changes, why bump it?

This is huge software. There are changes to every library in every
major release, major for us meaning, i.e., 7.0->7.1. That is why I bump
the numbers.

The interesting issue is that the version number changes for .so do
_not_ mean they only talk with servers of the same release. They will
talk to future servers of higher release numbers. This is done because
there is a backend protocol number that is passed from client to server
which determines how the server should behave with that client.

We can't always have new clients talking to older servers because the
old servers may not know the newer protocol. We could get fancy and
trade version numbers and try to get it working, but it has not been a
priority, and few have asked for it. Having old clients talking to new
databases has been enough for most users.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Tom Lane wrote:
>
> Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> > Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be
> > fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0
> > might fail if 2.1 were to be loaded under it (hypothetically)).
>
> If so, I claim RPM is broken.
>
> The whole point of major/minor version numbering for .so's is that
> a minor version bump is supposed to be binary-upward-compatible.
> If the RPM stuff has arbitrarily decided that it won't honor that
> definition, why do we bother with multiple numbers at all?
>
> > So, PostgreSQL 7.1 is slated to be libpq.so.2.2, then?
>
> To answer your question, there are no pending changes in libpq that
> would mandate a major version bump (ie, nothing binary-incompatible,
> AFAIK). We could ship it with the exact same version number, but then
> how are people to tell whether they have a 7.0 or 7.1 libpq?

And that is a very good point. Hey, I'm caught in the middle here :-).
I want to see PostgreSQL succeed and excel (which, to me, means becoming
the RDBMS of choice) on RPM-based Linux distributions, which I am sure
is a goal of others too. And I'm sure no one here is against that.

But, there is friction between RedHat's (to use the first example of a
distributor to pop into my head) needs and the needs of the PostgreSQL
group.

My gut feel is that RedHat may be better off shipping 7.0.x if the
library version numbers are a contributory problem. The data upgrade
problem is a bigger problem. To which RedHat might just want to stay at
7.0.x until either a tool is written to painlessly migrate or until the
next major RedHat is released.

Of course, that doesn't affect what I do as far as building 7.1 RPM's
for distribution from the PostgreSQL site (or by anyone who so desires
to distribute them). I have no choice for my own self but to stay on
the curve. I need TOAST and OUTER JOINS too much.

So, what I feel may be the best compromise is for RedHat (and myself) to
continue building 7.0.x RPM's with bugfixes, etc, while I build 7.1 ad
subsequent RPMset's for those who know what they're doing and not
blindly upgrading their systems.

Trond, do you have any comments on that? Or is the likely migration to
kernel 2.4 in the next RedHat going to make a compatability compromise
here moot?
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:

> My gut feel is that RedHat may be better off shipping 7.0.x if the
> library version numbers are a contributory problem.

We could provide compat-packages with just neeeded libraries.

> The data upgrade problem is a bigger problem. To which RedHat might
> just want to stay at 7.0.x until either a tool is written to
> painlessly migrate or until the next major RedHat is released.

We upgrade everything from 3.0.3 (we no longer support upgrades from
2.0 as we couldn't find a specific way to identify such a system and
we didn't want accidentaly upgrade other distributions), so there is
pain anyway.

> Of course, that doesn't affect what I do as far as building 7.1 RPM's
> for distribution from the PostgreSQL site (or by anyone who so desires
> to distribute them). I have no choice for my own self but to stay on
> the curve. I need TOAST and OUTER JOINS too much.

Others very likely have the same need. I'll be looking into issues
with these later.

> So, what I feel may be the best compromise is for RedHat (and myself) to
> continue building 7.0.x RPM's with bugfixes, etc, while I build 7.1 ad
> subsequent RPMset's for those who know what they're doing and not
> blindly upgrading their systems.

> Trond, do you have any comments on that? Or is the likely migration to
> kernel 2.4 in the next RedHat going to make a compatability compromise
> here moot?

No, the 2.4 kernel should go right in - I've been using it extensively
on my system until recently (the most recent pretest has problems with
flock for sendmail).

Anyway, I've had a look at psql in objdump:

Dynamic Section:
NEEDED libpq.so.2.1
NEEDED libcrypt.so.1
NEEDED libnsl.so.1
NEEDED libdl.so.2
NEEDED libm.so.6
NEEDED libutil.so.1
NEEDED libreadline.so.4.1
NEEDED libtermcap.so.2
NEEDED libncurses.so.5
NEEDED libc.so.6

[...]

It links against nice, round versions of most libraries but wants
specific versions of readline ad libpq.

--
Trond Eivind Glomsrød
Red Hat, Inc.

Bruce Momjian wrote:

> > I appreciate the fact that we are not here to make it easy for
> > distributors to package our software. I also appreciate the fact that
> > if you don't at least make an effort to work with major distributors
> > (and RedHat, TurboLinux, Caldera, and SuSE together comprise a major
> > userbase) that you run the risk of not being distributed in favor of an
> > inferior product.

> Let them. It is their decision. Frankly, I have seen this attitude
> before, and I don't like it. Just the mention that "Gee, if you don't
> cooperate, we may yank you," is really a veiled threat.

I don't even see it as a veiled threat, Bruce. It simply _is_ a
threat. There are other RDBMS choices. Currently PostgreSQL is the
Officially Sanctioned RDBMS for multiple Linux distributions. As our
capabilities increase, it will make us more and more attractive as the
Choice, Top Shelf Open Source RDBMS.

However, the upgrade gotcha has left a very bitter taste in more than
one user's mouth. I'll not say more about that now, as I've said quite
enough in the past. And I'm still trying to figure out enough of the
internals of the storage manager to try to write the migration tools
myself. But, I have other fish to fry right now, the biggest being
cross-distribution RPM's.

> > Linux is far from the only OS to be supported by
> > PostgreSQL, true. But Linux is also the most popular OS for PostgreSQL
> > deployment.

> True, it is the most popular, but that doesn't make the others less
> important.

No, it doesn't.

> This whole statement comes across as, "You run on Linux, and look, you
> took the time to run on other OS's too. How quaint."

I ran Unix before there was linux. I ran Unix years before Linus was
even out of High School. Well, that is if you count Tandy Xenix V7 and
System III as Unix. Or AT&T 3B1 SysVR2. Or Apollo DomainOS SR10.2. Or
Ultrix on a VAX 11/750 (running in tandem with VMS). And I'm considering
moving my most critical public servers from Linux over to OpenBSD. A
Linux bigot I'm not.

> > However, there are known problems that can bite people who are not using
> > RPM's and are not running Linux. Some of those problems are such that
> > it will take someone with more knowledge than I currently possess to

> Again, your comments where quite helpful. We need more of them. We
> need to hear more about the problems people are having with RPM's, and
> how to make them better.

Bruce, sometimes I fear my own lack of communications skills. If I can
make my wife fighting mad at me with me having no clue as to what I said
that made her mad, I fear I can make anyone mad, without knowing what I
said to do so. So, I guess you could say I'm a little paranoid about my
communications skills. So, I'm glad you considered my comments helpful
-- I was beginning to get worried.

> There must be a list of known problems. Let's hear them, so we can try
> to solve them as a group. However, in general, we do not make dramatic
> change to work around OS bugs, and do not plan to make major changes to
> work around the limitations of RPM's. My bet is that some middle layer
> can be created that will fix that for us.

Meet Mr. Middle Layer. :-) The PostgreSQL spec file that controls the
RPM build is one of the most complex ones in the RedHat distribution,
AFAIK. There's the middle layer. It does quite a bit of finagling
already.

And the work that Peter E is doing is helping my cause significantly.

Bruce, when I recover fully from the illness I've had the last few days,
I'll try to come up with a coherent listing of what I've had to work
around in the past. My current headache won't let me think straight
right now, which makes it likely that I won't effectively communicate
the issues.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

[BCC to Hackers -- cc: to PORTS, as, as Bruce correctly pointed out,
that's where this discussion belongs.]

Trond Eivind Glomsrød wrote:
> Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> > My gut feel is that RedHat may be better off shipping 7.0.x if the
> > library version numbers are a contributory problem.

> We could provide compat-packages with just neeeded libraries.

Yes, we could do that. And those libs could possibly just be the
symlinks (or even just a Provides: header).

> We upgrade everything from 3.0.3 (we no longer support upgrades from
> 2.0 as we couldn't find a specific way to identify such a system and
> we didn't want accidentaly upgrade other distributions), so there is
> pain anyway.

I tried going from 4.1 (the earliest one I have installation CD's for)
to pre-7.0 once. I don't recommend it.

> > Of course, that doesn't affect what I do as far as building 7.1 RPM's
> > for distribution from the PostgreSQL site (or by anyone who so desires
> > to distribute them). I have no choice for my own self but to stay on
> > the curve. I need TOAST and OUTER JOINS too much.

> Others very likely have the same need. I'll be looking into issues
> with these later.

Good. Let me know what you decide, if you don't mind.

> Anyway, I've had a look at psql in objdump:

> Dynamic Section:
> NEEDED libpq.so.2.1
> NEEDED libreadline.so.4.1
> [...]

> It links against nice, round versions of most libraries but wants
> specific versions of readline ad libpq.

And unfortunately PHP and other PostgreSQL clients also link against the
specific libpq version. This has caused pain for those installing the
PHP stuff from RPM which was linked against a RedHat 6.2 box with
PostgreSQL 6.5.3 installed -- onto a RedHat 6.2 box with PostgreSQL
7.0.2 installed. There is a failed dependency on libpq.so.2.0 -- even
though libpq.so.2.1 is there.

A symlink works around the problem, if the symlink is part of the RPM so
that it gets in the rpm dep database. Of course, this only causes
problems with RedHat 6.2 and earlier, as RH 7's PHP stuff was built
against 7.0.2 to start with. But, 7.1 with libpq.so.2.2 will cause
similar dep failures for PHP packages built against 7.0.2.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Trond Eivind Glomsrød wrote:
> Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> > Unfortunately RPM deems a dependency upon libpq.so.2.0 to not be
> > fulfilled by libpq.so.2.1 (how _can_ it know? A client linked to 2.0
> > might fail if 2.1 were to be loaded under it (hypothetically)).

> There usually are no such problems, and I'm not aware of any specific
> to postgresql either.

There have been reports to the pgsql-bugs list and to the PHP list about
this very issue.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

> However, the upgrade gotcha has left a very bitter taste in more than
> one user's mouth. I'll not say more about that now, as I've said quite
> enough in the past. And I'm still trying to figure out enough of the
> internals of the storage manager to try to write the migration tools
> myself. But, I have other fish to fry right now, the biggest being
> cross-distribution RPM's.

Actually, I would prefer to see how we can improve what we have before
making a binary conversion utility that will have to be updated for
every release.

> Meet Mr. Middle Layer. :-) The PostgreSQL spec file that controls the
> RPM build is one of the most complex ones in the RedHat distribution,
> AFAIK. There's the middle layer. It does quite a bit of finagling
> already.

Yes, I suspected the RPM was the middle layer. To the extent we can
make that easier, let's hear it. Tell us what you need to do, and what
you can't do, and see if any of us can figure out how to make things
easier.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

> And unfortunately PHP and other PostgreSQL clients also link against the
> specific libpq version. This has caused pain for those installing the
> PHP stuff from RPM which was linked against a RedHat 6.2 box with
> PostgreSQL 6.5.3 installed -- onto a RedHat 6.2 box with PostgreSQL
> 7.0.2 installed. There is a failed dependency on libpq.so.2.0 -- even
> though libpq.so.2.1 is there.
>
> A symlink works around the problem, if the symlink is part of the RPM so
> that it gets in the rpm dep database. Of course, this only causes
> problems with RedHat 6.2 and earlier, as RH 7's PHP stuff was built
> against 7.0.2 to start with. But, 7.1 with libpq.so.2.2 will cause
> similar dep failures for PHP packages built against 7.0.2.

For us, it would be great if libpq.so.2.1 linked against the
libpq.so.2.1, libpq.so.2.2, but not libpq.so.2.0. I would guess other
apps need this ability too. How do they handle it?

I saw someone installing pgaccess from RPM. It wanted tcl/tk 8.0, and
they had tcl/tk 8.3 installed, and it failed. Seems this is a common
RPM problem.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian wrote:
>
> > However, the upgrade gotcha has left a very bitter taste in more than
> > one user's mouth. I'll not say more about that now, as I've said quite
> > enough in the past. And I'm still trying to figure out enough of the
> > internals of the storage manager to try to write the migration tools
> > myself. But, I have other fish to fry right now, the biggest being
> > cross-distribution RPM's.
>
> Actually, I would prefer to see how we can improve what we have before
> making a binary conversion utility that will have to be updated for
> every release.
>
> > Meet Mr. Middle Layer. :-) The PostgreSQL spec file that controls the
> > RPM build is one of the most complex ones in the RedHat distribution,
> > AFAIK. There's the middle layer. It does quite a bit of finagling
> > already.
>
> Yes, I suspected the RPM was the middle layer. To the extent we can
> make that easier, let's hear it. Tell us what you need to do, and what
> you can't do, and see if any of us can figure out how to make things
> easier.

Ok, here goes:
* Location-agnostic installation. Documentation (which I'll be happy to
contribute) on that. Peter E is already working in this area. Getting
the installation that 'make install' spits out massaged into an FHS
compliant setup is the majority of the RPM's spec file.

* Upgrades that don't require an ASCII database dump for migration. This
can either be implemented as a program to do a pg_dump of an arbitrary
version of data, or as a binary migration utility. Currently, I'm
saving old executables to run under a special environment to pull a dump
-- but it is far from optimal. What if the OS upgrade behind 99% of the
upgrades makes it where those old executables can't run due to binary
incompatibility (say I'm going from RedHat 3.0.3 to RedHat 7 -- 3.0.3,
IIRC, as a.out...( and I know 3.0.3 didn't have PostgreSQL RPMs).)?
What I could actually do to prevent that problem is build all of
PostgreSQL's 6.1.x, 6.2.x, 6.3.x, 6.4.x, and 6.5.x and include the
necessary backend executables as part of the RPM.... But I think you see
the problem there. However, that would in my mind be better than the
current situation, albeit taking up a lot of space.

* A less source-centric mindset. Let's see, how to explain? The
regression tests are a good example. You need make. You need the source
installed, configured, and built in the usual location. You need
portions of contrib. RPM's need to be installable on compiler-crippled
servers for security. While the demand for regression testing on such a
box may not be there, it certainly does give the user something to use
to get standard output for bug reports. As a point, I run PostgreSQL in
production on a compilerless machine. No compiler == more security.
And Linux has enough security problems without a compiler being
available :-(. Oh, and I have no make on that machine either.

The documentation as well as many of the examples assume too much, IMHO,
about the install location and the install methodology.

I think I may have a solution for the library versioning problem.
Rather than symlink libpq.so->libpq.so.2->libpq.so.2.x, I'll copy
libpq.so.2.1 to libpq.so.2 and symlink libpq.so to that. A little more
code for me. There is no real danger in version confusion with RPM's
versioning and upgrade methodology, as long as you consistently use the
RPMset. The PostgreSQL version number is readily found from an RPM
database query, making the so version immaterial.

The upgrade issue is the hot trigger for me at this time. It is and has
been a major drain on my time and effort, as well as Trond's and others,
to get the RPM upgrade working even remotely smoothly. And I am willing
to code -- once I know how to go about doing it in the backend.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Bruce Momjian wrote:
> > A symlink works around the problem, if the symlink is part of the RPM so
> > that it gets in the rpm dep database. Of course, this only causes
> > problems with RedHat 6.2 and earlier, as RH 7's PHP stuff was built
> > against 7.0.2 to start with. But, 7.1 with libpq.so.2.2 will cause
> > similar dep failures for PHP packages built against 7.0.2.

> For us, it would be great if libpq.so.2.1 linked against the
> libpq.so.2.1, libpq.so.2.2, but not libpq.so.2.0. I would guess other
> apps need this ability too. How do they handle it?

If I were doing manual dependencies for the other packages, I would say:
Requires: libpq.so => 2.1

No as to whether that works or not, I don't know. I know it won't work
with RPM prior to 3.0.4 or so.

> I saw someone installing pgaccess from RPM. It wanted tcl/tk 8.0, and
> they had tcl/tk 8.3 installed, and it failed. Seems this is a common
> RPM problem.

Well, actually, there are times you might not want greater than a
certain version. And you as a packager can make certain dependency
requirements manually. However, this libpq.so.2.0 vs 2.1 failure was an
automatic dependency.

And, really, RPM shouldn't allow it for automatic requires. Suppose I
have an ancient client RPM that I want to install. Assuming for one
second that nothing else has changed on the system except the PostgreSQL
version, if the client was built against PostgreSQL 6.2.1 with
libpq.so.1, and I force the install of it even though libpq.so.2 is
installed, freakish things can happen. Been there and done that -- a
client linked against Postgres95 1.0.1 did really strange things when
libpq.so.2 was link loaded under it.

Worse things happen if you have a package that requires tcl 7.4 and you
have tcl 8.3.2 installed.

Not everyone is as generous as we are with upwards compatibility.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

> Ok, here goes:

Cool, a list.

> * Location-agnostic installation. Documentation (which I'll be happy to
> contribute) on that. Peter E is already working in this area. Getting
> the installation that 'make install' spits out massaged into an FHS
> compliant setup is the majority of the RPM's spec file.

Well, we certainly don't want to make changes that make things harder or
more confusing for non-RPM installs. How are they affected here?

> * Upgrades that don't require an ASCII database dump for migration. This
> can either be implemented as a program to do a pg_dump of an arbitrary
> version of data, or as a binary migration utility. Currently, I'm
> saving old executables to run under a special environment to pull a dump
> -- but it is far from optimal. What if the OS upgrade behind 99% of the
> upgrades makes it where those old executables can't run due to binary
> incompatibility (say I'm going from RedHat 3.0.3 to RedHat 7 -- 3.0.3,
> IIRC, as a.out...( and I know 3.0.3 didn't have PostgreSQL RPMs).)?
> What I could actually do to prevent that problem is build all of
> PostgreSQL's 6.1.x, 6.2.x, 6.3.x, 6.4.x, and 6.5.x and include the
> necessary backend executables as part of the RPM.... But I think you see
> the problem there. However, that would in my mind be better than the
> current situation, albeit taking up a lot of space.

I really don't see the issue here. We can compress ASCII dump files, so
the space need should not be too bad. Can't you just check to see if
there is enough space, and error out if there is not? If the 2GIG limit
is a problem, can't the split utility drop the files in <2gig chunks
that can be pasted together in a pipe on reload?

> * A less source-centric mindset. Let's see, how to explain? The
> regression tests are a good example. You need make. You need the source
> installed, configured, and built in the usual location. You need
> portions of contrib. RPM's need to be installable on compiler-crippled
> servers for security. While the demand for regression testing on such a
> box may not be there, it certainly does give the user something to use
> to get standard output for bug reports. As a point, I run PostgreSQL in
> production on a compilerless machine. No compiler == more security.
> And Linux has enough security problems without a compiler being
> available :-(. Oh, and I have no make on that machine either.

Well, no compiler? I can't see how we would do that without making
other OS installs harder. That is really the core of the issue. We
can't be making changes that make things harder for other OS's. Those
have to be isolated in the RPM, or in some other middle layer.

>
> The documentation as well as many of the examples assume too much, IMHO,
> about the install location and the install methodology.

Well, if we are not specific, things get very confusing for those other
OS's. Being specific about locations makes things easier. Seems we may
need to patch RPM installs to fix that. Certainly a pain, but I see no
other options.

>
> I think I may have a solution for the library versioning problem.
> Rather than symlink libpq.so->libpq.so.2->libpq.so.2.x, I'll copy
> libpq.so.2.1 to libpq.so.2 and symlink libpq.so to that. A little more
> code for me. There is no real danger in version confusion with RPM's
> versioning and upgrade methodology, as long as you consistently use the
> RPMset. The PostgreSQL version number is readily found from an RPM
> database query, making the so version immaterial.

Oh, that is good.

>
> The upgrade issue is the hot trigger for me at this time. It is and has
> been a major drain on my time and effort, as well as Trond's and others,
> to get the RPM upgrade working even remotely smoothly. And I am willing
> to code -- once I know how to go about doing it in the backend.

Please give us more information about how the current upgrade is a
problem. We don't hear that much from other OS's. How are RPM's
specific, and maybe we can get a plan for a solution.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

> And, really, RPM shouldn't allow it for automatic requires. Suppose I
> have an ancient client RPM that I want to install. Assuming for one
> second that nothing else has changed on the system except the PostgreSQL
> version, if the client was built against PostgreSQL 6.2.1 with
> libpq.so.1, and I force the install of it even though libpq.so.2 is
> installed, freakish things can happen. Been there and done that -- a
> client linked against Postgres95 1.0.1 did really strange things when
> libpq.so.2 was link loaded under it.
>
> Worse things happen if you have a package that requires tcl 7.4 and you
> have tcl 8.3.2 installed.
>
> Not everyone is as generous as we are with upwards compatibility.

And we aren't super-generous either. I am not sure how far back we go
in allowing old libpq apps to talk to new servers. We go one version at
least. So you could allow for the current version number, plus one
minor number greater, and know that would work.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Lamar Owen writes:

> Getting the installation that 'make install' spits out massaged into
> an FHS compliant setup is the majority of the RPM's spec file.

./configure --prefix=/usr --sysconfdir=/etc

Off you go... (I'll refrain from commenting further on the FHS.)

> * Upgrades that don't require an ASCII database dump for migration.

Let me ask you this question: When any given RPM-based Linux distribution
will update their system from ext2 to, say, ReiserFS across the board, how
are they going to do it? Sincere question.

> * A less source-centric mindset. Let's see, how to explain? The
> regression tests are a good example. You need make. You need the source
> installed, configured, and built in the usual location.

This is not an excuse, but almost every package behaves this way. Test
suites are designed to be run after "make all" and before "make install".
When you ship a binary package then you're saying to users "I did the
building and installation (and presumably everything else that the authors
recommend along the way) for you." RPM packages usually don't work very
well on systems that are not exactly like the one they were built on, so
this seems to be a fair assumption.

Getting the regression tests to work from anywhere is not very hard, but
it's not the most interesting project for most people. :-)

> I think I may have a solution for the library versioning problem.
> Rather than symlink libpq.so->libpq.so.2->libpq.so.2.x, I'll copy
> libpq.so.2.1 to libpq.so.2 and symlink libpq.so to that.

I'd still claim that if RPM thinks it's smarter than the dynamic loader,
then it's broken. All the shared libraries on Linux have a symlink from
more general to more specific names. PostgreSQL can't be the first to hit
this problem.

--
Peter Eisentraut peter_e(at)gmx(dot)net http://yi.org/peter-e/

[Since I've rested over the weekend, I hope I don't come across this
morning as an angry old snarl, like some of my previous posts on this
subject unfortunately have been.]

Bruce Momjian wrote:
> > * Location-agnostic installation. Documentation (which I'll be happy to
> > contribute) on that. Peter E is already working in this area. Getting
> > the installation that 'make install' spits out massaged into an FHS
> > compliant setup is the majority of the RPM's spec file.

> Well, we certainly don't want to make changes that make things harder or
> more confusing for non-RPM installs. How are they affected here?

They wouldn't be. Peter E has seemingly done an excellent job in this
area. I say seemingly because I haven't built an RPM from the 7.1 branch
yet, but from what he has posted, he seems to understand the issue.
Many thanks, Peter.

> > * Upgrades that don't require an ASCII database dump for migration. This
> > can either be implemented as a program to do a pg_dump of an arbitrary
> > version of data, or as a binary migration utility. Currently, I'm

> I really don't see the issue here.

At the risk of being redundant, here goes. As I've explained before,
the RPM upgrade environment, thanks to our standing with multiple
distributions as being shipped as a part of the OS, could be run as part
of a general-purpose OS upgrade. In the environment of the general
purpose OS upgrade, the RPM's installation scripts cannot fire up a
backend, nor can it assume one is running or is not running, nor can the
RPM installation scripts fathom from the run-time environment whether
they are being run from a command line or from the OS upgrade (except on
Linux Mandrake, which allows such usage).

Thus, if a system administrator upgrades a system, or if an end user who
has a pgaccess-customized data entry system for things as mundane as an
address list or recipe book, there is no opportunity to do a dump. The
dump has to be performed _after_ the RPM upgrade.

Now, this is far from optimal, I know. I _know_ that the user should
take pains with their data. I know that there should be a backup. I
also know that a user of PostgreSQL should realize that 'this is just
the way it is done' and do things Our Way.

I also know that few new users will do it 'Our Way'. No other package
that I am aware of requires the manual intervention that PostgreSQL
does, with the possible exception of upgrading to a different file
system -- but that is something most new users won't do, and is
something that is more difficult to automate.

However, over the weekend, while resting (I did absolutely NO computer
work this weekend -- too close to burnout), I had a brainstorm.

A binary migration tool does not need to be written, if a concession to
the needs of some users who just simply want to upgrade can be made.

Suppose we can package old backends (with newer network code to connect
to new clients). Suppose further that postmaster can be made
intelligent enough to fire up old backends for old data, using
PG_VERSION as a key. Suppose a NOTICE can be fired off warning the user
that 'The Database is running in Compatibility Mode -- some features may
not be available. Please perform a dump of your data, reinitialize the
database, and restore your data to access new features of version x.y'.

I'm highly considering doing just that from a higher level. It will not
be nearly as smooth, but doable.

Of course, that increases maintenance work, and I know it does. But I'm
trying to find a middle ground here, since providing a true migration
utility (even if it just produces a dump of the old data) seems out of
reach at this time.

We are currently forcing something like a popular word processing
program once did -- it's proprietary file format changed. It was coded
so that it could not even read the old files. But both the old and the
new versions could read and write an interchange format. People who
blindly upgraded their word processor were hit with a major problem.
There was even a notice in the README -- which could be read after the
program was installed.

While the majority of us use PostgreSQL as a server behind websites and
other clients, there will be a large number of new users who want to use
it for much more mundane tasks. Like address books, or personal
information management, or maybe even tax records. Frontends to
PostgreSQL, thanks to PostgreSQL's advanced features, are likely to span
the gamut -- we already have OnShore TimeSheet for time tracking and
payroll, as one example. And I even see database-backed intranet-style
web scripts being used on a client workstation for these sorts of
things. I personally do just that with my home Linux box -- I have a
number of AOLserver dynamic pages that use PostgreSQL for many mundane
tasks (a multilevel sermon database is one).

While I don't need handholding in the upgrade process, I have provided
support to users that do -- who are astonished at the way we upgrade.
Seamless upgrading won't help me personally -- but it will help
multitudes of users -- not just RPM users. As a newbie to PostgreSQL I
was bitten, giving me compassion on those who might be bitten.

> We can compress ASCII dump files, so
> the space need should not be too bad.

Space isn't the problem. The procedure is the problem. Even if the
user fails to do it Right, we should at least attempt to help them
recover, IMHO.

> > * A less source-centric mindset. Let's see, how to explain? The
> > regression tests are a good example. You need make. You need the source
[snip]
> > it certainly does give the user something to use
> > to get standard output for bug reports. As a point, I run PostgreSQL in

> Well, no compiler? I can't see how we would do that without making
> other OS installs harder. That is really the core of the issue. We
> can't be making changes that make things harder for other OS's. Those
> have to be isolated in the RPM, or in some other middle layer.

And I've done that in the past with the older serialized regression
tests.

I don't see how executing a shell script instead of executing a make
command would make it any harder for other OS users. I am not trying to
make it harder for other OS users. I _am_ trying to make it easier for
users who are getting funny results from queries to be able to run
regression tests as a standardized way to see where the problem lies.
Maybe there is a hardware issue -- regression testing might be the only
way to have a standard way to pinpoint the problem.

And telling someone who is having a problem with prepackaged binaries
'Run the regression tests by executing the script
/usr/lib/pgsql/tests/regress/regress.sh and pipe me the results' is much
easier to do than 'Find me a test case where this blow up, and pipe me a
backtrace/dump/whatever' for the new users. Plus that regression output
is a known quantity.

Or, to put it in a soundbite, regression testing can be the user's best
bug-zapping friend.

> > The documentation as well as many of the examples assume too much, IMHO,
> > about the install location and the install methodology.

> Well, if we are not specific, things get very confusing for those other
> OS's. Being specific about locations makes things easier. Seems we may
> need to patch RPM installs to fix that. Certainly a pain, but I see no
> other options.

I can do that, I guess. I currently ship the README.rpm as part of the
package -- but I continue to hear from people who have not read it, but
have read the online docs. I have even put the unpacked source RPM up
on the ftp site so that people can read the README right online.

> Please give us more information about how the current upgrade is a
> problem. We don't hear that much from other OS's. How are RPM's
> specific, and maybe we can get a plan for a solution.

RPM's are expected to 'rpm -U' and you can simply _use_ the new version,
with little to no preparation. At least that is the theory. And it
works for most packages.

--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Peter Eisentraut wrote:
> Lamar Owen writes:
> > Getting the installation that 'make install' spits out massaged into
> > an FHS compliant setup is the majority of the RPM's spec file.

> ./configure --prefix=/usr --sysconfdir=/etc
> Off you go... (I'll refrain from commenting further on the FHS.)

I know alot of people don't like LSB/FHS, but, like it or not, I have to
work with it. And, many many thanks for putting in the work on the
configuration as you have.

> > * Upgrades that don't require an ASCII database dump for migration.

> Let me ask you this question: When any given RPM-based Linux distribution
> will update their system from ext2 to, say, ReiserFS across the board, how
> are they going to do it? Sincere question.

Like the TRS-80 model III, whose TRSDOS 1.3 could not read the TRS-80
Model I's disks, written on TRSDOS 2.3 (TRSDOS's versioning was
absolutely horrendous). TRSDOS 1.3 included a CONVERT utility that
could read files from the old filesystem.

I'm sure that the newer distributions using ReiserFS as the primary
filesystem will include legacy Ext2/3 support, at least for read-only,
for many versions to come.

And that's my big beef -- a newer version of PostgreSQL can't even
pg_dump an old database. If that single function was supported, I would
have no problem with the upgrade whatsoever.

> > * A less source-centric mindset. Let's see, how to explain? The
> > regression tests are a good example. You need make. You need the source
> > installed, configured, and built in the usual location.

> This is not an excuse, but almost every package behaves this way. Test
> suites are designed to be run after "make all" and before "make install".
> When you ship a binary package then you're saying to users "I did the
> building and installation (and presumably everything else that the authors
> recommend along the way) for you."

Yes, and I do just that. Regression testing is a regular part of my
build process here.

> RPM packages usually don't work very
> well on systems that are not exactly like the one they were built on

Boy, don't I know it.....~;-/

> Getting the regression tests to work from anywhere is not very hard, but
> it's not the most interesting project for most people. :-)

I know. I'll probably do it myself, as that is something I _can_ do.

> > I think I may have a solution for the library versioning problem.
> > Rather than symlink libpq.so->libpq.so.2->libpq.so.2.x, I'll copy
> > libpq.so.2.1 to libpq.so.2 and symlink libpq.so to that.

> I'd still claim that if RPM thinks it's smarter than the dynamic loader,
> then it's broken. All the shared libraries on Linux have a symlink from
> more general to more specific names. PostgreSQL can't be the first to hit
> this problem.

RPM is getting it's .so dependency list straight from the mouth of the
dynamic loader itself. RPM uses shell scripts, customizable for each
system on which RPM runs, to determine the automatic dependencies --
those shell scripts run the dynamic loader to get the list of requires.
So, the dynamic loader itself is providing the list.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen writes:

> In the environment of the general purpose OS upgrade, the RPM's
> installation scripts cannot fire up a backend, nor can it assume one
> is running or is not running, nor can the RPM installation scripts
> fathom from the run-time environment whether they are being run from a
> command line or from the OS upgrade (except on Linux Mandrake, which
> allows such usage).

I don't understand why this is so. It seems perfectly possible that some
%preremovebeforeupdate starts a postmaster, runs pg_dumpall, saves the
file somewhere, then the %postinstallafterupdate runs the inverse
operation. Disk space is not a valid objection, you'll never get away
without 2x storage. Security is not a problem either. Are you not
upgrading in proper dependency order or what? Everybody does dump,
remove, install, undump; so can the RPMs.

Okay, so it's not as great as a new KDE starting up and asking "may I
update your configuration files?", but understand that the storage format
is optimized for performance, not easy processing by external tools or
something like that.

--
Peter Eisentraut peter_e(at)gmx(dot)net http://yi.org/peter-e/

[Explanation on why an RPM cannot dump a database during upgrade
follows. This is a lengthy explanation. If you don't want to read it,
please hit 'Delete' now. -- Also, I have blind copied Hackers, and cc:'d
PORTS, as that is where this discussion belongs, per Bruce's wishes.]

Peter Eisentraut wrote:
> Lamar Owen writes:
> > In the environment of the general purpose OS upgrade, the RPM's
> > installation scripts cannot fire up a backend, nor can it assume one

> I don't understand why this is so. It seems perfectly possible that some
> %preremovebeforeupdate starts a postmaster, runs pg_dumpall, saves the
> file somewhere, then the %postinstallafterupdate runs the inverse
> operation. Disk space is not a valid objection, you'll never get away
> without 2x storage. Security is not a problem either. Are you not
> upgrading in proper dependency order or what? Everybody does dump,
> remove, install, undump; so can the RPMs.

The RedHat installer (anaconda) is running in a terribly picky
environment. There a very few tools in this environment -- after all,
this is an installer we're talking about here. Starting a postmaster is
likely to fail, and fail big. Further, the anaconda install environment
is a chroot -- or, at least the environment the RPM scriptlets run in is
a chroot -- a chroot that is the active filesystem that is being
upgraded. This filesystem likely contains old libraries, old
executables, and other programs that may have a hard time running under
the limited installation kernel and the limited libraries available to
the installer.

And since packages are actively discouraged from probing whether they're
running in the anaconda chroot or not, it is not possible to start a
postmaster. Mandrake allows packages to probe this -- which I
personally think is a bad idea -- packages that need to know this sort
of information are usually packages that would be better off finding a
least common denominator upgrade path that will work the best. A single
upgrade path is much easier to maintain the two upgrade paths.

Sure, during a command line upgrade, I can probe for a postmaster, and
even start one -- but I dare say the majority of PostgreSQL RPM upgrades
don't happen from the command line. Even if I _can_ probe whether I'm
in the anaconda chroot or not, I _still_ have to have an upgrade path in
case this _is_ an OS upgrade.

Think about it: suppose I had a postmaster start up, and a pg_dumpall
runs during OS upgrade. Calculating free space is not possible -- you
are in the middle of an OS upgrade, and more packages may be selected
for installation than are already installed -- or, an upgrade to an
existing package may take more space than the previous version (XFree86
3.3.6 to XFree86 4.0.1 is a good example) -- you have no way of knowing
from the RPM installation scripts in the package how much free space
there will or won't be when the upgrade is complete. And anaconda
doesn't help you out with an ESTIMATED_SPACE_AFTER_INSTALL environment
variable.

And you really can't assume 2x space -- the user may have decided that
this machine that didn't have TeX installed needs TeX installed, and
Emacs, and, while it didn't have GNOME before, it needs it now.....
Sure, the user just got himself in a pickle -- but I'm not about to be
the scapegoat for _his_ pickle.

And I can't assume that the /var partition (where the dataset resides)
is separate, or that it even has enough space -- the user might be
dumping to another filesystem, or maybe onto tape. And, in the confines
of an RPM %pre scriptlet, I have no way of finding out.

Furthermore, I can't accurately predict how much space even a compressed
ASCII dump will take . Calculating the size of the dataset in PGDATA
does not accurately predict the size of the dumpfile.

As to using split or the like to split huge dumpfiles, that is a
necessity -- but the space calculation problem defeats the whole concept
of dump-during-upgrade. I cannot determine how much space I have, and I
cannot determine how much space I need -- and, if I overflow the
filesystem during an OS upgrade that is halfway complete (PostgreSQL
usually is upgraded about two thirds of the way through or so), then I
leave the user with a royally hosed system. I don't want that on my
shoulders, do you? :-)

Therefore, the pg_dumpall _has_ to occur _after_ the new version has
overwritten the old version, and _after_ the OS upgrade is completed --
unless the user has done what they should have done to begin with --
but, the fact of the matter is that many users simply won't do it Right.

You can't assume the user is going to be reasonable by your standard --
in fact, you have to do the opposite -- your standard of reasonable, and
the user's standard of reasonable, might be totally different things.

Incidentally, I originally attempted doing the dump inside the
preinstall, and found it to be an almost impossible task. The above
reasons might be solvable, but then there's this little problem: what if
you _are_ able to predict the space needed and the space available --
and there's not enough space available?

The PostgreSQL RPM's are not a single package, and anaconda has no way
of rolling back another part of an RPMset's installation if one part
fails. So, you can't just abort because you failed to dump -- the
package that needs the dump is the server subpackage -- and the main
package has already finished installation by that time. And you can't
roll it back.

And the user has a hosed PostgreSQL installation as a result.

As to why the package is split, well, it is highly useful to many people
to have a PostgreSQL _client_ installation that accesses a central
database server -- there is no need to have a postmaster and a full
backend when all you need is psql and the libraries and documentation
that goes along with psql.

RPM's have to deal with both a very difficult environment, and users who
might not be as technically savvy as those who install from source.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen wrote:

> As to why the package is split, well, it is highly useful to many people
> to have a PostgreSQL _client_ installation that accesses a central
> database server -- there is no need to have a postmaster and a full
> backend when all you need is psql and the libraries and documentation
> that goes along with psql.

My personal experience is that the way the PostgreSQL RPMs are split is very good. It meshes nicely with other dependencies so that I don't need to install extra RPMs on our servers. I for one would not like to see that change.

--
Karl DeBisschop kdebisschop(at)alert(dot)infoplease(dot)com
Learning Network Reference http://www.infoplease.com
Netsaint Plugin Developer kdebisschop(at)users(dot)sourceforge(dot)net

Karl DeBisschop wrote:
>
> Lamar Owen wrote:
>
> > As to why the package is split, well, it is highly useful to many people
> > to have a PostgreSQL _client_ installation that accesses a central
> > database server -- there is no need to have a postmaster and a full
> > backend when all you need is psql and the libraries and documentation
> > that goes along with psql.
>
> My personal experience is that the way the PostgreSQL RPMs are split is very good. It meshes nicely with other dependencies so that I don't need to install extra RPMs on our servers. I for one would not like to see that change.

And I agree -- and have no plans to change. If anything the RPMset will
increase in number, not decrease.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

> > Well, we certainly don't want to make changes that make things harder or
> > more confusing for non-RPM installs. How are they affected here?
>
> They wouldn't be. Peter E has seemingly done an excellent job in this
> area. I say seemingly because I haven't built an RPM from the 7.1 branch
> yet, but from what he has posted, he seems to understand the issue.
> Many thanks, Peter.

OK, glad that is done.

> > > * Upgrades that don't require an ASCII database dump for migration. This
> > > can either be implemented as a program to do a pg_dump of an arbitrary
> > > version of data, or as a binary migration utility. Currently, I'm
>
> > I really don't see the issue here.
>
> At the risk of being redundant, here goes. As I've explained before,
> the RPM upgrade environment, thanks to our standing with multiple
> distributions as being shipped as a part of the OS, could be run as part
> of a general-purpose OS upgrade. In the environment of the general
> purpose OS upgrade, the RPM's installation scripts cannot fire up a
> backend, nor can it assume one is running or is not running, nor can the
> RPM installation scripts fathom from the run-time environment whether
> they are being run from a command line or from the OS upgrade (except on
> Linux Mandrake, which allows such usage).

OK, maybe doing it in an RPM is the wrong way to go. If an old version
exists, maybe the RPM is only supposed to install the software in a
saved location, and the users must execute a command after the RPM
install that starts the old postmaster, does the dump, puts the new
PostgreSQL server in place, and reloads the database.

> > Well, no compiler? I can't see how we would do that without making
> > other OS installs harder. That is really the core of the issue. We
> > can't be making changes that make things harder for other OS's. Those
> > have to be isolated in the RPM, or in some other middle layer.
>
> And I've done that in the past with the older serialized regression
> tests.
>
> I don't see how executing a shell script instead of executing a make
> command would make it any harder for other OS users. I am not trying to
> make it harder for other OS users. I _am_ trying to make it easier for
> users who are getting funny results from queries to be able to run
> regression tests as a standardized way to see where the problem lies.
> Maybe there is a hardware issue -- regression testing might be the only
> way to have a standard way to pinpoint the problem.

You are basically saying that because you can ship without a compiler
sometimes, we are supposed to change the way our regression tests work.
Let's suppose SCO says they don't ship with a compiler, and wants us to
change our code to accomodate it. Should we? You can be certain we
would not, and in the RPM case, you get the same answer.

If the patch is trivial, we will work around OS limitations, but we do
not redesign code to work around OS limitations. We expect the OS to
get the proper features. That is what we do with NT. Cygwin provides
the needed features.

> > Please give us more information about how the current upgrade is a
> > problem. We don't hear that much from other OS's. How are RPM's
> > specific, and maybe we can get a plan for a solution.
>
> RPM's are expected to 'rpm -U' and you can simply _use_ the new version,
> with little to no preparation. At least that is the theory. And it
> works for most packages.

This is the "Hey, other people can do it, why can't you" issue. We are
looking for suggestions from Linux users in how this can be done.
Perhaps running a separate command after the RPM has been installed is
the only way to go.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian wrote:
> > At the risk of being redundant, here goes. As I've explained before,
> > the RPM upgrade environment, thanks to our standing with multiple
> > distributions as being shipped as a part of the OS, could be run as part

> OK, maybe doing it in an RPM is the wrong way to go. If an old version
> exists, maybe the RPM is only supposed to install the software in a
> saved location, and the users must execute a command after the RPM
> install that starts the old postmaster, does the dump, puts the new
> PostgreSQL server in place, and reloads the database.

That's more or less what's being done now. The RPM's preinstall script
(run before any files are overwritten from the new package) backs up the
required executables from the old installation. The RPM then overwrites
the necessary files, and then any old files left over are removed, along
with database removal of their records.

A script (actually, two scripts due to a bug in the first one) is
provided to dump the database using the old executables. Which works OK
as long as the new OS release is executable compatible with the old
release. Oliver Elphick originally wrote the script for the Debian
packages, and I adapted it to the RPM environment.

However, the dependency upon the new version of the OS being able to run
the old executables could be a killer in the future if executable
compatibility is removed -- after all, an upgrade might not be from the
immediately prior version of the OS.

> You are basically saying that because you can ship without a compiler
> sometimes, we are supposed to change the way our regression tests work.
> Let's suppose SCO says they don't ship with a compiler, and wants us to
> change our code to accomodate it. Should we? You can be certain we
> would not, and in the RPM case, you get the same answer.

> If the patch is trivial, we will work around OS limitations, but we do
> not redesign code to work around OS limitations. We expect the OS to
> get the proper features. That is what we do with NT. Cygwin provides
> the needed features.

No, I'm saying that someone running any OS might want to do this:
1.) They have two machines, a development machine and a production
machine. Due to budget constraints, the dev machine is an el cheapo
version of the production machine (for the sake of argument, let's say
dev is a Sun Ultra 1 bare-bones workstation, and the production is a
high end SMP Ultra server, both running the same version of Solaris).

2.) For greater security, the production machine has been severely
crippled WRT development tools -- if a cracker gets in, don't give him
any ammunition. Good procedure to follow for publicly exposed database
servers, like those that sit behind websites. Requiring such a server
to have a development system installed is a misfeature, IMHO.

3.) After compiling and testing PostgreSQL on dev, the user transfers
the binaries only over to production. All is well, at first.

4.) But then the load on production goes up -- and PostgreSQL starts
spitting errors and FATAL's. The problem cannot be duplicated on the
dev machine -- looks like a Solaris SMP issue.

5.) The user decides to run regression on production in parallel mode to
help debug the problem -- but cannot figure out how to do so without
installing make and other development tools on it, when he specifically
did not want those tools on there for security. Serial regression,
which is easily started in a no-make mode, doesn't expose the problem.

All I'm saying is that regression should be _runnable_ in all modes
without needing anything but a shell and the PostgreSQL binary
installation.

This is the problem -- it is not OS-specific.

> > RPM's are expected to 'rpm -U' and you can simply _use_ the new version,
> > with little to no preparation. At least that is the theory. And it
> > works for most packages.

> This is the "Hey, other people can do it, why can't you" issue. We are
> looking for suggestions from Linux users in how this can be done.
> Perhaps running a separate command after the RPM has been installed is
> the only way to go.

It's not really an RPM issue -- it's a PostgreSQL issue -- there have
been e-mails from users of other OS's -- even those that compile from
source -- expressing a desire for a smoother upgrade cycle. The RPM's,
Debian packages, and other binary packages just put the extant problem
in starker contrast. Until such occurs, I'll just have to continue
doing what I'm doing -- which I consider a stop-gap, not a solution.

And, BTW, welcome back from the summit. I heard that there was a little
'excitement' there :-).
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

> Bruce Momjian wrote:
> > > At the risk of being redundant, here goes. As I've explained before,
> > > the RPM upgrade environment, thanks to our standing with multiple
> > > distributions as being shipped as a part of the OS, could be run as part
>
> > OK, maybe doing it in an RPM is the wrong way to go. If an old version
> > exists, maybe the RPM is only supposed to install the software in a
> > saved location, and the users must execute a command after the RPM
> > install that starts the old postmaster, does the dump, puts the new
> > PostgreSQL server in place, and reloads the database.
>
> That's more or less what's being done now. The RPM's preinstall script
> (run before any files are overwritten from the new package) backs up the
> required executables from the old installation. The RPM then overwrites
> the necessary files, and then any old files left over are removed, along
> with database removal of their records.
>
> A script (actually, two scripts due to a bug in the first one) is
> provided to dump the database using the old executables. Which works OK
> as long as the new OS release is executable compatible with the old
> release. Oliver Elphick originally wrote the script for the Debian
> packages, and I adapted it to the RPM environment.
>
> However, the dependency upon the new version of the OS being able to run
> the old executables could be a killer in the future if executable
> compatibility is removed -- after all, an upgrade might not be from the
> immediately prior version of the OS.

That is a tough one. I see your point. How would the RPM do this
anyway? It is running the same version of the OS right? Did they move
the data files from the old OS to the new OS and now they want to
upgrade? Hmm.

> > You are basically saying that because you can ship without a compiler
> > sometimes, we are supposed to change the way our regression tests work.
> > Let's suppose SCO says they don't ship with a compiler, and wants us to
> > change our code to accomodate it. Should we? You can be certain we
> > would not, and in the RPM case, you get the same answer.
>
> > If the patch is trivial, we will work around OS limitations, but we do
> > not redesign code to work around OS limitations. We expect the OS to
> > get the proper features. That is what we do with NT. Cygwin provides
> > the needed features.
>
> No, I'm saying that someone running any OS might want to do this:
> 1.) They have two machines, a development machine and a production
> machine. Due to budget constraints, the dev machine is an el cheapo
> version of the production machine (for the sake of argument, let's say
> dev is a Sun Ultra 1 bare-bones workstation, and the production is a
> high end SMP Ultra server, both running the same version of Solaris).

Yes, but if we added capabilities every time someone wanted something so
it worked better in their environment, this software would be a mess,
right?

> > This is the "Hey, other people can do it, why can't you" issue. We are
> > looking for suggestions from Linux users in how this can be done.
> > Perhaps running a separate command after the RPM has been installed is
> > the only way to go.
>
> It's not really an RPM issue -- it's a PostgreSQL issue -- there have
> been e-mails from users of other OS's -- even those that compile from
> source -- expressing a desire for a smoother upgrade cycle. The RPM's,
> Debian packages, and other binary packages just put the extant problem
> in starker contrast. Until such occurs, I'll just have to continue
> doing what I'm doing -- which I consider a stop-gap, not a solution.

Yes, we all agree upgrades should be smoother. The problem is that the
cost/benefit analysis always pushed us away from improving it.

>
> And, BTW, welcome back from the summit. I heard that there was a little
> 'excitement' there :-).

Yes, it was very nice. I will post a summary to announce/general today.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian wrote:
> > However, the dependency upon the new version of the OS being able to run
> > the old executables could be a killer in the future if executable
> > compatibility is removed -- after all, an upgrade might not be from the
> > immediately prior version of the OS.

> That is a tough one. I see your point. How would the RPM do this
> anyway? It is running the same version of the OS right? Did they move
> the data files from the old OS to the new OS and now they want to
> upgrade? Hmm.

Well, let's suppose the following: J. Random User has a database server
that has been running smoothly for ages on RedHat 5.2, running
PostgreSQL 6.3.2. He has had no reason to upgrade since -- while MVCC
was a nice feature, he was really waiting for OUTER JOIN before
upgrading, as his server is lightly loaded and won't benefit greatly
from MVCC.

Likewise, he's not upgraded from RedHat 5.2, because until RedHat got
the 2.4 kernel into a distribution, he wasn't ready to upgrade, as he
needs improved NFS performance, available in Linux kernel 2.4. And he
wasn't about to go with a version of GCC that doesn't exist. So he
skips the whole RedHat 6.x series -- he doesn't want to mess with kernel
2.2 in any form, thanks to its abyssmal NFS performance.

So he waits on RedHat 7.2 to be released -- around October 2001 (if the
typical RedHat schedule holds). At this point, PostgreSQL 7.2.1 is the
standards bearer, with OUTER JOIN support that he craves, and robust WAL
for excellent recoverability, amongst other Neat Features(TM).

Now, by the time of RedHat 7.2, kernel 2.4 is up to .15 or so, with gcc
3.0 freshly (and officially) released, and glibc 2.2.5 finally fixing
the problems that had plagued both pre-2.2 glibc's AND the earliest 2.2
glibc's -- but, the upshot is that glibc 2.0 compatibility is toast.

Now, J Random slides in the new OS CD on a backup of his main server,
and upgrades. RedHat 7.2's installer is very smart -- if no packages
are left that use glibc 2.0, it doesn't install the compat-libs
necessary for glibc 2.0 apps to run.

The PostgreSQL RPMset's server subpackage preinstall script runs about
two-thirds of the way through the upgrade, and backs up the old 6.3.2
executables necessary to pull a dump. The old 6.3.2 rpm database
entries are removed, and, as far as the system is concerned, no
dependency upon glibc 2.0 remains, so no compat-libs get installed.

J Random checks out the new installation, and finds a conspicuous log
message telling him to read /usr/share/doc/postgresql-7.2.1/README.rpm.
He does so, and runs the (fixed by then) postgresql-dump script, which
attempts to start an old backend and do a pg_dumpall -- but, horrors,
the old postmaster can't start, glibc 2.0 is gone and glibc 2.2 blows
core loaded under postmaster-6.3.2. ARGGHHH....

That's the scenario I have nightmares about. Really.

> Yes, but if we added capabilities every time someone wanted something so
> it worked better in their environment, this software would be a mess,
> right?

Yes, it would. I'll work on a patch, and we'll see what it looks like.

> > been e-mails from users of other OS's -- even those that compile from
> > source -- expressing a desire for a smoother upgrade cycle. The RPM's,

> Yes, we all agree upgrades should be smoother. The problem is that the
> cost/benefit analysis always pushed us away from improving it.

I understand.

I'm looking at some point in time in the future doing a
'postgresql-upgrade' RPM that would include pre-built postmasters and
other binaries necessary to dump any previous version PostgreSQL (since
about 6.2.1 or so -- 6.2.1 was the first RedHat official PostgreSQL RPM,
although there were 6.1.1 RPM's before that, and there is still a
postgres95-1.09 RPM out there), linked to the current libs for that
RPM's OS release. It would be a large RPM (and the source RPM for it
would be _huge_, containing entire tarballs for at least 6.2.1, 6.3.2,
6.4.2, 6.5.3, and 7.0.3). But, this may be the only way to make this
work barring a real migration utility.

> > And, BTW, welcome back from the summit. I heard that there was a little
> > 'excitement' there :-).

> Yes, it was very nice. I will post a summary to announce/general today.

Good. And a welcome back to Tom as well, as he went too, IIRC.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:

> Now, J Random slides in the new OS CD on a backup of his main server,
> and upgrades. RedHat 7.2's installer is very smart -- if no packages
> are left that use glibc 2.0, it doesn't install the compat-libs
> necessary for glibc 2.0 apps to run.

Actually, glibc is a bad example of things to break - it has versioned
symbols, so postgresql is pretty likely to continue working (barring
doing extremely low-level stuff, like doing weird things to the loader
or depend on buggy behaviour (like Oracle did)).

Postgresql doesn't use C++ either (which is a horrible mess wrt. binary
compatibility - there is no such thing, FTTB).

However, if it depended on kernel specific behaviour (like things in
/proc, which may or may not have changed its output format) it could
break.

--
Trond Eivind Glomsrød
Red Hat, Inc.

Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> All I'm saying is that regression should be _runnable_ in all modes
> without needing anything but a shell and the PostgreSQL binary
> installation.

I think this'd be mostly a waste of effort. IMHO, 99% of the problems
the regression tests might expose will be exposed if they are run
against the RPMs by the RPM maker. (Something we have sometimes failed
to do in the past ;-).) The regress tests are not that good at
detecting environment-specific problems; in fact, they go out of their
way to suppress environmental differences. So I don't see any strong
need to support regression test running in binary distributions.
Especially not if we have to kluge around a lack of essential tools.

regards, tom lane