Re: proposal: psql –help reflecting service or URI usage

Hi,

I'd like to propose a patch to psql --help output:

Currently it is:

Usage:
psql [OPTION]... [DBNAME [USERNAME]]

...

Connection options:
-h, --host=HOSTNAME database server host or socket directory (default: "local socket")
-p, --port=PORT database server port (default: "5432")
-U, --username=USERNAME database user name (default: "paul")
-w, --no-password never prompt for password
-W, --password force password prompt (should happen automatically)

I'd like to change it to the following to reflect the psql ability to process a service name or a PostgreSQL URI:

Usage:
psql [OPTION]... [DBNAME [USERNAME]|service|uri]

...

Connection options:
-h, --host=HOSTNAME database server host or socket directory (default: "local socket")
-p, --port=PORT database server port (default: "5432")
-U, --username=USERNAME database user name (default: "paul")
-w, --no-password never prompt for password
-W, --password force password prompt (should happen automatically)
service=name service name as definited in pg_service.conf
uri connection URI (postgresql://...)

...

Attached is a patch for src/bin/psql/help.c for this. The file doc/src/sgml/ref/psql-ref.sgml does not seem to need any changes for this.

Any thoughts on this?

> On Feb 28, 2021, at 1:57 AM, Paul Förster <paul(dot)foerster(at)gmail(dot)com> wrote:
>
> Hi,
>
> I'd like to propose a patch to psql --help output:
>
> Currently it is:
>
> Usage:
> psql [OPTION]... [DBNAME [USERNAME]]
>
> ...
>
> Connection options:
> -h, --host=HOSTNAME database server host or socket directory (default: "local socket")
> -p, --port=PORT database server port (default: "5432")
> -U, --username=USERNAME database user name (default: "paul")
> -w, --no-password never prompt for password
> -W, --password force password prompt (should happen automatically)
>
> I'd like to change it to the following to reflect the psql ability to process a service name or a PostgreSQL URI:
>
> Usage:
> psql [OPTION]... [DBNAME [USERNAME]|service|uri]
>
> ...
>
> Connection options:
> -h, --host=HOSTNAME database server host or socket directory (default: "local socket")
> -p, --port=PORT database server port (default: "5432")
> -U, --username=USERNAME database user name (default: "paul")
> -w, --no-password never prompt for password
> -W, --password force password prompt (should happen automatically)
> service=name service name as definited in pg_service.conf

"definited" is a typo.

Should this say "as defined in pg_service.conf"? That's the default, but the user might have $PGSERVICEFILE set to something else. Perhaps you could borrow the wording of other options and use "(default: as defined in pg_service.conf)", or something like that, but of course being careful to still fit in the line length limit.

> uri connection URI (postgresql://...)
>
> ...
>
> Attached is a patch for src/bin/psql/help.c for this. The file doc/src/sgml/ref/psql-ref.sgml does not seem to need any changes for this.
>
> Any thoughts on this?

Other client applications follow the same pattern as psql, so if this change were adopted, it should apply to all of them.

Your proposal seems like something that would have been posted to the list before, possibly multiple times. Any chance you could dig up past conversations on this subject and post links here for context?

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi Mark,

> On 28. Feb, 2021, at 17:54, Mark Dilger <mark(dot)dilger(at)enterprisedb(dot)com> wrote:
>
> "definited" is a typo.

yes, definitely a typo, sorry. Thanks for pointing this out.

> Should this say "as defined in pg_service.conf"? That's the default, but the user might have $PGSERVICEFILE set to something else. Perhaps you could borrow the wording of other options and use "(default: as defined in pg_service.conf)", or something like that, but of course being careful to still fit in the line length limit.

I agree to all, thanks. What is the line length limit?

> Other client applications follow the same pattern as psql, so if this change were adopted, it should apply to all of them.

well, psql is central and IMHO the best place to start. I'd have to try out all of them then. What I do know, though, is that pg_isready does not understand a URI (why is that?), which is very unfortunate. So, I'd have to try them all out and supply patches for them all?

Still, supporting a feature and not documenting it in its help is IMHO not a good idea.

> Your proposal seems like something that would have been posted to the list before, possibly multiple times. Any chance you could dig up past conversations on this subject and post links here for context?

I don't know any past discussions here. I only subscribed today to psql-hackers. It might have been mentioned on psql-general, though. But I'm not sure. This idea popped into my mind just yesterday when I was playing around with psql and URIs and noticed that psql –help doesn't show them.

Cheers,
Paul

Hi Mark,

I revisited my first admittedly naive and faulty attempt.

> On 28. Feb, 2021, at 19:10, Paul Förster <paul(dot)foerster(at)gmail(dot)com> wrote:
>
>> but of course being careful to still fit in the line length limit.
> I agree to all, thanks. What is the line length limit?

still, what is the line length limit? Where do I find it?

I'd be very happy if you could take a look at this version. Thanks in advance and thanks very much for the input.

Cheers,
Paul

On Mon, Mar 1, 2021, at 10:32 AM, Paul Förster wrote:
> still, what is the line length limit? Where do I find it?
We try to limit it to 80 characters but it is not a hard limit. Long
descriptions should certainly be split into multiple lines.

The question is: how popular is service and connection URIs? We cannot
certainly include all possibilities in the --help that's why we have the
documentation. IMO we could probably include "connection string" that accepts 2
formats: (i) multiple keyword - value string and URIs ("service" is included in
the (i)).

Usage:
psql [OPTION]... [DBNAME [USERNAME]|CONNINFO]

or even

Usage:
psql [OPTION]... [DBNAME [USERNAME]]
psql [OPTION]... [CONNINFO]


Connection options:
-h, --host=HOSTNAME database server host or socket directory (default: "local socket")
-p, --port=PORT database server port (default: "9999")
-U, --username=USERNAME database user name (default: "euler")
-w, --no-password never prompt for password
-W, --password force password prompt (should happen automatically)

CONNINFO connection string to connect to (key = value strings
or connection URI)

I don't like the CONNINFO in the connection options. It seems a natural place
but DBNAME and USERNAME aren't included in it. Should we include it too?
Someone can argue that they are self-explanatory, hence, a description is not
necessary.

It might be a different topic but since we are talking about --help
improvements, I have some suggestions:

* an Example section could help newbies to Postgres command-line
tools to figure out how to inform the connection parameters. In this case, we
could include at least 3 examples: (i) -h, -p, -U parameters, (ii) key/value
connection string and (iii) connection URI.
* Connection options could be moved to the top (maybe after General options) if
we consider that it is more important than the other sections (you cannot
probably execute psql without using a connection parameter in production).

--
Euler Taveira
EDB https://www.enterprisedb.com/

> On Feb 28, 2021, at 10:10 AM, Paul Förster <paul(dot)foerster(at)gmail(dot)com> wrote:
>
> Hi Mark,
>
>> On 28. Feb, 2021, at 17:54, Mark Dilger <mark(dot)dilger(at)enterprisedb(dot)com> wrote:
>>
>> "definited" is a typo.
>
> yes, definitely a typo, sorry. Thanks for pointing this out.
>
>> Should this say "as defined in pg_service.conf"? That's the default, but the user might have $PGSERVICEFILE set to something else. Perhaps you could borrow the wording of other options and use "(default: as defined in pg_service.conf)", or something like that, but of course being careful to still fit in the line length limit.
>
> I agree to all, thanks. What is the line length limit?

The output from --help should fit in a terminal window with only 80 characters width. For example, in src/bin/scripts/createuser.c the line about --interactive is wrapped:

> printf(_(" -S, --no-superuser role will not be superuser (default)\n"));
> printf(_(" -V, --version output version information, then exit\n"));
> printf(_(" --interactive prompt for missing role name and attributes rather\n"
> " than using defaults\n"));
> printf(_(" --replication role can initiate replication\n"));

You can find counter-examples where applications do not follow this rule. pg_isready is one of them.

>> Other client applications follow the same pattern as psql, so if this change were adopted, it should apply to all of them.
>
> well, psql is central and IMHO the best place to start. I'd have to try out all of them then. What I do know, though, is that pg_isready does not understand a URI (why is that?), which is very unfortunate. So, I'd have to try them all out and supply patches for them all?

There is a pattern in the client applications that the --help output is less verbose than the docs (see, for example, /docs/13/reference-client.html) Your proposed patch makes psql's --help output a bit more verbose about this issue while leaving the other applications less so, without any obvious reason for the difference.

> Still, supporting a feature and not documenting it in its help is IMHO not a good idea.

Ok.

>> Your proposal seems like something that would have been posted to the list before, possibly multiple times. Any chance you could dig up past conversations on this subject and post links here for context?
>
> I don't know any past discussions here. I only subscribed today to psql-hackers. It might have been mentioned on psql-general, though. But I'm not sure. This idea popped into my mind just yesterday when I was playing around with psql and URIs and noticed that psql –help doesn't show them.

I mentioned looking at the mailing list archives, but neglected to give you a link: /list/

Over the years, many proposals get made and debated, with some accepted and some rejected. The rejected proposals often have some merit, and get suggested again, only to get rejected for the same reasons as the previous times they were suggested. So searching the archives before posting a patch can sometimes be enlightening. The difficulty in my experience is knowing what words and phrases to search for. It can be a bit time consuming trying to find a prior discussion on a topic.

I don't know of any specific discussion which rejected your patch idea.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi Mark,

sorry for the delay.

> On 01. Mar, 2021, at 17:02, Mark Dilger <mark(dot)dilger(at)enterprisedb(dot)com> wrote:
>
> The output from --help should fit in a terminal window with only 80 characters width. For example, in src/bin/scripts/createuser.c the line about --interactive is wrapped:

I see.

> You can find counter-examples where applications do not follow this rule. pg_isready is one of them.

yes, I noticed that.

> There is a pattern in the client applications that the --help output is less verbose than the docs (see, for example, /docs/13/reference-client.html) Your proposed patch makes psql's --help output a bit more verbose about this issue while leaving the other applications less so, without any obvious reason for the difference.

I could do the other tools too, that wouldn't be a problem. But I'm not good at writing docs. And I found that the man pages would have to be adapted too, which would be a big one for me as I'm no man guru.

> Over the years, many proposals get made and debated, with some accepted and some rejected. The rejected proposals often have some merit, and get suggested again, only to get rejected for the same reasons as the previous times they were suggested. So searching the archives before posting a patch can sometimes be enlightening. The difficulty in my experience is knowing what words and phrases to search for. It can be a bit time consuming trying to find a prior discussion on a topic.

I've searched the archives for quite some time and found tons of hits for the search term "help". But that's useless because all people ask for "help". :-) Beside that, I found nothing which even remotely discussed the topic.

But I generally see your points so I drop the proposal. It was only an idea after all :-) Thanks for your input.

Cheers,
Paul

Hi Euler,

> On 01. Mar, 2021, at 15:42, Euler Taveira <euler(at)eulerto(dot)com> wrote:
>
> We try to limit it to 80 characters but it is not a hard limit. Long descriptions should certainly be split into multiple lines.

got that, thanks.

> The question is: how popular is service and connection URIs?

well, we use them exclusively at work, because we have a lot of Patroni clusters which may fail/switch over and we don't have an haproxy or similar. So our usual way to connect is a URI with targetServerType set.

> We cannot certainly include all possibilities in the --help that's why we have the documentation. IMO we could probably include "connection string" that accepts 2 formats: (i) multiple keyword - value string and URIs ("service" is included in the (i)).
>
> Usage:
> psql [OPTION]... [DBNAME [USERNAME]|CONNINFO]
>
> Usage:
> psql [OPTION]... [DBNAME [USERNAME]]
> psql [OPTION]... [CONNINFO]
>
> Connection options:
> CONNINFO connection string to connect to (key = value strings
> or connection URI)

I could live with the second variant, though I'd still prefer two descriptions, one "service=name" and the second for the URI, as I initially suggested.

> It might be a different topic but since we are talking about --help improvements, I have some suggestions:
>
> * an Example section could help newbies to Postgres command-line tools to figure out how to inform the connection parameters. In this case, we could include at least 3 examples: (i) -h, -p, -U parameters, (ii) key/value connection string and (iii) connection URI.

there's an example in the USAGE/Connecting to a Database section of the man page already. Also, it is documented how an URI works, so I wouldn't include an example here. Just reflecting its existence in the syntax should do. Same goes for service names.

> * Connection options could be moved to the top (maybe after General options) if we consider that it is more important than the other sections (you cannot probably execute psql without using a connection parameter in production).

moving it up is IMHO merely a matter of personal taste. Making sure it's there was my initial point.

But as Mark pointed out, there's too much linked to it for me (man pages, docs, etc.). So I drop the proposal altogether. Thanks for you thoughts anyway.

Now we at least have this topic finally in the mail archives. ;-)

P.S.: Just curious, why do you right-pad your posts?

Cheers,
Paul

> On Mar 6, 2021, at 5:55 AM, Paul Förster <paul(dot)foerster(at)gmail(dot)com> wrote:
>
> Hi Mark,
>
> sorry for the delay.
>
>> On 01. Mar, 2021, at 17:02, Mark Dilger <mark(dot)dilger(at)enterprisedb(dot)com> wrote:
>>
>> The output from --help should fit in a terminal window with only 80 characters width. For example, in src/bin/scripts/createuser.c the line about --interactive is wrapped:
>
> I see.
>
>> You can find counter-examples where applications do not follow this rule. pg_isready is one of them.
>
> yes, I noticed that.
>
>> There is a pattern in the client applications that the --help output is less verbose than the docs (see, for example, /docs/13/reference-client.html) Your proposed patch makes psql's --help output a bit more verbose about this issue while leaving the other applications less so, without any obvious reason for the difference.
>
> I could do the other tools too, that wouldn't be a problem. But I'm not good at writing docs. And I found that the man pages would have to be adapted too, which would be a big one for me as I'm no man guru.

Fortunately, the man pages and html docs are generated from the same sources. Those sources are written in sgml, and the tools to build the docs must be installed. From the top directory, execute `make docs` and if it complains about missing tools you will need to install them. (The build target is 'docs', but the directory containing the docs is named 'doc'.)

>> Over the years, many proposals get made and debated, with some accepted and some rejected. The rejected proposals often have some merit, and get suggested again, only to get rejected for the same reasons as the previous times they were suggested. So searching the archives before posting a patch can sometimes be enlightening. The difficulty in my experience is knowing what words and phrases to search for. It can be a bit time consuming trying to find a prior discussion on a topic.
>
> I've searched the archives for quite some time and found tons of hits for the search term "help". But that's useless because all people ask for "help". :-) Beside that, I found nothing which even remotely discussed the topic.
>
> But I generally see your points so I drop the proposal. It was only an idea after all :-) Thanks for your input.

Oh, I'm quite sorry to hear that. The process of getting a patch accepted, especially the first time you submit one, can be discouraging. But the community greatly benefits from new contributors joining the effort, so I'd much rather you not withdraw the idea.

If you need help with certain portions of the submission, such as editing the docs, I can help with that.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Hi Mark,

> On 08. Mar, 2021, at 16:39, Mark Dilger <mark(dot)dilger(at)enterprisedb(dot)com> wrote:
>
> Fortunately, the man pages and html docs are generated from the same sources. Those sources are written in sgml, and the tools to build the docs must be installed. From the top directory, execute `make docs` and if it complains about missing tools you will need to install them. (The build target is 'docs', but the directory containing the docs is named 'doc'.)

so the help files I'd change would be doc/src/sgml/ref/psql-ref.sgml, doc/src/sgml/ref/pg_isready.sgml, etc.?

> Oh, I'm quite sorry to hear that. The process of getting a patch accepted, especially the first time you submit one, can be discouraging. But the community greatly benefits from new contributors joining the effort, so I'd much rather you not withdraw the idea.

I'd like to, and also I'd like to do all the bin/* tools (including wrapping the long line in pg_isready ;-)), as you suggested, but I don't know the process. In my first admittedly naive attempt, I just downloaded the source from /ftp/source/v13.2, unpacked it and made my changes there. Then I did a diff to the original and posted it here. I don't even know if this is the correct workflow. I saw gitgub being mentioned a couple of times but I don't have an account, nor do I even know how it works.

I was pretty surprised to see the lines in PWN:

"Paul Förster sent in a patch to mention database URIs in psql's --help output."
"Paul Förster sent in another revision of a patch to mention URIs and services in psql --help's output."

Is there a FAQ somewhere that describes how properly create patches, submit them and possibly get them released? Something like a step-by-step?

Is github a must-have here?

> If you need help with certain portions of the submission, such as editing the docs, I can help with that.

as you see above, I'm curious to learn, though doing it to all the tools will take some time for me.

Sorry, I'm a noob, not so much to C, but to the workflows here. Hence my questions may seem a little obvious to all the pros.

Cheers,
Paul

> On Mar 8, 2021, at 8:40 AM, Paul Förster <paul(dot)foerster(at)gmail(dot)com> wrote:
>
> Hi Mark,
>
>> On 08. Mar, 2021, at 16:39, Mark Dilger <mark(dot)dilger(at)enterprisedb(dot)com> wrote:
>>
>> Fortunately, the man pages and html docs are generated from the same sources. Those sources are written in sgml, and the tools to build the docs must be installed. From the top directory, execute `make docs` and if it complains about missing tools you will need to install them. (The build target is 'docs', but the directory containing the docs is named 'doc'.)
>
> so the help files I'd change would be doc/src/sgml/ref/psql-ref.sgml, doc/src/sgml/ref/pg_isready.sgml, etc.?

Yes

>> Oh, I'm quite sorry to hear that. The process of getting a patch accepted, especially the first time you submit one, can be discouraging. But the community greatly benefits from new contributors joining the effort, so I'd much rather you not withdraw the idea.
>
> I'd like to, and also I'd like to do all the bin/* tools (including wrapping the long line in pg_isready ;-)), as you suggested, but I don't know the process. In my first admittedly naive attempt, I just downloaded the source from /ftp/source/v13.2, unpacked it and made my changes there. Then I did a diff to the original and posted it here. I don't even know if this is the correct workflow. I saw gitgub being mentioned a couple of times but I don't have an account, nor do I even know how it works.
>
> I was pretty surprised to see the lines in PWN:
>
> "Paul Förster sent in a patch to mention database URIs in psql's --help output."
> "Paul Förster sent in another revision of a patch to mention URIs and services in psql --help's output."
>
> Is there a FAQ somewhere that describes how properly create patches, submit them and possibly get them released? Something like a step-by-step?
>
> Is github a must-have here?

No, github is not a must-have. I don't use a github account for my submissions. The project uses git for source code control, but that's not the same thing as requiring "github". The project switched from cvs to git a few years back.

If you can install git, using rpm or yum or whatever, then from the command line you can use

git clone https://git.postgresql.org/git/postgresql.git

and it will create a working directory for you. You can make modifications and commit them. When you are finished, you can run

git format-patch -v 1 master

and it will create a patch set containing all your changes relative to the public sources you cloned, and the patch will include your commit messages, which helps reviewers not only know what you changed, but why you made the changes, in your own words.

See https://wiki.postgresql.org/wiki/Development_information

>> If you need help with certain portions of the submission, such as editing the docs, I can help with that.
>
> as you see above, I'm curious to learn, though doing it to all the tools will take some time for me.
>
> Sorry, I'm a noob, not so much to C, but to the workflows here. Hence my questions may seem a little obvious to all the pros.

That's not a problem. If this gets too verbose for the list, we can take this off-list and I can still walk you through it.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company