PostgreSQL news

PostgreSQL Weekly News - January 2, 2022

Wed, 05 Jan 2022 00:00:00 +0000

PostgreSQL Weekly News - January 2, 2022

PostgreSQL Product News

Pgpool-II 4.2.7, 4.1.10, 4.0.17, and 3.7.22 a connection pooler and statement replication system for PostgreSQL, re le as ed.

parquet_s3_fdw 0.2.1, a foreign data wrapper for parquet files on S3, released.

Database Lab 3.0, a tool for fast cloning of large PostgreSQL databases to build non-production environments, released.

sqlite_fdw 2.1.1 released.

DynamoDB FDW 1.1.0 released.

pg_query_rewrite 0.0.3, a rewriter for certain types of PostgreSQL statements, released.

InfluxDB fdw 1.1.1 released https://github.com/pgspider/influxdb_fdw

pgspider v2.0, a cluster engine for distributed data based on PostgreSQL foreign data wrappers, released.

pg_builder 2.0.0 a PHP query builder for PostgreSQL, released.

JDBC FDW 0.1.0 released

griddb_fdw 2.1.1 released. https://github.com/pgspider/griddb_fdw

PostgreSQL Jobs for January

https://archives.postgresql.org/pgsql-jobs/2022-01/

PostgreSQL Local

Nordic PGDay 2022 will be held in Helsinki, Finland at the Hilton Helsinki Strand Hotel on March 22, 2022. here

pgDay Paris 2022 will be held in Paris, France on March 24, 2022.

FOSDEM PGDay 2022 will be held on line, on Feb 5-6, 2022. https://fosdem.org/2022/

Citus Con, a virtual global developer event, is happening April 12-13, 2022. The CFP is now open.

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Peter Eisentraut pushed:

doc: More documentation on regular expressions and SQL standard. Reviewed-by: Gilles Darold gilles@darold.net Discussion: /message-id/b7988566-daa2-80ed-2fdc-6f6630462d26@enterprisedb.com https://git.postgresql.org/pg/commitdiff/222b697ec077047024a96392a2f5cb9b1803ccf7
pg_dump: Refactor getIndexes(). Rearrange the version-dependent pieces in the new more modular style. Discussion: /message-id/flat/67a28a3f-7b79-a5a9-fcc7-947b170e66f0%40enterprisedb.com https://git.postgresql.org/pg/commitdiff/e2c52beecdea152ca680a22ef35c6a7da55aa30f
Fix typo in code comment. Reported-by: Kevin Zheng 1642644905@qq.com Discussion: /message-id/flat/17341-d913ddb626c5c08c%40postgresql.org https://git.postgresql.org/pg/commitdiff/962951be3ce319052df014e0d23b6e7626df587f
Fix incorrect format placeholders. https://git.postgresql.org/pg/commitdiff/dfaa346c7c00ff8a3fd8ea436a7d5be7527e67cb
Remove unused include. "fmgr.h" was used for load_external_function(), added by a05dc4d7fd57d4ae084c1f0801973e5c1a1aa26e, removed by f9143d102ffd0947ca904c62b1d3d6fd587e0c80. https://git.postgresql.org/pg/commitdiff/2f4fd1a73ba59247d4413f33d6178bbcbcfab60f
Remove unused include. "utils/builtins.h" was used for pg_strtouint64(), added by cff440d368690f94fbda1a475277e90ea2263843, removed by 3c6f8c011f85df7b35c32f4ccaac5c86c9064a4a. https://git.postgresql.org/pg/commitdiff/4965f75484aea9d273b11ad4957f57d8dc1ed4b0
Fix incorrect format placeholders. https://git.postgresql.org/pg/commitdiff/113fa3945f8969346d6a87b9a56d54afa3d34687

John Naylor pushed:

Add fast path for validating utf-8 text. Our previous validator used a traditional algorithm that performed comparison and branching one byte at a time. It's useful in that we always know exactly how many bytes we have validated, but that precision comes at a cost. Input validation can show up prominently in profiles of COPY FROM, and future improvements to COPY FROM such as parallelism or faster line parsing will put more pressure on input validation. Hence, add fast paths for both ASCII and multibyte utf-8: Use bitwise operations to check 16 bytes at a time for ASCII. If that fails, use a "shift-based" DFA on those bytes to handle the general case, including multibyte. These paths are relatively free of branches and thus robust against all kinds of byte patterns. With these algorithms, utf-8 validation is several times faster, depending on platform and the input byte distribution. The previous coding in pg_utf8_verifystr() is retained for short strings and for when the fast path returns an error. Review, performance testing, and additional hacking by: Heikki Linakangas, Vladimir Sitnikov, Amit Khandekar, Thomas Munro, and Greg Stark Discussion: /message-id/CAFBsxsEV_SzH%2BOLyCiyon%3DiwggSyMh_eF6A3LU2tiWf3Cy2ZQg%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/911588a3f816d875261d8f7d89e2517978831cd5

Tom Lane pushed:

Add a \getenv command to psql. \getenv fetches the value of an environment variable into a psql variable. This is the inverse of the \setenv command that was added over ten years ago. We'd not seen a compelling use-case for \getenv at the time, but upcoming regression test refactoring provides a sufficient reason to add it now. Discussion: https://postgr.es/m/1655733.1639871614@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/33d3eeadb21d2268104840cfef6bc2226ddfc680
Remove dynamic translation of regression test scripts, step 1. pg_regress has long had provisions for dynamically substituting path names into regression test scripts and result files, but use of that feature has always been a serious pain in the neck, mainly because updating the result files requires tedious manual editing. Let's get rid of that in favor of passing down the paths in environment variables. In addition to being easier to maintain, this way is capable of dealing with path names that require escaping at runtime, for example paths containing single-quote marks. (There are other stumbling blocks in the way of actually building in a path that looks like that, but removing this one seems like a good thing to do.) The key coding rule that makes that possible is to concatenate pieces of a dynamically-variable string using psql's \set command, and then use the :'variable' notation to quote and escape the string for the next level of interpretation. In hopes of making this change more transparent to "git blame", I've split it into two steps. This commit adds the necessary pg_regress.c support and changes all the *.source files in-place so that they no longer require any dynamic translation. The next commit will just "git mv" them into the regular sql/ and expected/ directories. Discussion: https://postgr.es/m/1655733.1639871614@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/d1029bb5a26cb84b116b0dee4dde312291359f2a
Remove dynamic translation of regression test scripts, step 2. "git mv" all the input/.source and output/.source files into the corresponding sql/ and expected/ directories. Then remove the pg_regress and Makefile infrastructure associated with dynamic translation. Discussion: https://postgr.es/m/1655733.1639871614@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/dc9c3b0ff21465fa89d71eecf5e6cc956d647eca
Merge dblink's paths test script into its main test. There's no longer any reason to fire up a separate psql run to create these functions. (Some refactoring in the main regression tests is also called for, but that will take more thought.) Discussion: https://postgr.es/m/1655733.1639871614@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/0e6e7f0806b2080cb31f33ff992ec2e4e35fa6f1
Add missing EmitWarningsOnPlaceholders() calls. Extensions that define any custom GUCs should call EmitWarningsOnPlaceholders after doing so, to help catch misspellings. Many of our contrib modules hadn't gotten the memo on that, though. Also add such calls to src/test/modules extensions that have GUCs. While these aren't really user-facing, they should illustrate good practice not faulty practice. Shinya Kato Discussion: https://postgr.es/m/524fa2c0a34f34b68fbfa90d0760d515@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/1fada5d81e6769ded832a4ca62ee9371bac3fb9f
Add help & tab-complete support for psql's \getenv. I forgot about these details in 33d3eeadb :-(. Noted by Christoph Berg. Discussion: https://postgr.es/m/YcI8i/mduMi91uXY@msg.df7cb.de https://git.postgresql.org/pg/commitdiff/0f2abd05441f524a67bc58ef5f0cc32054f7fb66
Rethink handling of settings with a prefix reserved by an extension. Commit 75d22069e made SET print a warning if you tried to set an unrecognized parameter within namespace previously reserved by an extension. It seems better for that to be an outright error though, for the same reason that we don't let you set unrecognized unqualified parameter names. In any case, the preceding implementation was inefficient and erroneous. Perform the check in a more appropriate spot, and be more careful about prefix-match cases. Discussion: https://postgr.es/m/116024.1640111629@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/2ed8a8cc5b634d33ea07d681c6b02213da07f792
Rename EmitWarningsOnPlaceholders() to MarkGUCPrefixReserved(). This seems like a clearer name for what it does now. Provide a compatibility macro so that extensions don't have to convert to the new name right away. Discussion: https://postgr.es/m/116024.1640111629@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/5609cc01c69b80f8788771dc6f5696a459469119
Revert changes about warnings/errors for placeholders. Revert commits 5609cc01c, 2ed8a8cc5, and 75d22069e until we have a less broken idea of how this should work in parallel workers. Per buildfarm. Discussion: https://postgr.es/m/1640909.1640638123@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/cab5b9ab2c066ba904f13de2681872dcda31e207
Fix issues in pgarch's new directory-scanning logic. The arch_filenames[] array elements were one byte too small, so that a maximum-length filename would get corrupted if another entry were made after it. (Noted by Thomas Munro, fix by Nathan Bossart.) Move these arrays into a palloc'd struct, so that we aren't wasting a few kilobytes of static data in each non-archiver process. Add a binaryheap_reset() call to make it plain that we start the directory scan with an empty heap. I don't think there's any live bug of that sort, but it seems fragile, and this is very cheap insurance. Cleanup for commit beb4e9ba1, so no back-patch needed. Discussion: https://postgr.es/m/CA+hUKGLHAjHuKuwtzsW7uMJF4BVPcQRL-UMZG_HM-g0y7yLkUg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/1fb17b1903414676bd371068739549cd2966fe87
Minor cleanup/optimization in pg_dump. In the wake of commits 05649b88c and 5209c0ba0, findComments() and findSecLabels() no longer use their "Archive *fout" arguments, so get rid of those. While doing that, I noticed that there's no very good reason why dumpCompositeTypeColComments() should be doing its own query to fetch the column names of the composite type, when the calling function has just fetched the same data. Tweak it to use that query result. This probably doesn't save a lot for most people, because since 5209c0ba0 we won't get into this code at all unless the composite type has at least one comment. Nonetheless, it's a wasted query. https://git.postgresql.org/pg/commitdiff/c7cf73eb7b9e7911748ebe117a7219f21e504121
pg_dump: make dumpPublication et al. less unlike sibling functions. dumpPublication, dumpPublicationNamespace, dumpPublicationTable, and dumpSubscription failed to check dataOnly. This is just a latent bug, because pg_backup_archiver.c would filter out the ArchiveEntry later; but they're wasting cycles in data-only dumps, and the omission might become a live bug someday. In any case, it's not good to have some dumpFoo functions do this and some not. On the same reasoning, make dumpPublicationNamespace follow the same pattern as every other dumpFoo function for checking the DUMP_COMPONENT_DEFINITION flag. (Since 5209c0ba0, we wouldn't even get here if that flag isn't set, so checking it is just pro forma right now. But it might not be so forever.) Since this is just cosmetic and/or future-proofing, no need for back-patch. https://git.postgresql.org/pg/commitdiff/5e65df64d631257ce60016bec0aca43f042b1d33
pg_dump: minor performance improvements from eliminating sub-SELECTs. Get rid of the "username_subquery" mechanism in favor of doing local lookups of role names from role OIDs. The PG backend isn't terribly smart about scalar SubLinks in SELECT output lists, so this offers a small performance improvement, at least in installations with more than a couple of users. In any case the old method didn't make for particularly readable SQL code. While at it, I removed the various custom warning messages about failing to find an object's owner, in favor of just fatal'ing in the local lookup function. AFAIK there is no reason any longer to treat that as anything but a catalog-corruption case, and certainly no reason to make translators deal with a dozen different messages where one would do. (If it turns out that fatal() is indeed a bad idea, we can back off to issuing pg_log_warning() and returning an empty string, resulting in the same behavior as before, except more consistent.) Also drop an entirely unnecessary sub-SELECT to check on the pg_depend status of a sequence relation: we already have a LEFT JOIN to fetch the row of interest in the FROM clause. Discussion: https://postgr.es/m/2460369.1640903318@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/d5e8930f50e31d836d84b353b9dadedd5007bb70
pg_dump: avoid unsafe function calls in getPolicies(). getPolicies() had the same disease I fixed in other places in commit e3fcbbd62, i.e., it was calling pg_get_expr() for expressions on tables that we don't necessarily have lock on. To fix, restrict the query to only collect interesting rows, rather than doing the filtering on the client side. Like the previous patch, apply to HEAD only for now. Discussion: https://postgr.es/m/2273648.1634764485@sss.pgh.pa.us Discussion: https://postgr.es/m/7d7eb6128f40401d81b3b7a898b6b4de@W2012-02.nidsa.loc https://git.postgresql.org/pg/commitdiff/3e6e86abca0138abd7265306beb6346dc2d9e221
Fix index-only scan plans when not all index columns can be returned. If an index has both returnable and non-returnable columns, and one of the non-returnable columns is an expression using a Var that is in a returnable column, then a query returning that expression could result in an index-only scan plan that attempts to read the non-returnable column, instead of recomputing the expression from the returnable column as intended. To fix, redefine the "indextlist" list of an IndexOnlyScan plan node as containing null Consts in place of any non-returnable columns. This solves the problem by preventing setrefs.c from falsely matching to such entries. The executor is happy since it only cares about the exposed types of the entries, and ruleutils.c doesn't care because a correct plan won't reference those entries. I considered some other ways to prevent setrefs.c from doing the wrong thing, but this way seems good since (a) it allows a very localized fix, (b) it makes the indextlist structure more compact in many cases, and (c) the indextlist is now a more faithful representation of what the index AM will actually produce, viz. nulls for any non-returnable columns. This is easier to hit since we introduced included columns, but it's possible to construct failing examples without that, as per the added regression test. Hence, back-patch to all supported branches. Per bug #17350 from Louis Jachiet. Discussion: https://postgr.es/m/17350-b5bdcf476e5badbb@postgresql.org https://git.postgresql.org/pg/commitdiff/4ace456776524839ef3279ab0bad8a2c9f6cc2a7

Amit Kapila pushed:

Move index vacuum routines to vacuum.c. An upcoming patch moves parallel vacuum code out of vacuumlazy.c. This code restructuring will allow both lazy vacuum and parallel vacuum to use index vacuum functions. Author: Masahiko Sawada Reviewed-by: Hou Zhijie, Amit Kapila Discussion: /message-id/20211030212101.ae3qcouatwmy7tbr%40alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/cc8b25712b5ed8809048c7e209882bb0981214d6
Move parallel vacuum code to vacuumparallel.c. This commit moves parallel vacuum related code to a new file commands/vacuumparallel.c so that any table AM supporting indexes can utilize parallel vacuum in order to call index AM callbacks (ambulkdelete and amvacuumcleanup) with parallel workers. Another reason for this refactoring is that the parallel vacuum isn't specific to heap so it doesn't make sense to keep this code in heap/vacuumlazy.c. Author: Masahiko Sawada, based on suggestion from Andres Freund Reviewed-by: Hou Zhijie, Amit Kapila, Haiying Tang Discussion: /message-id/20211030212101.ae3qcouatwmy7tbr%40alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/8e1fae193864527c931a704bd7908e4fbc983f5c
Fix compilation error introduced by commit 8e1fae1938. Author: Masahiko Sawada Discussion: https://postgr.es/m/E1n0HSK-00048l-RE@gemulon.postgresql.org https://git.postgresql.org/pg/commitdiff/94226d4506e66d6e7cbf4b391f1e7393c1962841

Michaël Paquier pushed:

Remove assertion for ALTER TABLE .. DETACH PARTITION CONCURRENTLY. One code path related to this flavor of ALTER TABLE was checking that the relation to detach has to be a normal table or a partitioned table, which would fail if using the command with a different relation kind. Views, sequences and materialized views cannot be part of a partition tree, so these would cause the command to fail anyway, but the assertion was triggered. Foreign tables can be part of a partition tree, and again the assertion would have failed. The simplest solution is just to remove this assertion, so as we get the same failure as the non-concurrent code path. While on it, add a regression test in postgres_fdw for the concurrent partition detach of a foreign table, as per a suggestion from Alexander Lakhin. Issue introduced in 71f4c8c. Reported-by: Alexander Lakhin Author: Michael Paquier, Alexander Lakhin Reviewed-by: Peter Eisentraut, Kyotaro Horiguchi Discussion: https://postgr.es/m/17339-a9e09aaf38a3457a@postgresql.org Backpatch-through: 14 https://git.postgresql.org/pg/commitdiff/2e577c94466fde77d24cd44dc47059cf9cf392a4
Correct comment and some documentation about REPLICA_IDENTITY_INDEX. catalog/pg_class.h was stating that REPLICA_IDENTITY_INDEX with a dropped index is equivalent to REPLICA_IDENTITY_DEFAULT. The code tells a different story, as it is equivalent to REPLICA_IDENTITY_NOTHING. The behavior exists since the introduction of replica identities, and fe7fd4e even added tests for this case but I somewhat forgot to fix this comment. While on it, this commit reorganizes the documentation about replica identities on the ALTER TABLE page, and a note is added about the case of dropped indexes with REPLICA_IDENTITY_INDEX. Author: Michael Paquier, Wei Wang Reviewed-by: Euler Taveira Discussion: https://postgr.es/m/OS3PR01MB6275464AD0A681A0793F56879E759@OS3PR01MB6275.jpnprd01.prod.outlook.com Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/fc95d35b9429096ec4d028d79dcf1fb8b5d4b16e
Fix incorrect field count in pg_control_checkpoint(). 18 columns are generated in this function, but we had enough space for 19 of them. Introduced by 4b0d28d. Author: Bharath Rupireddy Reviewed-by: Justin Pryzby, Euler Taveira Discussion: https://postgr.es/m/CALj2ACVQ=hAs=sT0n4xriimqRrrgECySfg_tSqA+26Rb_yfs2A@mail.gmail.com https://git.postgresql.org/pg/commitdiff/86d9888d2ead04a1a139bbaef9d7f4648022fe4b

Bruce Momjian pushed:

doc: clarify when expression indexes evaluate their expressions. Only non-HOT updates evaluate the index expression. Reported-by: Chris Lowder Discussion: https://postgr.es/m/163967385701.26064.15365003480975321072@wrigleys.postgresql.org Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/e2e1bbde46a3509c3b7e830196f4314242925247

Fujii Masao pushed:

postgres_fdw: Allow postgres_fdw.application_name to include escape sequences. application_name that used when postgres_fdw establishes a connection to a foreign server can be specified in either or both a connection parameter of a server object and GUC postgres_fdw.application_name. This commit allows those parameters to include escape sequences that begins with % character. Then postgres_fdw replaces those escape sequences with status information. For example, %d and %u are replaced with user name and database name in local server, respectively. This feature enables us to add information more easily to track remote transactions or queries, into application_name of a remote connection. Author: Hayato Kuroda Reviewed-by: Kyotaro Horiguchi, Masahiro Ikeda, Hou Zhijie, Fujii Masao Discussion: https://postgr.es/m/TYAPR01MB5866FAE71C66547C64616584F5EB9@TYAPR01MB5866.jpnprd01.prod.outlook.com Discussion: https://postgr.es/m/TYCPR01MB5870D1E8B949DAF6D3B84E02F5F29@TYCPR01MB5870.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/6e0cb3dec10e460288d68a128e3d79d16a230cdb
postgres_fdw: Revert unstable tests for postgres_fdw.application_name. Commit 6e0cb3dec1 added the tests that check that escape sequences in postgres_fdw.application_name setting are replaced with status information expectedly. But they were unstable and caused some buildfarm members to report the failure. This commit reverts those unstable tests. https://git.postgresql.org/pg/commitdiff/5e64ad369771b66bb3e916aade735defce6e65a1

Thomas Munro pushed:

Fix overly generic name in with.sql test. Avoid the name "test". In the 10 branch, this could clash with alter_table.sql, as seen in the build farm. That other instance was already renamed in later branches by commit 2cf8c7aa, but it's good to future-proof the name here too. Back-patch to 10. Reviewed-by: Tom Lane tgl@sss.pgh.pa.us Discussion: https://postgr.es/m/CA%2BhUKGJf4RAXUyAYVUcQawcptX%3DnhEco3SYpuPK5cCbA-F1eLA%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/8112bcf0cc602e00e95eab6c4bdc0eb73b5b547d

Daniel Gustafsson pushed:

Revert b2a459edf "Fix GRANTED BY support in REVOKE ROLE statements". The reverted commit attempted to fix SQL specification compliance for the cases which 6aaaa76bb left. This however broke existing behavior which takes precedence over spec compliance so revert. The introduced tests are left after the revert since the codepath isn't well covered. Per bug report 17346. Backpatch down to 14 where it was introduced. Reported-by: Andrew Bille andrewbille@gmail.com Discussion: https://postgr.es/m/17346-f72b28bd1a341060@postgresql.org https://git.postgresql.org/pg/commitdiff/e68570e388f08c2e36ce7d2a9564941b89db6549

Álvaro Herrera pushed:

Small cleanups related to PUBLICATION framework code. Discussion: https://postgr.es/m/202112302021.ca7ihogysgh3@alvherre.pgsql https://git.postgresql.org/pg/commitdiff/c9105dd3660f4a801e6f87a1ed68189bd30576bf

Andres Freund pushed:

ci: Add continuous integration for github repositories via cirrus-ci. Currently FreeBSD, Linux, macOS and Windows (Visual Studio) are tested. The main goal of this integration is to make it easier to test in-development patches across multiple platforms. This includes improving the testing done automatically by cfbot [1] for commitfest entries. It is not the goal to supersede the buildfarm. cirrus-ci [2] was chosen because it was already in use for cfbot, allows using full VMs, has good OS coverage and allows accessing the full test results without authentication (like a github account). It might be worth adding support for further CI providers, particularly ones supporting other git forges, in the future. To keep CI times tolerable, most platforms use pre-generated images. Some platforms use containers, others use full VMs. For instructions on how to enable the CI integration in a repository and further details, see src/tools/ci/README [1] http://cfbot.cputube.org/ [2] https://cirrus-ci.org/ Author: Andres Freund andres@anarazel.de Author: Thomas Munro tmunro@postgresql.org Author: Melanie Plageman melanieplageman@gmail.com Reviewed-By: Melanie Plageman melanieplageman@gmail.com Reviewed-By: Justin Pryzby pryzby@telsasoft.com Reviewed-By: Thomas Munro tmunro@postgresql.org Reviewed-By: Peter Eisentraut peter.eisentraut@enterprisedb.com Discussion: https://postgr.es/m/20211001222752.wrz7erzh4cajvgp6@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/93d97349461347d952e8cebdf62f5aa84b4bd20a

Magnus Hagander pushed:

Fix typo. Reported-By: Eric Mutta Backpatch-through: 10 Discussion: https://postgr.es/m/164052477973.21665.7888120874624887609@wrigleys.postgresql.org https://git.postgresql.org/pg/commitdiff/69872d0bbe64fcd67c4fb4c61e5c7bf6a3443a47

PostgreSQL Weekly News - December 19, 2021

Mon, 20 Dec 2021 00:00:00 +0000

PostgreSQL Weekly News - December 19, 2021

FOSDEM PGDay 2022 will be held on line, on Feb 5-6, 2022. https://fosdem.org/2022/

A PostgreSQL Transition Guide, containing much hard-won wisdom, and available in French and English, has been published

pgDay Paris 2022 will be held in Paris, France on March 24, 2022. The CfP is open through December 31, 2021 at midnight, Paris time.

Citus Con, a virtual global developer event, is happening April 12-13, 2022. The CFP is now open.

PostgreSQL Product News

Pgpool-II 4.3.0, a connection pooler and statement replication system for PostgreSQL, released.

Access-to-PostgreSQL v2.3 released.

check_pgbackrest 2.2, a Nagios-compatible monitor for pgBackRest, released. https://github.com/dalibo/check_pgbackrest/releases

DB Comparer 5.0 for PostgreSQL released.

Database .NET v33.6, a multi-database management tool, now with support for PostgreSQL, released.

pgAdmin4 6.3, a web- and native GUI control center for PostgreSQL, released.

pgFormatter 5.2, a formatter/beautifier for SQL code, released. https://github.com/darold/pgFormatter/blob/master/ChangeLog

MySQL-to-PostgreSQL v5.5 released.

PostgreSQL Jobs for December

https://archives.postgresql.org/pgsql-jobs/2021-12/

PostgreSQL Local

Nordic PGDay 2022 will be held in Helsinki, Finland at the Hilton Helsinki Strand Hotel on March 22, 2022. The CfP is open through December 31, 2021 here

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Michaël Paquier pushed:

Improve psql tab completion for views, FDWs, sequences and transforms. The following improvements are done: - Addition of type completion for ALTER SEQUENCE AS. - Ignore ALTER for transforms, as the command is not supported. - Addition of more completion for ALTER FOREIGN DATA WRAPPER. - Addition of options related to columns in ALTER VIEW. This is a continuation of the work done in 0cd6d3b. Author: Ken Kato Discussion: https://postgr.es/m/9497ae9ca1b31eb9b1e97aded1c2ab07@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/f44ceb46ec2d8da48f6e145bf462d5620c25e079
Centralize timestamp computation of control file on updates. This commit moves the timestamp computation of the control file within the routine of src/common/ in charge of updating the backend's control file, which is shared by multiple frontend tools (pg_rewind, pg_checksums and pg_resetwal) and the backend itself. This change has as direct effect to update the control file's timestamp when writing the control file in pg_rewind and pg_checksums, something that is helpful to keep track of control file updates for those operations, something also tracked by the backend at startup within its logs. This part is arguably a bug, as ControlFileData->time should be updated each time a new version of the control file is written, but this is a behavior change so no backpatch is done. Author: Amul Sul Reviewed-by: Nathan Bossart, Michael Paquier, Bharath Rupireddy Discussion: https://postgr.es/m/CAAJ_b97nd_ghRpyFV9Djf9RLXkoTbOUqnocq11WGq9TisX09Fw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/6fb7c5d67cdd55454fe6317f939a191f85e96473
Fix compatibility thinko for fstat() on standard streams in win32stat.c. GetFinalPathNameByHandleA() cannot be used in compilation environments where _WIN32_WINNT < 0x0600, meaning at least Windows XP used by some buildfarm members under MinGW that Postgres still needs to support. This was reported as a compilation warning by the buildfarm, but this is actually worse than the report as the code would have not worked. Instead, this switches to GetFileInformationByHandle() that is able to fail for standard streams and succeed for redirected ones, which is what we are looking for herein the code emulating fstat(). We also know that it is able to work in all the environments still supported, thanks to the existing logic of win32stat.c. Issue introduced by 10260c7, so backpatch down to 14. Reported-by: Justin Pryzby, via buildfarm member jacana Author: Michael Paquier Reviewed-by: Juan José Santamaría Flecha Discussion: https://postgr.es/m/20211129050122.GK17618@telsasoft.com Backpatch-through: 14 https://git.postgresql.org/pg/commitdiff/58651d8dd6a56af306a361e2c386db798164c0f1
Fix typos. Author: Lingjie Qiang Discussion: https://postgr.es/m/OSAPR01MB71654E773F62AC88DC1FC8CC80669@OSAPR01MB7165.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/98105e53e0ab472b7721a3e8d7b9f1750a635120
Fix flags of some GUCs and improve some descriptions. This commit fixes some issues with GUCs: - enable_incremental_sort was not marked as GUC_EXPLAIN, causing it to not be listed in the output of EXPLAIN (SETTINGS) if using a value different than the default, contrary to the other planner-level GUCs. - trace_recovery_messages missed GUC_NOT_IN_SAMPLE, like the other developer options. - ssl_renegotiation_limit should be marked as COMPAT_OPTIONS_PREVIOUS. While on it, this fixes one incorrect comment related to autovacuum_freeze_max_age, and improves the descriptions of some other GUCs, recently introduced. Extracted from a larger patch set by the same author. Author: Justin Pryzby Description: https://postgr.es/m/20211129030833.GJ17618@telsasoft.com https://git.postgresql.org/pg/commitdiff/be5455124b0f073ba3924ae2ba302a27b1686230
Improve psql tab completion for various DROP commands. The following improvements are done: - Handling of RESTRICT/CASCADE for DROP OWNED, matviews and policies. - Handling of DROP TRANSFORM This is a continuation of the work done in 0cd6d3b and f44ceb4. Author: Ken Kato Reviewed-by: Asif Rehman Discussion: https://postgr.es/m/0fafb73f3a0c6bcec817a25ca9d5a853@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/9270778f467dad0d78d3b9e435a89a6039322b2f
Fix comment grammar in slotfuncs.c. Author: Bharath Rupireddy Discussion: https://postgr.es/m/CALj2ACUkrNR2xTak+QaqxoTjPKGn8zXWripv7SR27t+Q5qF1Wg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/7799d4e3bdd14c90989d829a9b24e73d4ff4d4ad
Move into separate file all the SQL queries used in pg_upgrade tests. The existing pg_upgrade/test.sh and the buildfarm code have been holding the same set of SQL queries when doing cross-version upgrade tests to adapt the objects created by the regression tests before the upgrade (mostly, incompatible or non-existing objects need to be dropped from the origin, perhaps re-created). This moves all those SQL queries into a new, separate, file with a set of \if clauses to handle the version checks depending on the old version of the cluster to-be-upgraded. The long-term plan is to make the buildfarm code re-use this new SQL file, so as committers are able to fix any compatibility issues in the tests of pg_upgrade with a refresh of the core code, without having to poke at the buildfarm client. Note that this is only able to handle the main regression test suite, and that nothing is done yet for contrib modules yet (these have more issues like their database names). A backpatch down to 10 is done, adapting the version checks as this script needs to be only backward-compatible, so as it becomes possible to clean up a maximum amount of code within the buildfarm client. Author: Justin Pryzby, Michael Paquier Discussion: https://postgr.es/m/20201206180248.GI24052@telsasoft.com Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/0df9641d39057f431655b92b8a490b89c508a0b3
pg_waldump: Emit stats summary when interrupted by SIGINT. Previously, pg_waldump would not display its statistics summary if it got interrupted by SIGINT (or say a simple Ctrl+C). It gains with this commit a signal handler for SIGINT, trapping the signal to exit at the earliest convenience to allow a display of the stats summary before exiting. This makes the reports more interactive, similarly to strace -c. This new behavior makes the combination of the options --stats and --follow much more useful, so as the user will get a report for any invocation of pg_waldump in such a case. Information about the LSN range of the stats computed is added as a header to the report displayed. This implementation comes from a suggestion by Álvaro Herrera and myself, following a complaint by the author of this patch about --stats and --follow not being useful together originally. As documented, this is not supported on Windows, though its support would be possible by catching the terminal events associated to Ctrl+C, for example (this may require a more centralized implementation, as other tools could benefit from a common API). Author: Bharath Rupireddy Discussion: https://postgr.es/m/CALj2ACUUx3PcK2z9h0_m7vehreZAUbcmOky9WSEpe8TofhV=PQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/f2c52eeba919a1b191f60445001371bd7c53aaa9
Improve the description of various GUCs. This commit fixes a couple of inconsistencies in the descriptions of some GUCs, while making their wording more general regarding the units they rely on. For most of them, this removes the use of terms like "N seconds" or "N bytes", which may not apply easily to all the languages these strings are translated to (from my own experience, this works in French and English, less in Japanese). Per debate between the authors listed below. Author: Justin Pryzby, Michael Paquier Discussion: https://postgr.es/m/20211129030833.GJ17618@telsasoft.com https://git.postgresql.org/pg/commitdiff/03774f9bb304d49fae3379806115aaa5d1fafea2
Fix corruption of toast indexes with REINDEX CONCURRENTLY. REINDEX CONCURRENTLY run on a toast index or a toast relation could corrupt the target indexes rebuilt, as a backend running in parallel that manipulates toast values would directly release the lock on the toast relation when its local operation is done, rather than releasing the lock once the transaction that manipulated the toast values committed. The fix done here is simple: we now hold a ROW EXCLUSIVE lock on the toast relation when saving or deleting a toast value until the transaction working on them is committed, so as a concurrent reindex happening in parallel would be able to wait for any activity and see any new rows inserted (or deleted). An isolation test is added to check after the case fixed here, which is a bit fancy by design as it relies on allow_system_table_mods to rename the toast table and its index to fixed names. This way, it is possible to reindex them directly without any dependency on the OID of the underlying relation. Note that this could not use a DO block either, as REINDEX CONCURRENTLY cannot be run in a transaction block. The test is backpatched down to 13, where it is possible, thanks to c4a7a39, to use allow_system_table_mods in a test suite. Reported-by: Alexey Ermakov Analyzed-by: Andres Freund, Noah Misch Author: Michael Paquier Reviewed-by: Nathan Bossart Discussion: https://postgr.es/m/17268-d2fb426e0895abd4@postgresql.org Backpatch-through: 12 https://git.postgresql.org/pg/commitdiff/f99870dd867331f576a84e37438da86a866559c4
Improve parsing of options of CREATE/ALTER SUBSCRIPTION. This simplifies the code so as it is not necessary anymore for the caller of parse_subscription_options() to zero SubOpts, holding a bitmaps of the provided options as well as the default/parsed option values. This also simplifies some checks related to the options supported by a command when checking for incompatibilities. While on it, the errors generated for unsupported combinations with "slot_name = NONE" are reordered. This may generate a different errors compared to the previous major versions, but users have to go through all those errors to get a correct command in this case when using incorrect values for options "enabled" and "create\slot", so at the end the resulting command would remain the same. Author: Peter Smith Reviewed-by: Nathan Bossart Discussion: https://postgr.es/m/CAHut+PtXHfLgLHDDJ8ZN5f5Be_37mJoxpEsRg8LNmm4XCr06Rw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/00029deaf65aad47044d9290fe80f2f68601f7ac
Fix some typos with {a,an}. One of the changes impacts the documentation, so backpatch. Author: Peter Smith Discussion: https://postgr.es/m/CAHut+Pu6+c+r3mY24VT7u+H+E_s6vMr5OdRiZ8NT3EOa-E5Lmw@mail.gmail.com Backpatch-through: 14 https://git.postgresql.org/pg/commitdiff/5d08137076fd39694188ec4625013756aab889e1
Improve description of some WAL records with transaction commands. This commit improves the description of some WAL records for the Transaction RMGR: - Track remote_apply for a transaction commit. This GUC is user-settable, so this information can be useful for debugging. - Add replication origin information for PREPARE TRANSACTION, with the origin ID, LSN and timestamp - Same as above, for ROLLBACK PREPARED. This impacts the format of pg_waldump or anything using these description routines, so no backpatch is done. Author: Masahiko Sawada, Michael Paquier Discussion: https://postgr.es/m/CAD21AoD2dJfgsdxk4_KciAZMZQoUiCvmV9sDpp8ZuKLtKCNXaA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/c8b733c4c4b0c5b7aa93553aa5b7f2c1d0bf00bf
Remove assertion for replication origins in PREPARE TRANSACTION. When using replication origins, pg_replication_origin_xact_setup() is an optional choice to be able to set a LSN and a timestamp to mark the origin, which would be additionally added to WAL for transaction commits or aborts (including 2PC transactions). An assertion in the code path of PREPARE TRANSACTION assumed that this data should always be set, so it would trigger when using replication origins without setting up an origin LSN. Some tests are added to cover more this kind of scenario. Oversight in commit 1eb6d65. Per discussion with Amit Kapila and Masahiko Sawada. Discussion: https://postgr.es/m/YbbBfNSvMm5nIINV@paquier.xyz Backpatch-through: 11 https://git.postgresql.org/pg/commitdiff/ece8c76192fee0b78509688325631ceabca44ff5
Adjust behavior of some env settings for the TAP tests of MSVC. edc2332 has introduced in vcregress.pl some control on the environment variables LZ4, TAR and GZIP_PROGRAM to allow any TAP tests to be able use those commands. This makes the settings more consistent with src/Makefile.global.in, as the same default gets used for Make and MSVC builds. Each parameter can be changed in buildenv.pl, but as a default gets assigned after loading buldenv.pl, it is not possible to unset any of these, and using an empty value would not work with "||=" either. As some environments may not have a compatible command in their PATH (tar coming from MinGW is an issue, for one), this could break tests without an exit path to bypass any failing test. This commit changes things so as the default values for LZ4, TAR and GZIP_PROGRAM are assigned before loading buildenv.pl, not after. This way, we keep the same amount of compatibility as a GNU build with the same defaults, and it becomes possible to unset any of those values. While on it, this adds some documentation about those three variables in the section dedicated to the TAP tests for MSVC. Per discussion with Andrew Dunstan. Discussion: https://postgr.es/m/YbGYe483803il3X7@paquier.xyz Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/7acd01015c4a5edb253ea9468ccb71ef99cabd1a
Add option -N/--no-sync to pg_upgrade. This is an option consistent with what the other tools of src/bin/ (pg_checksums, pg_dump, pg_rewind and pg_basebackup) provide which is useful for leveraging the I/O effort when testing things. This is not to be used in a production environment. All the regression tests of pg_upgrade are updated to use this new option. This happens to cut at most a couple of seconds in environments constrained on I/O, by avoiding a flush of data folder for the new cluster upgraded. Author: Michael Paquier Reviewed-by: Peter Eisentraut Discussion: https://postgr.es/m/YbrhzuBmBxS/DkfX@paquier.xyz https://git.postgresql.org/pg/commitdiff/3d5ffccb6df323f528cf870c26d0d0517ffe3eaa
Fix typo in TAP tests of pg_receivewal. Introduced in d62bcc8, noticed while hacking in the area. https://git.postgresql.org/pg/commitdiff/22592e10b41a95f841642fa16127521989177bbc

Tom Lane pushed:

Replace random(), pg_erand48(), etc with a better PRNG API and algorithm. Standardize on xoroshiro128 as our basic PRNG algorithm, eliminating a bunch of platform dependencies as well as fundamentally-obsolete PRNG code. In addition, this API replacement will ease replacing the algorithm again in future, should that become necessary. xoroshiro128 is a few percent slower than the drand48 family, but it can produce full-width 64-bit random values not only 48-bit, and it should be much more trustworthy. It's likely to be noticeably faster than the platform's random(), depending on which platform you are thinking about; and we can have non-global state vectors easily, unlike with random(). It is not cryptographically strong, but neither are the functions it replaces. Fabien Coelho, reviewed by Dean Rasheed, Aleksander Alekseev, and myself Discussion: https://postgr.es/m/alpine.DEB.2.22.394.2105241211230.165418@pseudo https://git.postgresql.org/pg/commitdiff/3804539e48e794781c6145c7f988f5d507418fa8
Portability hack for pg_global_prng_state. PGDLLIMPORT is only appropriate for variables declared in the backend, not when the variable is coming from a library included in frontend code. (This isn't a particularly nice fix, but for now, use the same method employed elsewhere.) Discussion: https://postgr.es/m/E1mrWUD-000235-Hq@gemulon.postgresql.org https://git.postgresql.org/pg/commitdiff/11b500072e42c214462b973b0b05f1c68992226b
Simplify declaring variables exported from libpgcommon and libpgport. This reverts commits c2d1eea9e and 11b500072, as well as similar hacks elsewhere, in favor of setting up the PGDLLIMPORT macro so that it can just be used unconditionally. That can work because in frontend code, we need no marking in either the defining or consuming files for a variable exported from these libraries; and frontend code has no need to access variables exported from the core backend, either. While at it, write some actual documentation about the PGDLLIMPORT and PGDLLEXPORT macros. Patch by me, based on a suggestion from Robert Haas. Discussion: https://postgr.es/m/1160385.1638165449@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/e04a8059a74c125a8af94acdcb7b15b92188470a
Doc: improve documentation about ORDER BY in matviews. Remove the confusing use of ORDER BY in an example materialized view. It adds nothing to the example, but might encourage people to follow bad practice. Clarify REFRESH MATERIALIZED VIEW's note about whether view ordering is retained (it isn't). Maciek Sakrejda Discussion: https://postgr.es/m/CAOtHd0D-OvrUU0C=4hX28p4BaSE1XL78BAQ0VcDaLLt8tdUzsg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/4f33af23e7e3ac30b3cb9480981c3accf401ef01
Cope with cross-compiling when checking for a random-number source. Commit 16f96c74d neglected to consider the possibility of cross-compiling, causing cross-compiles to fail at the configure stage unless you'd selected --with-openssl. Since we're now more or less assuming that /dev/urandom is available everywhere, it seems reasonable to assume that the cross-compile target has it too, rather than failing. Per complaint from Vincas Dargis. Back-patch to v14 where this came in. Discussion: https://postgr.es/m/0dc14a31-acaf-8cae-0df4-a87339b22bd9@gmail.com https://git.postgresql.org/pg/commitdiff/b637101644aa84dccc7da4f30bad40452939b57a
psql: include intra-query "--" comments in what's sent to the server. psql's lexer has historically deleted dash-dash (single-line) comments from what's collected and sent to the server. This is inconsistent with what it does for slash-star comments, and people have complained before that they wish such comments would be captured in the server log. Undoing the decision completely seems like too big a behavioral change, however. In particular, comments on lines preceding the start of a query are generally not thought of as being part of that query. What we can do to improve the situation is to capture comments that are clearly within a query, that is after the first non-whitespace, non-comment token but before the query's ending semicolon or backslash command. This is a nearly trivial code change, and it affects only a few regression test results. (It is tempting to try to apply the same rule to slash-star comments. But it's hard to see how to do that without getting strange history behavior for comments that cross lines, especially if the user then starts a new query on the same line as the star-slash. In view of the lack of complaints, let's leave that case alone.) Discussion: https://postgr.es/m/CAJcOf-cAdMVr7azeYR7nWKsNp7qhORzc84rV6d7m7knG5Hrtsw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/83884682f4df96184549b91869a1cf79dafb4f94
psql: treat "--" comments between queries as separate history entries. If we've not yet collected any non-whitespace, non-comment token for a new query, flush the current input line to history before reading another line. This aligns psql's history behavior with the observation that lines containing only comments are generally not thought of as being part of the next query. psql's prompting behavior is consistent with that view, too, since it won't change the prompt until you enter something that's neither whitespace nor a "--" comment. Greg Nancarrow, simplified a bit by me Discussion: https://postgr.es/m/CAJcOf-cAdMVr7azeYR7nWKsNp7qhORzc84rV6d7m7knG5Hrtsw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/c2f654930e9f8119b9ed12caab6192d0aafe5ebd
psql: initialize comment-begin setting to a useful value by default. Readline's meta-# command is supposed to insert a comment marker at the start of the current line. However, the default marker is "#" which is entirely unhelpful for SQL. Set it to "-- " instead. (This setting can still be overridden in one's ~/.inputrc file, so this change won't affect people who have already taken steps to make the command useful.) Discussion: https://postgr.es/m/CAJcOf-cAdMVr7azeYR7nWKsNp7qhORzc84rV6d7m7knG5Hrtsw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/3d858af07ee67efda3778bdd655852afabf4a125
Avoid leaking memory during large-scale REASSIGN OWNED BY operations. The various ALTER OWNER routines tend to leak memory in CurrentMemoryContext. That's not a problem when they're only called once per command; but in this usage where we might be touching many objects, it can amount to a serious memory leak. Fix that by running each call in a short-lived context. (DROP OWNED BY likely has a similar issue, except that you'll probably run out of lock table space before noticing. REASSIGN is worth fixing since for most non-table object types, it won't take any lock.) Back-patch to all supported branches. Unfortunately, in the back branches this helps to only a limited extent, since the sinval message queue bloats quite a lot in this usage before commit 3aafc030a, consuming memory more or less comparable to what's actually leaked. Still, it's clearly a leak with a simple fix, so we might as well fix it. Justin Pryzby, per report from Guillaume Lelarge Discussion: https://postgr.es/m/CAECtzeW2DAoioEGBRjR=CzHP6TdL=yosGku8qZxfX9hhtrBB0Q@mail.gmail.com https://git.postgresql.org/pg/commitdiff/babe545caeba4c62feb3030940d93432721eea57
Add configure probe for rl_variable_bind(). Some exceedingly ancient readline libraries lack this function, causing commit 3d858af07 to fail. Per buildfarm (via Michael Paquier). Discussion: https://postgr.es/m/E1msTLm-0007Cm-Ri@gemulon.postgresql.org https://git.postgresql.org/pg/commitdiff/a7da41981021575e2359683d994eec6c9d246321
On Windows, close the client socket explicitly during backend shutdown. It turns out that this is necessary to keep Winsock from dropping any not-yet-sent data, such as an error message explaining the reason for process termination. It's pretty weird that the implicit close done by the kernel acts differently from an explicit close, but it's hard to argue with experimental results. Independently submitted by Alexander Lakhin and Lars Kanis (comments by me, though). Back-patch to all supported branches. Discussion: https://postgr.es/m/90b34057-4176-7bb0-0dbb-9822a5f6425b@greiz-reinsdorf.de Discussion: https://postgr.es/m/16678-253e48d34dc0c376@postgresql.org https://git.postgresql.org/pg/commitdiff/6051857fc953a62db318329c4ceec5f9668fd42a
Refactor pg_dump's tracking of object components to be dumped. Split the DumpableObject.dump bitmask field into separate bitmasks tracking which components are requested to be dumped (in the existing "dump" field) and which components exist for the particular object (in the new "components" field). This gets rid of some klugy and easily-broken logic that involved setting bits and later clearing them. More importantly, it restores the originally intended behavior that pg_dump's secondary data-gathering queries should not be executed for objects we have no interest in dumping. That optimization got broken when the dump flag was turned into a bitmask, because irrelevant bits tended to remain set in many cases. Since the "components" field starts from a minimal set of bits and is added onto as needed, ANDing it with "dump" provides a reliable indicator of what we actually have to dump, without having to complicate the logic that manages the request bits. This makes a significant difference in the number of queries needed when, for example, there are many functions in extensions. Discussion: https://postgr.es/m/2273648.1634764485@sss.pgh.pa.us Discussion: https://postgr.es/m/7d7eb6128f40401d81b3b7a898b6b4de@W2012-02.nidsa.loc https://git.postgresql.org/pg/commitdiff/5209c0ba0bfd16f23e38f707e487c0626e70564c
Rethink pg_dump's handling of object ACLs. Throw away most of the existing logic for this, as it was very inefficient thanks to expensive sub-selects executed to collect ACL data that we very possibly would have no interest in dumping. Reduce the ACL handling in the initial per-object-type queries to be just collection of the catalog ACL fields, as it was originally. Fetch pg_init_privs data separately in a single scan of that catalog, and do the merging calculations on the client side. Remove the separate code path used for pre-9.6 source servers; there is no good reason to treat them differently from newer servers that happen to have empty pg_init_privs. Discussion: https://postgr.es/m/2273648.1634764485@sss.pgh.pa.us Discussion: https://postgr.es/m/7d7eb6128f40401d81b3b7a898b6b4de@W2012-02.nidsa.loc https://git.postgresql.org/pg/commitdiff/0c9d84427f441602425b0e18be5cfe751d1d8ebe
Postpone calls of unsafe server-side functions in pg_dump. Avoid calling pg_get_partkeydef(), pg_get_expr(relpartbound), and regtypeout until we have lock on the relevant tables. The existing coding is at serious risk of failure if there are any concurrent DROP TABLE commands going on --- including drops of other sessions' temp tables. Arguably this is a bug fix that should be back-patched, but it's moderately invasive and we've not had all that many complaints about such failures. Let's just put it in HEAD for now. Discussion: https://postgr.es/m/2273648.1634764485@sss.pgh.pa.us Discussion: https://postgr.es/m/7d7eb6128f40401d81b3b7a898b6b4de@W2012-02.nidsa.loc https://git.postgresql.org/pg/commitdiff/e3fcbbd623b9ccc16cdbda374654d91a4727d173
Avoid per-object queries in performance-critical paths in pg_dump. Instead of issuing a secondary data-collection query against each table to be dumped, issue just one query, with a WHERE clause restricting it to be applied to only the tables we intend to dump. Likewise for indexes, constraints, and triggers. This greatly reduces the number of queries needed to dump a database containing many tables. It might seem that WHERE clauses listing many target OIDs could be inefficient, but at least on recent server versions this provides a very substantial speedup. (In principle the same thing could be done with other object types such as functions; but that would require significant refactoring of pg_dump, so those will be tackled in a different way in a following patch.) The new WHERE clauses depend on the unnest() function, which is only present in 8.4 and above. We could implement them differently for older servers, but there is an ongoing discussion that will probably result in dropping pg_dump support for servers before 9.2, so that seems like it'd be wasted work. For now, just bump the server version check to require >= 8.4, without stopping to remove any of the code that's thereby rendered dead. We'll mop that situation up soon. Patch by me, based on an idea from Andres Freund. Discussion: https://postgr.es/m/7d7eb6128f40401d81b3b7a898b6b4de@W2012-02.nidsa.loc https://git.postgresql.org/pg/commitdiff/9895961529ef8ff3fc12b39229f9a93e08bca7b7
Use PREPARE/EXECUTE for repetitive per-object queries in pg_dump. For objects such as functions, pg_dump issues the same secondary data-collection query against each object to be dumped. This can't readily be refactored to avoid the repetitive queries, but we can PREPARE these queries to reduce planning costs. This patch applies the idea to functions, aggregates, operators, and data types. While it could be carried further, the remaining sorts of objects aren't likely to appear in typical databases enough times to be worth worrying over. Moreover, doing the PREPARE is likely to be a net loss if there aren't at least some dozens of objects to apply the prepared query to. Discussion: https://postgr.es/m/7d7eb6128f40401d81b3b7a898b6b4de@W2012-02.nidsa.loc https://git.postgresql.org/pg/commitdiff/be85727a3df743a1f7e93b41dd7ac2b667e8ae04
Account for TOAST data while scheduling parallel dumps. In parallel mode, pg_dump tries to order the table-data-dumping jobs with the largest tables first. However, it was only consulting the pg_class.relpages value to determine table size. This ignores TOAST data, and so we could make poor scheduling decisions in cases where some large tables are mostly TOASTed data while others have very little. To fix, add in the relpages value for the TOAST table as well. This patch also fixes a potential integer-overflow issue that could result in poor scheduling on machines where off_t is only 32 bits wide. Such platforms are probably extinct in the wild, but we do still nominally support them, so repair. Per complaint from Hans Buschmann. Discussion: https://postgr.es/m/7d7eb6128f40401d81b3b7a898b6b4de@W2012-02.nidsa.loc https://git.postgresql.org/pg/commitdiff/65aaed22a849c0763f38f81338a1cad04ffc0e2c
On Windows, also call shutdown() while closing the client socket. Further experimentation shows that commit 6051857fc is not sufficient when using (some versions of?) OpenSSL. The reason is obscure, but calling shutdown(socket, SD_SEND) improves matters. Per testing by Andrew Dunstan and Alexander Lakhin. Back-patch as before. Discussion: https://postgr.es/m/af5e0bf3-6a61-bb97-6cba-061ddf22ff6b@dunslane.net https://git.postgresql.org/pg/commitdiff/ed52c3707bcf8858defb0d9de4b55f5c7f18fed7
Doc: improve xfunc-c-type-table. List types numeric and timestamptz, which don't seem to have ever been included here. Restore bigint, which was no-doubt-accidentally deleted in v12. Fix some errors, or at least obsolete usages (nobody declares float arguments as "float8*" anymore, even though they might be that under the hood). Re-alphabetize. Remove the seeming claim that this is a complete list of built-in types. Per question from Oskar Stenberg. Discussion: https://postgr.es/m/HE1PR03MB2971DE2527ECE1E99D6C19A8F96E9@HE1PR03MB2971.eurprd03.prod.outlook.com https://git.postgresql.org/pg/commitdiff/6f0e6ab04de5f52e4e0872d3ace2bb6a35e8b0b1
Create a new type category for "internal use" types. Historically we've put type "char" into the S (String) typcategory, although calling it a string is a stretch considering it can only store one byte. (In our actual usage, it's more like an enum.) This choice now seems wrong in view of the special heuristics that parse_func.c and parse_coerce.c have for TYPCATEGORY_STRING: it's not a great idea for "char" to have those preferential casting behaviors. Worse than that, recent patches inventing special-purpose types like pg_node_tree have assigned typcategory S to those types, meaning they also get preferential casting treatment that's designed on the assumption that they can hold arbitrary text. To fix, invent a new category TYPCATEGORY_INTERNAL for internal-use types, and assign that to all these types. I used code 'Z' for lack of a better idea ('I' was already taken). This change breaks one query in psql/describe.c, which now needs to explicitly cast a catalog "char" column to text before concatenating it with an undecorated literal. Also, a test case in contrib/citext now needs an explicit cast to convert citext to "char". Since the point of this change is to not have "char" be a surprisingly-available cast target, these breakages seem OK. Per report from Ian Campbell. Discussion: https://postgr.es/m/2216388.1638480141@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/07eee5a0dc642d26f44d65c4e6263304208e8583
Implement poly_distance(). geo_ops.c contains half a dozen functions that are just stubs throwing ERRCODE_FEATURE_NOT_SUPPORTED. Since it's been like that for more than twenty years, there's clearly not a lot of interest in filling in the stubs. However, I'm uncomfortable with deleting poly_distance(), since every other geometric type supports a distance-to-another-object- of-the-same-type function. We can easily add this capability by cribbing from poly_overlap() and path_distance(). It's possible that the (existing) test case for this will show some numeric instability, but hopefully the buildfarm will expose it if so. In passing, improve the documentation to try to explain why polygons are distinct from closed paths in the first place. Discussion: https://postgr.es/m/3426566.1638832718@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/c5c192d7bdfa78f19e735610488b1cc5ad6e41cc
Doc: de-document unimplemented geometric operators. In commit 791090bd7, I made an effort to fill in documentation for all geometric operators listed in pg_operator. However, it now appears that at least some of the omissions may have been intentional, because some of those operator entries point at unimplemented stub functions. Remove those from the docs again. (In HEAD, poly_distance stays, because c5c192d7b just added an implementation for it.) Per complaint from Anton Voloshin. Discussion: https://postgr.es/m/3426566.1638832718@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/922b23c13be075595c2abc67736b214cb90f84d9
Remove unimplemented/undocumented geometric functions & operators. Nobody has filled in these stubs for upwards of twenty years, so it's time to drop the idea that they might get implemented any day now. The associated pg_operator and pg_proc entries are just confusing wastes of space. Per complaint from Anton Voloshin. Discussion: https://postgr.es/m/3426566.1638832718@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/189699dd3680d85c74c3886b33d9a9f83301defd
Fix datatype confusion in logtape.c's right_offset(). This could only matter if (a) long is wider than int, and (b) the heap of free blocks exceeds UINT_MAX entries, which seems pretty unlikely. Still, it's a theoretical bug, so backpatch to v13 where the typo came in (in commit c02fdc922). In passing, also make swap_nodes() use consistent datatypes. Ma Liangzhu Discussion: https://postgr.es/m/17336-fc4e522d26a750fd@postgresql.org https://git.postgresql.org/pg/commitdiff/2de3c1015cb2556af501c630b1768a20f111fe95
Improve sift up/down code in binaryheap.c and logtape.c. Borrow the logic that's long been used in tuplesort.c: instead of physically swapping the data in two heap entries, keep the value that's being sifted up or down in a local variable, and just move the other values as necessary. This makes the code shorter as well as faster. It's not clear that any current callers are really time-critical enough to notice, but we might as well code heap maintenance the same way everywhere. Ma Liangzhu and Tom Lane Discussion: https://postgr.es/m/17336-fc4e522d26a750fd@postgresql.org https://git.postgresql.org/pg/commitdiff/a2ff18e89ff8f29677084bffd1e3de9ca6cd7224
Remove pg_dump/pg_dumpall support for dumping from pre-9.2 servers. Per discussion, we'll limit support for old servers to those branches that can still be built easily on modern platforms, which as of now is 9.2 and up. Remove over a thousand lines of code dedicated to dumping from older server versions. (As in previous changes of this sort, we aren't removing pg_restore's ability to read older archive files ... though it's fair to wonder how that might be tested nowadays.) This cleans up some dead code left behind by commit 989596152. Discussion: https://postgr.es/m/2923349.1634942313@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/30e7c175b81d53c0f60f6ad12d1913a6d7d77008
Remove pg_upgrade support for upgrading from pre-9.2 servers. Per discussion, we'll limit support for old servers to those branches that can still be built easily on modern platforms, which as of now is 9.2 and up. Discussion: https://postgr.es/m/2923349.1634942313@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/e469f0aaf3c586c8390bd65923f97d4b1683cd9f
Remove pg_dump's --no-synchronized-snapshots switch. Server versions for which there was a plausible reason to use this switch are all out of support now. Leaving it around would accomplish little except to let careless DBAs shoot themselves in the foot. Discussion: https://postgr.es/m/556122.1639520324@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/2a712066d0587f65fcecd44e884dc6a09958dbdd
Always use ReleaseTupleDesc after lookup_rowtype_tupdesc et al. The API spec for lookup_rowtype_tupdesc previously said you could use either ReleaseTupleDesc or DecrTupleDescRefCount. However, the latter choice means the caller must be certain that the returned tupdesc is refcounted. I don't recall right now whether that was always true when this spec was written, but it's certainly not always true since we introduced shared record typcaches for parallel workers. That means that callers using DecrTupleDescRefCount are dependent on typcache behavior details that they probably shouldn't be. Hence, change the API spec to say that you must call ReleaseTupleDesc, and fix the half-dozen callers that weren't. AFAICT this is just future-proofing, there's no live bug here. So no back-patch. Per gripe from Chapman Flack. Discussion: https://postgr.es/m/61B901A4.1050808@anastigmatix.net https://git.postgresql.org/pg/commitdiff/bbc227e951ecc59a29205be4952a623e7d1dd534
Clean up some more freshly-dead code in pg_dump and pg_upgrade. I missed a few things in 30e7c175b and e469f0aaf, as noted by Justin Pryzby. Discussion: https://postgr.es/m/2923349.1634942313@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/c49d926833fa6a987e3f9a66027f4a01d7a008be
Remove psql support for server versions preceding 9.2. Per discussion, we'll limit support for old servers to those branches that can still be built easily on modern platforms, which as of now is 9.2 and up. Aside from removing code that is dead per the assumption of server >= 9.2, I tweaked the startup warning for unsupported versions to complain about too-old servers as well as too-new ones. The warning that "Some psql features might not work" applies precisely to both cases. Discussion: https://postgr.es/m/2923349.1634942313@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/cf0cab868aa4758b7eec5f9412f2ec74acda7f45
Ensure casting to typmod -1 generates a RelabelType. Fix the code changed by commit 5c056b0c2 so that we always generate RelabelType, not something else, for a cast to unspecified typmod. Otherwise planner optimizations might not happen. It appears we missed this point because the previous experiments were done on type numeric: the parser undesirably generates a call on the numeric() length-coercion function, but then numeric_support() optimizes that down to a RelabelType, so that everything seems fine. It misbehaves for types that have a non-optimized length coercion function, such as bpchar. Per report from John Naylor. Back-patch to all supported branches, as the previous patch eventually was. Unfortunately, that no longer includes 9.6 ... we really shouldn't put this type of change into a nearly-EOL branch. Discussion: https://postgr.es/m/CAFBsxsEfbFHEkouc+FSj+3K1sHipLPbEC67L0SAe-9-da8QtYg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/9c356f4b2dd8f8ff49757287e387ab1d023e4449
Fix the public schema's permissions in a separate test script. In the wake of commit b073c3ccd, it's necessary to grant create permissions on the public schema to PUBLIC to get many of the core regression test scripts to pass. That commit did so via the quick-n-dirty expedient of adding the GRANT to the tablespace test, which runs first. This is problematic for single-machine replication testing, though. The least painful way to run the regression tests on such a setup is to skip the tablespace test, and that no longer works. To fix, let's invent a separate "test_setup" script to run first, and put the GRANT there. Revert b073c3ccd's changes to the tablespace.source files. In the future it might be good to try to reduce coupling between the various test scripts by having test_setup create widely-used objects, with the goal that most of the scripts could run after having run only test_setup. That's going to take some effort, so this commit just addresses my immediate pain point. Discussion: https://postgr.es/m/1363170.1639763559@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/944dc45d1b633c4d612cdff9f15153ed609eaa35
Remove some more dead code in pg_dump. Coverity complained that parts of dumpFunc() and buildACLCommands() were now unreachable, as indeed they are. Remove 'em. In passing, make dumpFunc's handling of protrftypes less gratuitously different from other fields. https://git.postgresql.org/pg/commitdiff/b1c169caf0678a82cf26b5656e01399f6153456b

Peter Geoghegan pushed:

vacuumlazy.c: Rename dead_tuples to dead_items. Commit 8523492d simplified what it meant for an item to be considered "dead" to VACUUM: TIDs collected in memory (in preparation for index vacuuming) must always come from LP_DEAD stub line pointers in heap pages, found following pruning. This formalized the idea that index vacuuming (and heap vacuuming) are optional processes. Unlike pruning, they can be delayed indefinitely, without any risk of that violating fundamental invariants. For example, leaving LP_DEAD items behind clearly won't add to the risk of transaction ID wraparound. You can't have transaction ID wraparound without transaction IDs. Renaming anything that references DEAD tuples (tuples with storage) reinforces all this. Code outside vacuumlazy.c continues to fudge the distinction between dead/deleted tuples, and LP_DEAD items. This is necessary because autovacuum scheduling is still mostly driven by "dead items/tuples" statistics. In the future we may find it useful to replace this model with something more sophisticated, as a step towards teaching autovacuum to perform more frequent vacuuming that targeting individual indexes that happen to be more prone to becoming bloated through version churn. In passing, simplify some function signatures that deal with VACUUM's dead_items array. Author: Peter Geoghegan pg@bowt.ie Reviewed-By: Masahiko Sawada sawada.mshk@gmail.com Discussion: https://postgr.es/m/CAH2-WzktGBg4si6DEdmq3q6SoXSDqNi6MtmB8CmmTmvhsxDTLA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/4f8d9d1217956798e761491d236af576b27f5e12
vacuumlazy.c: fix remaining "dead tuple" references. Oversight in commit 4f8d9d12. Reported-By: Masahiko Sawada sawada.mshk@gmail.com Discussion: https://postgr.es/m/CAD21AoDm38Em0bvRqeQKr4HPvOj65Y8cUgCP4idMk39iaLrxyw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/4bdfe6855901a4104dbdac2be53d465b626e244d
Standardize cleanup lock terminology. The term "super-exclusive lock" is a synonym for "buffer cleanup lock" that first appeared in nbtree many years ago. Standardize things by consistently using the term cleanup lock. This finishes work started by commit 276db875. There is no good reason to have two terms. But there is a good reason to only have one: to avoid confusion around why VACUUM acquires a full cleanup lock (not just an ordinary exclusive lock) in index AMs, during ambulkdelete calls. This has nothing to do with protecting the physical index data structure itself. It is needed to implement a locking protocol that ensures that TIDs pointing to the heap/table structure cannot get marked for recycling by VACUUM before it is safe (which is somewhat similar to how VACUUM uses cleanup locks during its first heap pass). Note that it isn't strictly necessary for index AMs to implement this locking protocol -- several index AMs use an MVCC snapshot as their sole interlock to prevent unsafe TID recycling. In passing, update the nbtree README. Cleanly separate discussion of the aforementioned index vacuuming locking protocol from discussion of the "drop leaf page pin" optimization added by commit 2ed5b87f. We now structure discussion of the latter by describing how individual index scans may safely opt out of applying the standard locking protocol (and so can avoid blocking progress by VACUUM). Also document why the optimization is not safe to apply during nbtree index-only scans. Author: Peter Geoghegan pg@bowt.ie Discussion: https://postgr.es/m/CAH2-WzngHgQa92tz6NQihf4nxJwRzCV36yMJO_i8dS+2mgEVKw@mail.gmail.com Discussion: https://postgr.es/m/CAH2-WzkHPgsBBvGWjz=8PjNhDefy7XRkDKiT5NxMs-n5ZCf2dA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/bcf60585e6e0e95f0b9e5d64c7a6edca99ec6e86

Amit Kapila pushed:

Add a view to show the stats of subscription workers. This commit adds a new system view pg_stat_subscription_workers, that shows information about any errors which occur during the application of logical replication changes as well as during performing initial table synchronization. The subscription statistics entries are removed when the corresponding subscription is removed. It also adds an SQL function pg_stat_reset_subscription_worker() to reset single subscription errors. The contents of this view can be used by an upcoming patch that skips the particular transaction that conflicts with the existing data on the subscriber. This view can be extended in the future to track other xact related statistics like the number of xacts committed/aborted for subscription workers. Author: Masahiko Sawada Reviewed-by: Greg Nancarrow, Hou Zhijie, Tang Haiying, Vignesh C, Dilip Kumar, Takamichi Osumi, Amit Kapila Discussion: https://postgr.es/m/CAD21AoDeScrsHhLyEPYqN3sydg6PxAPVBboK=30xJfUVihNZDA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/8d74fc96db5fd547e077bf9bf4c3b67f821d71cd
Doc: Add "Attach Partition" limitation during logical replication. ATTACHing a table into a partition tree whose root is published using a publication with publish_via_partition_root set to true does not result in the table's existing contents being replicated. This happens because subscriber doesn't consider replicating the newly attached partition as the root table is already in a 'ready' state. This behavior was introduced in PG13 (83fd4532a7) where we allowed to publish partition changes via ancestors. We can consider fixing this limitation in the future. Author: Amit Langote Reviewed-by: Hou Zhijie, Amit Kapila Backpatch-through: 13 Discussion: https://postgr.es/m/OS0PR01MB5716E97F00732B52DC2BBC2594989@OS0PR01MB5716.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/eb7828f54a44843a64a23d0891d7eb6018c0749e
Fix regression test failure caused by commit 8d74fc96db. The tests didn't considered that an error unrelated to apply changes, e.g. "replication origin with OID %d is already active ...", could occur on the table sync worker before starting to copy changes. To make the test robust we instead need to check the expected error and the source of error which will be either tablesync or apply worker. In passing remove the harmless option "streaming = off" from Create Subscription command as that is anyway the default. Per buildfarm member sidewinder. Author: Masahiko Sawada Reviewed-by: Hou Zhijie, Vignesh C, Amit Kapila Discussion: https://postgr.es/m/CAD21AoDeScrsHhLyEPYqN3sydg6PxAPVBboK=30xJfUVihNZDA@mail.gmail.com Discussion: https://postgr.es/m/E1mrtvV-0002Gz-73@gemulon.postgresql.org https://git.postgresql.org/pg/commitdiff/41e66fee051619ab84828814554f73556a958850
De-duplicate the result of pg_publication_tables view. We show duplicate values for child tables in publications that have both child and parent tables and are published with publish_via_partition_root as false which is not what the user would expect. We decided not to backpatch this as there is no user complaint about this and it doesn't seem to be a critical issue. Author: Hou Zhijie Reviewed-by: Bharath Rupireddy, Amit Langote, Amit Kapila Discussion: https://postgr.es/m/OS0PR01MB5716E97F00732B52DC2BBC2594989@OS0PR01MB5716.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/a61bff2bf479cfebda18a1655323eec1b19370de
Fix changing the ownership of ALL TABLES IN SCHEMA publication. Ensure that the new owner of ALL TABLES IN SCHEMA publication must be a superuser. The same is already ensured during CREATE PUBLICATION. Author: Vignesh C Reviewed-by: Nathan Bossart, Greg Nancarrow, Michael Paquier, Haiying Tang Discussion: https://postgr.es/m/CALDaNm0E5U-RqxFuFrkZrQeG7ae5trGa=xs=iRtPPHULtT4zOw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/1a2aaeb0db1bccd97140d479c4247127f6cb9378
Fix origin timestamp during decoding of ROLLBACK PREPARED operation. This happens because we were passing incorrect arguments to ReorderBufferFinishPrepared(). Author: Masahiko Sawada Reviewed-by: Vignesh C Backpatch-through: 14 Discussion: https://postgr.es/m/CAD21AoBqhUqgDZUhUVnnwKRubPDNJ6m6fJDPgok3E5cWJLL+pA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/e464cb7af317e216fef9bfe19a7c4df542817012
Fix double publish of child table's data. We publish the child table's data twice for a publication that has both child and parent tables and is published with publish_via_partition_root as true. This happens because subscribers will initiate synchronization using both parent and child tables, since it gets both as separate tables in the initial table list. Ensure that pg_publication_tables returns only parent tables in such cases. Author: Hou Zhijie Reviewed-by: Greg Nancarrow, Amit Langote, Vignesh C, Amit Kapila Backpatch-through: 13 Discussion: https://postgr.es/m/OS0PR01MB57167F45D481F78CDC5986F794B99@OS0PR01MB5716.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/5e97905a2c764d4ca36f5c6cccd0ebbf157b9df4
Improve parallel vacuum implementation. Previously, in parallel vacuum, we allocated shmem area of IndexBulkDeleteResult only for indexes where parallel index vacuuming is safe and had null-bitmap in shmem area to access them. This logic was too complicated with a small benefit of saving only a few bits per indexes. In this commit, we allocate a dedicated shmem area for the array of LVParallelIndStats that includes a parallel-safety flag, the index vacuum status, and IndexBulkdeleteResult. There is one array element for every index, even those indexes where parallel index vacuuming is unsafe or not worthwhile. This commit makes the code clear by removing all bitmap-related code. Also, add the check each index vacuum status after parallel index vacuum to make sure that all indexes have been processed. Finally, rename parallel vacuum functions to parallel_vacuum_* for consistency. Author: Masahiko Sawada, based on suggestions by Andres Freund Reviewed-by: Hou Zhijie, Amit Kapila Discussion: /message-id/20211030212101.ae3qcouatwmy7tbr%40alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/22bd3cbe0c284758d7174321f5596763095cdd55

Daniel Gustafsson pushed:

Extend configure_test_server_for_ssl to add extensions. In order to be able to test extensions with SSL connections, allow configure_test_server_for_ssl to create any extensions passed as an array. Each extension is created in all the test databases. Reviewed-by: Tom Lane tgl@sss.pgh.pa.us Reviewed-by: Andrew Dunstan andrew@dunslane.net Reviewed-by: Dagfinn Ilmari Mannsåker ilmari@ilmari.org Discussion: https://postgr.es/m/E23F9811-0C77-45DA-912F-D809AB140741@yesql.se https://git.postgresql.org/pg/commitdiff/879fc1a579cc2e2e1dbb79686668b4de2071ab83
Add TAP tests for contrib/sslinfo. This adds rudimentary coverage of the sslinfo extension into the SSL test harness. The output is validated by comparing with pg_stat_ssl to provide some level of test stability should the underlying certs be slightly altered. A new cert is added to provide an extension to test against. Reviewed-by: Tom Lane tgl@sss.pgh.pa.us Reviewed-by: Andrew Dunstan andrew@dunslane.net Reviewed-by: Dagfinn Ilmari Mannsåker ilmari@ilmari.org Discussion: https://postgr.es/m/E23F9811-0C77-45DA-912F-D809AB140741@yesql.se https://git.postgresql.org/pg/commitdiff/ae81776a23f78babc9707e22f95dea15aa2dbcd2
Use test-specific temp path for keys during SSL test. The SSL and SCRAM TAP test suites both use temporary copies of the supplied test keys in order to ensure correct permissions. These were however copied inside the tree using temporary filenames rather than a true temporary folder. Fix by using tmp_check supplied by PostgreSQL::Test::Utils. Spotted by Tom Lane during review of the nearby sslinfo TAP test patch. Reviewed-by: Tom Lane tgl@sss.pgh.pa.us Discussion: https://postgr.es/m/599244.1638041239@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/c113d8ad50d62bfcc16bbd5ceec91122e0046ede
Remove PF_USED_FOR_ASSERTS_ONLY from variables in general use. fsstate in process_pending_requests (in postgres_fdw.c) was added in 8998e3cafa2 as an assertion-only variable, 1ec7fca8592 stated using the variable outside of assertions. rd_index in get_index_column_opclass (in lsyscache.c) was introduced in 2a6368343ff, and then promptly used in the fix commit 7e041603904 shortly thereafter. This removes the PG_USED_FOR_ASSERTS_ONLY variable decoration from the above mentioned variables. Reviewed-by: Greg Nancarrow gregn4422@gmail.com Discussion: https://postgr.es/m/F959106C-0F21-43A5-B2AE-D007D51ACBEE@yesql.se https://git.postgresql.org/pg/commitdiff/ac0db34e0e5c7ee6f8b5c33c264de3e671fbd4f7
Disable unused-variable warning C4101 in MSVC. The C4101 warning for unused variable cannot be individually suppressed with PG_USED_FOR_ASSERTS_ONLY, and thus cause false-positive warnings for variables which are defined but only read/written in an assertion. Until a satisfactory solution for per-variable suppression like how we do for gcc and clang, disable the warning. Discussion: https://postgr.es/m/CAJcOf-c+KniGAp31pn8TC=9a-WHXpkX-3+8-2BkaCsZchhu=8w@mail.gmail.com https://git.postgresql.org/pg/commitdiff/e7122548a3f754060db1767582148b3559fe8d43
Extend the private key stat checking error handling. If the stat operation on the private key failed, the code assumed it was due to an ENOENT, which may or may not be true. Extend the check by printing a different error message on non-ENOENT errors for easier debugging. Per suggestion by Tom Lane due to an issue with the fairywren animal in the buildfarm. Discussion: https://postgr.es/m/1632478.1638305700@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/538724fc36e05339ea3734f1b886a67398fce71a
Remove mention of TimeLineID update from comments. Commit 4a92a1c3d removed the TimeLineID update from RecoveryInProgress, update comments accordingly. Author: Amul Sul sulamul@gmail.com Discussion: https://postgr.es/m/CAAJ_b96wyzs8N45jc-kYd-bTE02hRWQieLZRpsUtNbhap7_PuQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/018b800245c5d4db30d204fa42fada05a64eb287
Fix certificate paths to use perl2host. Commit c113d8ad50 moved the copying of certificates into a temporary path for the duration of the tests, instead of using the source tree. This broke the tests on msys as the absolute path wasn't adapted for the msys platform. Ensure to convert the path with perl2host before copying and passing in the connection string. While there also make certificate copying error handling uniform across all the test suites. Discussion: https://postgr.es/m/YacT3tm97xziSUFw@paquier.xyz https://git.postgresql.org/pg/commitdiff/c3b34a0ff4a00d00d6ea364c85201e155ca7ef6b
Fix path delimiters in connection string on Windows. The temporary path generated in commit c113d8ad5 cannot be passed as-is in the connection string on Windows since the path delimiting backslashes will be treated as escape characters. Fix by converting backslash to slash as in similar path usecases in other tests. Reported-by: Andres Freund andres@anarazel.de Discussion: https://postgr.es/m/20211202195130.e7pprpsx4ell22sp@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/49422ad0cc88c91a38522b2a7b222c2f2c939f82
Doc: Fix misleading wording of CRL parameters. ssl_crl_file and ssl_crl_dir are both used to for client certificate revocation, not server certificates. The description for the params could be easily misread to mean the opposite however, as evidenced by the bugreport leading to this fix. Similarly, expand sslcrl and and sslcrldir to explicitly mention server certificates. While there also mention sslcrldir where previously only sslcrl was discussed. Backpatch down to v10, with the CRL dir fixes down to 14 where they were introduced. Author: Kyotaro Horiguchi horikyota.ntt@gmail.com Reviewed-by: Peter Eisentraut peter.eisentraut@enterprisedb.com Discussion: https://postgr.es/m/20211202.135441.590555657708629486.horikyota.ntt@gmail.com Discussion: https://postgr.es/m/CABWY_HCBUCjY1EJHrEGePGEaSZ5b29apgTohCyygtsqe_ySYng@mail.gmail.com Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/fadac33bb8de1cb9005aed07cdd059ba1fa9c6f8

Álvaro Herrera pushed:

Increase size of shared memory for pg_commit_ts. Like 5364b357fb11 did for pg_commit, change the formula used to determine number of pg_commit_ts buffers, which helps performance with larger servers. Discussion: https://postgr.es/m/20210115220744.GA24457@alvherre.pgsql Reviewed-by: Noah Misch noah@leadboat.com Reviewed-by: Tomas Vondra tomas.vondra@enterprisedb.com https://git.postgresql.org/pg/commitdiff/4c83e59e01a89b0b19245b8e0317d87ae60226eb

Tomáš Vondra pushed:

Ignore BRIN indexes when checking for HOT udpates. When determining whether an index update may be skipped by using HOT, we can ignore attributes indexed only by BRIN indexes. There are no index pointers to individual tuples in BRIN, and the page range summary will be updated anyway as it relies on visibility info. This also removes rd_indexattr list, and replaces it with rd_attrsvalid flag. The list was not used anywhere, and a simple flag is sufficient. Patch by Josef Simanek, various fixes and improvements by me. Author: Josef Simanek Reviewed-by: Tomas Vondra, Alvaro Herrera Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/5753d4ee320b3f6fb2ff734667a1ce1d9d8615a1
Add bool to btree_gist documentation. Commit 57e3c516 added bool opclass to btree_gist, but update the list of data types in docs to reflect this change. Reported-by: Pavel Luzanov Discussion: https://postgr.es/m/CAE2gYzyDKJBZngssR84VGZEN=Ux=V9FV23QfPgo+7-yYnKKg4g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/4c6145b514fa62535f8a5029283de3a54d9cfd53
Move test for BRIN HOT behavior to stats.sql. The test added by 5753d4ee32 relies on statistics collector, and so it may occasionally fail when the UDP packet gets lost. Some machines may be susceptible to this, probably depending on load etc. Move the test to stats.sql, which is known to already have this issue and people know to ignore it. Reported-by: Justin Pryzby Discussion: https://postgr.es/m/CAFp7QwpMRGcDAQumN7onN9HjrJ3u4X3ZRXdGFT0K5G2JWvnbWg%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/fe60b67250a31cd1ac2a4882f12e199e30abd316

Peter Eisentraut pushed:

doc: Some additional information about when to use referential actions. https://git.postgresql.org/pg/commitdiff/5786fe154b53caef8b226ed863312d3608b32a51
Warning on SET of nonexisting setting with a prefix reserved by an extension. An extension can already de facto reserve a GUC prefix using EmitWarningsOnPlaceholders(). But this was only checked against settings that exist at the time the extension is loaded (or the extension chooses to call this). No diagnostic is given when a SET command later uses a nonexisting setting with a custom prefix. With this change, EmitWarningsOnPlaceholders() saves the prefixes it reserves in a list, and SET checks when it finds a "placeholder" setting whether it belongs to a reserved prefix and issues a warning in that case. Add a regression test that checks the patch using the "plpgsql" registered prefix. Author: Florin Irion florin.irion@enterprisedb.com Discussion: /message-id/flat/CA+HEvJDhWuuTpGTJT9Tgbdzm4QS4EzPAwDBScWK18H2Q=FVJFw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/75d22069e00d638d08c04e3aba71688f3fb002ed
Improve some comments in scanner files. Reviewed-by: John Naylor john.naylor@enterprisedb.com Discussion: /message-id/flat/b239564c-cad0-b23e-c57e-166d883cb97d@enterprisedb.com https://git.postgresql.org/pg/commitdiff/fb7f70112fd80f13a8f124f51c4992fe290d3836
Remove unused includes. These haven't been needed for a long time. Reviewed-by: John Naylor john.naylor@enterprisedb.com Discussion: /message-id/flat/b239564c-cad0-b23e-c57e-166d883cb97d@enterprisedb.com https://git.postgresql.org/pg/commitdiff/89d1c15d64602b0c27ed87c717f586ddf6cf310d
pg_dump: Add missing relkind case. Checking for RELKIND_MATVIEW was forgotten in guessConstraintInheritance(). This isn't a live problem, since it is checked in flagInhTables() which relkinds can have parents, and those entries will have numParents==0 after that. But after discussion it was felt that this place should be kept consistent with flagInhTables() and flagInhAttrs(). Reviewed-by: Michael Paquier michael@paquier.xyz Discussion: /message-id/flat/a574c8f1-9c84-93ad-a9e5-65233d6fc00f@enterprisedb.com https://git.postgresql.org/pg/commitdiff/a22d6a2cb62c4fc6d7c4b077d8014fd4ffaec426
Some RELKIND macro refactoring. Add more macros to group some RELKIND_* macros: - RELKIND_HAS_PARTITIONS() - RELKIND_HAS_TABLESPACE() - RELKIND_HAS_TABLE_AM() Reviewed-by: Michael Paquier michael@paquier.xyz Reviewed-by: Alvaro Herrera alvherre@alvh.no-ip.org Discussion: /message-id/flat/a574c8f1-9c84-93ad-a9e5-65233d6fc00f%40enterprisedb.com https://git.postgresql.org/pg/commitdiff/37b2764593c073ca61c2baebd7d85666e553928b
Fix inappropriate uses of PG_GETARG_UINT32(). The chr() function used PG_GETARG_UINT32() even though the argument is declared as (signed) integer. As a result, you can pass negative arguments to this function and it internally interprets them as positive. Ultimately ends up being harmless, but it seems wrong, so fix this and rearrange the internal error checking a bit to accommodate this. Another case was in the documentation, where example code used PG_GETARG_UINT32() with an argument declared as signed integer. Reviewed-by: Nathan Bossart bossartn@amazon.com Discussion: /message-id/flat/7e43869b-d412-8f81-30a3-809783edc9a3%40enterprisedb.com https://git.postgresql.org/pg/commitdiff/e9e63b7022ddd0aaaae7cd439daa234cf9e6a21c
Update snowball. Update to snowball tag v2.2.0. Minor changes only. https://git.postgresql.org/pg/commitdiff/bba962f0c052bfab79df79ac5629eac5eab5b842
pgcrypto: Remove explicit hex encoding/decoding from tests. This was from before the hex format was available in bytea. Now we can remove the extra explicit encoding/decoding calls and rely on the default output format. Discussion: /message-id/flat/17dcb4f7-7ac1-e2b6-d5f7-2dfba06cd9ee%40enterprisedb.com https://git.postgresql.org/pg/commitdiff/814e1d9ff7a853b16a544a244bfa92e8388be248
pgrowlocks: Fix incorrect format placeholders. Transaction IDs should be printed as unsigned, similar to xidout(). https://git.postgresql.org/pg/commitdiff/254c63e9eda0b006fb61b9dc23970a6381efd061
Allow specifying column list for foreign key ON DELETE SET actions. Extend the foreign key ON DELETE actions SET NULL and SET DEFAULT by allowing the specification of a column list, like CREATE TABLE posts ( ... FOREIGN KEY (tenant_id, author_id) REFERENCES users ON DELETE SET NULL (author_id) ); If a column list is specified, only those columns are set to null/default, instead of all the columns in the foreign-key constraint. This is useful for multitenant or sharded schemas, where the tenant or shard ID is included in the primary key of all tables but shouldn't be set to null. Author: Paul Martinez paulmtz@google.com Discussion: /message-id/flat/CACqFVBZQyMYJV=njbSMxf+rbDHpx=W=B7AEaMKn8dWn9OZJY7w@mail.gmail.com https://git.postgresql.org/pg/commitdiff/d6f96ed94e73052f99a2e545ed17a8b2fdc1fb8a
pg_checksums: Fix data type. Segment numbers should be int, not BlockNumber (see also buffile.c). Likely no harm, but better for consistency. https://git.postgresql.org/pg/commitdiff/bf9a55c10729988a3c7ffbe14108e29d90283939
Simplify the general-purpose 64-bit integer parsing APIs. pg_strtouint64() is a wrapper around strtoull/strtoul/_strtoui64, but it seems no longer necessary to have this indirection. msvc/Solution.pm claims HAVE_STRTOULL, so the "MSVC only" part seems unnecessary. Also, we have code in c.h to substitute alternatives for strtoull() if not found, and that would appear to cover all currently supported platforms, so having a further fallback in pg_strtouint64() seems unnecessary. Therefore, we could remove pg_strtouint64(), and use strtoull() directly in all call sites. However, it seems useful to keep a separate notation for parsing exactly 64-bit integers, matching the type definition int64/uint64. For that, add new macros strtoi64() and strtou64() in c.h as thin wrappers around strtol()/strtoul() or strtoll()/stroull(). This makes these functions available everywhere instead of just in the server code, and it makes the function naming notably different from the pg_strtointNN() functions in numutils.c, which have a different API. Discussion: /message-id/flat/a3df47c9-b1b4-29f2-7e91-427baf8b75a3%40enterprisedb.com https://git.postgresql.org/pg/commitdiff/3c6f8c011f85df7b35c32f4ccaac5c86c9064a4a

Robert Haas pushed:

Document that tar archives are now properly terminated. Commit 5a1007a5088cd6ddf892f7422ea8dbaef362372f changed the server behavior, but I didn't notice that the existing behavior was documented, and therefore did not update the documentation. This commit does that. I chose to mention that the behavior has changed rather than just removing the reference to a deviation from a standard. It seemed like that might be helpful to tool authors. Discussion: http://postgr.es/m/CA+TgmoaYZbz0=Yk797aOJwkGJC-LK3iXn+wzzMx7KdwNpZhS5g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/81fca310b38e7808dff9c01a26564e8f2db10893
Default to log_checkpoints=on, log_autovacuum_min_duration=10m. The idea here is that when a performance problem is known to have occurred at a certain point in time, it's a good thing if there is some information available from the logs to help figure out what might have happened around that time. This change attracted an above-average amount of dissent, because it means that a server with default settings will produce some amount of log output even if nothing has gone wrong. However, by my count, the mailing list discussion had about twice as many people in favor of the change as opposed. The reasons for believing that the extra log output is not an issue in practice are: (1) the rate at which messages can be generated by this setting is bounded to one every few minutes on a properly-configured system and (2) production systems tend to have a lot more junk in the log from that due to failed connection attempts, ERROR messages generated by application activity, and the like. Bharath Rupireddy, reviewed by Fujii Masao and by me. Many other people commented on the thread, but as far as I can see that was discussion of the merits of the change rather than review of the patch. Discussion: https://postgr.es/m/CALj2ACX-rW_OeDcp4gqrFUAkf1f50Fnh138dmkd0JkvCNQRKGA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/64da07c41a8c0a680460cdafc79093736332b6cf
Remove InitXLOGAccess(). It's not great that RecoveryInProgress() calls InitXLOGAccess(), because a status inquiry function typically shouldn't have the side effect of performing initializations. We could fix that by calling InitXLOGAccess() from some other place, but instead, let's remove it altogether. One thing InitXLogAccess() did is initialize wal_segment_size, but it doesn't need to do that. In the postmaster, PostmasterMain() calls LocalProcessControlFile(), and all child processes will inherit that value -- except in EXEC_BACKEND bulds, but then each backend runs SubPostmasterMain() which also calls LocalProcessControlFile(). The other thing InitXLOGAccess() did is update RedoRecPtr and doPageWrites, but that's not critical, because all code that uses them will just retry if it turns out that they've changed. The only difference is that most code will now see an initial value that is definitely invalid instead of one that might have just been way out of date, but that will only happen once per backend lifetime, so it shouldn't be a big deal. Patch by me, reviewed by Nathan Bossart, Michael Paquier, Andres Freund, Heikki Linnakangas, and Álvaro Herrera. Discussion: http://postgr.es/m/CA+TgmoY7b65qRjzHN_tWUk8B4sJqk1vj1d31uepVzmgPnZKeLg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/fa0e03c15a9f67671f0a6e0ec66d5e2ac7119c8a

Fujii Masao pushed:

postgres_fdw: Fix unexpected reporting of empty message. pgfdw_report_error() in postgres_fdw gets a message from PGresult or PGconn to report an error received from a remote server. Previously if it could get a message from neither of them, it reported empty message unexpectedly. The cause of this issue was that pgfdw_report_error() didn't handle properly the case where no message could be obtained and its local variable message_primary was set to '\0'. This commit improves pgfdw_report_error() so that it reports the message "could not obtain ..." when it gets no message and message_primary is set to '\0'. This is the same behavior as when message_primary is NULL. dblink_res_error() in dblink has the same issue, so this commit also improves it in the same way. Back-patch to all supported branches. Author: Fujii Masao Reviewed-by: Bharath Rupireddy Discussion: https://postgr.es/m/477c16c8-7ea4-20fc-38d5-ed3a77ed616c@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/557c39bba925d553c6bb12b5e80d1964d355583b
postgres_fdw: Report warning when timeout expires while getting query result. When aborting remote transaction or sending cancel request to a remote server, postgres_fdw calls pgfdw_get_cleanup_result() to wait for the result of transaction abort query or cancel request to arrive. It fails to get the result if the timeout expires or a connection trouble happens. Previously postgres_fdw reported no warning message even when the timeout expired or a connection trouble happened in pgfdw_get_cleanup_result(). This could make the troubleshooting harder when such an event occurred. This commit makes pgfdw_get_cleanup_result() tell its caller what trouble (timeout or connection error) occurred, on failure, and also makes its caller report the proper warning message based on that information. Author: Fujii Masao Reviewed-by: Bharath Rupireddy Discussion: https://postgr.es/m/15aa988c-722e-ad3e-c936-4420c5b2bfea@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/815d61fcd485e8c60dba22988bf5a90fc12df32d
doc: Add note about postgres_fdw.application_name. postgres_fdw.application_name can be any string of any length and contain even non-ASCII characters. However when it's passed to and used as application_name in a foreign server, it's truncated to less than NAMEDATALEN characters and any characters other than printable ASCII ones in it will be replaced with question marks. This commit adds these notes into the docs. Author: Hayato Kuroda Reviewed-by: Kyotaro Horiguchi, Fujii Masao Discussion: https://postgr.es/m/TYCPR01MB5870D1E8B949DAF6D3B84E02F5F29@TYCPR01MB5870.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/58e2e6eb67fec14c793c74207407e172d7e0291d

Andrew Dunstan pushed:

Silence perl complaint in ssl test. Perl's hex() function complains if its argument contains trailing white space (or in fact anything other than hex digits), so remove the offending text. https://git.postgresql.org/pg/commitdiff/d4596a20d046e800644d6027613c6a2cb5a6c35e
Enable settings used in TAP tests for MSVC builds. Certain settings from configuration or the Makefile infrastructure are used by the TAP tests, but were not being set up by vcregress.pl. This remedies those omissions. This should increase test coverage, especially on the buildfarm. Reviewed by Noah Misch Discussion: https://postgr.es/m/17093da5-e40d-8335-d53a-2bd803fc38b0@dunslane.net Backpatch to all live branches. https://git.postgresql.org/pg/commitdiff/edc2332550b2343bc9378540e11c8aa71f513a63
Check that we have a working tar before trying to use it. Issue exposed by commit edc2332550 and the buildfarm. Backpatch to release 14 where this usage started. https://git.postgresql.org/pg/commitdiff/f920f7e799c587228227ec94356c760e3f3d5f2b
Revert "Check that we have a working tar before trying to use it". This reverts commit f920f7e799c587228227ec94356c760e3f3d5f2b. The patch in effect fixed a problem we didn't have and caused another instead. Backpatch to release 14 like original Discussion: https://postgr.es/m/3655283.1638977975@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/745b99c6444f00befae77dc69c7a63529d751daf

Thomas Munro pushed:

Check for STATUS_DELETE_PENDING on Windows. 1. Update our open() wrapper to check for NT's STATUS_DELETE_PENDING and translate it to Unix-like errors. This is done with RtlGetLastNtStatus(), which is dynamically loaded from ntdll. A new file win32ntdll.c centralizes lookup of NT functions, in case we decide to add more in the future. 2. Remove non-working code that was trying to do something similar for stat(), and just reuse the open() wrapper code. As a side effect, stat() also gains resilience against "sharing violation" errors. 3. Since stat() is used very early in process startup, remove the requirement that the Win32 signal event has been created before pgwin32_open_handle() is reached. Instead, teach pg_usleep() to fall back to a non-interruptible sleep if reached before the signal event is available. This could be back-patched, but for now it's in master only. The problem has apparently been with us for a long time and generated only a few complaints. Proposed patches trigger it more often, which led to this investigation and fix. Reviewed-by: Andres Freund andres@anarazel.de Reviewed-by: Alexander Lakhin exclusion@gmail.com Reviewed-by: Juan José Santamaría Flecha juanjo.santamaria@gmail.com Discussion: https://postgr.es/m/CA%2BhUKGJz_pZTF9mckn6XgSv69%2BjGwdgLkxZ6b3NWGLBCVjqUZA%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/e2f0f8ed251d02c1eda79e1ca3cb3db2681e7a86
Change ProcSendSignal() to take pgprocno. Instead of referring to target backends by pid, use pgprocno. This means that we don't have to scan the ProcArray and we can drop some special case code for dealing with the startup process. Discussion: https://postgr.es/m/CA%2BhUKGLYRyDaneEwz5Uya_OgFLMx5BgJfkQSD%3Dq9HmwsfRRb-w%40mail.gmail.com Reviewed-by: Soumyadeep Chakraborty soumyadeep2007@gmail.com Reviewed-by: Ashwin Agrawal ashwinstar@gmail.com Reviewed-by: Andres Freund andres@anarazel.de https://git.postgresql.org/pg/commitdiff/a13db0e16404ae532fe037071c7fe2576a1f8890

Alexander Korotkov pushed:

Fix alignment in multirange_get_range() function. The multirange_get_range() function fails when two boundaries of the same range have different alignments. Fix that by adding proper pointer alignment. Reported-by: Alexander Lakhin Discussion: https://postgr.es/m/17300-dced2d01ddeb1f2f%40postgresql.org Backpatch-through: 14 https://git.postgresql.org/pg/commitdiff/5cc9c8374093ba0e427b3309e10077708c156b6a

Andres Freund pushed:

Make PG_TEST_USE_UNIX_SOCKETS work for tap tests on windows. We need to replace windows-style \ path separators with / when putting socket directories either in postgresql.conf or libpq connection strings, otherwise they are interpreted as escapes. Author: Andres Freund andres@anarazel.de Reviewed-By: Peter Eisentraut peter.eisentraut@enterprisedb.com Discussion: https://postgr.es/m/4da250a5-4222-1522-f14d-8a72bcf7e38e@enterprisedb.com https://git.postgresql.org/pg/commitdiff/45f52709d86ceaaf282a440f6311c51fc526340b
isolationtester: append session name to application_name. When writing / debugging an isolation test it sometimes is useful to see which session holds what lock etc. To make it easier, both as part of spec files and interactively, append the session name to application_name. Since b1907d688 application_name already contains the test name, this appends the session's name to that. insert-conflict-specconflict did something like this manually, which can now be removed. As we have done lately with other test infrastructure improvements, backpatch this change, to make it easier to backpatch tests. Author: Andres Freund andres@anarazel.de Reviewed-By: Michael Paquier michael@paquier.xyz Reviewed-By: Andrew Dunstan andrew@dunslane.net Discussion: https://postgr.es/m/20211211012052.2blmzcmxnxqawd2z@alap3.anarazel.de Backpatch: 10-, to make backpatching of tests easier. https://git.postgresql.org/pg/commitdiff/3f323956128ff8589ce4d3a14e8b950837831803

PostgreSQL Weekly News - November 28, 2021

Mon, 29 Nov 2021 00:00:00 +0000

PostgreSQL Weekly News - November 28, 2021

Person of the week

PostgreSQL Jobs for November

https://archives.postgresql.org/pgsql-jobs/2021-11/

PostgreSQL Local

Nordic PGDay 2022 will be held in Helsinki, Finland at the Hilton Helsinki Strand Hotel on March 22, 2022. The CfP is open through December 31, 2021 here

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Peter Geoghegan pushed:

Remove lazy_scan_heap parallel VACUUM comment block. This doesn't belong next to very high level discussion of the tasks that lazy_scan_heap performs. There is already a similar, longer comment block at the top of vacuumlazy.c that mentions lazy_scan_heap directly. https://git.postgresql.org/pg/commitdiff/97f5aef609ce51422934b7dbdba599a7de4dbafd
Go back to considering HOT on pages marked full. Commit 2fd8685e7f simplified the checking of modified attributes that takes place within heap_update(). This included a micro-optimization affecting pages marked PD_PAGE_FULL: don't even try to use HOT to save a few cycles on determining HOT safety. The assumption was that it won't work out this time around, since it can't have worked out last time around. Remove the micro-optimization. It could only ever save cycles that are consumed by the vast majority of heap_update() calls, which hardly seems worth the added complexity. It also seems quite possible that there are workloads that will do worse over time by repeated application of the micro-optimization, despite saving some cycles on average, in the short term. Author: Peter Geoghegan pg@bowt.ie Reviewed-By: Álvaro Herrera alvherre@alvh.no-ip.org Discussion: https://postgr.es/m/CAH2-WznU1L3+DMPr1F7o2eJBT7=3bAJoY6ZkWABAxNt+-afyTA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/1a6f5a0e876306293fda697e7820b404d5b93693
Update high level vacuumlazy.c comments. Update vacuumlazy.c file header comments (as well as comments above the lazy_scan_heap function) that were largely written before the introduction of the HOT optimization, when lazy_scan_heap did far less, and didn't actually prune during its initial heap pass. Since lazy_scan_heap now outsources far more work to lower level functions, it makes sense to introduce the function by talking about the high level invariant that dictates the order in which each phase takes place. Also deemphasize the case where we run out of memory for TIDs, since delaying that discussion makes it easier to talk about issues of central importance. Finally, remove discussion of parallel VACUUM from header comments. These don't add much, and are in the wrong place. https://git.postgresql.org/pg/commitdiff/12b5ade9023f3ecaddcbc423a22dc284c91c79f6
vacuumlazy.c: prefer the term "cleanup lock". The term "super-exclusive lock" is an acceptable synonym of "cleanup lock". Even still, switching from one term to the other in the same file is confusing. Standardize on "cleanup lock" within vacuumlazy.c. Per a complaint from Andres Freund. https://git.postgresql.org/pg/commitdiff/276db875d4f9be2911582f367596d444d6986c77

Fujii Masao pushed:

Report wait events for local shell commands like archive_command. This commit introduces new wait events for archive_command, archive_cleanup_command, restore_command and recovery_end_command. Author: Fujii Masao Reviewed-by: Bharath Rupireddy, Michael Paquier Discussion: https://postgr.es/m/4ca4f920-6b48-638d-08b2-93598356f5d3@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/1b06d7bac901e5fd20bba597188bae2882bf954b

Peter Eisentraut pushed:

Add ABI extra field to fmgr magic block. This allows derived products to intentionally make their fmgr ABI incompatible, with a clean error message. Discussion: /message-id/flat/55215fda-db31-a045-d6b7-d6f2d2dc9920%40enterprisedb.com https://git.postgresql.org/pg/commitdiff/d6d1dfcc99e3dd6e70e2a7024924e491bb7a9670
Fix incorrect format placeholders. Also choose better types for the underlying variables to make this more consistent. https://git.postgresql.org/pg/commitdiff/fb5961fd13b1262df280e400645bdf4ed192f058
Remove unneeded Python includes. Inluding <compile.h> and <eval.h> has not been necessary since Python 2.4, since they are included via <Python.h>. Morever, <eval.h> is being removed in Python 3.11. So remove these includes. Reviewed-by: Tom Lane tgl@sss.pgh.pa.us Discussion: /message-id/flat/84884.1637723223%40sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/99e4d24a9d77e7bb87e15b318e96dc36651a7da2
Update comments. Various places wanted to point out that tuple descriptors don't contain the variable-length fields of pg_attribute. This started when attacl was added, but more fields have been added since, and these comments haven't been kept up to date consistently. Reword so that the purpose is clearer and we don't have to keep updating them. https://git.postgresql.org/pg/commitdiff/36cb5e7c512bef394c9288786c62ef0eb1e891ba

Álvaro Herrera pushed:

Add missing words in comment. Reported by Zhihong Yu. Discussion: https://postgr.es/m/CALNJ-vR6uZivg_XkB1zKjEXeyZDEgoYanFXB-++1kBT9yZQoUw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/67385544ce672a9a53cfd51b39c1ff9048d65585
autovacuum: Improve wording in a couple places. A few strings (one WARNING and some memory context names) in the autovacuum code were written in a world where "worker" had no other possible meaning than "autovacuum worker", but that's long time gone. Be more specific about it. Also, change the WARNING from elog() to ereport(), to add translability. Author: Bharath Rupireddy bharath.rupireddyforpostgres@gmail.com Reviewed-by: Nathan Bossart bossartn@amazon.com Reviewed-by: Justin Pryzby pryzby@telsasoft.com Reviewed-by: Kyotaro Horiguchi horikyota.ntt@gmail.com Reviewed-by: Dilip Kumar dilipbalaut@gmail.com Reviewed-by: Masahiko Sawada sawada.mshk@gmail.com Discussion: https://postgr.es/m/CALj2ACX2UHp76dqdoZq92a7v4APFuV5wJQ+AUrb+2HURrKN=NQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/042412879e35791a65509f2786b4954a273466e5
Be more specific about OOM in XLogReaderAllocate. A couple of spots can benefit from an added errdetail(), which matches what we were already doing in other places; and those that cannot withstand errdetail() can get a more descriptive primary message. Author: Bharath Rupireddy bharath.rupireddyforpostgres@gmail.com Reviewed-by: Daniel Gustafsson daniel@yesql.se Reviewed-by: Julien Rouhaud rjuju123@gmail.com Discussion: https://postgr.es/m/CALj2ACV+cX1eM03GfcA=ZMLXh5fSn1X1auJLz3yuS1duPSb9QA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/2fed48f48f7f2f7a6d6f6d020f046efe3c249828
Fix determination of broken LSN in OVERWRITTEN_CONTRECORD. In commit ff9f111bce24 I mixed up inconsistent definitions of the LSN of the first record in a page, when the previous record ends exactly at the page boundary. The correct LSN is adjusted to skip the WAL page header; I failed to use that when setting XLogReaderState->overwrittenRecPtr, so at WAL replay time VerifyOverwriteContrecord would refuse to let replay continue past that record. Backpatch to 10. 9.6 also contains this bug, but it's no longer being maintained. Discussion: https://postgr.es/m/45597.1637694259@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/44bd3ed332d6ad3207f38b3b6deb6083f0baddf5
Document units for max_slot_wal_keep_size. The doc blurb failed to mention units, as well as lacking the point about changeability. Backpatch to 13. Reviewed-by: Kyotaro Horiguchi horikyota.ntt@gmail.com Reported by: b1000101@pm.me Discussion: https://postgr.es/m/163760291192.26193.10801700492025355788@wrigleys.postgresql.org https://git.postgresql.org/pg/commitdiff/013bb6c8c0b5b0ac7948d7126685008505b3aa58
Copy-edit vacuuumdb --analyze-in-stages doc blurb. I had made a few typos, and Nikolai Berkoff made a wording change suggestion. Discussion: https://postgr.es/m/VMwe7-sGegrQPQ7fJjSCdsEbESKeJFOb6G4DFxxNrf45I7DzHio7sNUH88wWRMnAy5a5G0-FB31dxPM47ldigW6WdiCPncHgqO9bNl6F240=@pm.me https://git.postgresql.org/pg/commitdiff/dd484c97f55be8336fcb41470768c5b8ae347d13
Harden be-gssapi-common.h for headerscheck. Surround the contents with a test that the feature is enabled by configure, to silence header checking tools on systems without GSSAPI installed. Backpatch to 12, where the file appeared. Discussion: https://postgr.es/m/202111161709.u3pbx5lxdimt@alvherre.pgsql https://git.postgresql.org/pg/commitdiff/f744519326e1ce4774d0966f7848601a8327eeaa

Tom Lane pushed:

Probe $PROVE not $PERL while checking for modules needed by TAP tests. Normally "prove" and "perl" come from the same Perl installation, but we support the case where they don't (mainly because the MSys buildfarm animals need this). In that case, AX_PROG_PERL_MODULES is completely the wrong thing to use, because it's checking what "perl" has. Instead, make a little TAP test script including the required modules, and run that under "prove". We don't need ax_prog_perl_modules.m4 at all after this change, so remove it. Back-patch to all supported branches, for the buildfarm's benefit. (In v10, this also back-patches the effects of commit 264eb03aa.) Andrew Dunstan and Tom Lane, per an observation by Noah Misch Discussion: https://postgr.es/m/E1moZHS-0002Cu-Ei@gemulon.postgresql.org https://git.postgresql.org/pg/commitdiff/c4fe3199a6d65212537a59eb0d7e6fad22b9e903
Fix pg_dump --inserts mode for generated columns with dropped columns. If a table contains a generated column that's preceded by a dropped column, dumpTableData_insert failed to account for the dropped column, and would emit DEFAULT placeholder(s) in the wrong column(s). This resulted in failures at restore time. The default COPY code path did not have this bug, likely explaining why it wasn't noticed sooner. While we're fixing this, we can be a little smarter about the situation: (1) avoid unnecessarily fetching the values of generated columns, (2) omit generated columns from the output, too, if we're using --column-inserts. While these modes aren't expected to be as high-performance as the COPY path, we might as well be as efficient as we can; it doesn't add much complexity. Per report from Дмитрий Иванов. Back-patch to v12 where generated columns came in. Discussion: https://postgr.es/m/CAPL5KHrkBniyQt5e1rafm5DdXvbgiiqfEQEJ9GjtVzN71Jj5pA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/0b126c6a4b00972f2f3533e1718bbe297e2851c2
Pacify perlcritic. Per buildfarm. https://git.postgresql.org/pg/commitdiff/db3a660c6327a6df81a55c4aa86e6c0837ecd505
Adjust pg_dump's priority ordering for casts. When a stored expression depends on a user-defined cast, the backend records the dependency as being on the cast's implementation function --- or indeed, if there's no cast function involved but just RelabelType or CoerceViaIO, no dependency is recorded at all. This is problematic for pg_dump, which is at risk of dumping things in the wrong order leading to restore failures. Given the lack of previous reports, the risk isn't that high, but it can be demonstrated if the cast is used in some view whose rowtype is then used as an input or result type for some other function. (That results in the view getting hoisted into the functions portion of the dump, ahead of the cast.) A logically bulletproof fix for this would require including the cast's OID in the parsed form of the expression, whence it could be extracted by dependency.c, and then the stored dependency would force pg_dump to do the right thing. Such a change would be fairly invasive, and certainly not back-patchable. Moreover, since we'd prefer that an expression using cast syntax be equal() to one doing the same thing by explicit function call, the cast OID field would have to have special ignored-by-comparisons semantics, making things messy. So, let's instead fix this by a very simple hack in pg_dump: change the object-type priority order so that casts are initially sorted before functions, immediately after types. This fixes the problem in a fairly direct way for casts that have no implementation function. For those that do, the implementation function will be hoisted to just before the cast by the dependency sorting step, so that we still have a valid dump order. (I'm not sure that this provides a full guarantee of no problems; but since it's been like this for many years without any previous reports, this is probably enough to fix it in practice.) Per report from Дмитрий Иванов. Back-patch to all supported branches. Discussion: https://postgr.es/m/CAPL5KHoGa3uvyKp6z6m48LwCnTsK+LRQ_mcA4uKGfqAVSEjV_A@mail.gmail.com https://git.postgresql.org/pg/commitdiff/b55f2b6926556115155930c4b2d006c173f45e65
Doc: improve documentation about nextval()/setval(). Clarify that the results of nextval and setval are not guaranteed persistent until the calling transaction commits. Some people seem to have drawn the opposite conclusion from the statement that these functions are never rolled back, so re-word to avoid saying it quite that way. Discussion: https://postgr.es/m/CAKU4AWohO=NfM-4KiZWvdc+z3c1C9FrUBR6xnReFJ6sfy0i=Lw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/4ac452e2285da347c75f5960ae211e183a87b57b

Michaël Paquier pushed:

Add SQL functions to monitor the directory contents of replication slots. This commit adds a set of functions able to look at the contents of various paths related to replication slots: - pg_ls_logicalsnapdir, for pg_logical/snapshots/ - pg_ls_logicalmapdir, for pg_logical/mappings/ - pg_ls_replslotdir, for pg_replslot/<slot_name>/ These are intended to be used by monitoring tools. Unlike pg_ls_dir(), execution permission can be granted to non-superusers. Roles members of pg_monitor gain have access to those functions. Bump catalog version. Author: Bharath Rupireddy Reviewed-by: Nathan Bossart, Justin Pryzby Discussion: https://postgr.es/m/CALj2ACWsfizZjMN6bzzdxOk1ADQQeSw8HhEjhmVXn_Pu+7VzLw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/1922d7c6e1a74178bd2f1d5aa5a6ab921b3fcd34
Add support for Visual Studio 2022 in build scripts. Documentation and any code paths related to VS are updated to keep the whole consistent. Similarly to 2017 and 2019, the version of VS and the version of nmake that we use to determine which code paths to use for the build are still inconsistent in their own way. Backpatch down to 10, so as buildfarm members are able to use this new version of Visual Studio on all the stable branches supported. Author: Hans Buschmann Discussion: https://postgr.es/m/1633101364685.39218@nidsa.net Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/b2265d305d81b0c1a2cec6c5b66a190a9e69e853
Remove useless LZ4 system call on failure when writing file header. If an error occurs when writing the LZ4 file header, LZ4F_compressEnd() was called in the error code path of write(), followed by LZ4F_freeCompressionContext() to finish the cleanup. The code as-is was not broken, but the LZ4F_compressEnd() proves to not be necessary as there are no contents to flush at this stage, so remove it. Per gripe from Jeevan Ladhe and Robert Haas. Discussion: https://postgr.es/m/CAOgcT0PE33wbD7giAT1OSkNJt=p-vu8huq++qh=ny9O=SCP5aA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/f79962d8264b8d205ce45a8aa11d1b37f9592a81
Fix fstat() emulation on Windows with standard streams. The emulation of fstat() in win32stat.c caused two issues with the existing in-core callers, failing on EINVAL when using a stream as argument: - psql's \copy would crash when using a stream. - pg_recvlogical would fail with -f -. The tests in copyselect.sql from the main test suite covers the first case, and there is a TAP test for the second case. However, in both cases, as the standard streams are always redirected, automated tests did not notice those issues, requiring a terminal on Windows to be reproducible. This issue has been introduced in bed9075, and the origin of the problem is that GetFileInformationByHandle() does not work directly on streams, so this commit adds an extra code path to emulate and return a set of stats that match best with the reality. Note that redirected streams rely on handles that can be queried with GetFileInformationByHandle(), but we can rely on GetFinalPathNameByHandleA() to detect this case. Author: Dmitry Koval, Juan José Santamaría Flecha Discussion: https://postgr.es/m/17288-6b58a91025a8a8a3@postgresql.org Backpatch-through: 14 https://git.postgresql.org/pg/commitdiff/10260c794b211117a56ee2eb2deacf609bcca25f
Block ALTER TABLE .. DROP NOT NULL on columns in replica identity index. Replica identities that depend directly on an index rely on a set of properties, one of them being that all the columns defined in this index have to be marked as NOT NULL. There was a hole in the logic with ALTER TABLE DROP NOT NULL, where it was possible to remove the NOT NULL property of a column part of an index used as replica identity, so block it to avoid problems with logical decoding down the road. The same check was already done columns part of a primary key, so the fix is straight-forward. Author: Haiying Tang, Hou Zhijie Reviewed-by: Dilip Kumar, Michael Paquier Discussion: https://postgr.es/m/OS0PR01MB6113338C102BEE8B2FFC5BD9FB619@OS0PR01MB6113.jpnprd01.prod.outlook.com Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/f0d43947a1b0c30f0bf2c117cd78bf95a3161268

David Rowley pushed:

Allow Memoize to operate in binary comparison mode. Memoize would always use the hash equality operator for the cache key types to determine if the current set of parameters were the same as some previously cached set. Certain types such as floating points where -0.0 and +0.0 differ in their binary representation but are classed as equal by the hash equality operator may cause problems as unless the join uses the same operator it's possible that whichever join operator is being used would be able to distinguish the two values. In which case we may accidentally return in the incorrect rows out of the cache. To fix this here we add a binary mode to Memoize to allow it to the current set of parameters to previously cached values by comparing bit-by-bit rather than logically using the hash equality operator. This binary mode is always used for LATERAL joins and it's used for normal joins when any of the join operators are not hashable. Reported-by: Tom Lane Author: David Rowley Discussion: https://postgr.es/m/3004308.1632952496@sss.pgh.pa.us Backpatch-through: 14, where Memoize was added https://git.postgresql.org/pg/commitdiff/e502150f7d0be41e3c8784be007fa871a32d8a7f
Flush Memoize cache when non-key parameters change. It's possible that a subplan below a Memoize node contains a parameter from above the Memoize node. If this parameter changes then cache entries may become out-dated due to the new parameter value. Previously Memoize was mistakenly not aware of this. We fix this here by flushing the cache whenever a parameter that's not part of the cache key changes. Bug: #17213 Reported by: Elvis Pranskevichus Author: David Rowley Discussion: https://postgr.es/m/17213-988ed34b225a2862@postgresql.org Backpatch-through: 14, where Memoize was added https://git.postgresql.org/pg/commitdiff/1050048a315790a505465bfcceb26eaf8dbc7e2e
Revert "Flush Memoize cache when non-key parameters change". This reverts commit 1050048a315790a505465bfcceb26eaf8dbc7e2e. https://git.postgresql.org/pg/commitdiff/dad20ad4709f602b4827a1ab2b0e715f36c548c3
Flush Memoize cache when non-key parameters change, take 2. It's possible that a subplan below a Memoize node contains a parameter from above the Memoize node. If this parameter changes then cache entries may become out-dated due to the new parameter value. Previously Memoize was mistakenly not aware of this. We fix this here by flushing the cache whenever a parameter that's not part of the cache key changes. Bug: #17213 Reported by: Elvis Pranskevichus Author: David Rowley Discussion: https://postgr.es/m/17213-988ed34b225a2862@postgresql.org Backpatch-through: 14, where Memoize was added https://git.postgresql.org/pg/commitdiff/411137a429210e432f923264a8e313a9872910ca

Amit Kapila pushed:

Rename SnapBuild* macros in slot.c. Same macro names for SnapBuildOnDiskNotChecksummedSize and SnapBuildOnDiskChecksummedSize are being used in slot.c and snapbuild.c. This patch renames them, in slot.c, to ReplicationSlotOnDiskNotChecksummedSize and ReplicationSlotOnDiskChecksummedSize similar to the other macros. This makes all macro names look consistent in slot.c. Author: Bharath Rupireddy Discussion: https://postgr.es/m/CALj2ACVZo-piDGzBOJRY4ob=_goFR6t9DhZMDMjJWN7LQs34Aw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/875e02c2dff34f1bc9f3832a4f83c34bf300eb9f

Robert Haas pushed:

Fix corner-case failure to detect improper timeline switch. rescanLatestTimeLine() contains a guard against switching to a timeline that forked off from the current one prior to the current recovery point, but that guard does not work if the timeline switch occurs before the first WAL recod (which must be the checkpoint record) is read. Without this patch, an improper timeline switch is therefore possible in such cases. This happens because rescanLatestTimeLine() relies on the global variable EndRecPtr to understand the current position of WAL replay. However, EndRecPtr at this point in the code contains the endpoint of the last-replayed record, not the startpoint or endpoint of the record being replayed now. Thus, before any records have been replayed, it's zero, which causes the sanity check to always pass. To fix, pass down the correct timeline explicitly. The EndRecPtr value we want is the one from the xlogreader, which will be the starting position of the record we're about to try to read, rather than the global variable, which is the ending position of the last record we successfully read. They're usually the same, but not in the corner case described here. No back-patch, because in v14 and earlier branhes, we were using the wrong TLI here as well as the wrong LSN. In master, that was fixed by commit 4a92a1c3d1c361ffb031ed05bf65b801241d7cdd, but that and it's prerequisite patches are too invasive to back-patch for such a minor issue. Patch by me, reviewed by Amul Sul. Discussion: http://postgr.es/m/CA+Tgmoao96EuNeSPd+hspRKcsCddu=b1h-QNRuKfY8VmfNQdfg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/e7ea2fa342b008ae97e794b0fa2ee538ddcee3b7
xlog.c: Remove global variables ReadRecPtr and EndRecPtr. In most places, the variables necessarily store the same value as the eponymous members of the XLogReaderState that we use during WAL replay, because ReadRecord() assigns the values from the structure members to the global variables just after XLogReadRecord() returns. However, XLogBeginRead() adjusts the structure members but not the global variables, so after XLogBeginRead() and before the completion of XLogReadRecord() the values can differ. Otherwise, they must be identical. According to my analysis, the only place where either variable is referenced at a point where it might not have the same value as the structure member is the refrence to EndRecPtr within XLogPageRead. Therefore, at every other place where we are using the global variable, we can just switch to using the structure member instead, and remove the global variable. However, we can, and in fact should, do this in XLogPageRead() as well, because at that point in the code, the global variable will actually store the start of the record we want to read - either because it's where the last WAL record ended, or because the read position has been changed using XLogBeginRead since the last record was read. The structure member, on the other hand, will already have been updated to point to the end of the record we just read. Elsewhere, the latter is what we use as an argument to emode_for_corrupt_record(), so we should do the same here. This part of the patch is perhaps a bug fix, but I don't think it has any important consequences, so no back-patch. The point here is just to continue to whittle down the entirely excessive use of global variables in xlog.c. Discussion: http://postgr.es/m/CA+Tgmoao96EuNeSPd+hspRKcsCddu=b1h-QNRuKfY8VmfNQdfg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/d2ddfa681db27a138acb63c8defa8cc6fa588922

Heikki Linnakangas pushed:

Fix missing space in docs. Author: Japin Li Discussion: /message-id/MEYP282MB1669C36E5F733C2EFBDCB80BB6619@MEYP282MB1669.AUSP282.PROD.OUTLOOK.COM https://git.postgresql.org/pg/commitdiff/373e55218972f840ad29cd8a4dabe4b17e98d28b

Andres Freund pushed:

Replace straggling uses of ReadRecPtr/EndRecPtr. d2ddfa681db removed ReadRecPtr/EndRecPtr, but two uses within an #ifdef WAL_DEBUG escaped. Discussion: https://postgr.es/m/20211124231206.gbadj5bblcljb6d5@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/3030903dfefb314ebb575834702904dc008eb5ca

Daniel Gustafsson pushed:

Fix GRANTED BY support in REVOKE ROLE statements. Commit 6aaaa76bb added support for the GRANTED BY clause in GRANT and REVOKE statements, but missed adding support for checking the role in the REVOKE ROLE case. Fix by checking that the parsed role matches the CURRENT_ROLE/CURRENT_USER requirement, and also add some tests for it. Backpatch to v14 where GRANTED BY support was introduced. Discussion: https://postgr.es/m/B7F6699A-A984-4943-B9BF-CEB84C003527@yesql.se Backpatch-through: 14 https://git.postgresql.org/pg/commitdiff/b2a459edfe645747744402f23de041e9c0a3cd93
Add test for REVOKE ADMIN OPTION. The REVOKE ADMIN OPTION FOR <role_name> syntax didn't have ample test coverage. Fix by adding coverage in the privileges test suite. Author: Mark Dilger mark.dilger@enterprisedb.com Discussion: https://postgr.es/m/333B0203-D19B-4335-AE64-90EB0FAF46F0@enterprisedb.com https://git.postgresql.org/pg/commitdiff/4597fd78d6dea2235cb948ea036c2d61057c415c

PostgreSQL Weekly News - November 21, 2021

Tue, 23 Nov 2021 00:00:00 +0000

PostgreSQL Weekly News - November 21, 2021

Nordic PGDay 2022 will be held in Helsinki, Finland at the Hilton Helsinki Strand Hotel on March 22, 2022. The CfP is open through December 31, 2021 here

PostgreSQL Product News

PGroonga 2.3.4 a full text search platform for all languages, released.

Pgpool-II 4.2.6, 4.1.9, 4.0.16, 3.7.21 and 3.6.28, a connection pooler and statement replication system for PostgreSQL, re l ea s ed.

Ora2Pg 23.0, a tool for migrating Oracle databases to PostgreSQL, released. https://github.com/darold/ora2pg/blob/master/changelog

BigAnimal, a managed PostgreSQL database on Azure, released.

pgAdmin4 6.2, a web- and native GUI control center for PostgreSQL, released.

PostgreSQL Jobs for November

https://archives.postgresql.org/pgsql-jobs/2021-11/

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Robert Haas pushed:

Fix thinko in bbsink_throttle_manifest_contents. Report and diagnosis by Dmitry Dolgov. Discussion: http://postgr.es/m/20211115162641.dmo6l32fklh64gnw@localhost https://git.postgresql.org/pg/commitdiff/1b098da2009362e0e8d9a1d0a6aac2f2bd3e2f0b
Move InitXLogInsert() call from InitXLOGAccess() to BaseInit(). At present, there is an undocumented coding rule that you must call RecoveryInProgress(), or do something else that results in a call to InitXLogInsert(), before trying to write WAL. Otherwise, the WAL construction buffers won't be initialized, resulting in failures. Since it's not good to rely on a status inquiry function like RecoveryInProgress() having the side effect of initializing critical data structures, instead do the initialization eariler, when the backend first starts up. Patch by me. Reviewed by Nathan Bossart and Michael Paquier. Discussion: http://postgr.es/m/CA+TgmoY7b65qRjzHN_tWUk8B4sJqk1vj1d31uepVzmgPnZKeLg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/e51c46991f0ee99cca222305619dee5543a1290a

Amit Kapila pushed:

Invalidate relcache when changing REPLICA IDENTITY index. When changing REPLICA IDENTITY INDEX to another one, the target table's relcache was not being invalidated. This leads to skipping update/delete operations during apply on the subscriber side as the columns required to search corresponding rows won't get logged. Author: Tang Haiying, Hou Zhijie Reviewed-by: Euler Taveira, Amit Kapila Backpatch-through: 10 Discussion: https://postgr.es/m/OS0PR01MB61133CA11630DAE45BC6AD95FB939@OS0PR01MB6113.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/354a1f8d220fbbb07b0ded32c5ade72646afb801
Fix parallel operations that prevent oldest xmin from advancing. While determining xid horizons, we skip over backends that are running Vacuum. We also ignore Create Index Concurrently, or Reindex Concurrently for the purposes of computing Xmin for Vacuum. But we were not setting the flags corresponding to these operations when they are performed in parallel which was preventing Xid horizon from advancing. The optimization related to skipping Create Index Concurrently, or Reindex Concurrently operations was implemented in PG-14 but the fix is the same for the Parallel Vacuum as well so back-patched till PG-13. Author: Masahiko Sawada Reviewed-by: Amit Kapila Backpatch-through: 13 Discussion: https://postgr.es/m/CAD21AoCLQqgM1sXh9BrDFq0uzd3RBFKi=Vfo6cjjKODm0Onr5w@mail.gmail.com https://git.postgresql.org/pg/commitdiff/0f0cfb494004befb0f6e89d3129347869420c509

Álvaro Herrera pushed:

Fix headerscheck failure in replication/worker_internal.h. Broken by 31c389d8de91 https://git.postgresql.org/pg/commitdiff/ad26ee28250c4cd357a7420161a2be321c3dd536

Michaël Paquier pushed:

Remove global variable "LastRec" in xlog.c. This variable is used only by StartupXLOG() now, so let's make it local to simplify the code. Author: Amul Sul Reviewed-by: Tom Lane, Michael Paquier Discussion: https://postgr.es/m/CAAJ_b96Qd023itERBRN9Z7P2saNDT3CYvGuMO8RXwndVNN6z7g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/f975fc3a3542005ed0dd689bdb5bd9ed4e1f4d52
Add table to regression tests for binary-compatibility checks in pg_upgrade. This commit adds to the main regression test suite a table with all the in-core data types (some exceptions apply). This table is not dropped, so as pg_upgrade would be able to check the binary compatibility of the types tracked in the table. If a new type is added in core, this part of the tests would need a refresh but the tests are designed to fail if that were to happen. As this is useful for upgrades and that these rely on the objects created in the regression test suite of the old version upgraded from, a backpatch down to 12 is done, which is the last point where a binary incompatible change has been done (7c15cef). This will hopefully be enough to find out if something gets broken during the development of a new version of Postgres, so as it is possible to take actions in pg_upgrade itself in this case (like 0ccfc28 for sql_identifier). An area that is not covered yet is related to external modules, which may create their own types. The testing infrastructure of pg_upgrade is not integrated yet with the external modules stored in core (src/test/modules/ or contrib/, all use the same database name for their tests so there would be an overlap). This could be improved in the future. Author: Justin Pryzby Reviewed-by: Jacob Champion, Peter Eisentraut, Tom Lane, Michael Paquier Discussion: https://postgr.es/m/20201206180248.GI24052@telsasoft.com Backpatch-through: 12 https://git.postgresql.org/pg/commitdiff/835bcba8b8d72a00cecc5431b67e70bbea93f947
Fix quoting of ACL item in table for upgrade binary compatibility checks. Per buildfarm member prion, that runs the regression tests under a role name that uses a hyphen. Issue introduced by 835bcba. Discussion: https://postgr.es/m/YZW4MvzCZ+hQ34vw@paquier.xyz Backpatch-through: 12 https://git.postgresql.org/pg/commitdiff/ac1c7458b17633d1e53a01393d12774c10cb6a91
Improve psql tab completion for transforms, domains and sequences. The following improvements are done: - Addition of some tab completion for CREATE DOMAIN. - Addition of some tab completion for CREATE TRANSFORM. - Addition of type completion for CREATE SEQUENCE AS. Author: Ken Kato Reviewed-by: Kyotaro Horiguchi, Michael Paquier Discussion: https://postgr.es/m/8d370135aef066659eef8e8fbfa6315b@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/0cd6d3b3c5aeac81903aa7de92e406f8567898a2

Peter Eisentraut pushed:

Fix incorrect format placeholders. https://git.postgresql.org/pg/commitdiff/303d4eb1c548f1d0821e168a6e7c7e9bd02c8088

Daniel Gustafsson pushed:

Doc: add see-also references to CREATE PUBLICATION. The "See also" section on the reference page for CREATE PUBLICATION didn't match the cross references on CREATE SUBSCRIPTION and their ALTER counterparts. Fixed by adding an xref to the CREATE and ALTER SUBSCRIPTION pages. Backpatch down to v10 where CREATE PUBLICATION was introduced. Author: Peter Smith smithpb2250@gmail.com Reviewed-by: Masahiko Sawada sawada.mshk@gmail.com Discussion: https://postgr.es/m/CAHut+PvGWd3-Ktn96c-z6uq-8TGVVP=TPOkEovkEfntoo2mRhw@mail.gmail.com Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/3374a87b62cc553fa65f57ade019dcf3104ae639
Improve publication error messages. Commit 81d5995b4b introduced more fine-grained errormessages for incorrect relkinds for publication, while unlogged and temporary tables were reported with using the same message. This provides separate error messages for these types of relpersistence. Author: Bharath Rupireddy bharath.rupireddyforpostgres@gmail.com Reviewed-by: Peter Eisentraut peter.eisentraut@enterprisedb.com Reviewed-by: Jeevan Ladhe jeevan.ladhe@enterprisedb.com Reviewed-by: Euler Taveira euler@eulerto.com Discussion: https://postgr.es/m/CALj2ACW9S=AswyQHjtO6WMcsergMkCBTtzXGrM8DX26DzfeTLQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/aa12781b0d039d93e1a851ece4bc75c3746cbd43

Tom Lane pushed:

Fix display of SQL-standard function's arguments in INSERT/SELECT. If a SQL-standard function body contains an INSERT ... SELECT statement, any function parameters referenced within the SELECT were always printed in $N style, rather than using the parameter name if any. While not strictly incorrect, this wasn't the intention, and it's inconsistent with the way that such parameters would be printed in any other kind of statement. The cause is that the recursion to get_query_def from get_insert_query_def neglected to pass down the context->namespaces list, passing constant NIL instead. This is a very ancient oversight, but AFAICT it had no visible consequences before commit e717a9a18 added an outermost namespace with function parameters. We don't allow INSERT ... SELECT as a sub-query, except in a top-level WITH clause, where it couldn't contain any outer references that might need to access upper namespaces. So although that's arguably a bug, I don't see any point in changing it before v14. In passing, harden the code added to get_parameter by e717a9a18 so that it won't crash if a PARAM_EXTERN Param appears in an unexpected place. Per report from Erki Eessaar. Code fix by me, regression test case by Masahiko Sawada. Discussion: https://postgr.es/m/AM9PR01MB8268347BED344848555167FAFE949@AM9PR01MB8268.eurprd01.prod.exchangelabs.com https://git.postgresql.org/pg/commitdiff/a8d8445a7b2f80f6d0bfe97b19f90bd2cbef8759
Handle close() failures more robustly in pg_dump and pg_basebackup. Coverity complained that applying get_gz_error after a failed gzclose, as we did in one place in pg_basebackup, is unsafe. I think it's right: it's entirely likely that the call is touching freed memory. Change that to inspect errno, as we do for other gzclose calls. Also, be careful to initialize errno to zero immediately before any gzclose() call where we care about the error status. (There are some calls where we don't, because we already failed at some previous step.) This ensures that we don't get a misleadingly irrelevant error code if gzclose() fails in a way that doesn't set errno. We could work harder at that, but it looks to me like all such cases are basically can't-happen if we're not misusing zlib, so it's not worth the extra notational cruft that would be required. Also, fix several places that simply failed to check for close-time errors at all, mostly at some remove from the close or gzclose itself; and one place that did check but didn't bother to report the errno. Back-patch to v12. These mistakes are older than that, but between the frontend logging API changes that happened in v12 and the fact that frontend code can't rely on %m before that, the patch would need substantial revision to work in older branches. It doesn't quite seem worth the trouble given the lack of related field complaints. Patch by me; thanks to Michael Paquier for review. Discussion: https://postgr.es/m/1343113.1636489231@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/3cac2c8caaefc642332e6994ce80032cc7d4cfdf
Clean up error handling in pg_basebackup's walmethods.c. The error handling here was a mess, as a result of a fundamentally bad design (relying on errno to keep its value much longer than is safe to assume) as well as a lot of just plain sloppiness, both as to noticing errors at all and as to reporting the correct errno. Moreover, the recent addition of LZ4 compression broke things completely, because liblz4 doesn't use errno to report errors. To improve matters, keep the error state in the DirectoryMethodData or TarMethodData struct, and add a string field so we can handle cases that don't set errno. (The tar methods already had a version of this, but it can be done more efficiently since all these cases use a constant error string.) Make the dir and tar methods handle errors in basically identical ways, which they didn't before. This requires copying errno into the state struct in a lot of places, which is a bit tedious, but it has the virtue that we can get rid of ad-hoc code to save and restore errno in a number of places ... not to mention that it fixes other places that should've saved/restored errno but neglected to. In passing, fix some pointlessly static buffers to be ordinary local variables. There remains an issue about exactly how to handle errors from fsync(), but that seems like material for its own patch. While the LZ4 problems are new, all the rest of this is fixes for old bugs, so backpatch to v10 where walmethods.c was introduced. Patch by me; thanks to Michael Paquier for review. Discussion: https://postgr.es/m/1343113.1636489231@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/248c3a937dd018a72095f407cff727c9f08db0c1
Add a planner support function for starts_with(). This fills in some gaps in planner support for starts_with() and the equivalent ^@ operator: * A condition such as "textcol ^@ constant" can now use a regular btree index, not only an SP-GiST index, so long as the index's collation is C. (This works just like "textcol LIKE 'foo%'".) * "starts_with(textcol, constant)" can be optimized the same as "textcol ^@ constant". * Fixed-prefix LIKE and regex patterns are now more like starts_with() in another way: if you apply one to an SPGiST-indexed column, you'll get an index condition using ^@ rather than two index conditions with >= and <. Per a complaint from Shay Rojansky. Patch by me; thanks to Nathan Bossart for review. Discussion: https://postgr.es/m/232599.1633800229@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/a148f8bc04b9980f019ea0d4b89311cf0bdc22b7
Provide a variant of simple_prompt() that can be interrupted by ^C. Up to now, you couldn't escape out of psql's \password command by typing control-C (or other local spelling of SIGINT). This is pretty user-unfriendly, so improve it. To do so, we have to modify the functions provided by pg_get_line.c; but we don't want to mess with psql's SIGINT handler setup, so provide an API that lets that handler cause the cancel to occur. This relies on the assumption that we won't do any major harm by longjmp'ing out of fgets(). While that's obviously a little shaky, we've long had the same assumption in the main input loop, and few issues have been reported. psql has some other simple_prompt() calls that could usefully be improved the same way; for now, just deal with \password. Nathan Bossart, minor tweaks by me Discussion: https://postgr.es/m/747443.1635536754@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/5f1148224bd78bcf3bf7d916b8fe85dd820c52c6
Use appropriate -Wno-warning switches when compiling bitcode. We use "clang" to compile bitcode files for LLVM inlining. That might be different from the build's main C compiler, so it needs its own set of compiler flags. To simplify configure, we don't bother adding any -W switches to that flag set; there's little need since the main build will show us any warnings. However, if we don't want to see unwanted warnings, we still have to add any -Wno-warning switches we'd normally use with clang. This escaped notice before commit 9ff47ea41, which tried to add -Wno-compound-token-split-by-macro; buildfarm animals using mismatched CC and CLANG still showed those warnings. I'm not sure why we never saw any effects from the lack of -Wno-unused-command-line-argument (maybe that's only activated by -Wall?). clang does not currently support -Wno-format-truncation or -Wno-stringop-truncation, although in the interests of future-proofing and consistency I included tests for those. Back-patch to v11 where we started building bitcode files. Discussion: https://postgr.es/m/2921539.1637254619@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/276517a96484f9e39a7a1095ab39fa76ef1ee8cc
Allow psql's other uses of simple_prompt() to be interrupted by ^C. This fills in the work left un-done by 5f1148224. \prompt can be canceled out of now, and so can password prompts issued during \connect. (We don't need to do anything for password prompts issued during startup, because we aren't yet trapping SIGINT at that point.) Nathan Bossart Discussion: https://postgr.es/m/747443.1635536754@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/46d665bc26ce57b5afecbc218c8fc3c6848211d8
Fix SP-GiST scan initialization logic for binary-compatible cases. Commit ac9099fc1 rearranged the logic in spgGetCache() that determines the index's attType (nominal input data type) and leafType (actual type stored in leaf index tuples). Turns out this broke things for the case where (a) the actual input data type is different from the nominal type, (b) the opclass's config function leaves leafType defaulted, and (c) the opclass has no "compress" function. (b) caused us to assign the actual input data type as leafType, and then since that's not attType, we complained that a "compress" function is required. For non-polymorphic opclasses, condition (a) arises in binary-compatible cases, such as using SP-GiST text_ops for a varchar column, or using any opclass on a domain over its nominal input type. To fix, use attType for leafType when the index's declared column type is different from but binary-compatible with attType. Do this only in the defaulted-leafType case, to avoid overriding any explicit selection made by the opclass. Per bug #17294 from Ilya Anfimov. Back-patch to v14. Discussion: https://postgr.es/m/17294-8f6c7962ce877edc@postgresql.org https://git.postgresql.org/pg/commitdiff/f4e7ae2b8a67ad6801726553a024a3306716ef80
Doc: update some things relevant to minimum Test::More version. Oversights in commit 405f32fc4. Also, add a tip (discovered the hard way) about getting Test::More 0.98 to pass its regression tests on recent Linux platforms. https://git.postgresql.org/pg/commitdiff/92e70796e91e2f9086fad0156e0e91513e54a66b
pg_receivewal, pg_recvlogical: allow canceling initial password prompt. Previously it was impossible to terminate these programs via control-C while they were prompting for a password. We can fix that trivially for their initial password prompts, by moving setup of the SIGINT handler from just before to just after their initial GetConnection() calls. This fix doesn't permit escaping out of later re-prompts, but those should be exceedingly rare, since the user's password or the server's authentication setup would have to have changed meanwhile. We considered applying a fix similar to commit 46d665bc2, but that seemed more complicated than it'd be worth. Moreover, this way is back-patchable, which that wasn't. The misbehavior exists in all supported versions, so back-patch to all. Tom Lane and Nathan Bossart Discussion: https://postgr.es/m/747443.1635536754@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/282b6d00abf5cebece6f94c796a4ed807a0176db

Andres Freund pushed:

Initialize backend status reporting during bootstrap. This allows a later commit to reduce the number of branches in performance sensitive functions during normal running, compared to a very minor saving during bootstrapping. Author: Melanie Plageman melanieplageman@gmail.com Reviewed-By: Andres Freund andres@anarazel.de Discussion: https://postgr.es/m/CAAKRu_Yeg+vh6SHNEo1+=O7e-BPX35cU0XQM=YwQRnkFyv_y+w@mail.gmail.com https://git.postgresql.org/pg/commitdiff/3b34645678d1a516c148e3e27c26325708e92f6f

Andrew Dunstan pushed:

Require version 0.98 of Test::More for TAP tests. This means that the subtest feature will be available for use. We expect that this change will make prairiedog go red until it is updated, but other buildfarm animals should be fine. Discussion: https://postgr.es/m/f5e1d308-4e33-37a7-bdf1-f6e0c75119de@dunslane.net https://git.postgresql.org/pg/commitdiff/405f32fc49609eb94fa39e7b5e7c1fe2bb2b73aa

PostgreSQL Weekly News - November 14, 2021

Mon, 15 Nov 2021 00:00:00 +0000

PostgreSQL Weekly News - November 14, 2021

PostgreSQL 14.1, 13.5, 12.9, 11.14, 10.19, and 9.6.24 released. This is the final release in the 9.6 series, so put those upgrade plans in action if you haven't already.

PostgreSQL Product News

Pgpool-II 4.3 beta1 a connection pooler and statement replication system for PostgreSQL, released

Odyssey 1.2, a multi-threaded connection pooler for PostgreSQL, released. https://github.com/yandex/odyssey/releases

pgbouncer 1.16.1, a connection pooler and more for PostgreSQL, released

PostgreSQL Jobs for November

https://archives.postgresql.org/pgsql-jobs/2021-11/

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

David Rowley pushed:

Fix incorrect hash equality operator bug in Memoize. In v14, because we don't have a field in RestrictInfo to cache both the left and right type's hash equality operator, we just restrict the scope of Memoize to only when the left and right types of a RestrictInfo are the same. In master we add another field to RestrictInfo and cache both hash equality operators. Reported-by: Jaime Casanova Author: David Rowley Discussion: https://postgr.es/m/20210929185544.GB24346%40ahch-to Backpatch-through: 14 https://git.postgresql.org/pg/commitdiff/39a3105678a247bbfdc132cd95db5b515b8cd7f6

Tom Lane pushed:

Reject extraneous data after SSL or GSS encryption handshake. The server collects up to a bufferload of data whenever it reads data from the client socket. When SSL or GSS encryption is requested during startup, any additional data received with the initial request message remained in the buffer, and would be treated as already-decrypted data once the encryption handshake completed. Thus, a man-in-the-middle with the ability to inject data into the TCP connection could stuff some cleartext data into the start of a supposedly encryption-protected database session. This could be abused to send faked SQL commands to the server, although that would only work if the server did not demand any authentication data. (However, a server relying on SSL certificate authentication might well not do so.) To fix, throw a protocol-violation error if the internal buffer is not empty after the encryption handshake. Our thanks to Jacob Champion for reporting this problem. Security: CVE-2021-23214 https://git.postgresql.org/pg/commitdiff/28e24125541545483093819efae9bca603441951
libpq: reject extraneous data after SSL or GSS encryption handshake. libpq collects up to a bufferload of data whenever it reads data from the socket. When SSL or GSS encryption is requested during startup, any additional data received with the server's yes-or-no reply remained in the buffer, and would be treated as already-decrypted data once the encryption handshake completed. Thus, a man-in-the-middle with the ability to inject data into the TCP connection could stuff some cleartext data into the start of a supposedly encryption-protected database session. This could probably be abused to inject faked responses to the client's first few queries, although other details of libpq's behavior make that harder than it sounds. A different line of attack is to exfiltrate the client's password, or other sensitive data that might be sent early in the session. That has been shown to be possible with a server vulnerable to CVE-2021-23214. To fix, throw a protocol-violation error if the internal buffer is not empty after the encryption handshake. Our thanks to Jacob Champion for reporting this problem. Security: CVE-2021-23222 https://git.postgresql.org/pg/commitdiff/160c0258802d10b0600d7671b1bbea55d8e17d45
Fix incorrect format placeholder. Per buildfarm warnings. https://git.postgresql.org/pg/commitdiff/b0cf5444f9a8d915b2e9b44790025f17a7dc107f
Fix instability in 026_overwrite_contrecord.pl test. We've seen intermittent failures in this test on slower buildfarm machines, which I think can be explained by assuming that autovacuum emitted some additional WAL. Disable autovacuum to stabilize it. In passing, use stringwise not numeric comparison to compare WAL file names. Doesn't matter at present, but they are hex strings not decimal ... Discussion: https://postgr.es/m/1372189.1636499287@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/b66767b56b1cd082f3499a7e5a21b480dd004f51
Doc: improve protocol spec for logical replication Type messages. protocol.sgml documented the layout for Type messages, but completely dropped the ball otherwise, failing to explain what they are, when they are sent, or what they're good for. While at it, do a little copy-editing on the description of Relation messages. In passing, adjust the comment for apply_handle_type() to make it clearer that we choose not to do anything when receiving a Type message, not that we think it has no use whatsoever. Per question from Stefen Hillman. Discussion: https://postgr.es/m/CAPgW8pMknK5pup6=T4a_UG=Cz80Rgp=KONqJmTdHfaZb0RvnFg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/c3b33698cf88550b017620f73b94b53029897f39
Fall back to unsigned int, not int, for socklen_t. It's a coin toss which of these is a better default assumption. However, of the machines we have in the buildfarm, the only ones relying on the fallback socklen_t definition are ancient HPUX, and on that platform unsigned int is the right choice. Minor tweak to ee3a1a5b6. Discussion: https://postgr.es/m/1440792.1636558888@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/01ec41a5fe4aa590dde18a2c551432aa1925caea
postgres_fdw: suppress casts on constants in limited cases. When deparsing an expression of the form "remote_var OP constant", we'd normally apply a cast to the constant to make sure that the remote parser thinks it's of the same type we do. However, doing so is often not necessary, and it causes problems if the user has intentionally declared the local column as being of a different type than the remote column. A plausible use-case for that is using text to represent a type that's an enum on the remote side. A comparison on such a column will get shipped as "var = 'foo'::text", which blows up on the remote side because there's no enum = text operator. But if we simply leave off the explicit cast, the comparison will do exactly what the user wants. It's possible to do this without major risk of semantic problems, by relying on the longstanding parser heuristic that "if one operand of an operator is of type unknown, while the other one has a known type, assume that the unknown operand is also of that type". Hence, this patch leaves off the cast only if (a) the operator inputs have the same type locally; (b) the constant will print as a string literal or NULL, both of which are initially taken as type unknown; and (c) the non-Const input is a plain foreign Var. Rule (c) guarantees that the remote parser will know the type of the non-Const input; moreover, it means that if this cast-omission does cause any semantic surprises, that can only happen in cases where the local column has a different type than the remote column. That wasn't guaranteed to work anyway, and this patch should represent a net usability gain for such cases. One point that I (tgl) remain slightly uncomfortable with is that we will ignore an implicit RelabelType when deciding if the non-Const input is a plain Var. That makes it a little squishy to argue that the remote should resolve the Const as being of the same type as its Var, because then our Const is not the same type as our Var. However, if we don't do that, then this hack won't work as desired if the user chooses to use varchar rather than text to represent some remote column. That seems useful, so do it like this for now. We might have to give up the RelabelType-ignoring bit if any problems surface. Dian Fay, with review and kibitzing by me Discussion: https://postgr.es/m/C9LU294V7K4F.34LRRDU449O45@lamia https://git.postgresql.org/pg/commitdiff/f8abb0f5e114d8c309239f0faa277b97f696d829
Make psql's \password default to CURRENT_USER, not PQuser(conn). The documentation says plainly that \password acts on "the current user" by default. What it actually acted on, or tried to, was the username used to log into the current session. This is not the same thing if one has since done SET ROLE or SET SESSION AUTHENTICATION. Aside from the possible surprise factor, it's quite likely that the current role doesn't have permissions to set the password of the original role. To fix, use "SELECT CURRENT_USER" to get the role name to act on. (This syntax works with servers at least back to 7.0.) Also, in hopes of reducing confusion, include the role name that will be acted on in the password prompt. The discrepancy from the documentation makes this a bug, so back-patch to all supported branches. Patch by me; thanks to Nathan Bossart for review. Discussion: https://postgr.es/m/747443.1635536754@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/d6eb5a0c258d3da5471814bcc6ed6498129fee16

Robert Haas pushed:

Minimal fix for unterminated tar archive problem. Commit 23a1c6578c87fca0e361c4f5f9a07df5ae1f9858 improved pg_basebackup's ability to parse tar archives, but also arranged to parse them only when we need to make some modification to the contents of the archive. That's a problem, because the server doesn't actually terminate tar archives. When the new parsing logic was engaged, pg_basebackup would properly terminate the tar file, but when it was skipped, pg_basebackup would just write whatever it got from the server, meaning that the terminator was missing. Most versions of tar are willing to overlook the missing terminator, but the AIX buildfarm animals were not. Fix by inventing a new kind of bbstreamer that just blindly adds a terminator, and using it whenever we don't parse the tar archive. Discussion: http://postgr.es/m/CA+TgmoZbNzsWwM4BE5Jb_qHncY817DYZwGf+2-7hkMQ27ZwsMQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/57b5a9646d97a3e8a5b6b6d86b375cc8da6ac85c
Have the server properly terminate tar archives. Earlier versions of PostgreSQL featured a version of pg_basebackup that wanted to edit tar archives but was too dumb to parse them properly. The server made things easier for the client by failing to add the two blocks of zero bytes that ought to end a tar file, leaving it up to the client to do that. But since commit 23a1c6578c87fca0e361c4f5f9a07df5ae1f9858, we don't need this hack any more, because pg_basebackup is now smarter and can parse tar files even if they are properly terminated! So change the server to always properly terminate the tar files. Older versions of pg_basebackup can't talk to new servers anyway, so there's no compatibility break. On the pg_basebackup side, we see still need to add the terminating zero bytes if we're talking to an older server, but not when the server is v15+. Hopefully at some point we'll be able to remove some of this compatibility cruft, but it seems best to hang on to it for now. In passing, add a file header comment to bbstreamer_tar.c, to make it clearer what's going on here. Discussion: http://postgr.es/m/CA+TgmoZbNzsWwM4BE5Jb_qHncY817DYZwGf+2-7hkMQ27ZwsMQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/5a1007a5088cd6ddf892f7422ea8dbaef362372f
More cleanup of 'ThisTimeLineID'. In XLogCtlData, rename the structure member ThisTimeLineID to InsertTimeLineID and update the comments to make clear that it's only expected to be set after recovery is complete. In StartupXLOG, replace the local variables ThisTimeLineID and PrevTimeLineID with new local variables replayTLI and newTLI. In the old scheme, ThisTimeLineID was the replay TLI until we created a new timeline, and after that the replay TLI was in PrevTimeLineID. Now, replayTLI is the TLI from which we last replayed WAL throughout the entire function, and newTLI is either that, or the new timeline created upon promotion. Remove some misleading comments from the comment block just above where recoveryTargetTimeLineGoal and friends are declared. It's become incorrect, not only because ThisTimeLineID as a variable is now gone, but also because the rmgr code does not care about ThisTimeLineID and has not since what used to be the TLI field in the page header was repurposed to store the page checksum. Add a comment GetFlushRecPtr that it's only supposed to be used in normal running, and an assertion to verify that this is so. Per some ideas from Michael Paquier and some of my own. Review by Michael Paquier also. Discussion: http://postgr.es/m/CA+TgmoY1a2d1AnVR3tJcKmGGkhj7GGrwiNwjtKr21dxOuLBzCQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/a27048cbcb582056bfbf15aa2f898b4a3ec74304
Fix thinko in assertion in basebackup.c. Commit 5a1007a5088cd6ddf892f7422ea8dbaef362372f tried to introduce an assertion that the block size was at least twice the size of a tar block, but I got the math wrong. My error was reported to me off-list. https://git.postgresql.org/pg/commitdiff/10eae82b27cebbb9586cda8baf8e3226496891d0
Improve performance of pgarch_readyXlog() with many status files. Presently, the archive_status directory was scanned for each file to archive. When there are many status files, say because archive_command has been failing for a long time, these directory scans can get very slow. With this change, the archiver remembers several files to archive during each directory scan, speeding things up. To ensure timeline history files are archived as quickly as possible, XLogArchiveNotify() forces the archiver to do a new directory scan as soon as the .ready file for one is created. Nathan Bossart, per a long discussion involving many people. It is not clear to me exactly who out of all those people reviewed this particular patch. Discussion: http://postgr.es/m/CA+TgmobhAbs2yabTuTRkJTq_kkC80-+jw=pfpypdOJ7+gAbQbw@mail.gmail.com Discussion: http://postgr.es/m/620F3CE1-0255-4D66-9D87-0EADE866985A@amazon.com https://git.postgresql.org/pg/commitdiff/beb4e9ba1652a04f66ff20261444d06f678c0b2d

Amit Kapila pushed:

Rename some enums to use TABLE instead of REL. Commit 5a2832465f introduced some enums to represent all tables in schema publications and used REL in their names. Use TABLE instead of REL in those enums to avoid confusion with other objects like SEQUENCES that can be part of a publication in the future. In the passing, (a) Change one of the newly introduced error messages to make it consistent for Create and Alter commands, (b) add missing alias in one of the SQL Statements that is used to print publications associated with the table. Reported-by: Tomas Vondra, Peter Smith Author: Vignesh C Reviewed-by: Hou Zhijie, Peter Smith Discussion: /message-id/CALDaNm0OANxuJ6RXqwZsM1MSY4s19nuH3734j4a72etDwvBETQ%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/b3812d0b9bcf00e8478186fc287940e17912248a

Fujii Masao pushed:

doc: Add index entries for pg_stat_statements configuration parameters. Author: Ken Kato Reviewed-by: Julien Rouhaud, Fujii Masao Discussion: https://postgr.es/m/699cfd8170178db087e54c954b21ece4@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/ec21779a5847ed89fab19431abbdba794d4a998e

Michaël Paquier pushed:

Make some comments use the term "ProcSignal" for consistency. The surroundings in procsignal.c prefer using "ProcSignal" rather than "procsignal". Author: Bharath Rupireddy Discussion: https://postgr.es/m/CALj2ACX99ghPmm1M_O4r4g+YsXFjCn=qF7PeDXntLwMpht_Gdg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/4cd046c203bbca2955182f78eabc06e831ffdbb1
Improve error messages for some callers of XLogReadRecord(). A couple of code paths related to logical decoding (WAL sender, slot advancing, etc.) use XLogReadRecord(), feeding on error messages generated by walreader.c on a failure. All those messages have no context, making it harder to spot from where an error could come even if these should not happen. All the other callers of XLogReadRecord() do that already. Reviewed-by: Kyotaro Horiguchi Discussion: https://postgr.es/m/YYnTH6OyOwQcAdkw@paquier.xyz https://git.postgresql.org/pg/commitdiff/c9c401a5e13accc4a3a775e3feeabdc5940c9178
Clean up compilation warnings coming from PL/Perl with clang-12~. clang-12 has introduced -Wcompound-token-split-by-macro, that is causing a large amount of warnings when building PL/Perl because of its interactions with upstream Perl. This commit adds one -Wno to CFLAGS at ./configure time if the flag is supported by the compiler to silence all those warnings. Upstream perl has fixed this issue, but it is going to take some time before this is spread across the buildfarm, and we have noticed that some animals would be useful with an extra -Werror to help with the detection of incorrect placeholders (see b0cf544), dangomushi being one. Reviewed-by: Tom Lane Discussion: https://postgr.es/m/YYr3qYa/R3Gw+Sbg@paquier.xyz Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/9ff47ea414c4e6ace347fc4e59866e38b9bbceaf
Fix buffer overrun in unicode string normalization with empty input. PostgreSQL 13 and newer versions are directly impacted by that through the SQL function normalize(), which would cause a call of this function to write one byte past its allocation if using in input an empty string after recomposing the string with NFC and NFKC. Older versions (v10~v12) are not directly affected by this problem as the only code path using normalization is SASLprep in SCRAM authentication that forbids the case of an empty string, but let's make the code more robust anyway there so as any out-of-core callers of this function are covered. The solution chosen to fix this issue is simple, with the addition of a fast-exit path if the decomposed string is found as empty. This would only happen for an empty string as at its lowest level a codepoint would be decomposed as itself if it has no entry in the decomposition table or if it has a decomposition size of 0. Some tests are added to cover this issue in v13~. Note that an empty string has always been considered as normalized (grammar "IS NF[K]{C,D} NORMALIZED", through the SQL function is_normalized()) for all the operations allowed (NFC, NFD, NFKC and NFKD) since this feature has been introduced as of 2991ac5. This behavior is unchanged but some tests are added in v13~ to check after that. I have also checked "make normalization-check" in src/common/unicode/, while on it (works in 13~, and breaks in older stable branches independently of this commit). The release notes should just mention this commit for v13~. Reported-by: Matthijs van der Vleuten Discussion: https://postgr.es/m/17277-0c527a373794e802@postgresql.org Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/098c134556664d37b78ae87853a82f4a9d23a2c8
Fix memory overrun when querying pg_stat_slru. pg_stat_get_slru() in pgstatfuncs.c would point to one element after the end of the array PgStat_SLRUStats when finishing to scan its entries. This had no direct consequences as no data from the extra memory area was read, but static analyzers would rightfully complain here. So let's be clean. While on it, this adds one regression test in the area reserved for system views. Reported-by: Alexander Kozhemyakin, via AddressSanitizer Author: Kyotaro Horiguchi Discussion: https://postgr.es/m/17280-37da556e86032070@postgresql.org Backpatch-through: 13 https://git.postgresql.org/pg/commitdiff/a45ed975c58fde7303eeae488b313bf0314383f7

Peter Eisentraut pushed:

Remove check for accept() argument types. This check was used to accommodate a staggering variety in particular in the type of the third argument of accept(). This is no longer of concern on currently supported systems. We can just use socklen_t in the code and put in a simple check that substitutes int for socklen_t if it's missing, to cover the few stragglers. Reviewed-by: Andres Freund andres@anarazel.de Discussion: /message-id/3538f4c4-1886-64f2-dcff-aaad8267fb82@enterprisedb.com https://git.postgresql.org/pg/commitdiff/ee3a1a5b636b69dde33d68c428dd56b3389a4538
Fix incorrect format placeholders. https://git.postgresql.org/pg/commitdiff/733e0391536dad99a2677ca5e19291854da5730f
doc: Add referential actions to CREATE/ALTER TABLE synopsis. The general constraint synopsis references "referential_action", but this was not further defined in the synopsis section. Compared to the level of detail that the synopsis gives to other subclauses, this should surely be there. extracted from a patch by Paul Martinez hellopfm@gmail.com Discussion: /message-id/flat/CACqFVBZQyMYJV=njbSMxf+rbDHpx=W=B7AEaMKn8dWn9OZJY7w@mail.gmail.com https://git.postgresql.org/pg/commitdiff/db9f287711ac49d9799f93f664d6d101ff8f5891

Jeff Davis pushed:

Add pg_checkpointer predefined role for CHECKPOINT command. Any user with the privileges of pg_checkpointer can issue a CHECKPOINT command. Reviewed-by: Stephen Frost Discussion: https://postgr.es/m/67a1d667e8ec228b5e07f232184c80348c5d93f4.camel%40j-davis.com https://git.postgresql.org/pg/commitdiff/4168a4745492cd54a0ffffc271b452525ef4dc60

Álvaro Herrera pushed:

Restore lock level to set vacuum flags. Commit 27838981be9d mistakenly reduced the lock level from exclusive to shared that is acquired to set PGPROC->statusFlags; this was reverted by dcfff74fb166, but failed to do so in one spot. Fix it. Backpatch to 14. Noted by Andres Freund. Discussion: https://postgr.es/m/20211111020724.ggsfhcq3krq5r4hb@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/0726c764bc4ec337a0216a546ce41c758d81600d

Peter Geoghegan pushed:

Update another obsolete reference in vacuumlazy.c. Addresses an oversight in commit 7ab96cf6. https://git.postgresql.org/pg/commitdiff/eb9baef8e92adf81c6adbe44f1d67878d850bff7
Update heap_page_prune() free space map comments. It is up to the heap_page_prune() caller to decide what to do about updating the FSM for a page following pruning. Update old comments that address what we might want to do as if it was the responsibility of heap_page_prune() itself. heap_page_prune() doesn't have enough high-level context to make a sensible choice. https://git.postgresql.org/pg/commitdiff/42f9427aa98a2245d29737e0f3b8aaf39a7f57ec
Explain pruning pgstats accounting subtleties. Add a comment explaining why the pgstats accounting used during opportunistic heap pruning operations (to maintain the current number of dead tuples in the relation) needs to compensate by subtracting away the number of new LP_DEAD items. This is needed so it can avoid completely forgetting about tuples that become LP_DEAD items during pruning -- they should still count. It seems more natural to discuss this issue at the only relevant call site (opportunistic pruning), since the same issue does not apply to the only other caller (the VACUUM call site). Move everything there too. Author: Peter Geoghegan pg@bowt.ie Discussion: https://postgr.es/m/CAH2-Wzm7f+A6ej650gi_ifTgbhsadVW5cujAL3punpupHff5Yg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/b0f7425ec2445678f32381de8bd3174d3cc2167e

Noah Misch pushed:

Report any XLogReadRecord() error in XlogReadTwoPhaseData(). Buildfarm members kittiwake and tadarida have witnessed errors at this site. The site discarded key facts. Back-patch to v10 (all supported versions). Reviewed by Michael Paquier and Tom Lane. Discussion: https://postgr.es/m/20211107013157.GB790288@rfd.leadboat.com https://git.postgresql.org/pg/commitdiff/335474691054a74d771f0e7c24d25e800e3a37b6

Andrew Dunstan pushed:

Report found versions of required perl modules. Configure tests for the presence of perl modules required for TAP tests, and that they meet specified minimum version requirements. This patch makes it report the version of the module that's actually found rather than just an 'ok' message. This will help in deciding if we can upgrade minimum requirements for these modules. Discussion: https://postgr.es/m/f5e1d308-4e33-37a7-bdf1-f6e0c75119de@dunslane.net https://git.postgresql.org/pg/commitdiff/1593998ae858902e805eb0f8bf3b019399044471

Daniel Gustafsson pushed:

Document PG_TEST_NOCLEAN in TAP test README. Commit 90627cf98 added support for retaining the data directory even on successful tests, but failed to document the environment variable which controls retention. This adds a small note to the TAP test README about PG_TEST_NOCLEAN which when set skips removing the data directories from successful tests. Reviewed-by: Tom Lane tgl@sss.pgh.pa.us Discussion: https://postgr.es/m/2B02C1B3-3F41-4E14-92B9-005D83623A0B@yesql.se https://git.postgresql.org/pg/commitdiff/05d8785af2a192d436df5b7734aacb4e0bab5da8

PostgreSQL Weekly News - November 7, 2021

Mon, 08 Nov 2021 00:00:00 +0000

PostgreSQL Weekly News - November 7, 2021

PG Build 2021 will be held online on 30 November and 1 December 2021 09:00-17:00 GMT. Details.

PostgreSQL Product News

PostgresDAC 3.11, a direct access component suite for PostgreSQL, released. http://microolap.com/products/connectivity/postgresdac/download/

JDBC 42.3.1 released.

ODB C++ ORM version 2.5.0-b.21 released.

DynamoDB FDW 1.0.0 released.

Babelfish, a MS SQL Server compatibility layer for PostgreSQL, released.

PostgreSQL Jobs for November

https://archives.postgresql.org/pgsql-jobs/2021-11/

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Tom Lane pushed:

plpgsql: report proper line number for errors in variable initialization. Previously, we pointed at the surrounding block's BEGIN keyword. If there are multiple variables being initialized in a DECLARE section, this isn't good enough: it can be quite confusing and unhelpful. We do know where the variable's declaration started, so it just takes a tiny bit more error-reporting infrastructure to use that. Discussion: https://postgr.es/m/713975.1635530414@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/acb2d7d5d2301f07d5857ee252995e62ce9e7055
Avoid O(N^2) behavior when the standby process releases many locks. When replaying a transaction that held many exclusive locks on the primary, a standby server's startup process would expend O(N^2) effort on manipulating the list of locks. This code was fine when written, but commit 1cff1b95a made repetitive list_delete_first() calls inefficient, as explained in its commit message. Fix by just iterating the list normally, and releasing storage only when done. (This'd be inadequate if we needed to recover from an error occurring partway through; but we don't.) Back-patch to v13 where 1cff1b95a came in. Nathan Bossart Discussion: https://postgr.es/m/CD2F0E7F-9822-45EC-A411-AE56F14DEA9F@amazon.com https://git.postgresql.org/pg/commitdiff/6301c3adabd947394682e37c933b0f3f83353b28
Doc: improve README files associated with TAP tests. Rearrange src/test/perl/README so that the first section is more clearly "how to run these tests", and the rest "how to write new tests". Add some basic info there about debugging test failures. Then, add cross-refs to that READNE from other READMEs that describe how to run TAP tests. Per suggestion from Kevin Burke, though this is not his original patch. Discussion: https://postgr.es/m/CAKcy5eiSbwiQnmCfnOnDCVC7B8fYyev3E=6pvvECP9pLE-Fcuw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/b21415595cace7f3a45cfeb3023359b4b4d56b85
Don't try to read a multi-GB pg_stat_statements file in one call. Windows fails on a request to read() more than INT_MAX bytes, and perhaps other platforms could have similar issues. Let's adjust this code to read at most 1GB per call. (One would not have thought the file could get that big, but now we have a field report of trouble, so it can. We likely ought to add some mechanism to limit the size of the query-texts file separately from the size of the hash table. That is not this patch, though.) Per bug #17254 from Yusuke Egashira. It's been like this for awhile, so back-patch to all supported branches. Discussion: https://postgr.es/m/17254-a926c89dc03375c2@postgresql.org https://git.postgresql.org/pg/commitdiff/a667b066837849c5e55e0d626f1f7c93e267b8b7
Avoid some other O(N^2) hazards in list manipulation. In the same spirit as 6301c3ada, fix some more places where we were using list_delete_first() in a loop and thereby risking O(N^2) behavior. It's not clear that the lists manipulated in these spots can get long enough to be really problematic ... but it's not clear that they can't, either, and the fixes are simple enough. As before, back-patch to v13. Discussion: https://postgr.es/m/CD2F0E7F-9822-45EC-A411-AE56F14DEA9F@amazon.com https://git.postgresql.org/pg/commitdiff/e9d9ba2a4ddc39e331dd8461b511aa59ec0dc8af
Avoid O(N^2) behavior in SyncPostCheckpoint(). As in commits 6301c3ada and e9d9ba2a4, avoid doing repetitive list_delete_first() operations, since that would be expensive when there are many files waiting to be unlinked. This is a slightly larger change than in those cases. We have to keep the list state valid for calls to AbsorbSyncRequests(), so it's necessary to invent a "canceled" field instead of immediately deleting PendingUnlinkEntry entries. Also, because we might not be able to process all the entries, we need a new list primitive list_delete_first_n(). list_delete_first_n() is almost list_copy_tail(), but it modifies the input List instead of making a new copy. I found a couple of existing uses of the latter that could profitably use the new function. (There might be more, but the other callers look like they probably shouldn't overwrite the input List.) As before, back-patch to v13. Discussion: https://postgr.es/m/CD2F0E7F-9822-45EC-A411-AE56F14DEA9F@amazon.com https://git.postgresql.org/pg/commitdiff/65c6cab1365ac33b11a49a2a193f6b3f9c53e487
Doc: be more precise about conflicts between relation names. Use verbiage like "The name of the table must be distinct from the name of any other relation (table, sequence, index, view, materialized view, or foreign table) in the same schema." in the reference pages for all those object types. The main change here is to mention materialized views explicitly; although a couple of these pages failed to say anything at all about name conflicts. Per suggestion from Daniel Westermann. Discussion: https://postgr.es/m/ZR0P278MB0920D0946509233459AF0DEFD2889@ZR0P278MB0920.CHEP278.PROD.OUTLOOK.COM https://git.postgresql.org/pg/commitdiff/af8c580e5cf32bb85dde70083a260c93a1f783eb
Doc: clean up some places that mentioned template1 but not template0. Improve old text that wasn't updated when we added template0 to the standard database set. Per suggestion from P. Luzanov. Discussion: https://postgr.es/m/163583775122.675.3700595100340939507@wrigleys.postgresql.org https://git.postgresql.org/pg/commitdiff/7d9ec0754afeabb9f336c5220ef415c3ea27a0b6
Fix variable lifespan in ExecInitCoerceToDomain(). This undoes a mistake in 1ec7679f1: domainval and domainnull were meant to live across loop iterations, but they were incorrectly moved inside the loop. The effect was only to emit useless extra EEOP_MAKE_READONLY steps, so it's not a big deal; nonetheless, back-patch to v13 where the mistake was introduced. Ranier Vilela Discussion: https://postgr.es/m/CAEudQAqXuhbkaAp-sGH6dR6Nsq7v28_0TPexHOm6FiDYqwQD-w@mail.gmail.com https://git.postgresql.org/pg/commitdiff/01fc6527034a6f70ed44a078af8f636b1ab64947
Ensure consistent logical replication of datetime and float8 values. In walreceiver, set the publisher's relevant GUCs (datestyle, intervalstyle, extra_float_digits) to the same values that pg_dump uses, and for the same reason: we need the output to be read the same way regardless of the receiver's settings. Without this, it's possible for subscribers to misinterpret transmitted values. Although this is clearly a bug fix, it's not without downsides: subscribers that are storing values into some other datatype, such as text, could get different results than before, and perhaps be unhappy about that. Given the lack of previous complaints, it seems best to change this only in HEAD, and to call it out as an incompatible change in v15. Japin Li, per report from Sadhuprasad Patro Discussion: https://postgr.es/m/CAFF0-CF=D7pc6st-3A9f1JnOt0qmc+BcBPVzD6fLYisKyAjkGA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/f3d4019da5d026f2c3fe5bd258becf6fbb6b4673
Blind attempt to silence SSL compile failures on hamerkop. Buildfarm member hamerkop has been failing for the last few days with errors that look like OpenSSL's X509-related symbols have not been imported into be-secure-openssl.c. It's unclear why this should be, but let's try adding an explicit #include of <openssl/x509v3.h>, as there has long been in fe-secure-openssl.c. Discussion: https://postgr.es/m/1051867.1635720347@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/24f9e49e430b4173d75a570e06abef8e3fd12c5e
Un-break pg_basebackup's MSVC build. Commit 23a1c6578 thought it'd be cute to refactor pg_basebackup/Makefile with a new variable BBOBJS, but our MSVC build system knows nothing of that. Per buildfarm. https://git.postgresql.org/pg/commitdiff/d8bf0a1c1d3429cafb3019f2773e2f3aa68f3b65
Second attempt to silence SSL compile failures on hamerkop. After further investigation, it seems the cause of the problem is our recent decision to start defining WIN32_LEAN_AND_MEAN. That causes <windows.h> to no longer include <wincrypt.h>, which means that the OpenSSL headers are unable to prevent conflicts with that header by #undef'ing the conflicting macros. Apparently, some other system header that be-secure-openssl.c #includes after the OpenSSL headers is pulling in <wincrypt.h>. It's obscure just where that happens and why we're not seeing it on other Windows buildfarm animals. However, it should work to move the OpenSSL #includes to the end of the list. For the sake of future-proofing, do likewise in fe-secure-openssl.c. In passing, remove useless double inclusions of <openssl/ssl.h>. Thanks to Thomas Munro for running down the relevant information. Discussion: https://postgr.es/m/1051867.1635720347@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/1241fcbd7e649414f09f9858ba73e63975dcff64
Disallow making an empty lexeme via array_to_tsvector(). The tsvector data type has always forbidden lexemes to be empty. However, array_to_tsvector() didn't get that memo, and would allow an empty-string array element to become an empty lexeme. This could result in dump/restore failures later, not to mention whatever semantic issues might be behind the original prohibition. However, other functions that take a plain text input directly as a lexeme value do not need a similar restriction, because they only match the string against existing tsvector entries. In particular it'd be a bad idea to make ts_delete() reject empty strings, since that is the most convenient way to clean up any bad data that might have gotten into a tsvector column via this bug. Reflecting on that, let's also remove the prohibition against NULL array elements in tsvector_delete_arr and tsvector_setweight_by_filter. It seems more consistent to ignore them, as an empty-string element would be ignored. There's a case for back-patching this, since it's clearly a bug fix. On balance though, it doesn't seem like something to change in a minor release. Jean-Christophe Arnu Discussion: https://postgr.es/m/CAHZmTm1YVndPgUVRoag2WL0w900XcoiivDDj-gTTYBsG25c65A@mail.gmail.com https://git.postgresql.org/pg/commitdiff/cbe25dcff73a297adbada9dc1d6cad3df18014e9
Blind attempt to fix MSVC pgcrypto build. Commit db7d1a7b0 pulled out Mkvcbuild.pm's custom support for building contrib/pgcrypto, but neglected to inform it that that module can now be built normally. Or at least I guess it can now be built normally. But this is definitely causing bowerbird to fail, since it's trying to test a module it hasn't built. https://git.postgresql.org/pg/commitdiff/3c2c391dc9f82fae181508ebcc2f7621ffefd024
Doc: add some notes about performance of the List functions. Per suggestion from Andres Freund. Discussion: https://postgr.es/m/20211104221248.pgo4h6wvnjl6uvkb@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/27ef132a805c8633ed8bb94ed70be995c681ab1f
contrib/sslinfo needs a fix too to make hamerkop happy. Re-ordering the #include's is a bit problematic here because libpq/libpq-be.h needs to include <openssl/ssl.h>. Instead, let's #undef the unwanted macro after all the #includes. This is definitely uglier than the other way, but it should work despite possible future header rearrangements. (A look at the openssl headers indicates that X509_NAME is the only conflicting symbol that we use.) In passing, remove a related but long-incorrect comment in pg_backup_archiver.h. Discussion: https://postgr.es/m/1051867.1635720347@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/568620dfd6912351b4127435eca5309f823abde8
Silence uninitialized-variable warning. Quite a few buildfarm animals are warning about this, and lapwing is actually failing (because -Werror). It's a false positive AFAICS, so no need to do more than zero the variable to start with. Discussion: https://postgr.es/m/YYXJnUxgw9dZKxlX@paquier.xyz https://git.postgresql.org/pg/commitdiff/c3ec4f8fe867807613c08fe16789434ab1a74a15

Michaël Paquier pushed:

Preserve opclass parameters across REINDEX CONCURRENTLY. The opclass parameter Datums from the old index are fetched in the same way as for predicates and expressions, by grabbing them directly from the system catalogs. They are then copied into the new IndexInfo that will be used for the creation of the new copy. This caused the new index to be rebuilt with default parameters rather than the ones pre-defined by a user. The only way to get back a new index with correct opclass parameters would be to recreate a new index from scratch. The issue has been introduced by 911e702. Author: Michael Paquier Reviewed-by: Zhihong Yu Discussion: https://postgr.es/m/YX0CG/QpLXcPr8HJ@paquier.xyz Backpatch-through: 13 https://git.postgresql.org/pg/commitdiff/add5cf28d48149459466b9aff374d78aebf17482
Add TAP test for pg_receivewal with timeline switch. pg_receivewal is able to follow a timeline switch, but this was not tested. This test uses an empty archive location with a restart done from a slot, making its implementation a tad simpler than if we would reuse an existing archive directory. Author: Ronan Dunklau Reviewed-by: Kyotaro Horiguchi, Michael Paquier Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan https://git.postgresql.org/pg/commitdiff/0f9b9938a0367313fcf6a32fcb7fb5be9e281198
Rework compression options of pg_receivewal. pg_receivewal includes since cada1af the option --compress, to allow the compression of WAL segments using gzip, with a value of 0 (the default) meaning that no compression can be used. This commit introduces a new option, called --compression-method, able to use as values "none", the default, and "gzip", to make things more extensible. The case of --compress=0 becomes fuzzy with this option layer, so we have made the choice to make pg_receivewal return an error when using "none" and a non-zero compression level, meaning that the authorized values of --compress are now [1,9] instead of [0,9]. Not specifying --compress with "gzip" as compression method makes pg_receivewal use the default of zlib instead (Z_DEFAULT_COMPRESSION). The code in charge of finding the streaming start LSN when scanning the existing archives is refactored and made more extensible. While on it, rename "compression" to "compression_level" in walmethods.c, to reduce the confusion with the introduction of the compression method, even if the tar method used by pg_basebackup does not rely on the compression method (yet, at least), but just on the compression level (this area could be improved more, actually). This is in preparation for an upcoming patch that adds LZ4 support to pg_receivewal. Author: Georgios Kokolatos Reviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip Kumar, Robert Haas Discussion: https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP5MwZXCNPybtsGouDsAD1o2L_I5E=@pm.me https://git.postgresql.org/pg/commitdiff/d62bcc8b07f921bad105c7a826702c117ea7be58
Fix some thinkos with pg_receivewal --compression-method. The option name was incorrect in one of the error messages, and the short option 'I' was used in the code but we did not intend things to be this way. While on it, fix the documentation to refer to a "method", and not a "level. Oversights in commit d62bcc8, that I have detected after more review of the LZ4 patch for pg_receivewal. https://git.postgresql.org/pg/commitdiff/9588622945754305836555273a6a3be814db315c
Add support for LZ4 compression in pg_receivewal. pg_receivewal gains a new option, --compression-method=lz4, available when the code is compiled with --with-lz4. Similarly to gzip, this gives the possibility to compress archived WAL segments with LZ4. This option is not compatible with --compress. The implementation uses LZ4 frames, and is compatible with simple lz4 commands. Like gzip, using --synchronous ensures that any data will be flushed to disk within the current .partial segment, so as it is possible to retrieve as much WAL data as possible even from a non-completed segment (this requires completing the partial file with zeros up to the WAL segment size supported by the backend after decompression, but this is the same as gzip). The calculation of the streaming start LSN is able to transparently find and check LZ4-compressed segments. Contrary to gzip where the uncompressed size is directly stored in the object read, the LZ4 chunk protocol does not store the uncompressed data by default. There is contentSize that can be used with LZ4 frames by that would not help if using an archive that includes segments compressed with the defaults of a "lz4" command, where this is not stored. So, this commit has taken the most extensible approach by decompressing the already-archived segment to check its uncompressed size, through a blank output buffer in chunks of 64kB (no actual performance difference noticed with 8kB, 16kB or 32kB, and the operation in itself is actually fast). Tests have been added to verify the creation and correctness of the generated LZ4 files. The latter is achieved by the use of command "lz4", if found in the environment. The tar-based WAL method in walmethods.c, used now only by pg_basebackup, does not know yet about LZ4. Its code could be extended for this purpose. Author: Georgios Kokolatos Reviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip Kumar Discussion: https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP5MwZXCNPybtsGouDsAD1o2L_I5E=@pm.me https://git.postgresql.org/pg/commitdiff/babbbb595d2322da095a1e6703171b3f1f2815cb
Improve psql tab completion for COMMENT. Completion is added for more object types, like domain constraints, text search-ish objects or policies. Moreover, the area is reorganized, changing the list of objects supported by COMMENT to be in the same order as the documentation to ease future additions. Author: Ken Kato Reviewed-by: Fujii Masao, Shinya Kato, Suraj Khamkar, Michael Paquier Discussion: https://postgr.es/m/6e0c2f3f657b229bea32d098d118f307@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/a5b336b8b9e04a93e7c8526302504d2e5201eb80

Álvaro Herrera pushed:

Handle XLOG_OVERWRITE_CONTRECORD in DecodeXLogOp. Failing to do so results in inability of logical decoding to process the WAL stream. Handle it by doing nothing. Backpatch all the way back. Reported-by: Petr Jelínek petr.jelinek@enterprisedb.com https://git.postgresql.org/pg/commitdiff/40c516bba864395c77bcfb1bae65ba9562ba8f71
Reword doc blurb for vacuumdb --analyze-in-stages. Make users aware that using it in a database with existing stats might cause transient problems. Author: Nikolai Berkoff nikolai.berkoff@pm.me Discussion: https://postgr.es/m/s-kSljtWXMWgMfGTztPTPcS80R8FHdOrBxDTnrQI6GMZbT7au1A4b0fzaSFtKwCI8nwN0MhgPLfVOTvJ7DwTjkip4P3d0o4VgrMJs4OLN-o=@pm.me https://git.postgresql.org/pg/commitdiff/00a354a13560dc529ac34a303c85c265aaf033b7
Document default and changeability of log_startup_progress_interval. Review for 9ce346eabf35. Author: Álvaro Herrera alvherre@alvh.no-ip.org Reviewed-by: Robert Haas robertmhaas@gmail.com Discussion: https://postgr.es/m/202110292123.bnf6axcp27vx@alvherre.pgsql https://git.postgresql.org/pg/commitdiff/e543906e217509ad95c1e341de4e874f027f871b
Pipeline mode disallows multicommand strings. ... so mention that in appropriate places of the libpq docs. Backpatch to 14. Reported-by: RekGRpth rekgrpth@gmail.com Discussion: https://postgr.es/m/17235-53bb38fc5be593dc@postgresql.org https://git.postgresql.org/pg/commitdiff/105c1de0197473dac8ada55dc8cf773d782224cb
Document that ALTER TABLE .. TYPE removes statistics. Co-authored-by: Nikolai Berkoff nikolai.berkoff@pm.me Discussion: https://postgr.es/m/vCc8XnwDmlP4ZnHBQLIVxzD405BiYHVC9qZlhIF7IsfxK0gC9mZ4PUUOH0-3y6kv5p-87-3_ljqT1KvQVAnb8OoWhPU3kcqWn2ZpmxRBCQg=@pm.me https://git.postgresql.org/pg/commitdiff/df80f9da5c6541e744eeb20eaca919c7fc189999
Avoid crash in rare case of concurrent DROP. When a role being dropped contains is referenced by catalog objects that are concurrently also being dropped, a crash can result while trying to construct the string that describes the objects. Suppress that by ignoring objects whose descriptions are returned as NULL. The majority of relevant codesites were already cautious about this already; we had just missed a couple. This is an old bug, so backpatch all the way back. Reported-by: Alexander Lakhin exclusion@gmail.com Discussion: https://postgr.es/m/17126-21887f04508cb5c8@postgresql.org https://git.postgresql.org/pg/commitdiff/d74b54b3ddf710926a44bf3f9c87c00e6f82d825

Daniel Gustafsson pushed:

Replace unicode characters in comments with ascii. The unicode characters, while in comments and not code, caused MSVC to emit compiler warning C4819: The file contains a character that cannot be represented in the current code page (number). Save the file in Unicode format to prevent data loss. Fix by replacing the characters in print.c with descriptive comments containing the codepoints and symbol names, and remove the character in brin_bloom.c which was a footnote reference copied from the paper citation. Per report from hamerkop in the buildfarm. Reviewed-by: Tom Lane tgl@sss.pgh.pa.us Discussion: https://postgr.es/m/340E4118-0D0C-4E85-8141-8C40EB22DA3A@yesql.se https://git.postgresql.org/pg/commitdiff/43a134f28b350c4b731db9dddf2f53c407a7077f

Amit Kapila pushed:

Replace XLOG_INCLUDE_XID flag with a more localized flag. Commit 0bead9af484c introduced XLOG_INCLUDE_XID flag to indicate that the WAL record contains subXID-to-topXID association. It uses that flag later to mark in CurrentTransactionState that top-xid is logged so that we should not try to log it again with the next WAL record in the current subtransaction. However, we can use a localized variable to pass that information. In passing, change the related function and variable names to make them consistent with what the code is actually doing. Author: Dilip Kumar Reviewed-by: Alvaro Herrera, Amit Kapila Discussion: https://postgr.es/m/E1mSoYz-0007Fh-D9@gemulon.postgresql.org https://git.postgresql.org/pg/commitdiff/71db6459e6e4ef623e98f3b1e3e9fed1bfb0ae3b
Move MarkCurrentTransactionIdLoggedIfAny() out of the critical section. We don't modify any shared state in this function which could cause problems for any concurrent session. This will make it look similar to the other updates for the same structure (TransactionState) which avoids confusion for future readers of code. Author: Dilip Kumar Reviewed-by: Amit Kapila Discussion: https://postgr.es/m/E1mSoYz-0007Fh-D9@gemulon.postgresql.org https://git.postgresql.org/pg/commitdiff/335397456b7e3f9f619038cb322fbfc9dd649d4f

Fujii Masao pushed:

pgbench: Improve error-handling in pgbench. Previously failures of initial connection and logfile open caused pgbench to proceed the benchmarking, report the incomplete results and exit with status 2. It didn't make sense to proceed the benchmarking even when pgbench could not start as prescribed. This commit improves pgbench so that early errors that occur when starting benchmark such as those failures should make pgbench exit immediately with status 1. Author: Yugo Nagata Reviewed-by: Fabien COELHO, Kyotaro Horiguchi, Fujii Masao Discussion: https://postgr.es/m/TYCPR01MB5870057375ACA8A73099C649F5349@TYCPR01MB5870.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/cd29be5459f0e138c0f19d49ee588feeda78e3c9
pgbench: Fix typo in comment. Discussion: https://postgr.es/m/f9041ec2-46b6-1b41-0e84-9c8a1e2d6bda@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/d8dba4d03068eeee1ea3ffc8e7c7b4fa3e35a7f4

Peter Geoghegan pushed:

Don't overlook indexes during parallel VACUUM. Commit b4af70cb, which simplified state managed by VACUUM, performed refactoring of parallel VACUUM in passing. Confusion about the exact details of the tasks that the leader process is responsible for led to code that made it possible for parallel VACUUM to miss a subset of the table's indexes entirely. Specifically, indexes that fell under the min_parallel_index_scan_size size cutoff were missed. These indexes are supposed to be vacuumed by the leader (alongside any parallel unsafe indexes), but weren't vacuumed at all. Affected indexes could easily end up with duplicate heap TIDs, once heap TIDs were recycled for new heap tuples. This had generic symptoms that might be seen with almost any index corruption involving structural inconsistencies between an index and its table. To fix, make sure that the parallel VACUUM leader process performs any required index vacuuming for indexes that happen to be below the size cutoff. Also document the design of parallel VACUUM with these below-size-cutoff indexes. It's unclear how many users might be affected by this bug. There had to be at least three indexes on the table to hit the bug: a smaller index, plus at least two additional indexes that themselves exceed the size cutoff. Cases with just one additional index would not run into trouble, since the parallel VACUUM cost model requires two larger-than-cutoff indexes on the table to apply any parallel processing. Note also that autovacuum was not affected, since it never uses parallel processing. Test case based on tests from a larger patch to test parallel VACUUM by Masahiko Sawada. Many thanks to Kamigishi Rei for her invaluable help with tracking this problem down. Author: Peter Geoghegan pg@bowt.ie Author: Masahiko Sawada sawada.mshk@gmail.com Reported-By: Kamigishi Rei iijima.yun@koumakan.jp Reported-By: Andrew Gierth andrew@tao11.riddles.org.uk Diagnosed-By: Andres Freund andres@anarazel.de Bug: #17245 Discussion: https://postgr.es/m/17245-ddf06aaf85735f36@postgresql.org Discussion: https://postgr.es/m/20211030023740.qbnsl2xaoh2grq3d@alap3.anarazel.de Backpatch: 14-, where the refactoring commit appears. https://git.postgresql.org/pg/commitdiff/9bacec15b67d1a643915858f054790f36b2b7871
Fix parallel amvacuumcleanup safety bug. Commit b4af70cb inverted the return value of the function parallel_processing_is_safe(), but missed the amvacuumcleanup test. Index AMs that don't support parallel cleanup at all were affected. The practical consequences of this bug were not very serious. Hash indexes are affected, but since they just return the number of blocks during hashvacuumcleanup anyway, it can't have had much impact. Author: Masahiko Sawada sawada.mshk@gmail.com Discussion: https://postgr.es/m/CAD21AoA-Em+aeVPmBbL_s1V-ghsJQSxYL-i3JP8nTfPiD1wjKw@mail.gmail.com Backpatch: 14-, where commit b4af70cb appears. https://git.postgresql.org/pg/commitdiff/c59278a1aa5ef2ee8a6d5d83bd987a7ce5c89e84
Add another old commit to git-blame-ignore-revs. Add another historic pgindent commit that was missed by the initial work done in commit 8e638845. https://git.postgresql.org/pg/commitdiff/581055c32fbb5018431265877754cbd8019bc012
Add various assertions to heap pruning code. These assertions document (and verify) our high level assumptions about how pruning can and cannot affect existing items from target heap pages. For example, one of the new assertions verifies that pruning does not set a heap-only tuple to LP_DEAD. Author: Peter Geoghegan pg@bowt.ie Reviewed-By: Andres Freund andres@anarazel.de Discussion: https://postgr.es/m/CAH2-Wz=vhvBx1GjF+oueHh8YQcHoQYrMi0F0zFMHEr8yc4sCoA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/5cd7eb1f1c32e1b95894f28b277b4e4b89add772
Add hardening to catch invalid TIDs in indexes. Add hardening to the heapam index tuple deletion path to catch TIDs in index pages that point to a heap item that index tuples should never point to. The corruption we're trying to catch here is particularly tricky to detect, since it typically involves "extra" (corrupt) index tuples, as opposed to the absence of required index tuples in the index. For example, a heap TID from an index page that turns out to point to an LP_UNUSED item in the heap page has a good chance of being caught by one of the new checks. There is a decent chance that the recently fixed parallel VACUUM bug (see commit 9bacec15) would have been caught had that particular check been in place for Postgres 14. No backpatch of this extra hardening for now, though. Author: Peter Geoghegan pg@bowt.ie Reviewed-By: Andres Freund andres@anarazel.de Discussion: https://postgr.es/m/CAH2-Wzk-4_raTzawWGaiqNvkpwDXxv3y1AQhQyUeHfkU=tFCeA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/e7428a99a13f973549aab30c57ec8380ddda1869
Update obsolete heap pruning comments. Add new comments that spell out what VACUUM expects from heap pruning: pruning must never leave behind DEAD tuples that still have tuple storage. This has at least been the case since commit 8523492d, which established the principle that vacuumlazy.c doesn't have to deal with DEAD tuples that still have tuple storage directly, except perhaps by simply retrying pruning (to handle a rare corner case involving concurrent transaction abort). In passing, update some references to old symbol names that were missed by the snapshot scalability work (specifically commit dc7420c2c9). https://git.postgresql.org/pg/commitdiff/f214960adde6028a39ba3014b1ab2b224faeefed
Update obsolete reference in vacuumlazy.c. Oversight in commit 7ab96cf6. https://git.postgresql.org/pg/commitdiff/02f9fd129432cab565b2a3cb9f3b3a5000dfe540

Peter Eisentraut pushed:

Fix incorrect format placeholder. https://git.postgresql.org/pg/commitdiff/ef6f047d2c87b91318364341c058dd6b715951b2
pgcrypto: Remove non-OpenSSL support. pgcrypto had internal implementations of some encryption algorithms, as an alternative to calling out to OpenSSL. These were rarely used, since most production installations are built with OpenSSL. Moreover, maintaining parallel code paths makes the code more complex and difficult to maintain. This patch removes these internal implementations. Now, pgcrypto is only built if OpenSSL support is configured. Reviewed-by: Daniel Gustafsson daniel@yesql.se Discussion: /message-id/flat/0b42f1df-8cba-6a30-77d7-acc241cc88c1%40enterprisedb.com https://git.postgresql.org/pg/commitdiff/db7d1a7b0530e8cbd045744e1c75b0e63fb6916f

Heikki Linnakangas pushed:

Fix snapshot reference leak if lo_export fails. If lo_export() fails to open the target file or to write to it, it leaks the created LargeObjectDesc and its snapshot in the top-transaction context and resource owner. That's pretty harmless, it's a small leak after all, but it gives the user a "Snapshot reference leak" warning. Fix by using a short-lived memory context and no resource owner for transient LargeObjectDescs that are opened and closed within one function call. The leak is easiest to reproduce with lo_export() on a directory that doesn't exist, but in principle the other lo_* functions could also fail. Backpatch to all supported versions. Reported-by: Andrew B Reviewed-by: Alvaro Herrera Discussion: /message-id/32bf767a-2d65-71c4-f170-122f416bab7e@iki.fi https://git.postgresql.org/pg/commitdiff/6b1b405ebfdce9da47f59d8d4144b1168709fbce
Update alternative expected output file. Previous commit added a test to 'largeobject', but neglected the alternative expected output file 'largeobject_1.source'. Per failure on buildfarm animal 'hamerkop'. Discussion: /message-id/DBA08346-9962-4706-92D1-230EE5201C10@yesql.se https://git.postgresql.org/pg/commitdiff/d5ab0681bf1bbf6c0c2cba9a2d55fe8e080597b6

Robert Haas pushed:

amcheck: Add additional TOAST pointer checks. Expand the checks of toasted attributes to complain if the rawsize is overlarge. For compressed attributes, also complain if compression appears to have expanded the attribute or if the compression method is invalid. Mark Dilger, reviewed by Justin Pryzby, Alexander Alekseev, Heikki Linnakangas, Greg Stark, and me. Discussion: http://postgr.es/m/8E42250D-586A-4A27-B317-8B062C3816A8@enterprisedb.com https://git.postgresql.org/pg/commitdiff/bd807be6935929bdefe74d1258ca08048f0aafa3
Introduce 'bbsink' abstraction to modularize base backup code. The base backup code has accumulated a healthy number of new features over the years, but it's becoming increasingly difficult to maintain and further enhance that code because there's no real separation of concerns. For example, the code that understands knows the details of how we send data to the client using the libpq protocol is scattered throughout basebackup.c, rather than being centralized in one place. To try to improve this situation, introduce a new 'bbsink' object which acts as a recipient for archives generated during the base backup progress and also for the backup manifest. This commit introduces three types of bbsink: a 'copytblspc' bbsink forwards the backup to the client using one COPY OUT operation per tablespace and another for the manifest, a 'progress' bbsink performs command progress reporting, and a 'throttle' bbsink performs rate-limiting. The 'progress' and 'throttle' bbsink types also forward the data to a successor bbsink; at present, the last bbsink in the chain will always be of type 'copytblspc'. There are plans to add more types of 'bbsink' in future commits. This abstraction is a bit leaky in the case of progress reporting, but this still seems cleaner than what we had before. Patch by me, reviewed and tested by Andres Freund, Sumanta Mukherjee, Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja, Mark Dilger, and Jeevan Ladhe. Discussion: https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com Discussion: https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com https://git.postgresql.org/pg/commitdiff/bef47ff85df18bf4a3a9b13bd2a54820e27f3614
Introduce 'bbstreamer' abstraction to modularize pg_basebackup. pg_basebackup knows how to do quite a few things with a backup that it gets from the server, like just write out the files, or compress them first, or even parse the tar format and inject a modified postgresql.auto.conf file into the archive generated by the server. Unforatunely, this makes pg_basebackup.c a very large source file, and also somewhat difficult to enhance, because for example the knowledge that the server is sending us a 'tar' file rather than some other sort of archive is spread all over the place rather than centralized. In an effort to improve this situation, this commit invents a new 'bbstreamer' abstraction. Each archive received from the server is fed to a bbstreamer which may choose to dispose of it or pass it along to some other bbstreamer. Chunks may also be "labelled" according to whether they are part of the payload data of a file in the archive or part of the archive metadata. So, for example, if we want to take a tar file, modify the postgresql.auto.conf file it contains, and the gzip the result and write it out, we can use a bbstreamer_tar_parser to parse the tar file received from the server, a bbstreamer_recovery_injector to modify the contents of postgresql.auto.conf, a bbstreamer_tar_archiver to replace the tar headers for the file modified in the previous step with newly-built ones that are correct for the modified file, and a bbstreamer_gzip_writer to gzip and write the resulting data. Only the objects with "tar" in the name know anything about the tar archive format, and in theory we could re-archive using some other format rather than "tar" if somebody wanted to write the code. These chances do add a substantial amount of code, but I think the result is a lot more maintainable and extensible. pg_basebackup.c itself shrinks by roughly a third, with a lot of the complexity previously contained there moving into the newly-added files. Patch by me. The larger patch series of which this is a part has been reviewed and tested at various times by Andres Freund, Sumanta Mukherjee, Dilip Kumar, Suraj Kharage, Dipesh Pandit, Tushar Ahuja, Mark Dilger, Sergei Kornilov, and Jeevan Ladhe. Discussion: https://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com Discussion: https://postgr.es/m/CA+TgmoZvqk7UuzxsX1xjJRmMGkqoUGYTZLDCH8SmU1xTPr1Xig@mail.gmail.com https://git.postgresql.org/pg/commitdiff/23a1c6578c87fca0e361c4f5f9a07df5ae1f9858
Don't set ThisTimeLineID when there's no reason to do so. In slotfuncs.c, pg_replication_slot_advance() needs to determine the LSN up to which the slot should be advanced, but that doesn't require us to update ThisTimeLineID, because none of the code called from here depends on it. If the replication slot is logical, pg_logical_replication_slot_advance will call read_local_xlog_page, which does use ThisTimeLineID, but also takes care of making sure it's up to date. If the replication slot is physical, the timeline isn't used for anything at all. In logicalfuncs.c, pg_logical_slot_get_changes_guts() has the same issue: the only code we're going to run that cares about timelines is in or downstream of read_local_xlog_page, which already makes sure that the correct value gets set. Hence, don't do it here. Patch by me, reviewed and tested by Michael Paquier, Amul Sul, and Álvaro Herrera. Discussion: https://postgr.es/m/CA+TgmobfAAqhfWa1kaFBBFvX+5CjM=7TE=n4r4Q1o2bjbGYBpA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/caf1f675b88d1aa67ea3fb642e8f38b470cc911e
Remove all use of ThisTimeLineID global variable outside of xlog.c. All such code deals with this global variable in one of three ways. Sometimes the same functions use it in more than one of these ways at the same time. First, sometimes it's an implicit argument to one or more functions being called in xlog.c or elsewhere, and must be set to the appropriate value before calling those functions lest they misbehave. In those cases, it is now passed as an explicit argument instead. Second, sometimes it's used to obtain the current timeline after the end of recovery, i.e. the timeline to which WAL is being written and flushed. Such code now calls GetWALInsertionTimeLine() or relies on the new out parameter added to GetFlushRecPtr(). Third, sometimes it's used during recovery to store the current replay timeline. That can change, so such code must generally update the value before each use. It can still do that, but must now use a local variable instead. The net effect of these changes is to reduce by a fair amount the amount of code that is directly accessing this global variable. That's good, because history has shown that we don't always think clearly about which timeline ID it's supposed to contain at any given point in time, or indeed, whether it has been or needs to be initialized at any given point in the code. Patch by me, reviewed and tested by Michael Paquier, Amul Sul, and Álvaro Herrera. Discussion: https://postgr.es/m/CA+TgmobfAAqhfWa1kaFBBFvX+5CjM=7TE=n4r4Q1o2bjbGYBpA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/e997a0c642860a96df0151cbeccfecbdf0450d08
Change ThisTimeLineID from a global variable to a local variable. StartupXLOG() still has ThisTimeLineID as a local variable, but the remaining code in xlog.c now needs to the relevant TimeLineID by some other means. Mostly, this means that we now pass it as a function parameter to a bunch of functions where we didn't previously. However, a few cases require special handling: - In functions that might be called by outside callers who wouldn't necessarily know what timeline to specify, we get the timeline ID from shared memory. XLogCtl->ThisTimeLineID can be used in most cases since recovery is known to have completed by the time those functions are called. In xlog_redo(), we can use XLogCtl->replayEndTLI. - XLogFileClose() needs to know the TLI of the open logfile. Do that with a new global variable openLogTLI. While someone could argue that this is just trading one global variable for another, the new one has a far more narrow purposes and is referenced in just a few places. - read_backup_label() now returns the TLI that it obtains by parsing the backup_label file. Previously, ReadRecord() could be called to parse the checkpoint record without ThisTimeLineID having been initialized. Now, the timeline is passed down, and I didn't want to pass an uninitialized variable; this change lets us avoid that. The old coding didn't seem to have any practical consequences that we need to worry about, but this is cleaner. - In BootstrapXLOG(), it's just a constant. Patch by me, reviewed and tested by Michael Paquier, Amul Sul, and Álvaro Herrera. Discussion: https://postgr.es/m/CA+TgmobfAAqhfWa1kaFBBFvX+5CjM=7TE=n4r4Q1o2bjbGYBpA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/4a92a1c3d1c361ffb031ed05bf65b801241d7cdd
Remove tests added by bd807be6935929bdefe74d1258ca08048f0aafa3. The buildfarm is unhappy. It's not obvious why it doesn't like these tests, but let's remove them until we figure it out. Discussion: http://postgr.es/m/462618.1636171009@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/ccf289745d3e50360653181dce6a277a1fc79730

Tomáš Vondra pushed:

Fix handling of NaN values in BRIN minmax multi. When calculating distance between float4/float8 values, we need to be a bit more careful about NaN values in order not to trigger assert. We consider NaN values to be equal (distace 0.0) and in infinite distance from all other values. On builds without asserts, this issue is mostly harmless - the ranges may be merged in less efficient order, but the index is still correct. Per report from Andreas Seltenreich. Backpatch to 14, where this new BRIN opclass was introduced. Reported-by: Andreas Seltenreich Discussion: https://postgr.es/m/87r1bw9ukm.fsf@credativ.de https://git.postgresql.org/pg/commitdiff/d91353f4b21f10417d142e6ac17a0490adae867c
Mark mystreamer variable as PG_USED_FOR_ASSERTS_ONLY. Silences warnings about unused variable, when built without asserts. https://git.postgresql.org/pg/commitdiff/dafcf887daa472b0a49bee7e07042372bc37cee4
Add bool GiST opclass to btree_gist. Adds bool opclass to btree_gist extension, to allow creating GiST indexes on bool columns. GiST indexes on a single bool column don't seem particularly useful, but this allows defining exclusion constraings involving a bool column, for example. Author: Emre Hasegeli Reviewed-by: Andrey Borodin Discussion: https://postgr.es/m/CAE2gYzyDKJBZngssR84VGZEN=Ux=V9FV23QfPgo+7-yYnKKg4g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/57e3c5160b24e61758f817feb7aac152cd695c6f
Fix gist_bool_ops to use gbtreekey2. Commit 57e3c5160b added a new GiST bool opclass, but it used gbtreekey4 to store the data, which left two bytes undefined, as reported by skink, our valgrind animal. There was a bit more confusion, because the opclass also used gbtreekey8 in the definition. Fix by defining a new gbtreekey2 struct, and using it in all the places. Discussion: https://postgr.es/m/CAE2gYzyDKJBZngssR84VGZEN=Ux=V9FV23QfPgo+7-yYnKKg4g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/e2fbb883720aa222f61eb9f3affad1c63bac7cbb

Alexander Korotkov pushed:

Reset lastOverflowedXid on standby when needed. Currently, lastOverflowedXid is never reset. It's just adjusted on new transactions known to be overflowed. But if there are no overflowed transactions for a long time, snapshots could be mistakenly marked as suboverflowed due to wraparound. This commit fixes this issue by resetting lastOverflowedXid when needed altogether with KnownAssignedXids. Backpatch to all supported versions. Reported-by: Stan Hu Discussion: https://postgr.es/m/CAMBWrQ%3DFp5UAsU_nATY7EMY7NHczG4-DTDU%3DmCvBQZAQ6wa2xQ%40mail.gmail.com Author: Kyotaro Horiguchi, Alexander Korotkov Reviewed-by: Stan Hu, Simon Riggs, Nikolay Samokhvalov, Andrey Borodin, Dmitry Dolgov https://git.postgresql.org/pg/commitdiff/05e6e78c1840d07154a4b52092178a2d1ad39445

Andres Freund pushed:

windows: Remove use of WIN32_LEAN_AND_MEAN from crashdump.c. Since 8162464a25e we do so in win32_port.h. But it likely didn't do much before that either, because at that point windows.h was already included via win32_port.h. Reported-By: Tom Lane Discussion: https://postgr.es/m/612842.1636237461@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/87bb606b20ae4e52fa45eda2d9914c19eb7eea5e

PostgreSQL Weekly News - October 31, 2021

Mon, 01 Nov 2021 00:00:00 +0000

PostgreSQL Weekly News - October 31, 2021

Happy Hallowe'en!

PostgreSQL Product News

pg_statement_rollback v1.3, an extension that adds server side transaction with rollback at statement level, released.

PostgreSQL Jobs for October

https://archives.postgresql.org/pgsql-jobs/2021-10/

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Michaël Paquier pushed:

Add replication command READ_REPLICATION_SLOT. The command is supported for physical slots for now, and returns the type of slot, its restart_lsn and its restart_tli. This will be useful for an upcoming patch related to pg_receivewal, to allow the tool to be able to stream from the position of a slot, rather than the last WAL position flushed by the backend (as reported by IDENTIFY_SYSTEM) if the archive directory is found as empty, which would be an advantage in the case of switching to a different archive locations with the same slot used to avoid holes in WAL segment archives. Author: Ronan Dunklau Reviewed-by: Kyotaro Horiguchi, Michael Paquier, Bharath Rupireddy Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan https://git.postgresql.org/pg/commitdiff/b4ada4e19fd7bedb433e46516ccd0ca4213d2719
Allow pg_receivewal to stream from a slot's restart LSN. Prior to this patch, when running pg_receivewal, the streaming start point would be the current location of the archives if anything is found in the local directory where WAL segments are written, and pg_receivewal would fall back to the current WAL flush location if there are no archives, as of the result of an IDENTIFY_SYSTEM command. If for some reason the WAL files from pg_receivewal were moved, it is better to try a restart where we left at, which is the replication slot's restart_lsn instead of skipping right to the current flush location, to avoid holes in the WAL backed up. This commit changes pg_receivewal to use the following sequence of methods to determine the starting streaming LSN: - Scan the local archives. - Use the slot's restart_lsn, if supported by the backend and if a slot is defined. - Fallback to the current flush LSN as reported by IDENTIFY_SYSTEM. To keep compatibility with older server versions, we only attempt to use READ_REPLICATION_SLOT if the backend version is at least 15, and fallback to the older behavior of streaming from the current flush LSN if the command is not supported. Some TAP tests are added to cover this feature. Author: Ronan Dunklau Reviewed-by: Kyotaro Horiguchi, Michael Paquier, Bharath Rupireddy Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan https://git.postgresql.org/pg/commitdiff/f61e1dd2cee6b1a1da75c2bb0ca3bc72f18748c1
Fix overly-lax regex pattern in TAP test of READ_REPLICATION_SLOT. The case checking for a NULL output when a slot does not exist was too lax, as it was passing for any output generated by the query. This fixes the matching pattern to be what it should be, matching only on "||". Oversight in b4ada4e. https://git.postgresql.org/pg/commitdiff/0db343dc13bc8657976c39ddbf7e0c7db8b2efff
doc: Fix grammar in page of pg_receivewal. Introduced by f61e1dd. Author: Kyotaro Horiguchi Discussion: https://postgr.es/m/20211026.112304.1962954080884317968.horikyota.ntt@gmail.com https://git.postgresql.org/pg/commitdiff/8af09daf5629e9b85f37cc23983819b8ccd11b43
Add test for copy of shared dependencies from template database. As 98ec35b has proved, there has never been any coverage in this area of the code. This commit adds a new TAP test with a template database that includes a small set of shared dependencies copied to a new database. The test is added in createdb, where we have never tested that -T generates a query with TEMPLATE, either. Reviewed-by: Tom Lane Discussion: https://postgr.es/m/YXDTl+PfSnqmbbkE@paquier.xyz https://git.postgresql.org/pg/commitdiff/70bfc5ae537c8bfeed4849b7d9f814de89a155fe
doc: Fix link to SELinux user guide in sepgsql page. Reported-by: Anton Voloshin Discussion: https://postgr.es/m/15a86d4e-a237-1acd-18a2-fd69730f1ab9@postgrespro.ru Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/cc1853b30048307d93f8aa30f4d64f88b527f04d
Add TAP test for archive_cleanup_command and recovery_end_command. This adds tests checking for the execution of both commands. The recovery test 002_archiving.pl is nicely adapted to that, as promotion is triggered already twice there, and even if any of those commands fail they don't affect recovery or promotion. A command success is checked using a file generated by an "echo" command, that should be able to work in all the buildfarm environments, even Msys (but we'll know soon about that). Command failure is tested with an "echo" command that points to a path that does not exist, scanning the backend logs to make sure that the failure happens. Both rely on the backend triggering the commands from the root of the data folder, making its logic more robust. Thanks to Neha Sharma for the extra tests on Windows. Author: Amul Sul, Michael Paquier Reviewed-by: Andres Freund, Euler Taveira Discussion: https://postgr.es/m/CAAJ_b95R_c4T5moq30qsybSU=eDzDHm=4SPiAWaiMWc2OW7=1Q@mail.gmail.com https://git.postgresql.org/pg/commitdiff/46dea2419ee7895a4eb3d048317682e6f18a17e1
Speed up TAP tests of pg_receivewal. This commit improves the speed of those tests by 25~30%, using some simple ideas to reduce the amount of data written by pg_receivewal: - Use a segment size of 1MB. While reducing the amount of data zeroed by pg_receivewal for the new segments, this improves the code coverage with a non-default segment size. - In the last test involving a slot's restart_lsn, generate a checkpoint to advance the redo LSN and the WAL retained by the slot created, reducing the number of segments that need to be archived. This counts for most of the gain. - Minimize the amount of data inserted into the dummy table. Reviewed-by: Ronan Dunklau Discussion: https://postgr.es/m/YXqYKAdVEqmyTltK@paquier.xyz https://git.postgresql.org/pg/commitdiff/d680992af5406245f769b697fbb4e130e6220664

Heikki Linnakangas pushed:

Clarify the logic in a few places in the new balanced merge code. In selectnewtape(), use 'nOutputTapes' rather than 'nOutputRuns' in the check for whether to start a new tape or to append a new run to an existing tape. Until 'maxTapes' is reached, nOutputTapes is always equal to nOutputRuns, so it doesn't change the logic, but it seems more logical to compare # of tapes with # of tapes. Also, currently maxTapes is never modified after the merging begins, but written this way, the code would still work if it was. (Although the nOutputRuns == nOutputTapes assertion would need to be removed and using nOutputRuns % nOutputTapes to distribute the runs evenly across the tapes wouldn't do a good job anymore). Similarly in mergeruns(), change to USEMEM(state->tape_buffer_mem) to account for the memory used for tape buffers. It's equal to availMem currently, but tape_buffer_mem is more direct and future-proof. For example, if we changed the logic to only allocate half of the remaining memory to tape buffers, USEMEM(state->tape_buffer_mem) would still be correct. Coverity complained about these. Hopefully this patch helps it to understand the logic better. Thanks to Tom Lane for initial analysis. https://git.postgresql.org/pg/commitdiff/166f94377c886516ca986ef8a623cd2e854fe911

Robert Haas pushed:

StartupXLOG: Call CleanupAfterArchiveRecovery after XLogReportParameters. This does a better job grouping related operations together, since all of the WAL records that we need to write prior to allowing WAL writes generally and written by a single uninterrupted stretch of code. Since CleanupAfterArchiveRecovery() just (1) runs recovery_end_command, (2) removes non-parent xlog files, and (3) archives any final partial segment, this should be safe, because all of those things are pretty much unrelated to the WAL record written by XLogReportParameters(). Amul Sul, per a suggestion from me Discussion: http://postgr.es/m/CAAJ_b97fysj6sRSQEfOHj-y8Jfd5uPqOgO74qast89B4WfD+TA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/a75dbf7f9ee6ff0c0e2ab4b224b04fc50c4e6577
StartupXLOG: Don't repeatedly disable/enable local xlog insertion. All the code that runs in the startup process to write WAL records before that's allowed generally is now consecutive, so there's no reason to shut the facility to write WAL locally off and then turn it on again three times in a row. Unfortunately, this requires a slight kludge in the checkpointer, which needs to separately enable writing WAL in order to write the checkpoint record. Because that code might run in the same process as StartupXLOG() if we are in single-user mode, we must save/restore the state of the LocalXLogInsertAllowed flag. Hopefully, we'll be able to eliminate this wart in further refactoring, but it's not too bad anyway. Amul Sul, with modifications by me. Discussion: http://postgr.es/m/CAAJ_b97fysj6sRSQEfOHj-y8Jfd5uPqOgO74qast89B4WfD+TA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/18e0913a420349d373cfd8e45b91b4777501fb74
Remove useless code from CreateReplicationSlot. According to the comments, we initialize sendTimeLineIsHistoric and sendTimeLine here for the benefit of WalSndSegmentOpen. However, the only way that can happen is if logical_read_xlog_page calls WALRead. And since logical_read_xlog_page initializes the same global variables internally, we don't need to also do it here. These initializations have been here since replication slots were introduced in commit 858ec11858a914d4c380971985709b6d6b7dd6fc. They were certainly useless at that time, too, because logical decoding didn't yet exist then, and physical replication doesn't examine any WAL at the time of slot creation. I haven't checked all the intermediate versions, but I suspect there's no point at which this code ever did anything useful. To reduce future confusion, remove the code. Since there's no functional defect, no back-patch. Discussion: http://postgr.es/m/CA+TgmobSWzacEs+r6C-7DrOPDHoDar4i9gzxB3SCBr5qjnLmVQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/902a2c280012557b85c7e0fce3f6f0e355cb2d69
Add enable_timeout_every() to fire the same timeout repeatedly. enable_timeout_at() and enable_timeout_after() can still be used when you want to fire a timeout just once. Patch by me, per a suggestion from Tom Lane. Discussion: http://postgr.es/m/2992585.1632938816@sss.pgh.pa.us Discussion: http://postgr.es/m/CA+TgmoYqSF5sCNrgTom9r3Nh=at4WmYFD=gsV-omStZ60S0ZUQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/732e6677a667c03b1551a855e3216644b0f125ec
Report progress of startup operations that take a long time. Users sometimes get concerned whe they start the server and it emits a few messages and then doesn't emit any more messages for a long time. Generally, what's happening is either that the system is taking a long time to apply WAL, or it's taking a long time to reset unlogged relations, or it's taking a long time to fsync the data directory, but it's not easy to tell which is the case. To fix that, add a new 'log_startup_progress_interval' setting, by default 10s. When an operation that is known to be potentially long-running takes more than this amount of time, we'll log a status update each time this interval elapses. To avoid undesirable log chatter, don't log anything about WAL replay when in standby mode. Nitin Jadhav and Robert Haas, reviewed by Amul Sul, Bharath Rupireddy, Justin Pryzby, Michael Paquier, and Álvaro Herrera. Discussion: https://postgr.es/m/CA+TgmoaHQrgDFOBwgY16XCoMtXxsrVGFB2jNCvb7-ubuEe1MGg@mail.gmail.com Discussion: https://postgr.es/m/CAMm1aWaHF7VE69572_OLQ+MgpT5RUiUDgF1x5RrtkJBLdpRj3Q@mail.gmail.com https://git.postgresql.org/pg/commitdiff/9ce346eabf350a130bba46be3f8c50ba28506969
Initialize variable to placate compiler. Per Nathan Bossart. Discussion: http://postgr.es/m/FECEE7FC-CB74-45A9-BB24-89FEE52A9585@amazon.com https://git.postgresql.org/pg/commitdiff/a030a0c5ccb113ccd09d0f0b82f1edb5e49ed607
When fetching WAL for a basebackup, report errors with a sensible TLI. The previous code used ThisTimeLineID, which need not even be initialized here, although it usually was in practice, because pg_basebackup issues IDENTIFY_SYSTEM before calling BASE_BACKUP, and that initializes ThisTimeLineID as a side effect. That's not really good enough, though, not only because we shoudn't be counting on side effects like that, but also because the TLI could change meanwhile. Fortunately, we have convenient access to more meaningful TLI values, so use those instead. Because of the way this logic is coded, the consequences of using a possibly-incorrect TLI here are no worse than a slightly confusing error message, I don't want to take any risk here, so no back-patch at least for now. Patch by me, reviewed by Kyotaro Horiguchi and Michael Paquier Discussion: http://postgr.es/m/CA+TgmoZRNWGWYDX9RgTXMG6_nwSdB=PB-PPRUbvMUTGfmL2sHQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/2f5c4397c39dea49c5608ba583868e26d767fc32
Fix race condition in startup progress reporting. Commit 9ce346eabf350a130bba46be3f8c50ba28506969 added startup progress reporting, but begin_startup_progress_phase has a race condition: the timeout for the previous phase might fire just before we reschedule the interrupt for the next phase. To avoid the race, disable the timeout, clear the flag, and then re-enable the timeout. Patch by me, reviewed by Nitin Jadhav. Discussion: https://postgr.es/m/CA+TgmoYq38i6iAzfRLVxA6Cm+wMCf4WM8wC3o_a+X_JvWC8bJg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/5ccceb2946d4104804f8dca67515b602f5e78cdd

Thomas Munro pushed:

Reject huge_pages=on if shared_memory_type=sysv. It doesn't work (it could, but hasn't been implemented). Back-patch to 12, where shared_memory_type arrived. Reported-by: Alexander Lakhin exclusion@gmail.com Reviewed-by: Alexander Lakhin exclusion@gmail.com Discussion: https://postgr.es/m/163271880203.22789.1125998876173795966@wrigleys.postgresql.org https://git.postgresql.org/pg/commitdiff/8781b0ce25e702ba4a4f032d00da7acdef8dbfe1

Daniel Gustafsson pushed:

Ensure that slots are zeroed before use. The previous coding relied on the memory for the slots being zeroed elsewhere, which while it was true in this case is not an contract which is guaranteed to hold. Explicitly clear the tts_isnull array to ensure that the slots are filled from a known state. Backpatch to v14 where the catalog multi-inserts were introduced. Reviewed-by: Michael Paquier michael@paquier.xyz Discussion: https://postgr.es/m/CAJ7c6TP0AowkUgNL6zcAK-s5HYsVHVBRWfu69FRubPpfwZGM9A@mail.gmail.com Backpatch-through: 14 https://git.postgresql.org/pg/commitdiff/e63ce9e8d6ac8dced20592c4134004640f9f5644
Fix VPATH builds for src/test/ssl targets. Commit b4c4a00ea refactored the gist of the sslfiles target into a separate makefile in order to override settings in Makefile.global. The invocation of this this file didn't however include the absolute path for VPATH builds, resulting in "make clean" failing. Fix by providing the path to the new makefile. Reported-by: Andres Freund andres@anarazel.de Discussion: https://postgr.es/m/20211026174152.jjcagswnbhxu7uqz@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/349cd8c582a1e666c9c804850cf5b532b86cd1b4
Fix typos in comments. Author: Peter Smith smithpb2250@gmail.com Discussion: https://postgr.es/m/CAHut+PsN_gmKu-KfeEb9NDARoTPbs4AN4PPu=6LZXFZRJ13SEw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/8af57ad81578f825ac8c46840c841833db205106

Fujii Masao pushed:

Improve HINT message that FDW reports when there are no valid options. The foreign data wrapper's validator function provides a HINT message with list of valid options for the object specified in CREATE or ALTER command, when the option given in the command is invalid. Previously postgresql_fdw_validator() and the validator functions for postgres_fdw and dblink_fdw worked in that way even there were no valid options in the object, which could lead to the HINT message with empty list (because there were no valid options). For example, ALTER FOREIGN DATA WRAPPER postgres_fdw OPTIONS (format 'csv') reported the following ERROR and HINT messages. This behavior was confusing. ERROR: invalid option "format" HINT: Valid options in this context are: There is no such issue in file_fdw. The validator function for file_fdw reports the HINT message "There are no valid options in this context." instead in that case. This commit improves postgresql_fdw_validator() and the validator functions for postgres_fdw and dblink_fdw so that they do likewise. For example, this change causes the above ALTER FOREIGN DATA WRAPPER command to report the following messages. ERROR: invalid option "nonexistent" HINT: There are no valid options in this context. Author: Kosei Masumura Reviewed-by: Bharath Rupireddy, Fujii Masao Discussion: https://postgr.es/m/557d06cebe19081bfcc83ee2affc98d3@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/5fedf7417b69295294b154a219edd8a26eaa6ab6

Jeff Davis pushed:

Allow GRANT on pg_log_backend_memory_contexts(). Remove superuser check, allowing any user granted permissions on pg_log_backend_memory_contexts() to log the memory contexts of any backend. Note that this could allow a privileged non-superuser to log the memory contexts of a superuser backend, but as discussed, that does not seem to be a problem. Reviewed-by: Nathan Bossart, Bharath Rupireddy, Michael Paquier, Kyotaro Horiguchi, Andres Freund Discussion: https://postgr.es/m/e5cf6684d17c8d1ef4904ae248605ccd6da03e72.camel@j-davis.com https://git.postgresql.org/pg/commitdiff/f0b051e322d530a340e62f2ae16d99acdbcb3d05
Grant memory views to pg_read_all_stats. Grant privileges on views pg_backend_memory_contexts and pg_shmem_allocations to the role pg_read_all_stats. Also grant on the underlying functions that those views depend on. Author: Bharath Rupireddy bharath.rupireddyforpostgres@gmail.com Reviewed-by: Nathan Bossart bossartn@amazon.com Discussion: https://postgr.es/m/CALj2ACWAZo3Ar_EVsn2Zf9irG+hYK3cmh1KWhZS_Od45nd01RA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/77ea4f94393eb4a16df32b573bf053bedaef2e09

Amit Kapila pushed:

Allow publishing the tables of schema. A new option "FOR ALL TABLES IN SCHEMA" in Create/Alter Publication allows one or more schemas to be specified, whose tables are selected by the publisher for sending the data to the subscriber. The new syntax allows specifying both the tables and schemas. For example: CREATE PUBLICATION pub1 FOR TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2; OR ALTER PUBLICATION pub1 ADD TABLE t1,t2,t3, ALL TABLES IN SCHEMA s1,s2; A new system table "pg_publication_namespace" has been added, to maintain the schemas that the user wants to publish through the publication. Modified the output plugin (pgoutput) to publish the changes if the relation is part of schema publication. Updates pg_dump to identify and dump schema publications. Updates the \d family of commands to display schema publications and \dRp+ variant will now display associated schemas if any. Author: Vignesh C, Hou Zhijie, Amit Kapila Syntax-Suggested-by: Tom Lane, Alvaro Herrera Reviewed-by: Greg Nancarrow, Masahiko Sawada, Hou Zhijie, Amit Kapila, Haiying Tang, Ajin Cherian, Rahila Syed, Bharath Rupireddy, Mark Dilger Tested-by: Haiying Tang Discussion: /message-id/CALDaNm0OANxuJ6RXqwZsM1MSY4s19nuH3734j4a72etDwvBETQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/5a2832465fd8984d089e8c44c094e6900d987fcd
Add tap tests for the schema publications. This adds additional tests for commit 5a2832465f ("Allow publishing the tables of schema.). This allows testing streaming of data in tables that are published via schema publications. Author: Vignesh C, Haiying Tang Reviewed-by: Greg Nancarrow, Hou Zhijie, Amit Kapila Discussion: /message-id/CALDaNm0OANxuJ6RXqwZsM1MSY4s19nuH3734j4a72etDwvBETQ%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/6b0f6f79eef2168ce38a8ee99c3ed76e3df5d7ad

Magnus Hagander pushed:

Clarify that --system reindexes system catalogs only. Make this more clear both in the help message and docs. Reviewed-By: Michael Paquier Backpatch-through: 9.6 Discussion: https://postgr.es/m/CABUevEw6Je0WUFTLhPKOk4+BoBuDrE-fKw3N4ckqgDBMFu4paA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/eff61383b982be8dc71d942340a839bea88a9eab

Peter Geoghegan pushed:

Further harden nbtree posting split code. Add more defensive checks around posting list split code. These should detect corruption involving duplicate table TIDs earlier and more reliably than any existing check. Follow up to commit 8f72bbac. Discussion: https://postgr.es/m/CAH2-WzkrSY_kjyd1_M5xJK1uM0govJXMxPn8JUSvwcUOiHuWVw@mail.gmail.com Backpatch: 13-, where nbtree deduplication was introduced. https://git.postgresql.org/pg/commitdiff/a5213adf3d351a31c5f5eae1a756a9d3555dc31c
Fix ordering of items in nbtree error message. Oversight in commit a5213adf. Backpatch: 13-, just like commit a5213adf. https://git.postgresql.org/pg/commitdiff/c2381b51049bad5dd1863ab1116b315bd7693b7c
Remove obsolete nbtree LP_DEAD item comments. Comments above _bt_findinsertloc() that talk about LP_DEAD items are now out of place. We already discuss index tuple deletion at an earlier point in the same comment block. Oversight in commit d168b666. https://git.postgresql.org/pg/commitdiff/4c6afd805b8db3492c8f409ecdba192d853fd571
Demote pg_unreachable() in heapam to an assertion. Commit d168b66682, which overhauled index deletion, added a pg_unreachable() to the end of a sort comparator used when sorting heap TIDs from an index page. This allows the compiler to apply optimizations that assume that the heap TIDs from the index AM must always be unique. That doesn't seem like a good idea now, given recent reports of corruption involving duplicate TIDs in indexes on Postgres
Demote to an assertion, just in case. Backpatch: 14-, where index deletion was overhauled. https://git.postgresql.org/pg/commitdiff/5f55fc5a346e1ab54f3d756e368d276b95be8c4a

Tom Lane pushed:

Improve contrib/amcheck's tests for CREATE INDEX CONCURRENTLY. Commits fdd965d07 and 3cd9c3b92 tested CREATE INDEX CONCURRENTLY by launching two separate pgbench runs concurrently. This was needed so that only a single client thread would run CREATE INDEX CONCURRENTLY, avoiding deadlock between two CICs. However, there's a better way, which is to use an advisory lock to prevent concurrent CICs. That's better in part because the test code is shorter and more readable, but mostly because it automatically scales things to launch an appropriate number of CICs relative to the number of INSERT transactions. As committed, typically half to three-quarters of the CIC transactions were pointless because the INSERT transactions had already stopped. In passing, remove background_pgbench, which was added to support these tests and isn't needed anymore. We can always put it back if we find a use for it later. Back-patch to v12; older pgbench versions lack the conditional-execution features needed for this method. Tom Lane and Andrey Borodin Discussion: https://postgr.es/m/139687.1635277318@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/7f580aa5d88a9b03d66fcb9a1d7c4fcd69d9e126
Speed up printing of integers in snprintf.c. Since the only possible divisors are 8, 10, and 16, it doesn't cost much code space to replace the division loop with three copies using constant divisors. On most machines, division by a constant can be done a lot more cheaply than division by an arbitrary value. A microbenchmark testing just snprintf("foo %d") with a 9-digit value showed about a 2X speedup for me (tgl). Most of Postgres isn't too dependent on the speed of snprintf, so that the effect in real-world cases is barely measurable. Still, a cycle saved is a cycle earned. Arjan van de Ven Discussion: https://postgr.es/m/40a4b32a-b841-4667-11b2-a0baedb12714@linux.intel.com Discussion: https://postgr.es/m/6e51c644-1b6d-956e-ac24-2d1b0541d532@linux.intel.com https://git.postgresql.org/pg/commitdiff/3c17926eedd51c4094db7c62f59950918044ab1c
Update time zone data files to tzdata release 2021e. DST law changes in Fiji, Jordan, Palestine, and Samoa. Historical corrections for Barbados, Cook Islands, Guyana, Niue, Portugal, and Tonga. Also, the Pacific/Enderbury zone has been renamed to Pacific/Kanton. The following zones have been merged into nearby, more-populous zones whose clocks have agreed since 1970: Africa/Accra, America/Atikokan, America/Blanc-Sablon, America/Creston, America/Curacao, America/Nassau, America/Port_of_Spain, Antarctica/DumontDUrville, and Antarctica/Syowa. https://git.postgresql.org/pg/commitdiff/937aafd6d5580b81134c7f303d04cf7561ad0309
Test and document the behavior of initialization cross-refs in plpgsql. We had a test showing that a variable isn't referenceable in its own initialization expression, nor in prior ones in the same block. It is referenceable in later expressions in the same block, but AFAICS there is no test case exercising that. Add one, and also add some error cases. Also, document that this is possible, since the docs failed to cover the point. Per question from tomás at tuxteam. I don't feel any need to back-patch this, but we should ensure we don't break it in future. Discussion: https://postgr.es/m/20211029121435.GA5414@tuxteam.de https://git.postgresql.org/pg/commitdiff/a2a731d6c9db0ba650aa6f7c4fe349ccf712f74d

Peter Eisentraut pushed:

Remove unused chunk from standalone-profile.xsl. unused since 1707a0d2aa6b2bcfe78f63836c769943a1a6b9e0 https://git.postgresql.org/pg/commitdiff/b8b62b4be28b8acd36d32d5db65162bbbcd3a754
uuid-ossp: Remove obsolete build connection with pgcrypto. unused since a8ed6bb8f4cf259b95c1bff5da09a8f4c79dca46 https://git.postgresql.org/pg/commitdiff/237c12aabe39a58f3f5364fd94e0ca8ae8824957
doc: Remove some obsolete pgcrypto documentation. The pgcrypto documentation contained acknowledgments of used external code, but some of this code has been moved to src/common/, so mentioning it with pgcrypto no longer makes sense, so remove it. https://git.postgresql.org/pg/commitdiff/e6c60719e6c6ee9bd396f430879e1de9079bf74c
pg_dump: Refactor messages. This reduces the number of separate messages for translation. https://git.postgresql.org/pg/commitdiff/fd2706589a7da4be6f6998befdf8e5fdea1565b8

PostgreSQL Weekly News - October 24, 2021

Sun, 24 Oct 2021 00:00:00 +0000

PostgreSQL Weekly News - October 24, 2021

PostgreSQL Product News

JDBC 42.3.0 released.

pgmetrics 1.12, a command-line tool for PostgreSQL metrics, released.

StackGres 1.0.0, a platform for running PostgreSQL on Kubernetes, released. https://stackgres.io/

pgexporter 0.2.0, a Prometheus exporter for PostgreSQL, released

pgAdmin4 6.1, a web- and native GUI control center for PostgreSQL, released.

PostgreSQL Jobs for October

https://archives.postgresql.org/pgsql-jobs/2021-10/

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Michaël Paquier pushed:

Fix portability issues in new TAP tests of psql. The tests added by c0280bc and d9ddc50 in 001_basic.pl have introduced commands calling directly psql, making them sensitive to the environment. One issue was that those commands forgot -X to not use a local .psqlrc, causing all those tests to fail if psql cannot properly parse this file. TAP tests should be designed so as they run in an isolated fashion, without any dependency on the environment where they are run. As PostgresNode::psql gives already all the facilities those new tests need, switch to that instead of calling plain psql commands where interactions with a backend are needed. The test is slightly refactored to be able to check after the expected patterns of stdout and stderr, keeping the same amount of coverage as previously. Reported-by: Peter Geoghegan Discussion: https://postgr.es/m/CAH2-Wzn8ftvcDPwomn+y04JJzbT=TG7TN=QsmSEATUOW-ZuvQQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/384f1abdb9b0f669279fcd57ba2173eb31724740
Reset properly snapshot export state during transaction abort. During a replication slot creation, an ERROR generated in the same transaction as the one creating a to-be-exported snapshot would have left the backend in an inconsistent state, as the associated static export snapshot state was not being reset on transaction abort, but only on the follow-up command received by the WAL sender that created this snapshot on replication slot creation. This would trigger inconsistency failures if this session tried to export again a snapshot, like during the creation of a replication slot. Note that a snapshot export cannot happen in a transaction block, so there is no need to worry resetting this state for subtransaction aborts. Also, this inconsistent state would very unlikely show up to users. For example, one case where this could happen is an out-of-memory error when building the initial snapshot to-be-exported. Dilip found this problem while poking at a different patch, that caused an error in this code path for reasons unrelated to HEAD. Author: Dilip Kumar Reviewed-by: Michael Paquier, Zhihong Yu Discussion: https://postgr.es/m/CAFiTN-s0zA1Kj0ozGHwkYkHwa5U0zUE94RSc_g81WrpcETB5=w@mail.gmail.com Backpatch-through: 9.6 https://git.postgresql.org/pg/commitdiff/409f9ca4471331be0f77b665ff3a1836a41de5b3
Block ALTER INDEX/TABLE index_name ALTER COLUMN colname SET (options). The grammar of this command run on indexes with column names has always been authorized by the parser, and it has never been documented. Since 911e702, it is possible to define opclass parameters as of CREATE INDEX, which actually broke the old case of ALTER INDEX/TABLE where relation-level parameters n_distinct and n_distinct_inherited could be defined for an index (see 76a47c0 and its thread where this point has been touched, still remained unused). Attempting to do that in v13~ would cause the index to become unusable, as there is a new dedicated code path to load opclass parameters instead of the relation-level ones previously available. Note that it is possible to fix things with a manual catalog update to bring the relation back online. This commit disables this command for now as the use of column names for indexes does not make sense anyway, particularly when it comes to index expressions where names are automatically computed. One way to properly support this case properly in the future would be to use column numbers when it comes to indexes, in the same way as ALTER INDEX .. ALTER COLUMN .. SET STATISTICS. Partitioned indexes were already blocked, but not indexes. Some tests are added for both cases. There was some code in ANALYZE to enforce n_distinct to be used for an index expression if the parameter was defined, but just remove it for now until/if there is support for this (note that index-level parameters never had support in pg_dump either, previously), so this was just dead code. Reported-by: Matthijs van der Vleuten Author: Nathan Bossart, Michael Paquier Reviewed-by: Vik Fearing, Dilip Kumar Discussion: https://postgr.es/m/17220-15d684c6c2171a83@postgresql.org Backpatch-through: 13 https://git.postgresql.org/pg/commitdiff/fdd88571454e2b00dbe446e8609c6e4294ca89ae
Fix build of MSVC with OpenSSL 3.0.0. The build scripts of Visual Studio would fail to detect properly a 3.0.0 build as the check on the second digit was failing. This is adjusted where needed, allowing the builds to complete. Note that the MSIs of OpenSSL mentioned in the documentation have not changed any library names for Win32 and Win64, making this change straight-forward. Reported-by: htalaco, via github Reviewed-by: Daniel Gustafsson Discussion: https://postgr.es/m/YW5XKYkq6k7OtrFq@paquier.xyz Backpatch-through: 9.6 https://git.postgresql.org/pg/commitdiff/41f30ecc29c89285d3eecd435906c4e9cb048be4
Fix corruption of pg_shdepend when copying deps from template database. Using for a new database a template database with shared dependencies that need to be copied over was causing a corruption of pg_shdepend because of an off-by-one computation error of the index number used for the values inserted with a slot. Issue introduced by e3931d0. Monitoring the rest of the code, there are no similar mistakes. Reported-by: Sven Klemm Author: Aleksander Alekseev Reviewed-by: Daniel Gustafsson, Michael Paquier Discussion: https://postgr.es/m/CAJ7c6TP0AowkUgNL6zcAK-s5HYsVHVBRWfu69FRubPpfwZGM9A@mail.gmail.com Backpatch-through: 14 https://git.postgresql.org/pg/commitdiff/98ec35b0bbf6003e89fc06aa140e12fd90bbad47
doc: Describe calculation method of streaming start for pg_receivewal. The documentation was imprecise about the starting LSN used for WAL streaming if nothing can be found in the local archive directory defined with the pg_receivewal command, so be more talkative on this matter. Extracted from a larger patch by the same author. Author: Ronan Dunklau, Michael Paquier Discussion: https://postgr.es/m/18708360.4lzOvYHigE@aivenronan Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/1e9475694b0ae2cf1204d01d2ef6ad86f3c7cac8

Heikki Linnakangas pushed:

Replace polyphase merge algorithm with a simple balanced k-way merge. The advantage of polyphase merge is that it can reuse the input tapes as output tapes efficiently, but that is irrelevant on modern hardware, when we can easily emulate any number of tape drives. The number of input tapes we can/should use during merging is limited by work_mem, but output tapes that we are not currently writing to only cost a little bit of memory, so there is no need to skimp on them. This makes sorts that need multiple merge passes faster. Discussion: /message-id/420a0ec7-602c-d406-1e75-1ef7ddc58d83%40iki.fi Reviewed-by: Peter Geoghegan, Zhihong Yu, John Naylor https://git.postgresql.org/pg/commitdiff/65014000b351d5725eb00d133416ab1b4f8245b1
Refactor LogicalTapeSet/LogicalTape interface. All the tape functions, like LogicalTapeRead and LogicalTapeWrite, now take a LogicalTape as argument, instead of LogicalTapeSet+tape number. You can create any number of LogicalTapes in a single LogicalTapeSet, and you don't need to decide the number upfront, when you create the tape set. This makes the tape management in hash agg spilling in nodeAgg.c simpler. Discussion: /message-id/420a0ec7-602c-d406-1e75-1ef7ddc58d83%40iki.fi Reviewed-by: Peter Geoghegan, Zhihong Yu, John Naylor https://git.postgresql.org/pg/commitdiff/c4649cce39a41b27db874e75ddd47adaec1b0ea4
Fix format modifier used in elog. The previous commit 65014000b3 changed the variable passed to elog from an int64 to a size_t variable, but neglected to change the modifier in the format string accordingly. Per failure on buildfarm member lapwing. https://git.postgresql.org/pg/commitdiff/0bd65a3905706927cdd6b3158b6457c1c854471b
Fix duplicate typedef LogicalTape. To make buildfarm member locust happy. https://git.postgresql.org/pg/commitdiff/aa3ac6453b28049b3198433b75228271b7612d4a
Fix parallel sort, broken by the balanced merge patch. The code for initializing the tapes on each merge iteration was skipped in a parallel worker. I put the !WORKER(state) check in wrong place while rebasing the patch. That caused failures in the index build in 'multiple-row-versions' isolation test, in multiple buildfarm members. On my laptop it was easier to reproduce by building an index on a larger table, so that you got a parallel sort more reliably. https://git.postgresql.org/pg/commitdiff/fc0f3b4cb0e882a9c5d51c302d4aa3591e4f80fd

Álvaro Herrera pushed:

Invalidate partitions of table being attached/detached. Failing to do that, any direct inserts/updates of those partitions would fail to enforce the correct constraint, that is, one that considers the new partition constraint of their parent table. Backpatch to 10. Reported by: Hou Zhijie houzj.fnst@fujitsu.com Author: Amit Langote amitlangote09@gmail.com Author: Álvaro Herrera alvherre@alvh.no-ip.org Reviewed-by: Nitin Jadhav nitinjadhavpostgres@gmail.com Reviewed-by: Pavel Borisov pashkin.elfe@gmail.com Discussion: https://postgr.es/m/OS3PR01MB5718DA1C4609A25186D1FBF194089%40OS3PR01MB5718.jpnprd01.prod.outlook.com https://git.postgresql.org/pg/commitdiff/d6f1e16c8fe27100e371a15aeeb498faa680ceed
Ensure correct lock level is used in ALTER ... RENAME. Commit 1b5d797cd4f7 intended to relax the lock level used to rename indexes, but inadvertently allowed any relation to be renamed with a lowered lock level, as long as the command is spelled ALTER INDEX. That's undesirable for other relation types, so retry the operation with the higher lock if the relation turns out not to be an index. After this fix, ALTER INDEX <sometable> RENAME will require access exclusive lock, which it didn't before. Author: Nathan Bossart bossartn@amazon.com Author: Álvaro Herrera alvherre@alvh.no-ip.org Reported-by: Onder Kalaci onderk@microsoft.com Discussion: https://postgr.es/m/PH0PR21MB1328189E2821CDEC646F8178D8AE9@PH0PR21MB1328.namprd21.prod.outlook.com https://git.postgresql.org/pg/commitdiff/c2c618ff1137f9ef58827f57e4ec0f97453e454e
Protect against collation variations in test. Discussion: https://postgr.es/m/YW/MYdSRQZtPFBWR@paquier.xyz https://git.postgresql.org/pg/commitdiff/cd124d205c42a623b68cd155ace94cc376851b78

Daniel Gustafsson pushed:

Fix sscanf limits in pg_basebackup and pg_dump. Make sure that the string parsing is limited by the size of the destination buffer. In pg_basebackup the available values sent from the server is limited to two characters so there was no risk of overflow. In pg_dump the buffer is bounded by MAXPGPATH, and thus the limit must be inserted via preprocessor expansion and the buffer increased by one to account for the terminator. There is no risk of overflow here, since in this case, the buffer scanned is smaller than the destination buffer. Backpatch the pg_basebackup fix to 11 where it was introduced, and the pg_dump fix all the way down to 9.6. Reviewed-by: Tom Lane Discussion: https://postgr.es/m/B14D3D7B-F98C-4E20-9459-C122C67647FB@yesql.se Backpatch-through: 11 and 9.6 https://git.postgresql.org/pg/commitdiff/1d7641d51a51aa00dff685022fab6c03be8f8af8
Fix bug in TOC file error message printing. If the blob TOC file cannot be parsed, the error message was failing to print the filename as the variable holding it was shadowed by the destination buffer for parsing. When the filename fails to parse, the error will print an empty string: ./pg_restore -d foo -F d dump pg_restore: error: invalid line in large object TOC file "": .. ..instead of the intended error message: ./pg_restore -d foo -F d dump pg_restore: error: invalid line in large object TOC file "dump/blobs.toc": .. Fix by renaming both variables as the shared name was too generic to store either and still convey what the variable held. Backpatch all the way down to 9.6. Reviewed-by: Tom Lane Discussion: https://postgr.es/m/A2B151F5-B32B-4F2C-BA4A-6870856D9BDE@yesql.se Backpatch-through: 9.6 https://git.postgresql.org/pg/commitdiff/998d060f3db79c6918cb4a547695be150833f9a4
Refactor the sslfiles Makefile target for ease of use. The Makefile handling of certificate and keypairs used for TLS testing had become quite difficult to work with. Adding a new cert without the need to regenerate everything was too complicated. This patch refactors the sslfiles make target such that adding a new certificate requires only adding a .config file, adding it to the top of the Makefile, and running make sslfiles. Improvements: - Interfile dependencies should be fixed, with the exception of the CRL dirs. - New certificates have serial numbers based on the current time, reducing the chance of collision. - The CA index state is created on demand and cleaned up automatically at the end of the Make run. - *.config files are now self-contained; one certificate needs one config file instead of two. - Duplication is reduced, and along with it some unneeded code (and possible copy-paste errors). - all configuration files underneath the conf/ directory. The target is moved to its own makefile in order to avoid colliding with global make settings. Author: Jacob Champion pchampion@vmware.com Reviewed-by: Michael Paquier michael@paquier.xyz Discussion: https://postgr.es/m/d15a9838344ba090e09fd866abf913584ea19fb7.camel@vmware.com https://git.postgresql.org/pg/commitdiff/b4c4a00eada3c512e819e9163114a5ad1606bc7e
Fix SSL tests on 32-bit Perl. The certificate serial number generation was changed in b4c4a00ea to use the current timestamp. The testharness must thus interrogate the cert for the serialnumber using "openssl x509" which emits the serial in hex format. Converting the serial to integer format to match whats in pg_stat_ssl requires a 64-bit capable Perl. This adds a fallback to checking for an integer when the tests with a 32-bit Perl. Per failure on buildfarm member prairiedog. Discussion: https://postgr.es/m/0D295F43-806D-4B3F-AB98-F941A19E0271@yesql.se https://git.postgresql.org/pg/commitdiff/0c04342b1d3dd5b24f795f94874163be8e21710e

Tom Lane pushed:

Remove bogus assertion in transformExpressionList(). I think when I added this assertion (in commit 8f889b108), I was only thinking of the use of transformExpressionList at top level of INSERT and VALUES. But it's also called by transformRowExpr(), which can certainly occur in an UPDATE targetlist, so it's inappropriate to suppose that p_multiassign_exprs must be empty. Besides, since the input is not expected to contain ResTargets, there's no reason it should contain MultiAssignRefs either. Hence this code need not be concerned about the state of p_multiassign_exprs, and we should just drop the assertion. Per bug #17236 from ocean_li_996. It's been wrong for years, so back-patch to all supported branches. Discussion: https://postgr.es/m/17236-3210de9bcba1d7ca@postgresql.org https://git.postgresql.org/pg/commitdiff/697dd1925f418c9f54ee1fd1cefbc613d6504b1f
Fix assignment to array of domain over composite. An update such as "UPDATE ... SET fld[n].subfld = whatever" failed if the array elements were domains rather than plain composites. That's because isAssignmentIndirectionExpr() failed to cope with the CoerceToDomain node that would appear in the expression tree in this case. The result would typically be a crash, and even if we accidentally didn't crash, we'd not correctly preserve other fields of the same array element. Per report from Onder Kalaci. Back-patch to v11 where arrays of domains came in. Discussion: https://postgr.es/m/PH0PR21MB132823A46AA36F0685B7A29AD8BD9@PH0PR21MB1328.namprd21.prod.outlook.com https://git.postgresql.org/pg/commitdiff/3e310d837a9b3de8ad977c0a3e2a769bcdf61cc9
pg_dump: Reorganize getTables(). Along the same lines as 047329624, ed2c7f65b and daa9fe8a5, reduce code duplication by having just one copy of the parts of the query that are the same across all server versions; and make the conditionals control the smallest possible amount of code. This also gets rid of the confusing assortment of different ways to accomplish the same result that we had here before. While at it, make sure all three relevant parts of the function list the fields in the same order. This is just neatnik-ism, of course. Discussion: https://postgr.es/m/1240992.1634419055@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/4438eb4a495c977d8ac485dd6e544c2b6e077deb
Improve pg_regress.c's infrastructure for issuing psql commands. Support issuing more than one "-c command" switch to a single psql invocation. This allows combining some things that formerly required two or more backend launches into a single session. In particular, we can issue DROP DATABASE as one of the -c commands without getting "DROP DATABASE cannot run inside a transaction block". In addition to reducing the number of sessions needed, this patch also suppresses "NOTICE: database "foo" does not exist, skipping" chatter that was formerly generated during pg_regress's DROP DATABASE (or ROLE) IF NOT EXISTS calls. That moves us another step closer to the ideal of not seeing any messages during successful build/test. This also eliminates some hard-coded restrictions on the length of the commands issued. I don't think we were anywhere near hitting those, but getting rid of the limit is comforting. Patch by me, but thanks to Nathan Bossart for starting the discussion. Discussion: https://postgr.es/m/DCBAE0E4-BD56-482F-8A70-7FD0DC0860BE@amazon.com https://git.postgresql.org/pg/commitdiff/f45dc59a38cab1d2af6baaedb79559fe2e9b3781
Doc: clarify a critical and undocumented aspect of simplehash.h. I just got burnt by trying to use pg_malloc instead of pg_malloc0 with this. Save the next hacker some time by not leaving this API detail undocumented. https://git.postgresql.org/pg/commitdiff/b1ce6c284366ce1dae120f5d10dd59e8804322ee
pg_dump: fix mis-dumping of non-global default privileges. Non-global default privilege entries should be dumped as-is, not made relative to the default ACL for their object type. This would typically only matter if one had revoked some on-by-default privileges in a global entry, and then wanted to grant them again in a non-global entry. Per report from Boris Korzun. This is an old bug, so back-patch to all supported branches. Neil Chen, test case by Masahiko Sawada Discussion: https://postgr.es/m/111621616618184@mail.yandex.ru Discussion: https://postgr.es/m/CAA3qoJnr2+1dVJObNtfec=qW4Z0nz=A9+r5bZKoTSy5RDjskMw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/2acc84c6fd299125702c8a8af13820abcc0d4891
Fix frontend version of sh_error() in simplehash.h. The code does not expect sh_error() to return, but the patch that made this header usable in frontend didn't get that memo. While here, plaster unlikely() on the tests that decide whether to invoke sh_error(), and add our standard copyright notice. Noted by Andres Freund. Back-patch to v13 where this frontend support came in. Discussion: https://postgr.es/m/0D54435C-1199-4361-9D74-2FBDCF8EA164@anarazel.de https://git.postgresql.org/pg/commitdiff/974aedcea46dfd0119eea2fbb2eeacd232596f05
In pg_dump, use simplehash.h to look up dumpable objects by OID. Create a hash table that indexes dumpable objects by CatalogId (that is, catalog OID + object OID). Use this to replace the former catalogIdMap array, as well as various other single- catalog index arrays, and also the extension membership map. In principle this should be faster for databases with many objects, since lookups are now O(1) not O(log N). However, it seems that these lookups are pretty much negligible in context, so that no overall performance change can be measured. But having only one lookup data structure to maintain makes the code simpler and more flexible, so let's do it anyway. Discussion: https://postgr.es/m/2595220.1634855245@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/92316a4582a5714d4e494aaf90360860e7fec37a
Fix minor memory leaks in pg_dump. I found these by running pg_dump under "valgrind --leak-check=full". The changes in flagInhIndexes() and getIndexes() replace allocation of an array of which we use only some elements by individual allocations of just the actually-needed objects. The previous coding wasted some memory, but more importantly it confused valgrind's leak tracking. collectComments() and collectSecLabels() remain major blots on the valgrind report, because they don't PQclear their query results, in order to avoid a lot of strdup's. That's a dubious tradeoff, but I'll leave it alone here; an upcoming patch will modify those functions enough to justify changing the tradeoff. https://git.postgresql.org/pg/commitdiff/70bef494000e4dbbeca0f0a40347ca1747aea701

Andres Freund pushed:

Adapt src/test/ldap/t/001_auth.pl to work with openldap 2.5. ldapsearch's deprecated -h/-p arguments were removed, need to use -H now - which has been around for over 20 years. As perltidy insists on reflowing the parameters anyway, change order and "phrasing" to yield a less confusing layout (per suggestion from Tom Lane). Discussion: https://postgr.es/m/20211009233850.wvr6apcrw2ai6cnj@alap3.anarazel.de Backpatch: 11-, where the tests were added. https://git.postgresql.org/pg/commitdiff/984f460e2f29e7ba9174cabb9f43a0d1dce543bf

Amit Kapila pushed:

Remove unused wait events. Commit 464824323e introduced the wait events which were neither used by that commit nor by follow-up commits for that work. Author: Masahiro Ikeda Backpatch-through: 14, where it was introduced Discussion: https://postgr.es/m/ff077840-3ab2-04dd-bbe4-4f5dfd2ad481@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/1607cd0b6c9919bf765198882ea48a98e901e1bc

Andrew Dunstan pushed:

Add module build directory to the PATH for TAP tests. For non-MSVC builds this is make's $(CURDIR), while for MSVC builds it is $topdir/$Config/$module. The directory is added as the second element in the PATH, so that the install location takes precedence, but the added PATH element takes precedence over the rest of the PATH. The reason for this is to allow tests to find built products that are not installed, such as the libpq_pipeline test driver. The libpq_pipeline test is adjusted to take advantage of this. Based on a suggestion from Andres Freund. Backpatch to release 14. Discussion: https://postgr.es/m/4941f5a5-2d50-1a0e-6701-14c5fefe92d6@dunslane.net https://git.postgresql.org/pg/commitdiff/f4ce6c4d3a30ec3a12c7f64b90a6fc82887ddd7b
Move Perl test modules to a better namespace. The five modules in our TAP test framework all had names in the top level namespace. This is unwise because, even though we're not exporting them to CPAN, the names can leak, for example if they are exported by the RPM build process. We therefore move the modules to the PostgreSQL::Test namespace. In the process PostgresNode is renamed to Cluster, and TestLib is renamed to Utils. PostgresVersion becomes simply PostgreSQL::Version, to avoid possible confusion about what it's the version of. Discussion: https://postgr.es/m/aede93a4-7d92-ef26-398f-5094944c2504@dunslane.net Reviewed by Erik Rijkers and Michael Paquier https://git.postgresql.org/pg/commitdiff/b3b4d8e68ae83f432f43f035c7eb481ef93e1583

Noah Misch pushed:

Avoid race in RelationBuildDesc() affecting CREATE INDEX CONCURRENTLY. CIC and REINDEX CONCURRENTLY assume backends see their catalog changes no later than each backend's next transaction start. That failed to hold when a backend absorbed a relevant invalidation in the middle of running RelationBuildDesc() on the CIC index. Queries that use the resulting index can silently fail to find rows. Fix this for future index builds by making RelationBuildDesc() loop until it finishes without accepting a relevant invalidation. It may be necessary to reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices. Back-patch to 9.6 (all supported versions). Noah Misch and Andrey Borodin, reviewed (in earlier versions) by Andres Freund. Discussion: https://postgr.es/m/20210730022548.GA1940096@gust.leadboat.com https://git.postgresql.org/pg/commitdiff/fdd965d074d46765c295223b119ca437dbcac973
Fix CREATE INDEX CONCURRENTLY for the newest prepared transactions. The purpose of commit 8a54e12a38d1545d249f1402f66c8cde2837d97c was to fix this, and it sufficed when the PREPARE TRANSACTION completed before the CIC looked for lock conflicts. Otherwise, things still broke. As before, in a cluster having used CIC while having enabled prepared transactions, queries that use the resulting index can silently fail to find rows. It may be necessary to reindex to recover from past occurrences; REINDEX CONCURRENTLY suffices. Fix this for future index builds by making CIC wait for arbitrarily-recent prepared transactions and for ordinary transactions that may yet PREPARE TRANSACTION. As part of that, have PREPARE TRANSACTION transfer locks to its dummy PGPROC before it calls ProcArrayClearTransaction(). Back-patch to 9.6 (all supported versions). Andrey Borodin, reviewed (in earlier versions) by Andres Freund. Discussion: https://postgr.es/m/01824242-AA92-4FE9-9BA7-AEBAFFEA3D0C@yandex-team.ru https://git.postgresql.org/pg/commitdiff/3cd9c3b921977272e6650a5efbeade4203c4bca2

PostgreSQL Weekly News - October 17, 2021

Mon, 18 Oct 2021 00:00:00 +0000

PostgreSQL Weekly News - October 17, 2021

PostgreSQL Product News

psycopg2 3.0.0, a Python connector for PostgreSQL, released

pg_partman 4.6.0, a management system for partitioned tables, released.

pgAdmin4 6.0, a web- and native GUI control center for PostgreSQL, released.

Percona Distribution for PostgreSQL Operator 1.0.0, a Kubernetes operator based on Crunchy Data's, for PostgreSQL, released.

PostgreSQL Jobs for October

https://archives.postgresql.org/pgsql-jobs/2021-10/

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Tom Lane pushed:

Doc: update testing recipe in src/test/perl/README. The previous text didn't provide any clear explanation of our policy around TAP test portability. The recipe for using perlbrew had some problems, too: it resulted in a non-shared libperl (preventing testing of plperl) and it caused some modules to be updated to current when the point of the recipe is to build an old environment. Discussion: https://postgr.es/m/E1mYY6Z-0006OL-QN@gemulon.postgresql.org https://git.postgresql.org/pg/commitdiff/3eb1f4d09745433c70ccac411cad24d0374b9c3b
Fix EXPLAIN of SEARCH BREADTH FIRST queries some more. Commit 3f50b8263 had an oversight: formerly, to deparse expressions attached to a plan node, it was only necessary to update the deparse_namespace ancestors list alongside calling set_deparse_plan. Now it's necessary to update the ancestors list first, because set_deparse_plan consults it, and one call site got that wrong. This error was masked in most cases because explain.c uses just one List object for the ancestors list, updating it in-place as the plan is scanned, so that we accidentally had the right List assigned to dpns->ancestors before it was needed. It would fail only if a WorkTableScan node were the first one that we tried to deparse a subexpression of. Per report from Markus Winand. Like the previous patch, back-patch to v14. Discussion: https://postgr.es/m/648B0505-AA57-42C2-A2DA-E551DE46FA15@winand.at https://git.postgresql.org/pg/commitdiff/39ae0ef8561362304ee512963aa51d5a705e5616
Make configure check for minimum required version of IPC::Run. Per the discussion around 3eb1f4d09, let's have configure verify that the available IPC::Run version is at least 0.79, the agreed-on minimum. It seems unlikely that this could bite anybody anymore, but it's useful as documentation. (Based on that, there's little need to back-patch.) For consistency, also supply a minimum version for the other Perl module we have an explicit check for, Time::HiRes. I used the version that ships with Perl 5.8.3. Discussion: https://postgr.es/m/E1mYY6Z-0006OL-QN@gemulon.postgresql.org https://git.postgresql.org/pg/commitdiff/4a235efddaa78ec78a47614ddc6161644e089290
Fix planner error with pulling up subquery expressions into function RTEs. If a function-in-FROM laterally references the output of some sub-SELECT earlier in the FROM clause, and we are able to flatten that sub-SELECT into the outer query, the expression(s) copied into the function RTE missed being processed by eval_const_expressions. This'd lead to trouble and probable crashes at execution if such expressions contained named-argument function call syntax or functions with defaulted arguments. The bug is masked if the query contains any explicit JOIN syntax, which may help explain why we'd not noticed. Per bug #17227 from Bernd Dorn. This is an oversight in commit 7266d0997, so back-patch to v13 where that came in. Discussion: https://postgr.es/m/17227-5a28ed1512189fa4@postgresql.org https://git.postgresql.org/pg/commitdiff/4d5f651f1d651c6fa79f9188e7b9a04654c7125a
Make pg_dump acquire lock on partitioned tables that are to be dumped. It was clearly the intent to do so all along, but the original coding fat-fingered this by checking the wrong array element. We fixed it in passing in 403a3d91c, but that later got reverted, and we forgot to keep this bug fix. Most of the time this'd be relatively harmless, since once we lock any of the partitioned table's leaf partitions, that would suffice to prevent major DDL on the partitioned table itself. However, a childless partitioned table would get dumped with no relevant lock whatsoever, possibly allowing dump failure or inconsistent output. Unlike 403a3d91c, there are no versioning concerns, since every server version that has partitioned tables will allow you to lock one. Back-patch to v10 where partitioned tables were introduced. Discussion: https://postgr.es/m/1018205.1634346327@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/e2ff7d9a83d4b489806281dc6dfce88510b40ad7
Avoid core dump in pg_dump when dumping from pre-8.3 server. Commit f0e21f2f6 missed adding a tgisinternal output column to getTriggers' query for pre-8.3 servers. Back-patch to v11, like that commit. https://git.postgresql.org/pg/commitdiff/40dfac4fc4776213a02291f13046d36e318f2629

Michaël Paquier pushed:

Clean up more code using "(expr) ? true : false". This is similar to fd0625c, taking care of any remaining code paths that are worth the cleanup. This also changes some cases using opposite expression patterns. Author: Justin Pryzby, Masahiko Sawada Discussion: https://postgr.es/m/CAD21AoCdF8dnUvr-BUWWGvA_XhKSoANacBMZb6jKyCk4TYfQ2Q@mail.gmail.com https://git.postgresql.org/pg/commitdiff/68f7c4b57a27dbcd3e93ba3ff7b0b49664b25e09
Add more $Test::Builder::Level in the TAP tests. Incrementing the level of the call stack reported is useful for debugging purposes as it allows to control which part of the test is exactly failing, especially if a test is structured with subroutines that call routines from Test::More. This adds more incrementations of $Test::Builder::Level where debugging gets improved (for example it does not make sense for some paths like pg_rewind where long subroutines are used). A note is added to src/test/perl/README about that, based on a suggestion from Andrew Dunstan and a wording coming from both of us. Usage of Test::Builder::Level has spread in 12, so a backpatch down to this version is done. Reviewed-by: Andrew Dunstan, Peter Eisentraut, Daniel Gustafsson Discussion: https://postgr.es/m/YV1CCFwgM1RV1LeS@paquier.xyz Backpatch-through: 12 https://git.postgresql.org/pg/commitdiff/f9c4cb686800d46ef9e9e90ed5133493b23962af
Fix tests of pg_upgrade across different major versions. This fixes a set of issues that cause different breakages or annoyances when using pg_upgrade's test.sh to do upgrades across different major versions: - test.sh is completely broken when using v14 as new version because of the removal of testtablespace/ as Makefile rule. Older versions of pg_regress don't support --make-tablespacedir, blocking the creation of the tablespace. In order to fix that, it is simple enough to create those directories in the script itself, but only do that when an old version is involved. This fix is needed on HEAD and REL_14_STABLE. - The script would fail when using PG <= v11 as old version because of WITH OIDS relations not supported in v12. In order to fix this, this steals a method from the buildfarm that uses a DO block to change all the relations marked as WITH OIDS, allowing pg_upgrade to pass. This is more portable than using ALTER TABLE queries on the relations causing issues. This is fixed down to v12, and authored originally by Andrew Dunstan. - Not using --extra-float-digits=0 with v11 as old version causes a lot of diffs in the dumps, making the whole unreadable. This gets only done when using v11 as old version. This is fixed down to v12. The buildfarm code uses that already. Note that the addition of --wal-segsize and --allow-group-access breaks the script when using v10 or older at initdb time as these got added in 11. 10 would be EOL'd next year and nobody has complained about those problems yet, so nothing is done about that. This means that this commit fixes upgrade tests using test.sh with v11 as minimum older version, up to HEAD, and that it is enough to apply this change down to 12. The old and new dumps still generate diffs, still require manual checks, and more could be done to reduce the noise, but this allows the tests to run with a rather minimal amount of them. I have tested this commit and test.sh with v11 as minimum across all the branches where this is applied. Note that this commit has no impact on the normal pg_upgrade test run with a simple "make check". Author: Justin Pryzby, Andrew Dunstan, Michael Paquier Discussion: https://postgr.es/m/20201206180248.GI24052@telsasoft.com Backpatch-through: 12 https://git.postgresql.org/pg/commitdiff/fa66b6dee0843d2bca5bf9c9b8b7be32defbffae
Fix use-after-free with multirange types in CREATE TYPE. The code was freeing the name of the multirange type function stored in the parse tree but it should not do that. Event triggers could for example look at such a corrupted parsed tree with a ddl_command_end event. Author: Alex Kozhemyakin, Sergey Shinderuk Reviewed-by: Peter Eisentraut, Michael Paquier Discussion: https://postgr.es/m/d5042d46-b9cd-6efb-219a-71ed0cf45bc8@postgrespro.ru Backpatch-through: 14 https://git.postgresql.org/pg/commitdiff/5b0e7fe1d67235a092be1132bc5c97f1d7f29aaf

Peter Geoghegan pushed:

amcheck: Skip unlogged relations in Hot Standby. Have verify_heapam.c treat unlogged relations as if they were simply empty when in Hot Standby mode. This brings it in line with verify_nbtree.c, which has handled unlogged relations in the same way since bugfix commit 6754fe65a4. This was an oversight in commit 866e24d47d, which extended contrib/amcheck to check heap relations. In passing, lower the verbosity used when reporting that a relation has been skipped like this, from NOTICE to DEBUG1. This is appropriate because the skipping behavior is only an implementation detail, needed to work around the fact that unlogged tables don't have smgr-level storage for their main fork when in Hot Standby mode. Affected unlogged relations should be considered "trivially verified", not skipped over. They are verified in the same sense that a totally empty relation can be verified. This behavior seems least surprising overall, since unlogged relations on a replica will initially be empty if and when the replica is promoted and Hot Standby ends. Author: Mark Dilger mark.dilger@enterprisedb.com Reviewed-By: Peter Geoghegan pg@bowt.ie Discussion: https://postgr.es/m/CAH2-Wzk_pukOFY7JmdiFLsrz+Pd3V8OwgC1TH2Vd5BH5ZgK4bA@mail.gmail.com Backpatch: 14-, where heapam verification was introduced. https://git.postgresql.org/pg/commitdiff/292698f158ddb3f9a88f536e6eecb9e55d9619c9
Doc: normalize vacuum_multixact_failsafe_age ID. Author: Pavel Luzanov p.luzanov@postgrespro.ru Discussion: https://postgr.es/m/c71a3cfc-a267-3d9f-1b44-fbd668d0ab10@postgrespro.ru Backpatch: 14-, where the failsafe was introduced. https://git.postgresql.org/pg/commitdiff/00c61a74bcdbc04a3db721d53c7aff62244da198
pg_amcheck: avoid unhelpful verification attempts. Avoid calling contrib/amcheck functions with relations that are unsuitable for checking. Specifically, don't attempt verification of temporary relations, or indexes whose pg_index entry indicates that the index is invalid, or not ready. These relations are not supported by any of the contrib/amcheck functions, for reasons that are pretty fundamental. For example, the implementation of REINDEX CONCURRENTLY can add its own "transient" pg_index entries, which has rather unclear implications for the B-Tree verification functions, at least in the general case -- so they just treat it as an error. It falls to the amcheck caller (in this case pg_amcheck) to deal with the situation at a higher level. pg_amcheck now simply treats these conditions as additional "visibility concerns" when it queries system catalogs. This is a little arbitrary. It seems to have the least problems among any of the available alternatives. Author: Mark Dilger mark.dilger@enterprisedb.com Reported-By: Alexander Lakhin exclusion@gmail.com Reviewed-By: Peter Geoghegan pg@bowt.ie Reviewed-By: Robert Haas robertmhaas@gmail.com Bug: #17212 Discussion: https://postgr.es/m/17212-34dd4a1d6bba98bf@postgresql.org Backpatch: 14-, where pg_amcheck was introduced. https://git.postgresql.org/pg/commitdiff/d2bf06db377967b0d671ae372d513806e2a28052
Remove unstable pg_amcheck tests. Recent pg_amcheck bugfix commit d2bf06db added a test case that the buildfarm has shown to be non-portable. It doesn't particularly seem worth keeping anyway. Remove it. Discussion: https://postgr.es/m/CAH2-Wz=7HKJ9WzAh7+M0JfwJ1yfT9qoE+KPa3P7iGToPOtGhXg@mail.gmail.com Backpatch: 14-, just like the original commit. https://git.postgresql.org/pg/commitdiff/cd3f429d9565b2e5caf0980ea7c707e37bc3b317
Remove obsolete nbtree deduplication comments. Follow up to commit 2903f140. https://git.postgresql.org/pg/commitdiff/b76c1d6e849779e4a5a6c24d159a42125e522154

Fujii Masao pushed:

Make autovacuum launcher more responsive to pg_log_backend_memory_contexts(). Previously when pg_log_backend_memory_contexts() sent the request to the autovacuum launcher, it could take more than several seconds to log its memory contexts. Because the function (HandleAutoVacLauncherInterrupts) to process any new interrupts that autovacuum launcher received didn't handle the request for logging of memory contexts. This commit changes the function so that it handles the request, to make autovacuum launcher more responsitve to pg_log_backend_memory_contexts(). Back-patch to v14 where pg_log_backend_memory_contexts() was added. Author: Koyu Tanigawa Reviewed-by: Bharath Rupireddy, Atsushi Torikoshi Discussion: https://postgr.es/m/0aae3e074face409b35153451be5cc11@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/e3e29cec10d15bbcedc6b41887d8f4e138d719bd

Peter Eisentraut pushed:

psql: More tests. Add some basic tests for command-line option handling and help output, similar to what we have for other command-line programs. This also creates a place to put some more one-off test cases later. Discussion: /message-id/2570e2ae-fa0f-aac9-f72f-bb59a9983a20@enterprisedb.com https://git.postgresql.org/pg/commitdiff/c0280bc3edeb9e9958efc14083b6f301d2ef79d5
psql: Add test for handling of replication commands. Add a test for the clean handling of unsupported replication command responses. This was once accidentally broken, and it seems unusual enough that it's easy to forget when testing manually. Discussion: /message-id/2570e2ae-fa0f-aac9-f72f-bb59a9983a20@enterprisedb.com https://git.postgresql.org/pg/commitdiff/67c069848a998de1436cad2d67baedbf31c3a28c
psql: Fix test. The test didn't work on platforms where getopt() doesn't support non-option arguments before options. https://git.postgresql.org/pg/commitdiff/d9ddc50bafc062ec1ae7f98b886b7950102d87fc
psql: Fix some scan-build warnings. A repeated complaint was that scan-build thought that if the \timing setting changes during processing of a query, the post-processing might read garbage time values. This is probably not possible right now, but it's not entirely inconceivable given the code structure. So silence this warning with small restructuring that makes this more robust. The other warnings were a few dead stores that are easy to remove. Discussion: /message-id/2570e2ae-fa0f-aac9-f72f-bb59a9983a20@enterprisedb.com https://git.postgresql.org/pg/commitdiff/390edeeb570c01de1a14e2985ffed96de001e42e
Fix incorrect format placeholder. https://git.postgresql.org/pg/commitdiff/780054bf31a0a6ba781f46c454f0116efee8a74c

Robert Haas pushed:

Refactor basebackup.c's _tarWriteDir() function. Sometimes, we replace a symbolic link that we find in the data directory with an actual directory within the tarfile that we create. _tarWriteDir was responsible both for making this substitution and also for writing the tar header for the resulting directory into the tar file. Make it do only the first of those things, and rename to convert_link_to_directory. Substantially larger refactoring of this source file is planned, but this little bit seemed to make sense to commit independently. Discussion: http://postgr.es/m/CA+Tgmobz6tuv5tr-WxURe5JA1vVcGz85k4kkvoWxcyHvDpEqFA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/967a17fe2fa77b61061c8fb1183f64a5df4e080a
Refactor some end-of-recovery code out of StartupXLOG(). Create a new function PerformRecoveryXLogAction() and move the code which either writes an end-of-recovery record or requests a checkpoint there. Also create a new function CleanupAfterArchiveRecovery() to perform a few tasks that we want to do after we've actually exited archive recovery but before we start accepting new WAL writes. More refactoring of this file is planned, but this commit is just straightforward code movement to make StartupXLOG() a little bit shorter and a little bit easier to understand. Robert Haas and Amul Sul Discussion: http://postgr.es/m/CAAJ_b97abMuq=470Wahun=aS1PHTSbStHtrjjPaD-C0YQ1AqVw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/6df1543abfed6f6a86b76a48fa11a6f019111c01
Postpone some end-of-recovery operations related to allowing WAL. CreateOverwriteContrecordRecord(), UpdateFullPageWrites(), PerformRecoveryXLogAction(), and CleanupAfterArchiveRecovery() are moved somewhat later in StartupXLOG(). This is preparatory work for a future patch that wants to allow recovery to end at one time and only later start to allow WAL writes. To do that, it's necessary to separate code that has to do with allowing WAL writes from other things that need to happen simply because recovery is ending, such as initializing shared memory data structures that depend on information that might not be accurate before redo is complete. This commit does not achieve that goal, but it is a step in that direction. For example, there are a few different bits of code that write things into WAL once we have finished recovery, and with this change, those bits of code are closer to each other than previously, with fewer unrelated bits of code interspersed. Robert Haas and Amul Sul Discussion: http://postgr.es/m/CAAJ_b97abMuq=470Wahun=aS1PHTSbStHtrjjPaD-C0YQ1AqVw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/811051c2e7af1b030467760baf7ee0f4a22bc992
shm_mq: Update mq_bytes_written less often. Do not update shm_mq's mq_bytes_written until we have written an amount of data greater than 1/4th of the ring size, unless the caller of shm_mq_send(v) requests a flush at the end of the message. This reduces the number of calls to SetLatch(), and also the number of CPU cache misses, considerably, and thus makes shm_mq significantly faster. Dilip Kumar, reviewed by Zhihong Yu and Tomas Vondra. Some minor cosmetic changes by me. Discussion: http://postgr.es/m/CAFiTN-tVXqn_OG7tHNeSkBbN+iiCZTiQ83uakax43y1sQb2OBA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/46846433a03dff4f2e08c8a161e54a842da360d6

Etsuro Fujita pushed:

postgres_fdw: Move comments about elog level in (sub)abort cleanup. The comments were misplaced when adding postgres_fdw. Fix that by moving the comments to more appropriate functions. Author: Etsuro Fujita Backpatch-through: 9.6 Discussion: https://postgr.es/m/CAPmGK164sAXQtC46mDFyu6d-T25Mzvh5qaRNkit06VMmecYnOA%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/8c7be8688309dd7f30f313d3b312b31f64e93497

Álvaro Herrera pushed:

Change recently added test code for stability. The test code added with ff9f111bce24 fails under valgrind, and probably other slow cases too, because if (say) autovacuum runs in between and produces WAL of its own, the large INSERT fails to account for that in the LSN calculations. Rewrite to use a DO loop. Per complaint from Andres Freund Backpatch to all branches. Discussion: https://postgr.es/m/20211013180338.5guyqzpkcisqugrl@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/010e5233733aedf86634e1719d9536c42e18a27d

Jeff Davis pushed:

Check criticalSharedRelcachesBuilt in GetSharedSecurityLabel(). An extension may want to call GetSecurityLabel() on a shared object before the shared relcaches are fully initialized. For instance, a ClientAuthentication_hook might want to retrieve the security label on a role. Discussion: https://postgr.es/m/ecb7af0b26e3be1d96d291c8453a86f1f82d9061.camel@j-davis.com Backpatch-through: 9.6 https://git.postgresql.org/pg/commitdiff/7821a0bf2096df659671924fbeef0ebc66449292

Andrew Dunstan pushed:

Fix PostgresNode install_path sanity tests that fail on Windows. Backpatch to 14 where install_path was introduced. https://git.postgresql.org/pg/commitdiff/15124d0e22ed2280e4603638e5baede190ae584c

PostgreSQL Weekly News - October 10, 2021

Mon, 11 Oct 2021 00:00:00 +0000

PostgreSQL Weekly News - October 10, 2021

PostgreSQL Product News

pgCluu 3.2, a Perl program to audit PostgreSQL performance, released.

PGroonga 2.3.2 a full text search platform for all languages, released.

PostgreSQL Jobs for October

https://archives.postgresql.org/pgsql-jobs/2021-10/

PostgreSQL in the News

Planet PostgreSQL: https://planet.postgresql.org/

PostgreSQL Weekly News is brought to you this week by David Fetter

Submit news and announcements by Sunday at 3:00pm PST8PDT to david@fetter.org.

Applied Patches

Michaël Paquier pushed:

Fix snapshot builds during promotion of hot standby node with 2PC. Some specific logic is done at the end of recovery when involving 2PC transactions: 1) Call RecoverPreparedTransactions(), to recover the state of 2PC transactions into memory (re-acquire locks, etc.). 2) ShutdownRecoveryTransactionEnvironment(), to move back to normal operations, mainly cleaning up recovery locks and KnownAssignedXids (including any 2PC transaction tracked previously). 3) Switch XLogCtl->SharedRecoveryState to RECOVERY_STATE_DONE, which is the tipping point for any process calling RecoveryInProgress() to check if the cluster is still in recovery or not. Any snapshot taken between steps 2) and 3) would be empty, causing any transaction relying on a snapshot at this point to potentially corrupt data as there could still be some 2PC transactions to track, with RecentXmin moving backwards on successive calls to GetSnapshotData() in the same transaction. As SharedRecoveryState is the point to take into account to know if it is safe to discard KnownAssignedXids, this commit moves step 2) after step 3), so as we can never finish with empty snapshots. This exists since the introduction of hot standby, so backpatch all the way down. The window with incorrect snapshots is extremely small, but I have seen it when running 023_pitr_prepared_xact.pl, as did buildfarm member fairywren. Thomas Munro also found it independently. Special thanks to Andres Freund for taking the time to analyze this issue. Reported-by: Thomas Munro, Michael Paquier Analyzed-by: Andres Freund Discussion: https://postgr.es/m/20210422203603.fdnh3fu2mmfp2iov@alap3.anarazel.de Backpatch-through: 9.6 https://git.postgresql.org/pg/commitdiff/8a4237908c0fe73dd41d4d7c7a6314f17dfd7a6f
Fix warning in TAP test of pg_verifybackup. Oversight in a3fcbcd. Reported-by: Thomas Munro Discussion: https://postgr.es/m/CA+hUKGKnajZEwe91OTjro9kQLCMGGFHh2vvFn8tgHgbyn4bF9w@mail.gmail.com Backpatch-through: 13 https://git.postgresql.org/pg/commitdiff/ec2133a447318ac6d78887e91940d69e6d92a435
Refactor per-destination file rotation in logging collector. stderr and csvlog have been using duplicated code when it came to the rotation of their file by size, age or if forced by a user request (pg_ctl logrotate or the SQL function pg_rotate_logfile). The main difference between both is that stderr requires its file to always be opened, so as it is possible to have a redirection route if the logging collector is not ready yet to do its work if alternate destinations are enabled. Also, if csvlog gets disabled, we need to close properly its meta-data stored in the logging collector (last file name for current_logfiles and fd currently open for business). Except for those points, the code is the same in terms of error handling and if a file should be created or just continued. This change makes the code simpler overall, and it will help in the introduction of more file-based log destinations. This refactoring is similar to the work done in 5b0b699. Most of the duplication originates from fd801f4. Some of the TAP tests of pg_ctl check the case of a forced log rotation, but this is somewhat limited as there is no coverage for log_rotation_age or log_rotation_size (these may not be worth the extra resources to run either), and no coverage for reload of log_destination with different combinations of stderr and csvlog. I have tested all those cases separately for this refactoring. Author: Michael Paquier Discussion: https://postgr.es/m/CAH7T-aqswBM6JWe4pDehi1uOiufqe06DJWaU5=X7dDLyqUExHg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/5c6e33f071537d9831db57471a06d39a175b535a
Fix compilation warning in syslogger.c. Oversight in 5c6e33f. Author: Nathan Bossart Discussion: https://postgr.es/m/DD8AD4CE-63B7-44BE-A3D2-14A4E4B19C26@amazon.com https://git.postgresql.org/pg/commitdiff/05c4248ad1bf0c2721ce9445f6908da9ece36ff8
Refactor fallback to stderr for csvlog to handle better WIN32 service case. send_message_to_server_log() would force a redirection of a log entry to stderr in some cases for csvlog, like the syslogger not being available yet. If this happens, csvlog would fall back to stderr to log some information rather than nothing. The code was organized so as stderr is done before csvlog, with csvlog checking that stderr did not happen yet with a reversed condition. With this code organization, it could be possible to lose some messages if running Postgres as a service on WIN32, as there is no usable stderr, and the handling of the StringInfoData holding the message for stderr was rather confusing because of that. This commit moves the csvlog handling to be before stderr, as as we are able to track down if it is necessary to log something to stderr. The reduces the handling of stderr to be in a single code path, adding a fallback to event logs for a WIN32 service. This also simplifies the way we handle the StringInfoData for stderr, making easier the integration of new file-based log destinations. I got to play with services and event logs on Windows while checking this change. Reviewed-by: Chris Bandy Discussion: https://postgr.es/m/YV0vwBovEKf1WXkl@paquier.xyz https://git.postgresql.org/pg/commitdiff/8b76f89c37973082b3d64f5a27937efcca9d65f6

Daniel Gustafsson pushed:

Replace occurrences of InvalidXid with InvalidTransactionId. While Xid is a known shortening of TransactionId, InvalidXid is not defined in the code. Fix comments which mistakenly were using the shorter version. Author: Bharath Rupireddy bharath.rupireddyforpostgres@gmail.com Discussion: https://postgr.es/m/CALj2ACUQzdigML868nV4cojfELPkEzNLNOk7b91Pho4JB90fng@mail.gmail.com https://git.postgresql.org/pg/commitdiff/941921b875c7710e2b070c02c7819f2510808fdd
Provide error hint if TAP tests are not enabled. The error message for trying to run the TAP tests in a tree not configured with --enable-tap-tests is quite terse, and could be made more helpful to new developers onboarding to postgres. This adds a small hint on how to get the tests running in such cases. Author: Kevin Burke kevin@burke.dev Discussion: https://postgr.es/m/CAKcy5ejKVYwUXguQcd6i9KHDm7cM7FzjQ+aayaPveoa_woyQpQ@mail.gmail.com https://git.postgresql.org/pg/commitdiff/b5cb4db91327c2cef66207bde9cbcb592b91f93c
Provide error hint on exit() check when building libpq. Commit dc227eb82 introduced a restriction on libpq that no functions which invoke exit() are allowed to be called. This was further refined and fixed in e45b0dfa1f and 2f7bae2f92 and 792259591. While this is well documented in the Makefile, the error message emitted when the check failed was terse, without hints for new developers without prior context. This adds an error hint to assist new developers onboarding to postgres. Author: Rachel Heaton rheaton@vmware.com Co-authored-by: Jacob Champion pchampion@vmware.com Discussion: https://postgr.es/m/CADJcwiVL20955HCNzDqz9BEDr6A77pz6-nac5sbZVvhAEMijLg@mail.gmail.com https://git.postgresql.org/pg/commitdiff/e9bc0441f1446f6614fa6712841acec91890e089
Fix duplicate words in comments. Remove accidentally duplicated words in code comments. Author: Dagfinn Ilmari Mannsåker ilmari@ilmari.org Discussion: https://postgr.es/m/87bl45t0co.fsf@wibble.ilmari.org https://git.postgresql.org/pg/commitdiff/7111e332c57ddb562d0ce26a4e08761a0baafb65
Fix check for trapping exit() calls in libpq. Commit e9bc0441f added an errorhint on the exit() check for libpq, but accidentally changed the nm commandline to use -a instead of -A. These options are similar enough to hide it in testing, but -a can also show debugger symbols which isn't what we want. Fix by reverting the check back to using -A again. Reported-by: Anton Voloshin a.voloshin@postgrespro.ru Discussion: https://postgr.es/m/bd2c8409-d6b3-5de9-ba0f-40c1381f630f@postgrespro.ru https://git.postgresql.org/pg/commitdiff/de744e9efbc55288572d1e81168c74ea85a4b90a

Peter Eisentraut pushed:

Update Unicode map text files. A couple of newer ones are available. There are no functional differences, but let's get them in anyway, so that there is no surprise diff next time someone wants to do some actual work in this area. https://git.postgresql.org/pg/commitdiff/ce27c8953e8e48c69c690c0e5795cde40ed59fd2
Make Unicode makefile parallel-safe. Fix the rules so that each rule is parallel safe, using the same trickery that we use elsewhere in the tree for rules that produce more than one output file. Refactor the whole makefile so that there is less repetition. Discussion: /message-id/18e34084-aab1-1b4c-edd1-c4f9fb04f714%40enterprisedb.com https://git.postgresql.org/pg/commitdiff/e752727195798c324e769cfebf9dc4baa1c6bb0c
Fix loop variable signedness. https://git.postgresql.org/pg/commitdiff/ba216d3b54ac334729c505ec8a725db3826290a2
Improve order in file. Move support functions for new PublicationTable node to more sensible locations in the files. https://git.postgresql.org/pg/commitdiff/d942887039a608c91084a942fe10571c6f6be35a

Tom Lane pushed:

Doc: fix minor issues in GiST support function documentation. gist.sgml and xindex.sgml hadn't been fully updated for the addition of a sortsupport support function (commit 16fa9b2b3). xindex.sgml also missed that the compress and decompress support functions are optional, an apparently far older oversight. In passing, fix gratuitous inconsistencies in wording and capitalization. Noted by E. Rogov. Back-patch to v14; the residual issues before that aren't significant enough to bother with. Discussion: https://postgr.es/m/163335322905.12519.5711557029494638051@wrigleys.postgresql.org https://git.postgresql.org/pg/commitdiff/36d1a848a86afd2855215af2a112b9bde999354a
Update our mapping of Windows time zone names some more. Per discussion, let's just follow CLDR's default zone mappings faithfully. There are two changes here that are clear improvements: * Mapping "Greenwich Standard Time" to Atlantic/Reykjavik is actually a better fit than using London, because Iceland hasn't observed DST since 1968, so this is more nearly what people might expect. * Since the "Samoa" zone is specified to be UTC+13:00, we must map it to Pacific/Apia not Pacific/Samoa; the latter refers to American Samoa which is now on the other side of the date line. The rest of these changes look like they're choosing the most populous IANA zone as representative. Whatever the details, we're just going to say "if you don't like this mapping, complain to CLDR". Discussion: https://postgr.es/m/3266414.1633045628@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/c1aa3b3c0d2125cb04df8ed0387448d8aeb9519c
Doc: improve description of UNION/INTERSECT/EXCEPT syntax. queries.sgml failed to mention the rather important point that INTERSECT binds more tightly than UNION or EXCEPT. I thought it could also use more discussion of the role of parentheses in these constructs. Per gripe from Christopher Painter-Wakefield. Discussion: https://postgr.es/m/163338891727.12510.3939775743980651160@wrigleys.postgresql.org https://git.postgresql.org/pg/commitdiff/f3fec23dbdead113700fb1b401b681fa24f1e4f4
Doc: improve timezone/README's recipe for tracking Windows zones. We should now cite CLDR as primary reference for the zone name mapping. Discussion: https://postgr.es/m/3266414.1633045628@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/db692b0c84908b4ef5ea4c15fa2d742582ad2cf9
Fix null-pointer crash in postgres_fdw's conversion_error_callback. Commit c7b7311f6 adjusted conversion_error_callback to always use information from the query's rangetable, to avoid doing catalog lookups in an already-failed transaction. However, as a result of the utterly inadequate documentation for make_tuple_from_result_row, I failed to realize that fsstate could be NULL in some contexts. That led to a crash if we got a conversion error in such a context. Fix by falling back to the previous coding when fsstate is NULL. Improve the commentary, too. Per report from Andrey Borodin. Back-patch to 9.6, like the previous patch. Discussion: https://postgr.es/m/08916396-55E4-4D68-AB3A-BD6066F9E5C0@yandex-team.ru https://git.postgresql.org/pg/commitdiff/3071bbfe44f36019710190a9273ad2bd4a947878
plperl: update ppport.h to Perl 5.34.0. Also apply the changes suggested by running perl ppport.h --compat-version=5.8.0 And remove some no-longer-required NEED_foo declarations. Dagfinn Ilmari Mannsåker Discussion: https://postgr.es/m/87y278s6iq.fsf@wibble.ilmari.org https://git.postgresql.org/pg/commitdiff/05798c9f7f08908bdd06c82d934da67535b72005
Adjust configure to insist on Perl version >= 5.8.3. Previously it only checked for version >= 5.8.0, although the documentation has said that the minimum version is 5.8.3 since commit dea6ba939. Per the discussion leading up to that commit, I (tgl) left it that way intentionally because you could, at the time, do some bare-bones stuff with 5.8.0. But we aren't actually testing against anything older than 5.8.3, so who knows if that's still true. It's pretty unlikely that anyone would care anyway, so let's just make configure's version check match the docs. Dagfinn Ilmari Mannsåker Discussion: https://postgr.es/m/87y278s6iq.fsf@wibble.ilmari.org Discussion: https://postgr.es/m/16894.1501392088@sss.pgh.pa.us https://git.postgresql.org/pg/commitdiff/92e6a98c3636948e7ece9a3260f9d89dd60da278
Update test/perl/README to insist on Perl version >= 5.8.3, too. Oversight in previous commit, noted by Daniel Gustafsson. Discussion: https://postgr.es/m/87y278s6iq.fsf@wibble.ilmari.org https://git.postgresql.org/pg/commitdiff/93fb39eca643a33dd6e3c8818fc7899aa67a8103
Doc: update our claims about the minimum recommended AIX version. We currently have buildfarm members testing back to AIX 7.1, but not before, and older AIX versions are long out of support from IBM. So say that 7.1 is the oldest supported version. Discussion: https://postgr.es/m/87y278s6iq.fsf@wibble.ilmari.org https://git.postgresql.org/pg/commitdiff/08e2daf06c71881415ebd19105a8fe53f6eb2f8f
Doc: improve documentation for ^@ starts-with operator. This operator wasn't formally documented anywhere. To give it a natural home, relabel the functions-string-other table as "Other String Functions and Operators", which is more parallel to the functions-string-sql table anyway. While here, add cross-references to the pattern match and text search sections. It seems moderately likely that people would come to this section looking for those (but I don't want to actually list them in these tables). Discussion: https://postgr.es/m/CADT4RqB13KQHOJqqQ+WXmYtJrukS2UiFdtfTvT-XA3qYLyB6Cw@mail.gmail.com https://git.postgresql.org/pg/commitdiff/2ae5d72f004f599c351ee31e8da5fb3e40303760

Andres Freund pushed:

windows: Define WIN32_LEAN_AND_MEAN to make compilation faster. windows.h includes a lot of other headers, slowing down compilation significantly. WIN32_LEAN_AND_MEAN reduces that a bit. It'd be better to remove the include of windows.h (as well as indirect inclusions of it) from such a central place, but until then... Discussion: https://postgr.es/m/20210921193035.pqzay43vpyv7in43@alap3.anarazel.de https://git.postgresql.org/pg/commitdiff/8162464a25e5314e753c580389f76a9b7f69445b
Fix TestLib::slurp_file() with offset on windows. 3c5b0685b921 used setFilePointer() to set the position of the filehandle, but passed the wrong filehandle, always leaving the position at 0. Instead of just fixing that, remove use of setFilePointer(), we have a perl fd at this point, so we can just use perl's seek(). Additionally, the perl filehandle wasn't closed, just the windows filehandle. Reviewed-By: Andrew Dunstan andrew@dunslane.net Author: Andres Freund andres@anarazel.de Discussion: https://postgr.es/m/20211003173038.64mmhgxctfqn7wl6@alap3.anarazel.de Backpatch: 9.6-, like 3c5b0685b921 https://git.postgresql.org/pg/commitdiff/2f74db1236fe83e6665e5b0ddad4454c69495614

Bruce Momjian pushed:

doc: remove URL for ICU explorer/locexp. The old URL was HTTP 404 and the git link didn't build. Also update two other ICU links. If we ever get a good link we will add it back. Reported-by: Anton Voloshin Author: Laurenz Albe Backpatch-through: 10 https://git.postgresql.org/pg/commitdiff/e8259439066c5fa4f3266f30434d5a52b8347bd1

Fujii Masao pushed:

psql: Improve tab-completion for LOCK TABLE. This commit makes psql support the tab-completion for ONLY and NOWAIT keywords of LOCK TABLE command. Author: Koyu Tanigawa Reviewed-by: Shinya Kato, Fujii Masao Discussion: https://postgr.es/m/a322684daa36319e6ebc60b541000a3a@oss.nttdata.com https://git.postgresql.org/pg/commitdiff/0b0d277c35533baecc8d1a9356f71de5f2ee0bd8
doc: Document pg_encoding_to_char() and pg_char_to_encoding(). Previously both functions were not described anywhere in the docs. But since they have been around since 7.0 and mentioned in the description for system catalog like pg_database, it's reasonable to add short descriptions for them. Author: Ian Lawrence Barwick Reviewed-by: Laurenz Albe, Fujii Masao Discussion: https://postgr.es/m/CAB8KJ=infievn4q1N4X7Vx8w4_RMPPG0pLvxhSDjy5WQOSHW9g@mail.gmail.com https://git.postgresql.org/pg/commitdiff/f6b5d05ba9a4ac7c5ebec76045c6e0afcf7c9eec
Make recovery report error message when invalid page header is found. Commit 0668719801 changed XLogPageRead() so that it validated the page header, if invalid page header was found reset the error message and retried reading the page, to fix the scenario where streaming standby got stuck at a continuation record. This change hid the error message about invalid page header, which would make it harder for users to investigate what the actual issue was found in WAL. To fix the issue, this commit makes XLogPageRead() report the error message when invalid page header is found. When not in standby mode, an invalid page header should cause recovery to end, not retry reading the page, so XLogPageRead() doesn't need to validate the page header for the retry. Instead, ReadPageInternal() should be responsible for the validation in that case. Therefore this commit changes XLogPageRead() so that if not in standby mode it doesn't validate the page header for the retry. Reported-by: Yugo Nagata Author: Yugo Nagata, Kyotaro Horiguchi Reviewed-by: Ranier Vilela, Fujii Masao Discussion: https://postgr.es/m/20210718045505.32f463ed6c227111038d8ae4@sraoss.co.jp https://git.postgresql.org/pg/commitdiff/68601985e699adeb267636fd19d3d6113554bd1f

Amit Kapila pushed:

Remove obsolete comment in snapbuild.c. Commits 955a684e04 and a975ff4980 removed the usage of running xacts information from serialized snapshots but forgot to remove the corresponding comment. Author: Masahiko Sawada Discussion: https://postgr.es/m/CAD21AoBifOr7RS=jRe7YCavc646y9omChv6zkWXvJeZcjS9mXA@mail.gmail.com https://git.postgresql.org/pg/commitdiff/826584fa5284341c601f3c49804dfa9c02295554

Robert Haas pushed:

Flexible options for BASE_BACKUP. Previously, BASE_BACKUP used an entirely hard-coded syntax, but that's hard to extend. Instead, adopt the same kind of syntax we've used for SQL commands such as VACUUM, ANALYZE, COPY, and EXPLAIN, where it's not necessary for all of the option names to be parser keywords. In the new syntax, most of the options now take an optional Boolean argument. To match our practice in other in places, the options which the old syntax called NOWAIT and NOVERIFY_CHECKSUMS options are in the new syntax called WAIT and VERIFY_CHECKUMS, and the default value is false. In the new syntax, the FAST option has been replaced by a CHECKSUM option whose value may be 'fast' or 'spread'. This commit does not remove support for the old syntax. It just adds the new one as an additional option, and makes pg_basebackup prefer the new syntax when the server is new enough to support it. Patch by me, reviewed and tested by Fabien Coelho, Sergei Kornilov, Fujii Masao, and Tushar Ahuja. Discussion: http://postgr.es/m/CA+TgmobAczXDRO_Gr2euo_TxgzaH1JxbNxvFx=HYvBinefNH8Q@mail.gmail.com Discussion: http://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com https://git.postgresql.org/pg/commitdiff/0ba281cb4bf9f5f65529dfa4c8282abb734dd454
Flexible options for CREATE_REPLICATION_SLOT. Like BASE_BACKUP, CREATE_REPLICATION_SLOT has historically used a hard-coded syntax. To improve future extensibility, adopt a flexible options syntax here, too. In the new syntax, instead of three mutually exclusive options EXPORT_SNAPSHOT, USE_SNAPSHOT, and NOEXPORT_SNAPSHOT, there is now a single SNAPSHOT option with three possible values: 'export', 'use', and 'nothing'. This commit does not remove support for the old syntax. It just adds the new one as an additional option, makes pg_receivewal, pg_recvlogical, and walreceiver processes use it. Patch by me, reviewed by Fabien Coelho, Sergei Kornilov, and Fujii Masao. Discussion: http://postgr.es/m/CA+TgmobAczXDRO_Gr2euo_TxgzaH1JxbNxvFx=HYvBinefNH8Q@mail.gmail.com Discussion: http://postgr.es/m/CA+TgmoZGwR=ZVWFeecncubEyPdwghnvfkkdBe9BLccLSiqdf9Q@mail.gmail.com https://git.postgresql.org/pg/commitdiff/0266e98c6b865246c3031bbf55cb15f330134e30

Dean Rasheed pushed:

Fix corner-case loss of precision in numeric_power(). This fixes a loss of precision that occurs when the first input is very close to 1, so that its logarithm is very small. Formerly, during the initial low-precision calculation to estimate the result weight, the logarithm was computed to a local rscale that was capped to NUMERIC_MAX_DISPLAY_SCALE (1000). However, the base may be as close as 1e-16383 to 1, hence its logarithm may be as small as 1e-16383, and so the local rscale needs to be allowed to exceed 16383, otherwise all precision is lost, leading to a poor choice of rscale for the full-precision calculation. Fix this by removing the cap on the local rscale during the initial low-precision calculation, as we already do in the full-precision calculation. This doesn't change the fact that the initial calculation is a low-precision approximation, computing the logarithm to around 8 significant digits, which is very fast, especially when the base is very close to 1. Patch by me, reviewed by Alvaro Herrera. Discussion: https://postgr.es/m/CAEZATCV-Ceu%2BHpRMf416yUe4KKFv%3DtdgXQAe5-7S9tD%3D5E-T1g%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/e54a758d24dab056bb7f50d26c57a3c8761cc44a

Etsuro Fujita pushed:

Add missing word to comment in joinrels.c. Author: Amit Langote Backpatch-through: 13 Discussion: https://postgr.es/m/CA%2BHiwqGQNbtamQ_9DU3osR1XiWR4wxWFZurPmN6zgbdSZDeWmw%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/700c73312841bd1a89263f238556ce8d8d916258
postgres_fdw: Fix comments in connection.c. Commit 27e1f1456 missed updating some comments. Reviewed-by: Bharath Rupireddy Backpatch-through: 14 Discussion: https://postgr.es/m/CAPmGK15Q2Nm6U%2Ba_GwskrWFEVBZ9_3VKOvRrprGufpx91M_3Sw%40mail.gmail.com https://git.postgresql.org/pg/commitdiff/972c7c6567fbb02a59b94ede80b17805de1bc03c