Re: BUG #18254: could not load library llvmjit.so，Undefined symbol "_ZN4llvm12MemoryBuffer7getFileERKNS_5TwineEbbbSt

The following bug has been logged on the website:

Bug reference: 18254
Logged by: Xiaoming Yang
Email address: yangxiaoming2019(at)gmail(dot)com
PostgreSQL version: 16.1
Operating system: FreeBSD14
Description:

when I run the sql
```sql
explain analyze select .. from ...;
```

I got the error info
```text
SQL Error [XX000]: ERROR: could not load library
"/usr/local/lib/postgresql/llvmjit.so":
/usr/local/lib/postgresql/llvmjit.so: Undefined symbol
"_ZN4llvm12MemoryBuffer7getFileERKNS_5TwineEbbbSt8optionalINS_5AlignEE"
Where: parallel worker
ERROR: could not load library "/usr/local/lib/postgresql/llvmjit.so":
/usr/local/lib/postgresql/llvmjit.so: Undefined symbol
"_ZN4llvm12MemoryBuffer7getFileERKNS_5TwineEbbbSt8optionalINS_5AlignEE"
Where: parallel worker
ERROR: could not load library "/usr/local/lib/postgresql/llvmjit.so":
/usr/local/lib/postgresql/llvmjit.so: Undefined symbol
"_ZN4llvm12MemoryBuffer7getFileERKNS_5TwineEbbbSt8optionalINS_5AlignEE"
Where: parallel worker
``

I build postgresql on my FreeBSD system, the system & postgresql info:
```text
# build config
./configure --prefix=/usr/local --with-openssl --with-lz4
--with-wal-blocksize=64 --with-blocksize=32 --with-llvm

# uname -a
FreeBSD xxxx 14.0-RELEASE FreeBSD 14.0-RELEASE #0
releng/14.0-n265380-f9716eee8ab4: Fri Nov 10 05:57:23 UTC 2023
root(at)releng1(dot)nyi(dot)freebsd(dot)org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
amd64

# pg_config
BINDIR = /usr/local/bin
DOCDIR = /usr/local/share/doc/postgresql
HTMLDIR = /usr/local/share/doc/postgresql
INCLUDEDIR = /usr/local/include
PKGINCLUDEDIR = /usr/local/include/postgresql
INCLUDEDIR-SERVER = /usr/local/include/postgresql/server
LIBDIR = /usr/local/lib
PKGLIBDIR = /usr/local/lib/postgresql
LOCALEDIR = /usr/local/share/locale
MANDIR = /usr/local/share/man
SHAREDIR = /usr/local/share/postgresql
SYSCONFDIR = /usr/local/etc/postgresql
PGXS = /usr/local/lib/postgresql/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--prefix=/usr/local' '--with-openssl' '--with-lz4'
'--with-wal-blocksize=64' '--with-blocksize=32' '--with-llvm'
'CFLAGS=-I/usr/local/include/python3.11/' 'LLVM_CONFIG=llvm-config16'
CC = gcc
CPPFLAGS = -I/usr/local/include -I/usr/local/include
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Werror=vla -Wendif-labels
-Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type
-Wshadow=compatible-local -Wformat-security -fno-strict-aliasing -fwrapv
-fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation
-I/usr/local/include/python3.11/
CFLAGS_SL = -fPIC -DPIC
LDFLAGS = -L/usr/local/llvm16/lib -L/usr/local/lib -Wl,--as-needed
-Wl,-R'/usr/local/lib'
LDFLAGS_EX =
LDFLAGS_SL =
LIBS = -lpgcommon -lpgport -llz4 -lssl -lcrypto -lz -lreadline -lpthread
-lexecinfo -lm
VERSION = PostgreSQL 16.1

# llvm-config16 --version
16.0.6

# postgresql.conf
jit_provider = 'llvmjit'
```

Demangle the symbol
`_ZN4llvm12MemoryBuffer7getFileERKNS_5TwineEbbbSt8optionalINS_5AlignEE` is
`llvm::MemoryBuffer::getFile(llvm::Twine const&, bool, bool, bool,
std::optional<llvm::Align>)`

And, I have another PostgreSQL on Rokey 9 Linux, it's work well.
The Linux version info:
```text
# uname -a
Linux xxx 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Nov 8
17:36:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

# build config
./configure --with-openssl --with-zstd -with-lz4 --with-systemd
--with-python --with-llvm

# postgresql.conf
jit_provider = 'llvmjit'
# pg_config
BINDIR = /usr/local/bin
DOCDIR = /usr/local/share/doc/postgresql
HTMLDIR = /usr/local/share/doc/postgresql
INCLUDEDIR = /usr/local/include
PKGINCLUDEDIR = /usr/local/include/postgresql
INCLUDEDIR-SERVER = /usr/local/include/postgresql/server
LIBDIR = /usr/local/lib
PKGLIBDIR = /usr/local/lib/postgresql
LOCALEDIR = /usr/local/share/locale
MANDIR = /usr/local/share/man
SHAREDIR = /usr/local/share/postgresql
SYSCONFDIR = /usr/local/etc/postgresql
PGXS = /usr/local/lib/postgresql/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--prefix=/usr/local' '--with-openssl' '-with-lz4'
'--with-systemd' '--with-python' '--with-llvm'
CC = gcc
CPPFLAGS = -D_GNU_SOURCE
CFLAGS = -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Werror=vla -Wendif-labels
-Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type
-Wshadow=compatible-local -Wformat-security -fno-strict-aliasing -fwrapv
-fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation
-O2
CFLAGS_SL = -fPIC
LDFLAGS = -L/usr/lib64 -Wl,--as-needed
-Wl,-rpath,'/usr/local/lib',--enable-new-dtags
LDFLAGS_EX =
LDFLAGS_SL =
LIBS = -lpgcommon -lpgport -llz4 -lssl -lcrypto -lz -ledit -lm
VERSION = PostgreSQL 16.1

# llvm-config --version
16.0.6

# postgresql.conf
jit_provider = 'llvmjit'
```

Linux Version run out:
```text
-- some info

Planning Time: 1.596 ms
JIT:
Functions: 75
Options: Inlining true, Optimization true, Expressions true, Deforming
true
Timing: Generation 2.626 ms, Inlining 221.238 ms, Optimization 161.173
ms, Emission 132.760 ms, Total 517.797 ms
Execution Time: 1416.203 ms
```

On Thu, Dec 21, 2023 at 9:31 AM PG Bug reporting form
<noreply(at)postgresql(dot)org> wrote:
> ERROR: could not load library "/usr/local/lib/postgresql/llvmjit.so":
> /usr/local/lib/postgresql/llvmjit.so: Undefined symbol
> "_ZN4llvm12MemoryBuffer7getFileERKNS_5TwineEbbbSt8optionalINS_5AlignEE"

> Demangle the symbol
> `_ZN4llvm12MemoryBuffer7getFileERKNS_5TwineEbbbSt8optionalINS_5AlignEE` is
> `llvm::MemoryBuffer::getFile(llvm::Twine const&, bool, bool, bool,
> std::optional<llvm::Align>)`

My OS and LLVM (installed with pkg) are the same as yours,
though I'm building the master branch of PostgreSQL... hmm, I see a
slightly different name here:

tmunro(at)build1:~/projects/postgresql $ nm -DC
~/install/lib/postgresql/llvmjit.so | grep
'llvm::MemoryBuffer::getFile'
U llvm::MemoryBuffer::getFile(llvm::Twine const&,
bool, bool, bool, std::__1::optional<llvm::Align>)@LLVM_16

tmunro(at)build1:~/projects/postgresql $ nm -DC
/usr/local/llvm16/lib/libLLVM.so | grep
'llvm::MemoryBuffer::getFile(llvm::Twine'
00000000038ba2d0 T llvm::MemoryBuffer::getFile(llvm::Twine const&,
bool, bool, bool, std::__1::optional<llvm::Align>)@@LLVM_16

Note "std::__1::optional". Could this be a clue about C++ standard
library mismatch, or something like that? Google is telling me that
stdlib=libc++ (LLVM's standard library) has the __1 namespace, but
stdlib=libstdc++ (GNU's standard library) lacks it, but I didn't look
too hard...

I got the reason, why not found the symbol.

On my FreeBSD system, I have 2 version clang++, the system path
`/usr/bin/clang++` and the `pkg` version `/usr/local/bin/clang++16`.

When I just only set the environment `LLVM_CONFIG=llvm-config16`, run
`./configure --prefix=/usr/local --with-openssl --with-lz4
--with-wal-blocksize=64 --with-blocksize=32 --with-llvm` out put this
configure result:
```text
configure: using compiler=FreeBSD clang version 16.0.6
(https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)
configure: using CFLAGS=-Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Werror=vla
-Werror=unguarded-availability-new -Wendif-labels
-Wmissing-format-attribute -Wcast-function-type -Wformat-security
-fno-strict-aliasing -fwrapv -fexcess-precision=standard
-Wno-unused-command-line-argument -Wno-compound-token-split-by-macro
-Wno-cast-function-type-strict -I/usr/local/include/python3.11/
configure: using CPPFLAGS= -I/usr/local/include
configure: using LDFLAGS= -L/usr/local/llvm16/lib -L/usr/local/lib 
-Wl,--as-needed
configure: using CXX=/usr/bin/clang++
configure: using CXXFLAGS=-Wall -Wpointer-arith
-Werror=unguarded-availability-new -Wendif-labels
-Wmissing-format-attribute -Wcast-function-type -Wformat-security
-fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2
configure: using CLANG=/usr/bin/clang
configure: using BITCODE_CFLAGS= -fno-strict-aliasing -fwrapv
-fexcess-precision=standard -Xclang -no-opaque-pointers
-Wno-unused-command-line-argument -Wno-compound-token-split-by-macro -O2
configure: using BITCODE_CXXFLAGS= -fno-strict-aliasing -fwrapv
-fexcess-precision=standard -Xclang -no-opaque-pointers -O2
```

I changed the environments:
```bash
export LLVM_CONFIG=/usr/local/bin/llvm-config16
export CC=/usr/local/bin/clang16
export CXX=/usr/local/bin/clang++16
export CLANG=/usr/local/bin/clang16
```

configure out put:
```text
configure: using compiler=clang version 16.0.6
configure: using CFLAGS=-Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Werror=vla
-Werror=unguarded-availability-new -Wendif-labels
-Wmissing-format-attribute -Wcast-function-type -Wformat-security
-fno-strict-aliasing -fwrapv -fexcess-precision=standard
-Wno-unused-command-line-argument -Wno-compound-token-split-by-macro
-Wno-cast-function-type-strict -I/usr/local/include/python3.11/
configure: using CPPFLAGS= -I/usr/local/include
configure: using LDFLAGS= -L/usr/local/llvm16/lib -L/usr/local/lib 
-Wl,--as-needed
configure: using CXX=/usr/local/bin/clang++16
configure: using CXXFLAGS=-Wall -Wpointer-arith
-Werror=unguarded-availability-new -Wendif-labels
-Wmissing-format-attribute -Wcast-function-type -Wformat-security
-fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2
configure: using CLANG=/usr/local/bin/clang16
configure: using BITCODE_CFLAGS= -fno-strict-aliasing -fwrapv
-fexcess-precision=standard -Wno-unused-command-line-argument
-Wno-compound-token-split-by-macro -O2
configure: using BITCODE_CXXFLAGS= -fno-strict-aliasing -fwrapv
-fexcess-precision=standard -O2
 ```

This issue may be reason with the system path clang++ pair of `pkg`
version llvm libraries.
On the FreeBSD system, if with the `--with-llvm` configure argument,
should build with the llvm's clang and clang++. I guess the system path
`/usr/bin/clang++` has some difference with `pkg` version
`/usr/local/bin/clang++16`, or `LD_LIBRARY_PATH` not contained the real
`/usr/local/llvm16/lib` path.

My new build version, it's work well.
```text
...
Planning Time: 2.444 ms
JIT:
    Functions: 75
    Options: Inlining true, Optimization true, Expressions true,
Deforming true
    Timing: Generation 3.608 ms, Inlining 248.625 ms, Optimization
208.667 ms, Emission 167.912 ms, Total 628.812 ms
Execution Time: 10882.537 ms
```

On 2023/12/21 05:02, Thomas Munro wrote:
> On Thu, Dec 21, 2023 at 9:31 AM PG Bug reporting form
> <noreply(at)postgresql(dot)org> wrote:
>> ERROR: could not load library "/usr/local/lib/postgresql/llvmjit.so":
>> /usr/local/lib/postgresql/llvmjit.so: Undefined symbol
>> "_ZN4llvm12MemoryBuffer7getFileERKNS_5TwineEbbbSt8optionalINS_5AlignEE"
>> Demangle the symbol
>> `_ZN4llvm12MemoryBuffer7getFileERKNS_5TwineEbbbSt8optionalINS_5AlignEE` is
>> `llvm::MemoryBuffer::getFile(llvm::Twine const&, bool, bool, bool,
>> std::optional<llvm::Align>)`
> My OS and LLVM (installed with pkg) are the same as yours,
> though I'm building the master branch of PostgreSQL... hmm, I see a
> slightly different name here:
>
> tmunro(at)build1:~/projects/postgresql $ nm -DC
> ~/install/lib/postgresql/llvmjit.so | grep
> 'llvm::MemoryBuffer::getFile'
> U llvm::MemoryBuffer::getFile(llvm::Twine const&,
> bool, bool, bool, std::__1::optional<llvm::Align>)@LLVM_16
>
> tmunro(at)build1:~/projects/postgresql $ nm -DC
> /usr/local/llvm16/lib/libLLVM.so | grep
> 'llvm::MemoryBuffer::getFile(llvm::Twine'
> 00000000038ba2d0 T llvm::MemoryBuffer::getFile(llvm::Twine const&,
> bool, bool, bool, std::__1::optional<llvm::Align>)@@LLVM_16
>
> Note "std::__1::optional". Could this be a clue about C++ standard
> library mismatch, or something like that? Google is telling me that
> stdlib=libc++ (LLVM's standard library) has the __1 namespace, but
> stdlib=libstdc++ (GNU's standard library) lacks it, but I didn't look
> too hard...