Re: PGXLOG variable worthwhile?

> -----Original Message-----
> From: Bruce Momjian [mailto:pgman(at)candle(dot)pha(dot)pa(dot)us]
> Sent: 12 September 2002 06:27
> To: Justin Clift
> Cc: PostgreSQL Hackers Mailing List
> Subject: Re: [HACKERS] PGXLOG variable worthwhile?
>
> Also, I have heard symlinks are available in native Windows
> but the interface to them isn't clearly visible. Can someone
> clarify that?

Well there are 'shortcuts' but I wouldn't want to trust my xlog
directory to one.

Even if I did, iirc, unless you are using the shell api, they just
appear to be regular files anyway (for example, in Cygwin vi, I can edit
a shortcut to a directory).

Regards, Dave.

Dave Page wrote:
>
>>-----Original Message-----
>>From: Bruce Momjian [mailto:pgman(at)candle(dot)pha(dot)pa(dot)us]
>>
>>Also, I have heard symlinks are available in native Windows
>>but the interface to them isn't clearly visible. Can someone
>>clarify that?
>
>
> Well there are 'shortcuts' but I wouldn't want to trust my xlog
> directory to one.

These are Shell OLE links. As Dave points out, it requires the
shell to interpret the shortcut.

>
> Even if I did, iirc, unless you are using the shell api, they just
> appear to be regular files anyway (for example, in Cygwin vi, I can edit
> a shortcut to a directory).
>
> Regards, Dave.

In Windows 2000 and Windows XP with an NTFS filesystem,
Microsoft has added Reparse Points, which allow for the
implementation of symbolic links for directories. Microsoft
calls them "Junctions". I *believe* the function used for
creating reparse points is DeviceIoControl() with the
FSCTL_SET_REPARSE_POINT I/O control code. I don't have quick
access to 2K or XP, but it is clearly not supported by Win32 on
95/98/ME.

Here's a link discussing the features of NTFS5 and Reparse Points:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnw2kmag00/html/NTFSPart1.asp

Mike Mascari
mascarm(at)mascari(dot)com

Mike Mascari wrote:
<snip>
> In Windows 2000 and Windows XP with an NTFS filesystem,
> Microsoft has added Reparse Points, which allow for the
> implementation of symbolic links for directories. Microsoft
> calls them "Junctions". I *believe* the function used for
> creating reparse points is DeviceIoControl() with the
> FSCTL_SET_REPARSE_POINT I/O control code. I don't have quick
> access to 2K or XP, but it is clearly not supported by Win32 on
> 95/98/ME.
>
> Here's a link discussing the features of NTFS5 and Reparse Points:
>
> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnw2kmag00/html/NTFSPart1.asp

That's really useful info. Reparse points under Win2k (mount points to
the rest of us) are definitely something to try out in the future then.
:)

Seems like the NT4 users are left out in the cold though until we add
some kind of ability for PostgreSQL to not look at the filesystem for
info about where to put the xlog files.

Regards and best wishes,

Justin Clift

> Mike Mascari
> mascarm(at)mascari(dot)com

--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi

On Thu, 12 Sep 2002, Justin Clift wrote:

> Mike Mascari wrote:
> <snip>
> > In Windows 2000 and Windows XP with an NTFS filesystem,
> > Microsoft has added Reparse Points, which allow for the
> > implementation of symbolic links for directories. Microsoft
> > calls them "Junctions". I *believe* the function used for
> > creating reparse points is DeviceIoControl() with the
> > FSCTL_SET_REPARSE_POINT I/O control code. I don't have quick
> > access to 2K or XP, but it is clearly not supported by Win32 on
> > 95/98/ME.
> >
> > Here's a link discussing the features of NTFS5 and Reparse Points:
> >
> > http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnw2kmag00/html/NTFSPart1.asp
>
> That's really useful info. Reparse points under Win2k (mount points to
> the rest of us) are definitely something to try out in the future then.
> :)
>
> Seems like the NT4 users are left out in the cold though until we add
> some kind of ability for PostgreSQL to not look at the filesystem for
> info about where to put the xlog files.

This isn't true. With the resource kit, you get the gnu utils, and ln
works a charm under NT4 with ntfs. And not just for directories, but
files as well. Unless Microsoft somehow removed that functionality in the
intervening years since I've used NT. (wouldn't put it past them, but I
doubt they have.)

"scott.marlowe" wrote:
<snip>
> > Seems like the NT4 users are left out in the cold though until we add
> > some kind of ability for PostgreSQL to not look at the filesystem for
> > info about where to put the xlog files.
>
> This isn't true. With the resource kit, you get the gnu utils, and ln
> works a charm under NT4 with ntfs. And not just for directories, but
> files as well. Unless Microsoft somehow removed that functionality in the
> intervening years since I've used NT. (wouldn't put it past them, but I
> doubt they have.)

The reference point that I'm working from is this:

- Am testing out the third beta of the Native PostgreSQL port for
Windows, on NT4 SP6 at present.
- Have an internal RAID array of Seagate Cheetah 10kRPM drives. When
installing the PGDATA directory on one drive it gives a certain kind of
performance, and I'm interested in testing the performance of the Native
PostgreSQL port for Windows with the xlog directory being located on
another drive.
- Have tried doing normal shortcuts, and have also tried using the
cygwin "ln" command to create the appropriate soft link. Both
approaches create a shortcut object of the correct name pointing to the
correct place on the new drive, but the only thing that appears to
follow this shortcut is when I click on them using Windows Explorer.
The Native PostgreSQL port for Windows doesn't, and neither do a few
other applications I tested.

Would it be correct to say that the 'ln' command in the MS Resource Kit
creates this kind of shortcut too, as the Reparse Points feature doesn't
seem to be possible under NT4?

Can only think of two real solutions at present, one being for us to add
a PGXLOG environment variable or similar ability (GUC parameter
perhaps?), and the other would be for the Native PostgreSQL for Windows
port to follow these shortcuts.

Not if any of these is all that easy, or maybe there is another solution
that would work (apart from ignoring the problem).

:-)

Regards and best wishes,

Justin Clift

--
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi

On Fri, 13 Sep 2002, Justin Clift wrote:

> "scott.marlowe" wrote:
> <snip>
> > > Seems like the NT4 users are left out in the cold though until we add
> > > some kind of ability for PostgreSQL to not look at the filesystem for
> > > info about where to put the xlog files.
> >
> > This isn't true. With the resource kit, you get the gnu utils, and ln
> > works a charm under NT4 with ntfs. And not just for directories, but
> > files as well. Unless Microsoft somehow removed that functionality in the
> > intervening years since I've used NT. (wouldn't put it past them, but I
> > doubt they have.)
>
> The reference point that I'm working from is this:
>
> - Am testing out the third beta of the Native PostgreSQL port for
> Windows, on NT4 SP6 at present.
> - Have an internal RAID array of Seagate Cheetah 10kRPM drives. When
> installing the PGDATA directory on one drive it gives a certain kind of
> performance, and I'm interested in testing the performance of the Native
> PostgreSQL port for Windows with the xlog directory being located on
> another drive.
> - Have tried doing normal shortcuts, and have also tried using the
> cygwin "ln" command to create the appropriate soft link. Both
> approaches create a shortcut object of the correct name pointing to the
> correct place on the new drive, but the only thing that appears to
> follow this shortcut is when I click on them using Windows Explorer.
> The Native PostgreSQL port for Windows doesn't, and neither do a few
> other applications I tested.
>
> Would it be correct to say that the 'ln' command in the MS Resource Kit
> creates this kind of shortcut too, as the Reparse Points feature doesn't
> seem to be possible under NT4?

I wouldn't assume that. It's been years since I tested it, but back then,
the command line and all program I used could see the link created by ln
that came with the resource kit. They were distinctly different from the
shortcut type of links, in that they seems transparent like short cuts in
unix generally are.

Do you have the resource kit or the gnu utils from it?

Looking at this url:

http://unxutils.sourceforge.net/

the part for ln.exe says it makes real hard links on ntfs (which means
they would be on the same drive.) So I'm not sure if ntfs supports soft
links across volumes transparently or not now.

scott.marlowe wrote:
> > Seems like the NT4 users are left out in the cold though until we add
> > some kind of ability for PostgreSQL to not look at the filesystem for
> > info about where to put the xlog files.
>
> This isn't true. With the resource kit, you get the gnu utils, and ln
> works a charm under NT4 with ntfs. And not just for directories, but
> files as well. Unless Microsoft somehow removed that functionality in the
> intervening years since I've used NT. (wouldn't put it past them, but I
> doubt they have.)

Yes, this is what I remember, that Cygwin had symlinks, and at that time
that was the only Win32 OS we supported. Now, with native Win32 port
coming, we have to figure out what is available.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

scott.marlowe wrote:
> On Fri, 13 Sep 2002, Justin Clift wrote:
>
>>Would it be correct to say that the 'ln' command in the MS Resource Kit
>>creates this kind of shortcut too, as the Reparse Points feature doesn't
>>seem to be possible under NT4?
>
>
> I wouldn't assume that. It's been years since I tested it, but back then,
> the command line and all program I used could see the link created by ln
> that came with the resource kit. They were distinctly different from the
> shortcut type of links, in that they seems transparent like short cuts in
> unix generally are.
>
> Do you have the resource kit or the gnu utils from it?

The situation appears to be this:

1. Soft links are available on NTFS 5 (2K/XP) as Reparse Points
via the DeviceIoControl() function for any application using the
standard C library routines.

2. Soft links are available on any filesystem under
95/98/ME/NT4/2K/XP as OLE streams (.lnk files) for Shell-aware
applications.

3. Hard links are available on NTFS 5 (2K/XP) via the
CreateHardLink() API.

See:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/createhardlink.asp

4. Hard links are available on NTFS (NT3.1/NT4) via the
BackupWrite() API by writing a special stream to the NTFS.

Example:

http://www.mvps.org/win32/ntfs/lnw.cpp

The cygwin implementation of link():

http://sources.redhat.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/syscalls.cc?rev=1.149.2.23&content-type=text/x-cvsweb-markup&cvsroot=src

1. Will use CreateHardLink() if on 2K/XP
2. Will try to use the BackupWrite() method
3. Failing #2 will just copy the file

See how fun Microsoft makes things?

Mike Mascari
mascarm(at)mascari(dot)com

I wrote:
> scott.marlowe wrote:
>>
>> I wouldn't assume that. It's been years since I tested it, but back
>> then, the command line and all program I used could see the link
>> created by ln that came with the resource kit. They were distinctly
>> different from the shortcut type of links, in that they seems
>> transparent like short cuts in unix generally are.
>>
>> Do you have the resource kit or the gnu utils from it?
>
>
> The situation appears to be this:
>
> 1. Soft links are available on NTFS 5 (2K/XP) as Reparse Points via the
> DeviceIoControl() function for any application using the standard C
> library routines.
>
> 2. Soft links are available on any filesystem under 95/98/ME/NT4/2K/XP
> as OLE streams (.lnk files) for Shell-aware applications.
>
> 3. Hard links are available on NTFS 5 (2K/XP) via the CreateHardLink() API.

<snip>

> 4. Hard links are available on NTFS (NT3.1/NT4) via the BackupWrite()
> API by writing a special stream to the NTFS.

I also believe (I could be wrong) that for directories, the only
two methods of links are the Soft link methods above. So PGXLOG
cannot use soft links on a non-XP/2K machine unless it is
"Shell-Aware". For example, in a cygwin bash command window:

mkdir dir1
ln dir1 dir2 <- Error using Cygwin implementation
ln -s dir1 dir2 <- Creates a Shell short-cut (NT4)
echo "Hello" > dir1/test.txt
cat dir2/test.txt
"Hello" <- Cygwin's cat(bash?) is shell short-cut aware

Now, in a Windows NT command prompt:

notepad dir2\test.txt <- Notepad can't find file
notepad dir2.lnk <- Displays link contents

That means for a native port with a different PGXLOG directory
running on NT4, the only choice *using links* is to make the
native port shell short-cut aware.

I could be wrong but I don't think so.

Mike Mascari
mascarm(at)mascari(dot)com