Re: Committing Resources to Win32

Lists: pgsql-hackers-win32
From: "Merlin Moncure" <merlin(dot)moncure(at)rcsonline(dot)com>
To: "Joshua D(dot) Drake" <jd(at)commandprompt(dot)com>
Cc: <pgsql-hackers-win32(at)postgresql(dot)org>
Subject: Re: Committing Resources to Win32
Date: 2003-11-10 21:39:58
Message-ID: 303E00EBDD07B943924382E153890E5434AA05@cuthbert.rcsinc.local
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

Joshua D. Drake wrote:
> aren't good enough?". We could always use Borland or Intel.

Borland released (not sure if they still do) a C++ compiler that is free
for download (but closed source). Borland's compiler is a pretty decent
C++ compiler, not sure how well it handles C, it does support TLS
though.

> I believe that Apache 2.x is not completly threaded.. don't they use
> some kind of hybrid model?

Apache by nature has a lot more performance issues wrt threading and
process management. Postgres is more generally I/O bound. My
professional opinion is that for win32 performance we should be looking
at how the file system calls, especially synching, will run rather than
ipc and threading.

> Hmmm... maybe this all becomes moot if we also do connection pooling?

This would serialize all database access...IMO not worth considering
this angle.

TLS variables are supported by the Win32 API, and all C compilers for
windows support the Win32 API. __declspec(thread) is just syntactic
sugar which automate the API TLS commands. Converting the variable
access syntax is likely just one part of the overall move to the
threaded approach. A win32 centric move to threading will marginalize
the port (again, IMO).

Regards,
Merlin


From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To:
Cc: pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: Committing Resources to Win32
Date: 2003-11-10 22:10:35
Message-ID: 3FB00CDB.1080504@dunslane.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

Merlin Moncure wrote:

>Joshua D. Drake wrote:
>
>
>
>
>
>>I believe that Apache 2.x is not completly threaded.. don't they use
>>some kind of hybrid model?
>>
>>
>
>Apache by nature has a lot more performance issues wrt threading and
>process management. Postgres is more generally I/O bound. My
>professional opinion is that for win32 performance we should be looking
>at how the file system calls, especially synching, will run rather than
>ipc and threading.
>

Yes, Web requests by the nature of the protocol are short-lived
connections - if you are setting up and breaking down connections at a
very fast clip then forking a server per connection is going to be very
expensive. That is not a common usage pattern with a database server and
a well written app, IMNSHO.

Apache's APR took them years to get right, too.

>>Hmmm... maybe this all becomes moot if we also do connection pooling?
>>
>>
>
>This would serialize all database access...IMO not worth considering
>this angle.
>

Why does it serialize access? I have a web server app with connection
pooling - it has a pool of 20 connections that it keeps around for when
they are needed - they can all be active at any given time. And a good
connection pool manager sets up the connections ahead of time so that
each request sees *no* startup time.

>
>TLS variables are supported by the Win32 API, and all C compilers for
>windows support the Win32 API. __declspec(thread) is just syntactic
>sugar which automate the API TLS commands. Converting the variable
>access syntax is likely just one part of the overall move to the
>threaded approach. A win32 centric move to threading will marginalize
>the port (again, IMO).
>
>
>

*nod*

Joshua, can you tell us any more about the nature of your client's
app(s)? Speculating like this in the dark is a bit fruitless.

cheers

andrew


From: Marsh Ray <marsh-pg(at)mysteray(dot)com>
To: Merlin Moncure <merlin(dot)moncure(at)rcsonline(dot)com>, pgsql-hackers-win32(at)postgresql(dot)org, jd(at)commandprompt(dot)com
Subject: Re: Committing Resources to Win32
Date: 2003-11-10 22:45:13
Message-ID: 3FB014F9.1050002@mysteray.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

Merlin Moncure wrote:

>Joshua D. Drake wrote:
>
>
>>aren't good enough?". We could always use Borland or Intel.
>>
>>
>
>Borland released (not sure if they still do) a C++ compiler that is free
>for download (but closed source).
>
And Intel won't sell you their Windows compiler unless you already own
Microsoft's.

- Marsh


From: "Joshua D(dot) Drake" <jd(at)commandprompt(dot)com>
To: Andrew Dunstan <andrew(at)dunslane(dot)net>
Cc: pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: Committing Resources to Win32
Date: 2003-11-10 22:57:09
Message-ID: 3FB017C5.8050602@commandprompt.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32


> *nod*
>
> Joshua, can you tell us any more about the nature of your client's
> app(s)? Speculating like this in the dark is a bit fruitless.
>
I can't legally tell you much but what I can tell you is:

There application creates a great deal of processes that open and close.

On Linux which has a very light process model the performance hit is
nominal. On platforms like Win32 or Solaris where processes are
expensive, under heavy load you can see a pretty significant increase in
performance by going to a threaded model.

They are currently running our Cygwin installation which combined with
connection pooling has provided "ok" performance but nothing
worth writing home about (especially considering they compared against
Linux).

I am not going to lie, from a Windows perspective I am a little bit of a
PHB. I don't develop (personally) on Windows. However the
customer requirements are simple:

Command Prompt needs to provide a native Win32 PostgreSQL version that
supplies a reasonable proximity of performance
per the Linux native version. The Win32 native version must also
maintain the same level of transactibility as the Linux version.

My customer is a house of Windows C/C++ and they are telling me that
using CreateProcess will not generate that proximity.

They and I could be totally on crack, but my own research suggests
pretty much the same thing and the Windows programmers
that I have talked to that are not associated with this customer also
say the same thing.

What this all comes down to for us is this:

Can we (the community) develop a Win32 native version using
CreateProcess that will scale and perform at a level that
is acceptable to wide general use. Understanding that for many
operations PostgreSQL on Linux will perform as well if
not faster than the other well known database with the letter O in their
name.

Sincerely,

Joshua Drake

> cheers
>
> andrew
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faqs/FAQ.html

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-222-2783 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
Editor-N-Chief - PostgreSQl.Org - http://www.postgresql.org


From: "Joshua D(dot) Drake" <jd(at)commandprompt(dot)com>
To: Marsh Ray <marsh-pg(at)mysteray(dot)com>
Cc: Merlin Moncure <merlin(dot)moncure(at)rcsonline(dot)com>, pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: Committing Resources to Win32
Date: 2003-11-10 22:58:11
Message-ID: 3FB01803.4080802@commandprompt.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32


> And Intel won't sell you their Windows compiler unless you already own
> Microsoft's.
>
That still leaves Borland as an option. So what if it is not Open Source.

Option 2:

What would it take to add the functionality we need to minGW?

> - Marsh
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 7: don't forget to increase your free space map settings

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-222-2783 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
Editor-N-Chief - PostgreSQl.Org - http://www.postgresql.org


From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To: pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: Committing Resources to Win32
Date: 2003-11-11 00:04:19
Message-ID: 3FB02783.6000606@dunslane.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

Joshua D. Drake wrote:

>
>>
>> Joshua, can you tell us any more about the nature of your client's
>> app(s)? Speculating like this in the dark is a bit fruitless.
>>
> I can't legally tell you much but what I can tell you is:
>
> There application creates a great deal of processes that open and close.
> On Linux which has a very light process model the performance hit is
> nominal. On platforms like Win32 or Solaris where processes are
> expensive, under heavy load you can see a pretty significant increase
> in performance by going to a threaded model.
>
> They are currently running our Cygwin installation which combined with
> connection pooling has provided "ok" performance but nothing
> worth writing home about (especially considering they compared against
> Linux).

I think the short answer is that we should do better than the Cygwin
version (a lot better, probably) - anecdotal information about the
previous PeerDirect port suggested significant performance gains, but
not as well as a threaded version. I personally ran that version very
happily for months, but not under heavy connection load. If they see big
improvements with threads on Solaris they would probably see bigger
improvements on Windows, I suspect, although Solaris process creation
cost is notoriously high (as it is on Windows).

>
> I am not going to lie, from a Windows perspective I am a little bit of
> a PHB. I don't develop (personally) on Windows. However the
> customer requirements are simple:
>
> Command Prompt needs to provide a native Win32 PostgreSQL version that
> supplies a reasonable proximity of performance
> per the Linux native version. The Win32 native version must also
> maintain the same level of transactibility as the Linux version.
>
> My customer is a house of Windows C/C++ and they are telling me that
> using CreateProcess will not generate that proximity.
>
> They and I could be totally on crack, but my own research suggests
> pretty much the same thing and the Windows programmers
> that I have talked to that are not associated with this customer also
> say the same thing.

You are not on crack. The question is how we can get there, within the
generally accepted parameters of the PostgreSQL project. Those include,
as I understand it,
. use of Open Source software tools only to build, and
. maintaining a single body of code

>
>
> What this all comes down to for us is this:
>
> Can we (the community) develop a Win32 native version using
> CreateProcess that will scale and perform at a level that
> is acceptable to wide general use. Understanding that for many
> operations PostgreSQL on Linux will perform as well if
> not faster than the other well known database with the letter O in
> their name.

Depends right now on your definition of "wide use" ;-) The "O" entity's
connection times are also notoriously slow. I think we can get
acceptable performance for what I would regard as typical usage
patterns. But possibly not right now for your client's usage pattern.

cheers

andrew


From: "Jim Jones" <jjones(at)aantix(dot)com>
To: <pgsql-hackers-win32(at)postgresql(dot)org>
Subject: Re: Committing Resources to Win32
Date: 2003-11-11 00:15:29
Message-ID: 001b01c3a7e8$e9f22e4000000a@chatserver
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

If you guys have already discussed the following possibilities, I
apologize. I joined the list late.

The correct way to create a high performance server on the Windows
platform (NT, Advanced Server, etc) is to utilize IO Completion Ports.
The kernel will manage your connection pool and threading for you. Any
object that is serializable can utilize this model (sockets, files,
etc).

If one thread becomes busy with processing, a second worker thread is
awoken, and it performs the processing. Sockets and threads are all
preallocated to increase performance.

Here are a couple informative articles on the subject :

Writing scalable server applications using IOCP
http://www.codeproject.com/internet/iocp.asp

Developing a Truly Scalable Winsock Server using IO Completion Ports
http://www.codeproject.com/internet/winsockiocp.asp

The Microsoft SDK comes with a free compiler, but you will have access
to only the API functions, none of the frameworks (MFC, ATL, etc).

Jim

-----Original Message-----
From: pgsql-hackers-win32-owner(at)postgresql(dot)org
[mailto:pgsql-hackers-win32-owner(at)postgresql(dot)org] On Behalf Of Joshua D.
Drake
Sent: Monday, November 10, 2003 4:57 PM
To: Andrew Dunstan
Cc: pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: [pgsql-hackers-win32] Committing Resources to Win32

> *nod*
>
> Joshua, can you tell us any more about the nature of your client's
> app(s)? Speculating like this in the dark is a bit fruitless.
>
I can't legally tell you much but what I can tell you is:

There application creates a great deal of processes that open and close.

On Linux which has a very light process model the performance hit is
nominal. On platforms like Win32 or Solaris where processes are
expensive, under heavy load you can see a pretty significant increase in

performance by going to a threaded model.

They are currently running our Cygwin installation which combined with
connection pooling has provided "ok" performance but nothing
worth writing home about (especially considering they compared against
Linux).

I am not going to lie, from a Windows perspective I am a little bit of a

PHB. I don't develop (personally) on Windows. However the
customer requirements are simple:

Command Prompt needs to provide a native Win32 PostgreSQL version that
supplies a reasonable proximity of performance
per the Linux native version. The Win32 native version must also
maintain the same level of transactibility as the Linux version.

My customer is a house of Windows C/C++ and they are telling me that
using CreateProcess will not generate that proximity.

They and I could be totally on crack, but my own research suggests
pretty much the same thing and the Windows programmers
that I have talked to that are not associated with this customer also
say the same thing.

What this all comes down to for us is this:

Can we (the community) develop a Win32 native version using
CreateProcess that will scale and perform at a level that
is acceptable to wide general use. Understanding that for many
operations PostgreSQL on Linux will perform as well if
not faster than the other well known database with the letter O in their

name.

Sincerely,

Joshua Drake

> cheers
>
> andrew
>
>
> ---------------------------(end of
broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faqs/FAQ.html

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-222-2783 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
Editor-N-Chief - PostgreSQl.Org - http://www.postgresql.org

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org


From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To: jjones(at)aantix(dot)com
Cc: pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: Committing Resources to Win32
Date: 2003-11-11 02:18:23
Message-ID: 3FB046EF.40707@dunslane.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32


Using completion ports assumes your app is threaded in the first
instance, doesn't it?

Unless I am mistaken, at the moment
. we don't have a server app at all
. our code base does not currently support threading, and we want to
keep to a single code base
- corollary: we don't know if supporting POSIX threads and Windows
threads in a single code base is possible or sensible
. we don't know of any OpenSource tools that support compiling threaded
apps with TLS on Windows (specifically, MinGW does not, apparently)

So I suspect we are a very long way from being able to use Completion Ports.

cheers

andrew

Jim Jones wrote:

>If you guys have already discussed the following possibilities, I
>apologize. I joined the list late.
>
>The correct way to create a high performance server on the Windows
>platform (NT, Advanced Server, etc) is to utilize IO Completion Ports.
>The kernel will manage your connection pool and threading for you. Any
>object that is serializable can utilize this model (sockets, files,
>etc).
>
>If one thread becomes busy with processing, a second worker thread is
>awoken, and it performs the processing. Sockets and threads are all
>preallocated to increase performance.
>
>Here are a couple informative articles on the subject :
>
>Writing scalable server applications using IOCP
>http://www.codeproject.com/internet/iocp.asp
>
>Developing a Truly Scalable Winsock Server using IO Completion Ports
>http://www.codeproject.com/internet/winsockiocp.asp
>
>The Microsoft SDK comes with a free compiler, but you will have access
>to only the API functions, none of the frameworks (MFC, ATL, etc).
>
>Jim
>
>
>-----Original Message-----
>From: pgsql-hackers-win32-owner(at)postgresql(dot)org
>[mailto:pgsql-hackers-win32-owner(at)postgresql(dot)org] On Behalf Of Joshua D.
>Drake
>Sent: Monday, November 10, 2003 4:57 PM
>To: Andrew Dunstan
>Cc: pgsql-hackers-win32(at)postgresql(dot)org
>Subject: Re: [pgsql-hackers-win32] Committing Resources to Win32
>
>
>
>
>>*nod*
>>
>>Joshua, can you tell us any more about the nature of your client's
>>app(s)? Speculating like this in the dark is a bit fruitless.
>>
>>
>>
>I can't legally tell you much but what I can tell you is:
>
>There application creates a great deal of processes that open and close.
>
>
>On Linux which has a very light process model the performance hit is
>nominal. On platforms like Win32 or Solaris where processes are
>expensive, under heavy load you can see a pretty significant increase in
>
>performance by going to a threaded model.
>
>They are currently running our Cygwin installation which combined with
>connection pooling has provided "ok" performance but nothing
>worth writing home about (especially considering they compared against
>Linux).
>
>I am not going to lie, from a Windows perspective I am a little bit of a
>
>PHB. I don't develop (personally) on Windows. However the
>customer requirements are simple:
>
>Command Prompt needs to provide a native Win32 PostgreSQL version that
>supplies a reasonable proximity of performance
>per the Linux native version. The Win32 native version must also
>maintain the same level of transactibility as the Linux version.
>
>My customer is a house of Windows C/C++ and they are telling me that
>using CreateProcess will not generate that proximity.
>
>They and I could be totally on crack, but my own research suggests
>pretty much the same thing and the Windows programmers
>that I have talked to that are not associated with this customer also
>say the same thing.
>
>What this all comes down to for us is this:
>
>Can we (the community) develop a Win32 native version using
>CreateProcess that will scale and perform at a level that
>is acceptable to wide general use. Understanding that for many
>operations PostgreSQL on Linux will perform as well if
>not faster than the other well known database with the letter O in their
>
>name.
>
>Sincerely,
>
>Joshua Drake
>
>
>
>
>
>
>>cheers
>>
>>andrew
>>
>>
>>---------------------------(end of
>>
>>
>broadcast)---------------------------
>
>
>>TIP 5: Have you checked our extensive FAQ?
>>
>> http://www.postgresql.org/docs/faqs/FAQ.html
>>
>>
>
>
>
>


From: Marsh Ray <marsh-pg(at)mysteray(dot)com>
To: jjones(at)aantix(dot)com
Cc: pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: Committing Resources to Win32
Date: 2003-11-11 12:22:25
Message-ID: 3FB0D481.5090907@mysteray.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

Jim Jones wrote:

>If you guys have already discussed the following possibilities, I
>apologize. I joined the list late.
>
>The correct way to create a high performance server on the Windows
>platform (NT, Advanced Server, etc) is to utilize IO Completion Ports.
>
>
I would think that you could probably get 80% of the performance with
ordinary threads and ordinary blocking IO, and without the great
increase in complexity required to implement asynchronous IO. If someone
then wants to go after a possible 25% performance gain, they could
submit a working set of modifications, or wait a few months for the
hardware to catch up.

- Marsh


From: "Joshua D(dot) Drake" <jd(at)commandprompt(dot)com>
To: Marsh Ray <marsh-pg(at)mysteray(dot)com>
Cc: jjones(at)aantix(dot)com, pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: Committing Resources to Win32
Date: 2003-11-11 16:27:17
Message-ID: 3FB10DE5.6020001@commandprompt.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32


>>
>>
> I would think that you could probably get 80% of the performance with
> ordinary threads and ordinary blocking IO, and without the great
> increase in complexity required to implement asynchronous IO. If
> someone then wants to go after a possible 25% performance gain, they
> could submit a working set of modifications, or wait a few months for
> the hardware to catch up.
>
This argument doesn't work for several reasons:

1. Hardware generally does not catch up because:
A. Users demand increases and thus the load on the hardware
increases.
B. Users don't purchase new hardware every couple of months.
2. If we were talking about a 5% performance gain that "might" be
acceptable but
the 20% gain you are talking about is huge. Think about it:

I have several customers right now that push over 250,000 (some as
high as 750,000)
transactions an hour. They are pushing the hardware hard as it is,
especially with the
limitations of Vacuum. If they were to move to the Win32 version (it
doesn't matter why),
they would loose 50,000 transactions an hour based on your argument.

That is completely illegitamate and would make PostgreSQL look like
a toy on Windows.
We already have PostgreSQL as a toy on Windows, it uses Cygwin.

> - Marsh
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if
> your
> joining column's datatypes do not match

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-222-2783 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
Editor-N-Chief - PostgreSQl.Org - http://www.postgresql.org


From: Marsh Ray <marsh-pg(at)mysteray(dot)com>
To: "Joshua D(dot) Drake" <jd(at)commandprompt(dot)com>
Cc: Marsh Ray <marsh-pg(at)mysteray(dot)com>, jjones(at)aantix(dot)com, pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: Committing Resources to Win32
Date: 2003-11-12 03:20:17
Message-ID: 3FB1A6F1.6000104@mysteray.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

Joshua D. Drake wrote:

> Marsh wrote:
>
>> I would think that you could probably get 80% of the performance with
>> ordinary threads and ordinary blocking IO, and without the great
>> increase in complexity required to implement asynchronous IO. If
>> someone then wants to go after a possible 25% performance gain, they
>> could submit a working set of modifications, or wait a few months for
>> the hardware to catch up.
>>
> This argument doesn't work for several reasons:
>
> 1. Hardware generally does not catch up because:
> A. Users demand increases and thus the load on the hardware
> increases.
> B. Users don't purchase new hardware every couple of months.
> 2. If we were talking about a 5% performance gain that "might" be
> acceptable but
> the 20% gain you are talking about is huge. Think about it:
>
> I have several customers right now that push over 250,000 (some as
> high as 750,000)
> transactions an hour. They are pushing the hardware hard as it is,
> especially with the
> limitations of Vacuum. If they were to move to the Win32 version (it
> doesn't matter why),
> they would loose 50,000 transactions an hour based on your argument.
>
> That is completely illegitamate and would make PostgreSQL look like
> a toy on Windows.
> We already have PostgreSQL as a toy on Windows, it uses Cygwin.

OK, but are you using asych IO on any platform now?

Note that the original poster proposed it as a possible performance
enhancement, and I'm not claiming that a threaded blocking-IO
implementation on Windows is going to be slower than the current model
on Unix.

The hypothetical 25% gain really is a WAG that would probably only be
achievable for some current worst-case scenarios, but the point is it
represents a few months of hardware advance. If somebody's servers are
really that saturated, they will be looking at hardware upgrades sooner
rather than later, and a similar effect might achieved by throwing RAM
at the problem.

Ideally you would have a single process with exactly as many threads as
there are CPUs (maybe x2 for Intel HyperThreading chips). Each thread
would have affinity to a specific processor. All IO would be done
asynchronously and use coroutines/fibers for task switching.

Due to the complexity involved in implementing an app this way, it's
usually not done. Process- or thread-per-connection with blocking IO is
conceptionally much simpler and usually not that much slower. Efficient
CPU scheduling is one of the main functions of the mainstream OS, right?

- Marsh


From: "Joshua D(dot) Drake" <jd(at)commandprompt(dot)com>
To: pgsql-hackers-win32(at)postgresql(dot)org
Subject: CreateProcess vs Create Thread
Date: 2003-11-13 17:29:51
Message-ID: 3FB3BF8F.2050809@commandprompt.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

Hello,

O.k. there has been a lot of discussion in regards to what the best
method to take with PostgreSQL Win32 is.
It seems that the community is set on using CreateProcess. If this is
the case I have a couple of further questions:

1. Are we just a bunch of Unix coders trying to develop a product for
Win32 or are there any people a part
of the Win32 port project that actually have REAL, hardcore experience
coding at the system level for Win32?
I am not trying to be an antagonist but I it is a legitamate question.

2. Have we built a test case? For example:

We have a client that connects to a server. For every connection we
start a new process and write 1k to disk from the client.

Connect in varying degrees from 5 connection to say 500 connections.

Do it with CreateProcess and the with CreateThread... Time the
difference, and review the load on the server during the tests...

What is the difference? Maybe it is nominal enough not to worry
about it... Maybe it's not.

Sincerely,

Joshua D. Drake

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC - S/JDBC
Postgresql support, programming, shared hosting and dedicated hosting.
+1-503-222-2783 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
PostgreSQL.Org - Editor-N-Chief - http://www.postgresql.org


From: "Steve Tibbett" <stevex(at)stevex(dot)org>
To: "'Joshua D(dot) Drake'" <jd(at)commandprompt(dot)com>, <pgsql-hackers-win32(at)postgresql(dot)org>
Subject: Re: CreateProcess vs Create Thread
Date: 2003-11-13 17:37:24
Message-ID: 3F61636001A863FD@mta3.wss.scd.yahoo.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

CreateProcess has to come first since all the work that happens for the
multithreaded version also has to happen for the CreateProcess version, and
it's just natural that it will come first.

It makes a lot of sense for Postgres to get the CreateProcess version
working and stable before embarking on threads - Win32 and threading are two
separate and mostly unrelated tasks, but threading on Win32 can't happen
until Win32 is stable.

Just my two cents...

- Steve

-----Original Message-----
From: pgsql-hackers-win32-owner(at)postgresql(dot)org
[mailto:pgsql-hackers-win32-owner(at)postgresql(dot)org] On Behalf Of Joshua D.
Drake
Sent: November 13, 2003 12:30 PM
To: pgsql-hackers-win32(at)postgresql(dot)org
Subject: [pgsql-hackers-win32] CreateProcess vs Create Thread

Hello,

O.k. there has been a lot of discussion in regards to what the best
method to take with PostgreSQL Win32 is.
It seems that the community is set on using CreateProcess. If this is the
case I have a couple of further questions:

1. Are we just a bunch of Unix coders trying to develop a product for
Win32 or are there any people a part
of the Win32 port project that actually have REAL, hardcore experience
coding at the system level for Win32?
I am not trying to be an antagonist but I it is a legitamate question.

2. Have we built a test case? For example:

We have a client that connects to a server. For every connection we
start a new process and write 1k to disk from the client.

Connect in varying degrees from 5 connection to say 500 connections.

Do it with CreateProcess and the with CreateThread... Time the
difference, and review the load on the server during the tests...

What is the difference? Maybe it is nominal enough not to worry about
it... Maybe it's not.

Sincerely,

Joshua D. Drake

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC - S/JDBC
Postgresql support, programming, shared hosting and dedicated hosting.
+1-503-222-2783 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
PostgreSQL.Org - Editor-N-Chief - http://www.postgresql.org

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo(at)postgresql(dot)org)


From: Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us>
To: "Joshua D(dot) Drake" <jd(at)commandprompt(dot)com>
Cc: pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: CreateProcess vs Create Thread
Date: 2003-11-13 17:42:18
Message-ID: 200311131742.hADHgIT16699@candle.pha.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

Joshua D. Drake wrote:
> Hello,
>
> O.k. there has been a lot of discussion in regards to what the best
> method to take with PostgreSQL Win32 is.
> It seems that the community is set on using CreateProcess. If this is
> the case I have a couple of further questions:
>
> 1. Are we just a bunch of Unix coders trying to develop a product for
> Win32 or are there any people a part
> of the Win32 port project that actually have REAL, hardcore experience
> coding at the system level for Win32?
> I am not trying to be an antagonist but I it is a legitimate question.

Not sure. We have a working Win32 port from PeerDirect, and I have one
from SRA, so we know it can be done whether we have hard-core guys or
not.

> 2. Have we built a test case? For example:
>
> We have a client that connects to a server. For every connection we
> start a new process and write 1k to disk from the client.
>
> Connect in varying degrees from 5 connection to say 500 connections.
>
> Do it with CreateProcess and the with CreateThread... Time the
> difference, and review the load on the server during the tests...
>
> What is the difference? Maybe it is nominal enough not to worry
> about it... Maybe it's not.

I certainly would like to see that test with a skelaton C program.

You can use PeerDirect's beta version for the CreateProcess test.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073


From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To: pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: CreateProcess vs Create Thread
Date: 2003-11-13 18:04:02
Message-ID: 3FB3C792.4020106@dunslane.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

Joshua D. Drake wrote:

> Hello,
>
> O.k. there has been a lot of discussion in regards to what the best
> method to take with PostgreSQL Win32 is.
> It seems that the community is set on using CreateProcess. If this is
> the case I have a couple of further questions:
>
> 1. Are we just a bunch of Unix coders trying to develop a product for
> Win32 or are there any people a part
> of the Win32 port project that actually have REAL, hardcore experience
> coding at the system level for Win32?
> I am not trying to be an antagonist but I it is a legitamate question.

Speaking for myself I am a Unix coder and proud of it :-) I don't know
how many times I have said in my professional life "I don't do Windows".
That's why I chose to make my major contribution to the project the
porting of initdb to C - it had to be done (and it took me quite a few
hours) but it didn't require a deep level of understanding of Windows.

OTOH there is clearly a deeper level of understanding and experience
around than I have.

I think the answer is a mixture.

cheers

andrew


From: Marsh Ray <marsh(at)mysteray(dot)com>
To: "Joshua D(dot) Drake" <jd(at)commandprompt(dot)com>
Cc: pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: CreateProcess vs Create Thread
Date: 2003-11-14 01:58:24
Message-ID: 3FB436C0.7050206@mysteray.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

Joshua D. Drake wrote:

> 1. Are we just a bunch of Unix coders trying to develop a product for
> Win32 or are there any people a part of the Win32 port project that
> actually have REAL, hardcore experience coding at the system level for
> Win32?

I've got exactly 10 years' experience as a Win32 coder, most of it on
the server side. I haven't yet committed any time to the effort, but I
suppose if I keep yacking on this list, then I'll have to.

I really think what Steve Tibbett said is right on the money:

>CreateProcess has to come first since all the work that happens for the
>multithreaded version also has to happen for the CreateProcess version, and
>it's just natural that it will come first.
>It makes a lot of sense for Postgres to get the CreateProcess version
>working and stable before embarking on threads - Win32 and threading are two
>separate and mostly unrelated tasks, but threading on Win32 can't happen
>until Win32 is stable.
>
I'll guess that once any Win32 version is out there with a nice
installer and documentation, it will quickly become popular. It probably
won't too slow for most uses, although benchmarkers might form lasting
first impressions from it. This will probably reveal a demand for a
threaded version.

- Marsh


From: Marsh Ray <marsh-pg(at)mysteray(dot)com>
To:
Cc: pgsql-hackers-win32(at)postgresql(dot)org
Subject: Re: CreateProcess vs Create Thread
Date: 2003-11-14 02:00:59
Message-ID: 3FB4375B.5050005@mysteray.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers-win32

Joshua D. Drake wrote:

> 1. Are we just a bunch of Unix coders trying to develop a product for
> Win32 or are there any people a part of the Win32 port project that
> actually have REAL, hardcore experience coding at the system level for
> Win32?

I've got exactly 10 years' experience as a Win32 coder, most of it on
the server side. I haven't yet committed any time to the effort, but I
suppose if I keep yacking on this list, then I'll have to.

I really think what Steve Tibbett said is right on the money:

>CreateProcess has to come first since all the work that happens for the
>multithreaded version also has to happen for the CreateProcess version, and
>it's just natural that it will come first.
>It makes a lot of sense for Postgres to get the CreateProcess version
>working and stable before embarking on threads - Win32 and threading are two
>separate and mostly unrelated tasks, but threading on Win32 can't happen
>until Win32 is stable.
>
I'll guess that once any Win32 version is out there with a nice
installer and documentation, it will quickly become popular. It probably
won't too slow for most uses, although benchmarkers might form lasting
first impressions from it. This will probably reveal a demand for a
threaded version.

- Marsh