Re: improve Chinese locale performance

(Replying on phone, please forgive bad quoting)

Isn't this pretty much what adopting ICU is supposed to give us? OS-independent collations?

I'd be interested in seeing the rest data for this performance report, partly as I'd like to see how ICU collations would compare when ICU is crudely hacked into place for testing.

On Tue, Jul 23, 2013 at 9:42 AM, Craig Ringer <craig(at)2ndquadrant(dot)com> wrote:
> (Replying on phone, please forgive bad quoting)
>
> Isn't this pretty much what adopting ICU is supposed to give us? OS-independent collations?

Yes.

> I'd be interested in seeing the rest data for this performance report, partly as I'd like to see how ICU collations would compare when ICU is crudely hacked into place for testing.

I pretty much lost interest in ICU upon reading that they use UTF-16
as their internal format.

http://userguide.icu-project.org/strings#TOC-Strings-in-ICU

What that would mean for us is that instead of copying the input
strings into a temporary buffer and passing the buffer to strcoll(),
we'd need to convert them to ICU's representation (which means writing
twice as many bytes as the length of the input string in cases where
the input string is mostly single-byte characters) and then call ICU's
strcoll() equivalent. I agree that it might be worth testing, but I
can't work up much optimism. It seems to me that something that
operates directly on the server encoding could run a whole lot faster.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 07/23/2013 09:42 PM, Craig Ringer wrote:
> (Replying on phone, please forgive bad quoting)
>
> Isn't this pretty much what adopting ICU is supposed to give us? OS-independent collations?
>
Yes, we need OS-independent collations.

> I'd be interested in seeing the rest data for this performance report, partly as I'd like to see how ICU collations would compare when ICU is crudely hacked into place for testing.
>

On Tue, Jul 23, 2013 at 10:34:21AM -0400, Robert Haas wrote:
> I pretty much lost interest in ICU upon reading that they use UTF-16
> as their internal format.
>
> http://userguide.icu-project.org/strings#TOC-Strings-in-ICU

The utf-8 support has been steadily improving:

For example, icu::Collator::compareUTF8() compares two utf-8 strings
incrementally, without converting all of the two strings to UTF-16 if
there is an early base letter difference.

http://userguide.icu-project.org/strings/utf-8

For all other encodings you should be able to use an iterator. As to
performance I have no idea.

The main issue with strxfrm() is its lame API. If it supported
returning prefixes you'd be set, but as it is you need >10MB of memory
just to transform a 10MB string, even if only the first few characers
would be enough to sort...

Mvg,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> He who writes carelessly confesses thereby at the very outset that he does
> not attach much importance to his own thoughts.
-- Arthur Schopenhauer

On Sun, Jul 28, 2013 at 5:39 AM, Martijn van Oosterhout
<kleptog(at)svana(dot)org> wrote:
> On Tue, Jul 23, 2013 at 10:34:21AM -0400, Robert Haas wrote:
>> I pretty much lost interest in ICU upon reading that they use UTF-16
>> as their internal format.
>>
>> http://userguide.icu-project.org/strings#TOC-Strings-in-ICU
>
> The utf-8 support has been steadily improving:
>
> For example, icu::Collator::compareUTF8() compares two utf-8 strings
> incrementally, without converting all of the two strings to UTF-16 if
> there is an early base letter difference.
>
> http://userguide.icu-project.org/strings/utf-8
>
> For all other encodings you should be able to use an iterator. As to
> performance I have no idea.
>
> The main issue with strxfrm() is its lame API. If it supported
> returning prefixes you'd be set, but as it is you need >10MB of memory
> just to transform a 10MB string, even if only the first few characers
> would be enough to sort...

Yep, definitely. And by ">10MB" you mean ">90MB", at least on my Mac,
which is really outrageous.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 07/23/2013 09:42 PM, Craig Ringer wrote:
> (Replying on phone, please forgive bad quoting)
>
> Isn't this pretty much what adopting ICU is supposed to give us? OS-independent collations?
>
> I'd be interested in seeing the rest data for this performance report, partly as I'd like to see how ICU collations would compare when ICU is crudely hacked into place for testing.
>
I think of a new idea.
Add a compare method column to pg_collation.
Every collation has its own compare function or null.
When function varstr_cmp is called, if specified collation
has compare function, call it instead of strcoll().

How about this?

Regards.

Quan Zongliang

On Wed, Sep 4, 2013 at 11:02 PM, Quan Zongliang <quanzongliang(at)gmail(dot)com> wrote:
> I think of a new idea.
> Add a compare method column to pg_collation.
> Every collation has its own compare function or null.
> When function varstr_cmp is called, if specified collation
> has compare function, call it instead of strcoll().

I think we're going to need to have two kinds of collations:
OS-derived collations (which get all of their smarts from the OS), and
PG-internal collations (which use PG-aware code for everything).
Which I suspect is a bit more involved than what you're imagining, but
mixing and matching doesn't seem likely to end well.

However, what you're proposing might serve as a useful demonstration
of how much performance there is to be gained here.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 09/06/2013 01:02 AM, Robert Haas wrote:
> On Wed, Sep 4, 2013 at 11:02 PM, Quan Zongliang <quanzongliang(at)gmail(dot)com> wrote:
>> I think of a new idea.
>> Add a compare method column to pg_collation.
>> Every collation has its own compare function or null.
>> When function varstr_cmp is called, if specified collation
>> has compare function, call it instead of strcoll().
>
> I think we're going to need to have two kinds of collations:
> OS-derived collations (which get all of their smarts from the OS), and
> PG-internal collations (which use PG-aware code for everything).
> Which I suspect is a bit more involved than what you're imagining, but
> mixing and matching doesn't seem likely to end well.
>
> However, what you're proposing might serve as a useful demonstration
> of how much performance there is to be gained here.
>
Understood.

I just try to speed up text compare, not redesign locale.

Do you have a plan to do this?

Thank you.

Quan Zongliang

On Mon, Sep 9, 2013 at 5:22 AM, Quan Zongliang <quanzongliang(at)gmail(dot)com> wrote:
> Understood.
>
> I just try to speed up text compare, not redesign locale.
>
> Do you have a plan to do this?

Not any time soon, anyway.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company