Re: BUG #18240: Undefined behaviour in cash_mul_flt8() and friends

The following bug has been logged on the website:

Bug reference: 18240
Logged by: Alexander Lakhin
Email address: exclusion(at)gmail(dot)com
PostgreSQL version: 16.1
Operating system: Ubuntu 22.04
Description:

The following multiplication:
SELECT 1_000_000_000::money * 1_000_000_000::float8;

gives the incorrect result:
-$92,233,720,368,547,758.08

UBSan detects undefined behaviour:
cash.c:669:11: runtime error: 1e+20 is outside the range of representable
values of type 'long'
#0 0x55d66011b73a in cash_mul_flt8
.../src/backend/utils/adt/cash.c:669:11

The same is observed with float4:
cash.c:719:11: runtime error: 1e+20 is outside the range of representable
values of type 'long'
#0 0x55f2adc46072 in cash_mul_flt4
.../src/backend/utils/adt/cash.c:719:11

And with float8 * money...

Reproduced on REL9_6_0 (but the defect is much older, AFAICS) .. HEAD.

On Mon, Dec 11, 2023 at 06:00:02AM +0000, PG Bug reporting form wrote:
> The following multiplication:
> SELECT 1_000_000_000::money * 1_000_000_000::float8;
>
> gives the incorrect result:
> -$92,233,720,368,547,758.08

Yep, good catch. Reproduced here.
--
Michael

Michael Paquier <michael(at)paquier(dot)xyz> writes:
> On Mon, Dec 11, 2023 at 06:00:02AM +0000, PG Bug reporting form wrote:
>> The following multiplication:
>> SELECT 1_000_000_000::money * 1_000_000_000::float8;
>> gives the incorrect result:
>> -$92,233,720,368,547,758.08

> Yep, good catch. Reproduced here.

Yeah, approximately none of cash.c pays any attention to the risks
of overflow/underflow. Improving that situation would be a good
finger exercise for some aspiring hacker, perhaps. Although I bet
somebody will ask again why it is that we continue to support the
money type.

regards, tom lane

Hello Michael and Tom,

11.12.2023 18:05, Tom Lane wrote:
> Michael Paquier <michael(at)paquier(dot)xyz> writes:
>> On Mon, Dec 11, 2023 at 06:00:02AM +0000, PG Bug reporting form wrote:
>>> The following multiplication:
>>> SELECT 1_000_000_000::money * 1_000_000_000::float8;
>>> gives the incorrect result:
>>> -$92,233,720,368,547,758.08
>> Yep, good catch. Reproduced here.
> Yeah, approximately none of cash.c pays any attention to the risks
> of overflow/underflow. Improving that situation would be a good
> finger exercise for some aspiring hacker, perhaps. Although I bet
> somebody will ask again why it is that we continue to support the
> money type.

Thank you for looking at that!
I've tried a simple operator-testing query (with already known problematic
operators spelled out):
SELECT format('SELECT ''2000000000''::%s %s ''2000000000''::%s;',
              tl.typname, oprname, tr.typname) FROM pg_operator
              INNER JOIN pg_type tl ON tl.oid = oprleft
              INNER JOIN pg_type tr ON tr.oid = oprright WHERE (
/* 2000000000 xxx 2000000000 */
NOT(tl.typname = 'money' AND oprname = '*' AND tr.typname = 'float4') AND
NOT(tl.typname = 'float4' AND oprname = '*' AND tr.typname = 'money') AND
NOT(tl.typname = 'money' AND oprname = '*' AND tr.typname = 'float8') AND
NOT(tl.typname = 'float8' AND oprname = '*' AND tr.typname = 'money') AND
NOT(tl.typname = 'int4' AND oprname = '<<' AND tr.typname = 'int4') AND
NOT(tl.typname = 'int4' AND oprname = '>>' AND tr.typname = 'int4') AND
NOT(tl.typname = 'int8' AND oprname = '<<' AND tr.typname = 'int4') AND
NOT(tl.typname = 'int8' AND oprname = '>>' AND tr.typname = 'int4') AND
/* 2000000000 xxx 0.0000000002 */
NOT(tl.typname = 'money' AND oprname = '/' AND tr.typname = 'float4') AND
NOT(tl.typname = 'money' AND oprname = '/' AND tr.typname = 'float8') AND
/* 32767 xxx 32767 */
NOT(tl.typname = 'int2' AND oprname = '<<' AND tr.typname = 'int4') AND
NOT(tl.typname = 'int2' AND oprname = '>>' AND tr.typname = 'int4')
)
\gexec

and found no new UBSan-detected anomalies with "extraordinary" numbers.

Best regards,
Alexander

On Mon, Dec 11, 2023 at 10:05:53AM -0500, Tom Lane wrote:
> Yeah, approximately none of cash.c pays any attention to the risks
> of overflow/underflow. Improving that situation would be a good
> finger exercise for some aspiring hacker, perhaps. Although I bet
> somebody will ask again why it is that we continue to support the
> money type.

AFAIK, we discourage the use of money in the wiki for quite a few
years:
https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_money

And numeric has much better code coverage and support. I am wondering
whether we've reached the point where it would be better to remove it
entirely from the tree, and just tell people to use numeric. This has
a cost for upgrades, where we should cross check for its use but there
is already check_for_data_type_usage() to do this job so the facility
is there.
--
Michael

On Tue, Dec 12, 2023 at 10:03:43AM +0100, Michael Paquier wrote:
> On Mon, Dec 11, 2023 at 10:05:53AM -0500, Tom Lane wrote:
> > Yeah, approximately none of cash.c pays any attention to the risks
> > of overflow/underflow. Improving that situation would be a good
> > finger exercise for some aspiring hacker, perhaps. Although I bet
> > somebody will ask again why it is that we continue to support the
> > money type.
>
> AFAIK, we discourage the use of money in the wiki for quite a few
> years:
> https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_money
>
> And numeric has much better code coverage and support. I am wondering
> whether we've reached the point where it would be better to remove it
> entirely from the tree, and just tell people to use numeric. This has
> a cost for upgrades, where we should cross check for its use but there
> is already check_for_data_type_usage() to do this job so the facility
> is there.

Yes, probably.

--
Bruce Momjian <bruce(at)momjian(dot)us> https://momjian.us
EDB https://enterprisedb.com

Only you can decide what is important to you.

On Tue, 12 Dec 2023 at 22:03, Michael Paquier <michael(at)paquier(dot)xyz> wrote:
> AFAIK, we discourage the use of money in the wiki for quite a few
> years:
> https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use_money
>
> And numeric has much better code coverage and support. I am wondering
> whether we've reached the point where it would be better to remove it
> entirely from the tree, and just tell people to use numeric. This has
> a cost for upgrades, where we should cross check for its use but there
> is already check_for_data_type_usage() to do this job so the facility
> is there.

We could extract it into a contrib module.

That might help reduce new usages of it and would also allow people
who have large tables using the money type but can't realistically
wait out a table rewrite to upgrade to a newer version of Postgres.

David

David Rowley <dgrowleyml(at)gmail(dot)com> writes:
> On Tue, 12 Dec 2023 at 22:03, Michael Paquier <michael(at)paquier(dot)xyz> wrote:
>> And numeric has much better code coverage and support. I am wondering
>> whether we've reached the point where it would be better to remove it
>> entirely from the tree, and just tell people to use numeric.

> We could extract it into a contrib module.

Perhaps, but ...

> That might help reduce new usages of it and would also allow people
> who have large tables using the money type but can't realistically
> wait out a table rewrite to upgrade to a newer version of Postgres.

... I doubt that a contrib module would solve the problem for people
who can't afford a rewrite. pg_upgrade requires that datatype OIDs
stay the same, which is something I don't believe a contrib module
could manage. We've run into that before if memory serves. Is it
time to do something about that? Perhaps we could allow extension
modules to use binary_upgrade_set_next_pg_type_oid, and then somehow
reserve the money and _money OIDs forever?

regards, tom lane

On Tue, Dec 12, 2023 at 09:37:58AM -0500, Tom Lane wrote:
> ... I doubt that a contrib module would solve the problem for people
> who can't afford a rewrite. pg_upgrade requires that datatype OIDs
> stay the same, which is something I don't believe a contrib module
> could manage. We've run into that before if memory serves. Is it
> time to do something about that? Perhaps we could allow extension
> modules to use binary_upgrade_set_next_pg_type_oid, and then somehow
> reserve the money and _money OIDs forever?

You mean with an integration in binary dumps? I don't see why it
would not work and this facility may prove to be useful for
out-of-core extension, assuming that this can be made transparent
based on some new .control properties on the upgrade cluster, I
guess..

Now, I am not sure that we're making users a favor in keeping around a
data type that's kind of obsolete these days, particularly when all of
them basically require handling of fractional cents when doing serious
calculations. That's a trap in disguise, even if tying data to a
locale for the unit sounds appealing for some applications.
--
Michael

Michael Paquier <michael(at)paquier(dot)xyz> writes:
> On Tue, Dec 12, 2023 at 09:37:58AM -0500, Tom Lane wrote:
>> ... Perhaps we could allow extension
>> modules to use binary_upgrade_set_next_pg_type_oid, and then somehow
>> reserve the money and _money OIDs forever?

> Now, I am not sure that we're making users a favor in keeping around a
> data type that's kind of obsolete these days, particularly when all of
> them basically require handling of fractional cents when doing serious
> calculations. That's a trap in disguise, even if tying data to a
> locale for the unit sounds appealing for some applications.

Yeah, I wouldn't propose this if money were the only type we were
thinking of getting rid of; likely not worth the trouble. But I've
got one eye on the geometric types as well --- I think it would be
great all around if we could kick those out of the core distribution
and let them be maintained independently. But I fear we can't do that
without some method whereby pg_upgrade is still possible across the
change.

regards, tom lane

On 12/16/23 10:19 AM, Tom Lane wrote:
> Michael Paquier <michael(at)paquier(dot)xyz> writes:
>> On Tue, Dec 12, 2023 at 09:37:58AM -0500, Tom Lane wrote:
>>> ... Perhaps we could allow extension
>>> modules to use binary_upgrade_set_next_pg_type_oid, and then somehow
>>> reserve the money and _money OIDs forever?
>
>> Now, I am not sure that we're making users a favor in keeping around a
>> data type that's kind of obsolete these days, particularly when all of
>> them basically require handling of fractional cents when doing serious
>> calculations. That's a trap in disguise, even if tying data to a
>> locale for the unit sounds appealing for some applications.
>
> Yeah, I wouldn't propose this if money were the only type we were
> thinking of getting rid of; likely not worth the trouble. But I've
> got one eye on the geometric types as well --- I think it would be
> great all around if we could kick those out of the core distribution
> and let them be maintained independently. But I fear we can't do that
> without some method whereby pg_upgrade is still possible across the
> change.

"money" poses a bit more risk than the geometric types (I do want to
come back to geometric types later), and it'd be good if we could make
some headway on removing it. Just this past week I came across a case
where "money" was being discussed as being valid, and had to give the
"it's kinda, sorta deprecated" talk.

I'm of the strong opinion that we should get rid of money. I personally
haven't encountered it in the wild -- I'm sure it's there, but it seems
limited -- and most apps that seriously deal with money will either user
either "numeric" or an integer-based type.

I am sensitive to the upgrade piece, as we don't want someone relying on
that behavior to be indefinitely stuck. But perhaps part of the
deprecation plan is to just keep the casts around[1] for a few releases,
without exposing the type, and prevent new creations of the type? We do
state that "A money value can be cast to numeric without loss of
precision"[1], but I do agree with the concerns on a pg_upgrade
operation being stuck on a costly rewrite operation.

Would it make sense to block an upgrade via pg_upgrade if we detect a
money type unless it's rewritten to a different type? Users should still
be able to user other upgrades methods if need be, and handle the
rewrite during that time.

Jonathan

[1] /docs/current/datatype-money.html

"Jonathan S. Katz" <jkatz(at)postgresql(dot)org> writes:
> I'm of the strong opinion that we should get rid of money. I personally
> haven't encountered it in the wild -- I'm sure it's there, but it seems
> limited -- and most apps that seriously deal with money will either user
> either "numeric" or an integer-based type.

Yeah, maybe we should just do it. I was pleasantly surprised by how
little push-back we got from nuking the 32-bit datetime types a few
releases ago; perhaps this one would likewise not have much of a
constituency.

> I am sensitive to the upgrade piece, as we don't want someone relying on
> that behavior to be indefinitely stuck. But perhaps part of the
> deprecation plan is to just keep the casts around[1] for a few releases,
> without exposing the type, and prevent new creations of the type?

I don't really see a way to do that, especially not if we don't want
to put a large amount of effort into it. We can certainly make
pg_upgrade reject "money" columns, and tell people they need to
rewrite those before they upgrade not after.

regards, tom lane

On Sun, Dec 24, 2023 at 01:21:15PM -0500, Tom Lane wrote:
> "Jonathan S. Katz" <jkatz(at)postgresql(dot)org> writes:
>> I'm of the strong opinion that we should get rid of money. I personally
>> haven't encountered it in the wild -- I'm sure it's there, but it seems
>> limited -- and most apps that seriously deal with money will either user
>> either "numeric" or an integer-based type.
>
> Yeah, maybe we should just do it. I was pleasantly surprised by how
> little push-back we got from nuking the 32-bit datetime types a few
> releases ago; perhaps this one would likewise not have much of a
> constituency.

Yeah, that's hard to say. Perhaps it would a good idea to ask more
widely how much people have been relying on that and how large these
data sets are. Another thing that we could attempt is doing something
about the the table rewrites on type changes to require an EOL. Like
more CONCURRENTLY flavors in ALTER TABLE?

>> I am sensitive to the upgrade piece, as we don't want someone relying on
>> that behavior to be indefinitely stuck. But perhaps part of the
>> deprecation plan is to just keep the casts around[1] for a few releases,
>> without exposing the type, and prevent new creations of the type?
>
> I don't really see a way to do that, especially not if we don't want
> to put a large amount of effort into it. We can certainly make
> pg_upgrade reject "money" columns, and tell people they need to
> rewrite those before they upgrade not after.

At least the infra to be able to achieve that should be here.
--
Michael

On Sun, 2023-12-24 at 13:21 -0500, Tom Lane wrote:
> "Jonathan S. Katz" <jkatz(at)postgresql(dot)org> writes:
> > I'm of the strong opinion that we should get rid of money. I personally
> > haven't encountered it in the wild -- I'm sure it's there, but it seems
> > limited -- and most apps that seriously deal with money will either user
> > either "numeric" or an integer-based type.
>
> Yeah, maybe we should just do it. I was pleasantly surprised by how
> little push-back we got from nuking the 32-bit datetime types a few
> releases ago; perhaps this one would likewise not have much of a
> constituency.

There are questions on Stackoverflow:
https://stackoverflow.com/search?q=%5Bpostgresql%5D+money

But of course it is hard to say if these are production uses or just
experiments.

Yours,
Laurenz Albe

On Mon, Dec 25, 2023 at 09:52:53AM +0900, Michael Paquier wrote:
> At least the infra to be able to achieve that should be here.

Looking at that, I can see that Peter has added a few tests to test
the predictability of plans generated with non-hashable types, and
that these are based on money. See 6dd8b0080787.

One possible pick to replace money for these tests is tsvector, that
cannot be hashed and has an equality operator. Perhaps these had
better be replaced anyway?
--
Michael

Michael Paquier <michael(at)paquier(dot)xyz> writes:
> Looking at that, I can see that Peter has added a few tests to test
> the predictability of plans generated with non-hashable types, and
> that these are based on money. See 6dd8b0080787.

> One possible pick to replace money for these tests is tsvector, that
> cannot be hashed and has an equality operator. Perhaps these had
> better be replaced anyway?

Hm. A quick check in pg_opclass shows that these are the types that
currently have btree but not hash opclasses:

# select ob.opcintype::regtype from (select * from pg_opclass where opcmethod = 403) ob
left join (select * from pg_opclass where opcmethod = 405) oh using(opcintype) where oh.opcintype is null;
opcintype
-------------
bit
bit varying
money
tsvector
tsquery
(5 rows)

I'm a little nervous about using tsvector or tsquery, as it seems
pretty plausible that somebody would get around to making hash
support for them someday. Perhaps the same argument could be made
about bit or varbit, but I'd bet a lot less on that happening,
as those are backwater-ish types (not even in the standard
anymore, IIRC). So I'd think about using one of those.

Or we could build a for-test-purposes-only datatype, but that
could require a lot of scaffolding work.

regards, tom lane

On Mon, Dec 25, 2023 at 10:58:59AM -0500, Tom Lane wrote:
> I'm a little nervous about using tsvector or tsquery, as it seems
> pretty plausible that somebody would get around to making hash
> support for them someday. Perhaps the same argument could be made
> about bit or varbit, but I'd bet a lot less on that happening,
> as those are backwater-ish types (not even in the standard
> anymore, IIRC). So I'd think about using one of those.

Okay, I've used varbit then.

> Or we could build a for-test-purposes-only datatype, but that
> could require a lot of scaffolding work.

It took me some time to come back to this thread, apologies for the
delay.

I have been looking at all that, and finished for now with the
attached patch that removes the dependency to "money" in the main
regression test suite in all the relevant places without losing
coverage, switching these to use varbit (well, mostly):
- Tests for hashing and hash-based plans (6dd8b0080787).
- A set of tests in rules used money historically but these can be
switched to numeric.
- Two tests in stats_ext relied on cash_words() to generate different
strings, but the same variance can be achieved without it.
- One test in create_table relied on money as its cast is not
immutable. Here it is possible to use a timestamptz to achieve the
same, with something like to_date() to force a function execution
when defining a partition boundary for one of the partitions (from
7c079d7417a8).

With this patch, the tests related to money that remain are for
type_sanity, window.sql and of course money which are all related to
the data type so they could be wiped out once the type is itself
removed.

While looking at the whole picture, an issue with the direct removal
of money is how we should handle btree_gin and btree_gist which have
operators based on money. We try to keep things compatible at
run-time, but could this be worth a hard break in these modules,
dropping the older sql scripts used in the modules if we don't have
access to money anymore at runtime? These are not popular modules..
Any thoughts about that?

For now, please find attached a patch to adjust the regression tests
to depend less on money, which does not depend on the type removal.

Comments are welcome.
--
Michael

On 1/10/24 8:12 PM, Michael Paquier wrote:

> While looking at the whole picture, an issue with the direct removal
> of money is how we should handle btree_gin and btree_gist which have
> operators based on money. We try to keep things compatible at
> run-time, but could this be worth a hard break in these modules,
> dropping the older sql scripts used in the modules if we don't have
> access to money anymore at runtime? These are not popular modules..
> Any thoughts about that?

Both modules are pretty popular. Personally, I used it in scheduling
apps that involved range types + exclusion constraints. The data I've
seen suggests btree_gist / btree_gin are widely deployed.

That said, I don't know how much of these modules are used with the
money type specifically. My guess is that it's more common to combine it
with something like an {int,bool}/range type than a money type.

It sounds like we'd have to tread a bit lightly because of this, even if
money is not frequently (or at all) used with btree_gist/gin?

Jonathan

"Jonathan S. Katz" <jkatz(at)postgresql(dot)org> writes:
> On 1/10/24 8:12 PM, Michael Paquier wrote:
>> While looking at the whole picture, an issue with the direct removal
>> of money is how we should handle btree_gin and btree_gist which have
>> operators based on money. We try to keep things compatible at
>> run-time, but could this be worth a hard break in these modules,
>> dropping the older sql scripts used in the modules if we don't have
>> access to money anymore at runtime? These are not popular modules..
>> Any thoughts about that?

> Both modules are pretty popular.

Yeah, I think it's a serious error to assume they aren't used plenty.
Not the money part, but other scalar types.

> It sounds like we'd have to tread a bit lightly because of this, even if
> money is not frequently (or at all) used with btree_gist/gin?

What'd have to happen is that people would have to upgrade to a
version of btree_gin/btree_gist that deletes its money support
before they could pg_upgrade into a core version that lacks money.
So we'd have to ship that version at least one major release before
nuking the core support.

I think the shortest timeline we could possibly do this on is:

v17: label money as deprecated and due for removal in the SGML docs

v18: remove support in btree_gin/btree_gist (and any other affected
extensions)

v19: remove it from core

Note that in v18, people could still use money even in
btree_gin/btree_gist, just by installing a non-default extension
version. So their C code for money would have to stay.

regards, tom lane

On 1/12/24 11:33 AM, Tom Lane wrote:
> "Jonathan S. Katz" <jkatz(at)postgresql(dot)org> writes:

>> It sounds like we'd have to tread a bit lightly because of this, even if
>> money is not frequently (or at all) used with btree_gist/gin?
>
> What'd have to happen is that people would have to upgrade to a
> version of btree_gin/btree_gist that deletes its money support
> before they could pg_upgrade into a core version that lacks money.
> So we'd have to ship that version at least one major release before
> nuking the core support.

Hm -- what about people who skip versions (e.g. 16 => 19)? Would they
have to stop at 18 first to perform the upgrade? And does this only
affect people who use btree_gist/gin for moeny, or all btree_gist/gin users?

> I think the shortest timeline we could possibly do this on is:
>
> v17: label money as deprecated and due for removal in the SGML docs

We had apparently deprecation notice as late as 8.2[1], but this warning
is missing in the docs :(

> v18: remove support in btree_gin/btree_gist (and any other affected
> extensions)

If the btree_gist/gin issue only affects people who use it in
combination with money, I'd at least think we consider this for v17, and
make clear that a users with money have to take action before upgrading.

If we stick with just adding the deprecation notice in the v17 docs, I'd
also suggest we emit a log warning when loading the catalog if the user
has an active use of a "money" type in a table. (I understand that may a
pain to do, but at least wanted to suggested it).

> v19: remove it from core
>
> Note that in v18, people could still use money even in
> btree_gin/btree_gist, just by installing a non-default extension
> version. So their C code for money would have to stay.

Yeah, but for what we support directly in PostgreSQL, we will have made
best effort. If someone _really_ wants to do something with "money" in
the way you describe above, then they're committed to it.

Thanks,

Jonathan

[1] /docs/8.2/datatype-money.html

On Fri, Jan 12, 2024 at 10:50:25PM -0500, Jonathan S. Katz wrote:
> On 1/12/24 11:33 AM, Tom Lane wrote:
>> What'd have to happen is that people would have to upgrade to a
>> version of btree_gin/btree_gist that deletes its money support
>> before they could pg_upgrade into a core version that lacks money.
>> So we'd have to ship that version at least one major release before
>> nuking the core support.
>
> Hm -- what about people who skip versions (e.g. 16 => 19)? Would they have
> to stop at 18 first to perform the upgrade? And does this only affect people
> who use btree_gist/gin for moeny, or all btree_gist/gin users?

That's exactly why I am worried about a schedule being too aggressive
here. Less runs of pg_upgrade means less headaches and less downtime.
I'd guess that we should have at least 3~4 years between a deprecation
in the docs and the actual removal. It seems like this is what we
should do now on HEAD, aiming at a complete removal in 21~22 when 17
gets out of support (even if pg_upgrade support is down to 10 years).

>> v19: remove it from core
>>
>> Note that in v18, people could still use money even in
>> btree_gin/btree_gist, just by installing a non-default extension
>> version. So their C code for money would have to stay.
>
> Yeah, but for what we support directly in PostgreSQL, we will have made best
> effort. If someone _really_ wants to do something with "money" in the way
> you describe above, then they're committed to it.

By the way, let me know if you have any objections about the patch I
have posted upthread that reduces the dependency to money in the main
regression test suite, replacing things with varbit while keeping
coverage the same. I have a few second thoughts about the change in
create_table, so I'm tempted to leave that part out for now, switching
the rest.
--
Michael

On Sun, Jan 14, 2024 at 09:55:03AM +0900, Michael Paquier wrote:
> By the way, let me know if you have any objections about the patch I
> have posted upthread that reduces the dependency to money in the main
> regression test suite, replacing things with varbit while keeping
> coverage the same. I have a few second thoughts about the change in
> create_table, so I'm tempted to leave that part out for now, switching
> the rest.

This one has been backpatched as 31acee4b66f9 for now, but I've left
out the test case in create_table with partitions. I kind of felt
that I was missing something, and it did not prevent to switch all the
others.

I'll spawn a thread on -hackers about what to do next, with probably a
patch to propose a deprecation notice in the docs for v17.
--
Michael

On Mon, Jan 15, 2024 at 04:54:51PM +0900, Michael Paquier wrote:
> I'll spawn a thread on -hackers about what to do next, with probably a
> patch to propose a deprecation notice in the docs for v17.

Did this thread ever materialize? I searched around a bit and couldn't
find anything.

--
nathan

For future reference, this bug should be fixed by commit 22b0ccd.

--
nathan