Re: Insecure DNS servers on PG infrastructure

I just noted that cvs.postgresql.org and svr1.postgresql.org are not
running the latest bind release, which means that they are vulnerable to
the DNS cache poisoning attack recently discovered by Dan Kaminsky.
Vixie and co think this is a pretty big deal, so folks might want to
update sooner rather than later.
http://www.kb.cert.org/vuls/id/800113

BTW, there is an excellent end-to-end test available for whether the
security fix (port randomization) is actually working for you:

dig @server-to-test porttest.dns-oarc.net in txt

This takes a few seconds (they've arranged it to force multiple queries
from the tested server) and gives you back a readout of how many ports
those queries arrived from and the spread in the port addresses.
A good result looks about like this:

;; ANSWER SECTION:
porttest.dns-oarc.net. 60 IN CNAME z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net. 60 IN TXT "66.207.139.134 is GOOD: 26 queries in 2.3 seconds from 26 ports with std dev 17102.06"

If it says FAIR or POOR then you have an unpatched server or there
is something interfering with the port randomization. If the server
is behind a NAT firewall then the latter is entirely likely.

regards, tom lane

On Fri, 2008-07-25 at 11:02 -0400, Tom Lane wrote:
> I just noted that cvs.postgresql.org and svr1.postgresql.org are not
> running the latest bind release, which means that they are vulnerable to
> the DNS cache poisoning attack recently discovered by Dan Kaminsky.
> Vixie and co think this is a pretty big deal, so folks might want to
> update sooner rather than later.
> http://www.kb.cert.org/vuls/id/800113

Dave and Magnus are on vacation. I believe the only other people that
would have access to those boxes are Stefan and Marc. I have pinged
Stefan.

Joshua D. Drake

--
The PostgreSQL Company since 1997: http://www.commandprompt.com/
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate

On Fri, Jul 25, 2008 at 11:02:03AM -0400, Tom Lane wrote:
> I just noted that cvs.postgresql.org and svr1.postgresql.org are not
> running the latest bind release, which means that they are vulnerable to
> the DNS cache poisoning attack recently discovered by Dan Kaminsky.
> Vixie and co think this is a pretty big deal, so folks might want to
> update sooner rather than later.

This is an extremely big deal. The numbers I've seen suggest windows
somewhere around 10 minutes. If the systems above are doing
recursion, then they need to be patched right away. (If they're
running both authority and recursive services in the same BIND
instance, I suggest that the practice be abandoned immediately.)

> BTW, there is an excellent end-to-end test available for whether the
> security fix (port randomization) is actually working for you:
>
> dig @server-to-test porttest.dns-oarc.net in txt
>
> This takes a few seconds (they've arranged it to force multiple queries
> from the tested server) and gives you back a readout of how many ports
> those queries arrived from and the spread in the port addresses.

Note the TTL, BTW, so that running it twice in a row gives exactly the
same answer. You need to wait for the cache to expire to repeat your tests.

> If it says FAIR or POOR then you have an unpatched server or there
> is something interfering with the port randomization. If the server
> is behind a NAT firewall then the latter is entirely likely.

There's no reason that a NAT should do that, if the device is
competently built: if you randomise source ports on the inside, the
NAT device could just use the same port on the outside.

Much worse is the bad habit many got into recently of nailing the
source port to 53, and then only opening that on the firewall on the
way out. Some packages have had this behaviour turned on by default.
Setting the source port this way will foil the patch that is included
in BIND. You need to make your recursive queries from a random port,
and the port needs to be unpredictable by someone watching your
resolver.

If you want to try another recursive-only resolver (one that is much
more recent and so was never vulnerable to this attack), have a look
at unbound (find it at unbound.net) It can do DNSSEC validation for
you too.

BTW, the strategies that BIND and others are using to foil the
Kaminsky attack merely reduce the liklihood, and don't prevent it
completely. There's not a lot of source ports over which to
randomize. The real solution to all this is DNSSEC, but deploying it
isn't trivial, and the root isn't signed yet. (.ORG will be soon,
though!)

A

--
Andrew Sullivan
ajs(at)commandprompt(dot)com
+1 503 667 4564 x104
http://www.commandprompt.com/

Andrew Sullivan <ajs(at)commandprompt(dot)com> writes:
> On Fri, Jul 25, 2008 at 11:02:03AM -0400, Tom Lane wrote:
>> If it says FAIR or POOR then you have an unpatched server or there
>> is something interfering with the port randomization. If the server
>> is behind a NAT firewall then the latter is entirely likely.

> There's no reason that a NAT should do that, if the device is
> competently built: if you randomise source ports on the inside, the
> NAT device could just use the same port on the outside.

I'm not convinced that that's true. If the router is trying to forward
UDP messages arriving from several "inside" IP addresses using only one
"outside" address, it has to deal with the possibility of collisions,
ie two "inside" addresses using the same port number at about the same
time. So it doesn't surprise me that it rewrites the port numbers.
If it assigned randomly-generated substitute numbers there'd be no
problem, but with no prior knowledge that would be a good idea you can
hardly blame the router authors for not indulging in extra complexity.

What I do know is that my own firewall hardware (a Netopia T1 router
that's two or three years old) *was* rewriting UDP port numbers on
requests from a machine that was sharing a NAT address with others.
After remapping to give that machine its own "outside" IP address,
it stopped doing so. BTW the porttest.dns-oarc.net service was
invaluable in testing this; I'd probably have thought that just
installing the new BIND made me safe, if I hadn't had a way to test it.

regards, tom lane

On Fri, Jul 25, 2008 at 04:44:32PM -0400, Tom Lane wrote:

> I'm not convinced that that's true. If the router is trying to forward
> UDP messages arriving from several "inside" IP addresses using only one
> "outside" address, it has to deal with the possibility of collisions,
> ie two "inside" addresses using the same port number at about the same
> time.

This is true. They can't arrive at exactly the same time, though,
which means that different strategies can be used. It's certainly
true, however, that one of the strategies may well be to rewrite port
numbers.

In some sense, rewriting to the same port number makes things quite a
bit worse for the router, because rather than just remembering "oh,
port O1 was port I1 and port O2 was port I2", the router has to
remember which {staticport,Iport} pair belongs with which inside
address. So more state is needed. (Now everyone can be amazed at
just how fast a hand can be made to wave. But this is the gist of the
argument.)

> What I do know is that my own firewall hardware (a Netopia T1 router
> that's two or three years old) *was* rewriting UDP port numbers on
> requests from a machine that was sharing a NAT address with others.

It is a problem, for sure, and the OARC test is a big help. Yay
OARC (full disclosure: my former employer isa major OARC sponsor).

A

--
Andrew Sullivan
ajs(at)commandprompt(dot)com
+1 503 667 4564 x104
http://www.commandprompt.com/

Tom Lane wrote:
> Andrew Sullivan <ajs(at)commandprompt(dot)com> writes:
>> On Fri, Jul 25, 2008 at 11:02:03AM -0400, Tom Lane wrote:
>>> If it says FAIR or POOR then you have an unpatched server or there
>>> is something interfering with the port randomization. If the server
>>> is behind a NAT firewall then the latter is entirely likely.
>
>> There's no reason that a NAT should do that, if the device is
>> competently built: if you randomise source ports on the inside, the
>> NAT device could just use the same port on the outside.

Tom can you check if this has been resolved? If not I am going to start
paging people.

Joshua D. Drake

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> Tom can you check if this has been resolved? If not I am going to start
> paging people.

The machines I have access to (cvs and svr1) are still unpatched
according to the "dig porttest" test. Personally, though, I'd be
more interested in getting someone to work on fixing the mailing
lists. I've received nothing since about midnight EDT, and I know
that there has been traffic.

regards, tom lane

Andrew Sullivan wrote:
> On Fri, Jul 25, 2008 at 11:02:03AM -0400, Tom Lane wrote:
>> I just noted that cvs.postgresql.org and svr1.postgresql.org are not
>> running the latest bind release, which means that they are vulnerable to
>> the DNS cache poisoning attack recently discovered by Dan Kaminsky.
>> Vixie and co think this is a pretty big deal, so folks might want to
>> update sooner rather than later.
>
> This is an extremely big deal. The numbers I've seen suggest windows
> somewhere around 10 minutes. If the systems above are doing
> recursion, then they need to be patched right away. (If they're
> running both authority and recursive services in the same BIND
> instance, I suggest that the practice be abandoned immediately.)

cvs.postgresql.org is not running bind at all - what it is using are two
(purely) recursive resolvers upstream. One of them is only going to get
upgraded tomorrow(some changes need to be rolled out in a staged
fashion) the other one was done a while ago - I have simply removed that
one from the resolv.conf for the time being.

Stefan

Tom Lane wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> > Tom can you check if this has been resolved? If not I am going to start
> > paging people.
>
> The machines I have access to (cvs and svr1) are still unpatched
> according to the "dig porttest" test. Personally, though, I'd be
> more interested in getting someone to work on fixing the mailing
> lists. I've received nothing since about midnight EDT, and I know
> that there has been traffic.

Maybe the mail servers have been DDoS'ed again?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera wrote:
> Tom Lane wrote:
>> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
>>> Tom can you check if this has been resolved? If not I am going to start
>>> paging people.
>> The machines I have access to (cvs and svr1) are still unpatched
>> according to the "dig porttest" test. Personally, though, I'd be
>> more interested in getting someone to work on fixing the mailing
>> lists. I've received nothing since about midnight EDT, and I know
>> that there has been traffic.
>
> Maybe the mail servers have been DDoS'ed again?

I did have a bunch of alerts throw recovery around 2:00am but then where
is all the traffic?

>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Friday, July 25, 2008 11:02:03 -0400 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> I just noted that cvs.postgresql.org and svr1.postgresql.org are not

Was out camping, just saw this ... svr1.postgresql.org's name server is
shutdown/disabled now, since its not needed ... and all of the primary name
servers have had recursion disabled since Andrew pointed out to me awhile back
how to do so ...

- --
Marc G. Fournier Hub.Org Hosting Solutions S.A. (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (FreeBSD)

iEYEARECAAYFAkiNSnwACgkQ4QvfyHIvDvOs2QCfTJPaWg2si8wbdmGSVy4p8Xdx
GIsAoIl4TDOy4axlpAKDfmgwgOoxPb4s
=Ro1U
-----END PGP SIGNATURE-----