BUG #14709: inconsistent answers with foreign data wrappers to mysql

The following bug has been logged on the website:

Bug reference: 14709
Logged by: Mark Manley
Email address: mark(dot)manley(at)tapad(dot)com
PostgreSQL version: 9.6.2
Operating system: CentOS 7.3
Description:

Greetings.

When using a foreign data wrapper from our 9.6.2 database to a MySQL 5.7
database, we get inconsistent results. We're joining two tables from
another server:

dsp_users
dsp_accounts

This table is local in our Postgres server:

audiences

This query returns only the first record, which is using simple joins:

select audiences.* FROM audiences
INNER JOIN dsp_users ON dsp_users.id = audiences.created_by
INNER JOIN dsp_accounts ON dsp_accounts.id = audiences.account_id
WHERE audiences.name like '%Test%'

and I get only the first hit.

When I run the composite queries under it, removing the like clause from the
predicate, I get all the results I would expect:

1 |23842583561190673 |XXXXXXXX |sg1 |Test audience
| |1012 |269 |2017-05-03
15:04:57 |
2 |23842583562050673 |XXXXXXXX |sg1 |Test audience
| |1012 |269 |2017-05-03
15:25:59 |
10 |23842688223720004 |XXXXXXXX |sg1 |Audience DTAC
Test 1 | |1013 |307 |2017-06-09
16:40:54 |

with the clause:

1 |23842583561190673 |XXXXXXXX |sg1 |Test audience
| |1012 |269 |2017-05-03
15:04:57 |

If I rewrite the query to turn the foreign MySQL tables into a subselect,
then I get the correct results:

select a.* from audiences a
where exists (select 1 from audiences a, dsp_users du, dsp_accounts da
where
a.created_by = du.id
and a.account_id = da.id)

1 |23842583561190673 |XXXXXXXX |sg1 |Test audience
| |1012 |269 |2017-05-03
15:04:57 |
2 |23842583562050673 |XXXXXXXX |sg1 |Test audience
| |1012 |269 |2017-05-03
15:25:59 |
10 |23842688223720004 |XXXXXXXX |sg1 |Audience DTAC
Test 1 | |1013 |307 |2017-06-09
16:40:54 |

As you can see, I get inconsistent results if I use a normal join versus a
subselect. It's as though the like filter in the predicate is breaking.
This behaviour does not happen with using likes joining local tables only.

Anyway, wanted you guys to know.

Thanks!

mark(dot)manley(at)tapad(dot)com writes:
> When using a foreign data wrapper from our 9.6.2 database to a MySQL 5.7
> database, we get inconsistent results.

My first thought is that you're looking at a bug in the mysql FDW,
probably incorrect pushdown of conditions to the mysql server. You
should enable logging on the mysql server and look at the queries
generated by the FDW, comparing those to EXPLAIN output to see which
conditions get checked locally as opposed to being sent to the remote.

If you can identify what the FDW is doing wrong, reporting that to
the FDW's authors might get some results. (I don't know if they
read this bugs list.)

regards, tom lane

>>>>> "mark" == mark manley <mark(dot)manley(at)tapad(dot)com> writes:

mark> When using a foreign data wrapper from our 9.6.2 database to a
mark> MySQL 5.7 database, we get inconsistent results. We're joining
mark> two tables from another server:

mysql_fdw is (and has _always_ been) broken for anything beyond the most
trivial queries:

https://github.com/EnterpriseDB/mysql_fdw/blob/master/mysql_fdw.c#L734

https://github.com/EnterpriseDB/mysql_fdw/issues/73

Since mysql_fdw is not part of postgresql you should take this up with
the maintainers. However, the linked issue has been outstanding for 18
months, so probably best not hold your breath.

--
Andrew (irc:RhodiumToad)

Hi,

I informed the maintainer of mysql_fdw and they will start looking into
this issue.

Thanks,

On Mon, Jun 19, 2017 at 4:13 AM, Andrew Gierth <andrew(at)tao11(dot)riddles(dot)org(dot)uk>
wrote:

> >>>>> "mark" == mark manley <mark(dot)manley(at)tapad(dot)com> writes:
>
> mark> When using a foreign data wrapper from our 9.6.2 database to a
> mark> MySQL 5.7 database, we get inconsistent results. We're joining
> mark> two tables from another server:
>
> mysql_fdw is (and has _always_ been) broken for anything beyond the most
> trivial queries:
>
> https://github.com/EnterpriseDB/mysql_fdw/blob/master/mysql_fdw.c#L734
>
> https://github.com/EnterpriseDB/mysql_fdw/issues/73
>
> Since mysql_fdw is not part of postgresql you should take this up with
> the maintainers. However, the linked issue has been outstanding for 18
> months, so probably best not hold your breath.
>
> --
> Andrew (irc:RhodiumToad)
>
>
> --
> Sent via pgsql-bugs mailing list (pgsql-bugs(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-bugs
>

--
Rushabh Lathia
www.EnterpriseDB.com

Thank you all. :)

On Mon, Jun 19, 2017 at 3:57 AM, Rushabh Lathia <rushabh(dot)lathia(at)gmail(dot)com>
wrote:

> Hi,
>
> I informed the maintainer of mysql_fdw and they will start looking into
> this issue.
>
> Thanks,
>
>
> On Mon, Jun 19, 2017 at 4:13 AM, Andrew Gierth <
> andrew(at)tao11(dot)riddles(dot)org(dot)uk> wrote:
>
>> >>>>> "mark" == mark manley <mark(dot)manley(at)tapad(dot)com> writes:
>>
>> mark> When using a foreign data wrapper from our 9.6.2 database to a
>> mark> MySQL 5.7 database, we get inconsistent results. We're joining
>> mark> two tables from another server:
>>
>> mysql_fdw is (and has _always_ been) broken for anything beyond the most
>> trivial queries:
>>
>> https://github.com/EnterpriseDB/mysql_fdw/blob/master/mysql_fdw.c#L734
>>
>> https://github.com/EnterpriseDB/mysql_fdw/issues/73
>>
>> Since mysql_fdw is not part of postgresql you should take this up with
>> the maintainers. However, the linked issue has been outstanding for 18
>> months, so probably best not hold your breath.
>>
>> --
>> Andrew (irc:RhodiumToad)
>>
>>
>> --
>> Sent via pgsql-bugs mailing list (pgsql-bugs(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-bugs
>>
>
>
>
> --
> Rushabh Lathia
> www.EnterpriseDB.com
>

It will be helpful if you can share a test case that can be used to verify
our fix?

-- Ahsan

On Mon, Jun 19, 2017 at 3:57 PM, Mark Manley <mark(dot)manley(at)tapad(dot)com> wrote:

> Thank you all. :)
>
>
> On Mon, Jun 19, 2017 at 3:57 AM, Rushabh Lathia <rushabh(dot)lathia(at)gmail(dot)com>
> wrote:
>
>> Hi,
>>
>> I informed the maintainer of mysql_fdw and they will start looking into
>> this issue.
>>
>> Thanks,
>>
>>
>> On Mon, Jun 19, 2017 at 4:13 AM, Andrew Gierth <
>> andrew(at)tao11(dot)riddles(dot)org(dot)uk> wrote:
>>
>>> >>>>> "mark" == mark manley <mark(dot)manley(at)tapad(dot)com> writes:
>>>
>>> mark> When using a foreign data wrapper from our 9.6.2 database to a
>>> mark> MySQL 5.7 database, we get inconsistent results. We're joining
>>> mark> two tables from another server:
>>>
>>> mysql_fdw is (and has _always_ been) broken for anything beyond the most
>>> trivial queries:
>>>
>>> https://github.com/EnterpriseDB/mysql_fdw/blob/master/mysql_fdw.c#L734
>>>
>>> https://github.com/EnterpriseDB/mysql_fdw/issues/73
>>>
>>> Since mysql_fdw is not part of postgresql you should take this up with
>>> the maintainers. However, the linked issue has been outstanding for 18
>>> months, so probably best not hold your breath.
>>>
>>> --
>>> Andrew (irc:RhodiumToad)
>>>
>>>
>>> --
>>> Sent via pgsql-bugs mailing list (pgsql-bugs(at)postgresql(dot)org)
>>> To make changes to your subscription:
>>> http://www.postgresql.org/mailpref/pgsql-bugs
>>>
>>
>>
>>
>> --
>> Rushabh Lathia
>> www.EnterpriseDB.com
>>
>
>

--
Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

The fix for this issue is committed in mysql_fdw github repository, please
see the following mysql_fdw github entry for details...

https://github.com/EnterpriseDB/mysql_fdw/issues/73

Thanks Mark for providing the test case.

On Mon, Jun 19, 2017 at 4:01 PM, Ahsan Hadi <ahsan(dot)hadi(at)enterprisedb(dot)com>
wrote:

> It will be helpful if you can share a test case that can be used to verify
> our fix?
>
> -- Ahsan
>
>
> On Mon, Jun 19, 2017 at 3:57 PM, Mark Manley <mark(dot)manley(at)tapad(dot)com>
> wrote:
>
>> Thank you all. :)
>>
>>
>> On Mon, Jun 19, 2017 at 3:57 AM, Rushabh Lathia <rushabh(dot)lathia(at)gmail(dot)com
>> > wrote:
>>
>>> Hi,
>>>
>>> I informed the maintainer of mysql_fdw and they will start looking into
>>> this issue.
>>>
>>> Thanks,
>>>
>>>
>>> On Mon, Jun 19, 2017 at 4:13 AM, Andrew Gierth <
>>> andrew(at)tao11(dot)riddles(dot)org(dot)uk> wrote:
>>>
>>>> >>>>> "mark" == mark manley <mark(dot)manley(at)tapad(dot)com> writes:
>>>>
>>>> mark> When using a foreign data wrapper from our 9.6.2 database to a
>>>> mark> MySQL 5.7 database, we get inconsistent results. We're joining
>>>> mark> two tables from another server:
>>>>
>>>> mysql_fdw is (and has _always_ been) broken for anything beyond the most
>>>> trivial queries:
>>>>
>>>> https://github.com/EnterpriseDB/mysql_fdw/blob/master/mysql_fdw.c#L734
>>>>
>>>> https://github.com/EnterpriseDB/mysql_fdw/issues/73
>>>>
>>>> Since mysql_fdw is not part of postgresql you should take this up with
>>>> the maintainers. However, the linked issue has been outstanding for 18
>>>> months, so probably best not hold your breath.
>>>>
>>>> --
>>>> Andrew (irc:RhodiumToad)
>>>>
>>>>
>>>> --
>>>> Sent via pgsql-bugs mailing list (pgsql-bugs(at)postgresql(dot)org)
>>>> To make changes to your subscription:
>>>> http://www.postgresql.org/mailpref/pgsql-bugs
>>>>
>>>
>>>
>>>
>>> --
>>> Rushabh Lathia
>>> www.EnterpriseDB.com
>>>
>>
>>
>
>
> --
> Ahsan Hadi
> Snr Director Product Development
> EnterpriseDB Corporation
> The Enterprise Postgres Company
>
> Phone: +92-51-8358874 <(051)%208358874>
> Mobile: +92-333-5162114 <0333%205162114>
>
> Website: www.enterprisedb.com
> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> Follow us on Twitter: http://www.twitter.com/enterprisedb
>
> This e-mail message (and any attachment) is intended for the use of the
> individual or entity to whom it is addressed. This message contains
> information from EnterpriseDB Corporation that may be privileged,
> confidential, or exempt from disclosure under applicable law. If you are
> not the intended recipient or authorized to receive this for the intended
> recipient, any use, dissemination, distribution, retention, archiving, or
> copying of this communication is strictly prohibited. If you have received
> this e-mail in error, please notify the sender immediately by reply e-mail
> and delete this message.
>

--
Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

Thank you. I have confirmed your fix works:

iris_staging=# select audiences.* FROM audiences
INNER JOIN mwm_users ON mwm_users.id = audiences.created_by
INNER JOIN mwm_accounts ON mwm_accounts.id = audiences.account_id and
audiences.name like '%name%';
id | foreign_audience_id | advertising_platform_name |
location | name | description | account
_id | created_by | created_at
----+---------------------------------+---------------------------+----------+-----------+--------------------+--------
----+------------+-------------------------------
1 | mock_facebook_audience(happy)-1 | FACEBOOK | nj1
| some name | some foreign descr | 1
012 | 269 | 2017-05-02 22:49:24.984224+00
2 | mock_facebook_audience(happy)-2 | FACEBOOK | nj1
| some name | some foreign descr | 1
012 | 269 | 2017-05-02 22:49:49.771061+00
3 | mock_facebook_audience(happy)-1 | FACEBOOK | nj1
| some name | some foreign descr | 1
012 | 269 | 2017-05-04 18:38:54.569716+00
(3 rows)

Thanks again!

On Wed, Jun 21, 2017 at 1:50 AM, Ahsan Hadi <ahsan(dot)hadi(at)enterprisedb(dot)com>
wrote:

> The fix for this issue is committed in mysql_fdw github repository, please
> see the following mysql_fdw github entry for details...
>
> https://github.com/EnterpriseDB/mysql_fdw/issues/73
>
> Thanks Mark for providing the test case.
>
> On Mon, Jun 19, 2017 at 4:01 PM, Ahsan Hadi <ahsan(dot)hadi(at)enterprisedb(dot)com>
> wrote:
>
>> It will be helpful if you can share a test case that can be used to
>> verify our fix?
>>
>> -- Ahsan
>>
>>
>> On Mon, Jun 19, 2017 at 3:57 PM, Mark Manley <mark(dot)manley(at)tapad(dot)com>
>> wrote:
>>
>>> Thank you all. :)
>>>
>>>
>>> On Mon, Jun 19, 2017 at 3:57 AM, Rushabh Lathia <
>>> rushabh(dot)lathia(at)gmail(dot)com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I informed the maintainer of mysql_fdw and they will start looking into
>>>> this issue.
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> On Mon, Jun 19, 2017 at 4:13 AM, Andrew Gierth <
>>>> andrew(at)tao11(dot)riddles(dot)org(dot)uk> wrote:
>>>>
>>>>> >>>>> "mark" == mark manley <mark(dot)manley(at)tapad(dot)com> writes:
>>>>>
>>>>> mark> When using a foreign data wrapper from our 9.6.2 database to a
>>>>> mark> MySQL 5.7 database, we get inconsistent results. We're joining
>>>>> mark> two tables from another server:
>>>>>
>>>>> mysql_fdw is (and has _always_ been) broken for anything beyond the
>>>>> most
>>>>> trivial queries:
>>>>>
>>>>> https://github.com/EnterpriseDB/mysql_fdw/blob/master/mysql_fdw.c#L734
>>>>>
>>>>> https://github.com/EnterpriseDB/mysql_fdw/issues/73
>>>>>
>>>>> Since mysql_fdw is not part of postgresql you should take this up with
>>>>> the maintainers. However, the linked issue has been outstanding for 18
>>>>> months, so probably best not hold your breath.
>>>>>
>>>>> --
>>>>> Andrew (irc:RhodiumToad)
>>>>>
>>>>>
>>>>> --
>>>>> Sent via pgsql-bugs mailing list (pgsql-bugs(at)postgresql(dot)org)
>>>>> To make changes to your subscription:
>>>>> http://www.postgresql.org/mailpref/pgsql-bugs
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Rushabh Lathia
>>>> www.EnterpriseDB.com
>>>>
>>>
>>>
>>
>>
>> --
>> Ahsan Hadi
>> Snr Director Product Development
>> EnterpriseDB Corporation
>> The Enterprise Postgres Company
>>
>> Phone: +92-51-8358874 <(051)%208358874>
>> Mobile: +92-333-5162114 <0333%205162114>
>>
>> Website: www.enterprisedb.com
>> EnterpriseDB Blog: http://blogs.enterprisedb.com/
>> Follow us on Twitter: http://www.twitter.com/enterprisedb
>>
>> This e-mail message (and any attachment) is intended for the use of the
>> individual or entity to whom it is addressed. This message contains
>> information from EnterpriseDB Corporation that may be privileged,
>> confidential, or exempt from disclosure under applicable law. If you are
>> not the intended recipient or authorized to receive this for the intended
>> recipient, any use, dissemination, distribution, retention, archiving, or
>> copying of this communication is strictly prohibited. If you have received
>> this e-mail in error, please notify the sender immediately by reply e-mail
>> and delete this message.
>>
>
>
>
> --
> Ahsan Hadi
> Snr Director Product Development
> EnterpriseDB Corporation
> The Enterprise Postgres Company
>
> Phone: +92-51-8358874 <+92%2051%208358874>
> Mobile: +92-333-5162114 <+92%20333%205162114>
>
> Website: www.enterprisedb.com
> EnterpriseDB Blog: http://blogs.enterprisedb.com/
> Follow us on Twitter: http://www.twitter.com/enterprisedb
>
> This e-mail message (and any attachment) is intended for the use of the
> individual or entity to whom it is addressed. This message contains
> information from EnterpriseDB Corporation that may be privileged,
> confidential, or exempt from disclosure under applicable law. If you are
> not the intended recipient or authorized to receive this for the intended
> recipient, any use, dissemination, distribution, retention, archiving, or
> copying of this communication is strictly prohibited. If you have received
> this e-mail in error, please notify the sender immediately by reply e-mail
> and delete this message.
>