Re: BUG #15671: The copy command does not load all the lines from a .csv file.

Lists: Postg스포츠 토토 사이트SQL : Postg스포츠 토토 사이트SQL 메일 링리스트 : 2019-03-06 이후 PGSQL-BUGS 15:46
From: PG Bug reporting form <noreply(at)postgresql(dot)org>
To: pgsql-bugs(at)lists(dot)postgresql(dot)org
Cc: ezenchenko(at)gmail(dot)com
Subject: BUG #15671: The copy command does not load all the lines from a .csv file.
Date: 2019-03-06 15:03:50
Message-ID: 15671-013f74cfbd977f66@postgresql.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-bugs

The following bug has been logged on the website:

Bug reference: 15671
Logged by: Эдуард Зенченко
Email address: ezenchenko(at)gmail(dot)com
PostgreSQL version: 10.4
Operating system: Debian 4.9
Description:

Hello!
There is a .csv file in windows-1251 encoding.
The file has 1165988 lines, along with the title.
I need to upload it to the table.
I execute the command in PgAdmin 4:

drop table test_csv;
create table test_csv (addr text, houseid text);
copy test_csv (addr, houseid) FROM '/tmp/gis_fias_kra.csv' (DELIMITER ';',
FORMAT CSV, NULL '\', HEADER, ENCODING 'win1251');

In response, I get:
COPY 993130
Query returned successfully in 7 secs 352 msec.

In table 993130 records.
The file is located in a directory on the server.

Why aren't all the lines from the .csv file loaded? This is very similar to
an error in the application.
The same file was loaded using pgloader, and all lines were loaded.


From: Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
To: ezenchenko(at)gmail(dot)com, pgsql-bugs(at)lists(dot)postgresql(dot)org
Subject: Re: BUG #15671: The copy command does not load all the lines from a .csv file.
Date: 2019-03-06 15:46:48
Message-ID: CAMkU=1z-68inLP+LPb+fc-ZnWDS3_QnH_SBkeusL0OsiUip+-w@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: Postg스포츠 토토 사이트SQL : Postg스포츠 토토 사이트SQL 메일 링리스트 : 2019-03-06 이후 PGSQL-BUGS 15:46

On Wed, Mar 6, 2019 at 10:34 AM PG Bug reporting form <
noreply(at)postgresql(dot)org> wrote:

> The following bug has been logged on the website:
>
> Bug reference: 15671
> Logged by: Эдуард Зенченко
> Email address: ezenchenko(at)gmail(dot)com
> PostgreSQL version: 10.4
> Operating system: Debian 4.9
> Description:
>
> Hello!
> There is a .csv file in windows-1251 encoding.
> The file has 1165988 lines, along with the title.
> I need to upload it to the table.
> I execute the command in PgAdmin 4:
>
> drop table test_csv;
> create table test_csv (addr text, houseid text);
> copy test_csv (addr, houseid) FROM '/tmp/gis_fias_kra.csv' (DELIMITER ';',
> FORMAT CSV, NULL '\', HEADER, ENCODING 'win1251');
>
> In response, I get:
> COPY 993130
> Query returned successfully in 7 secs 352 msec.
>
> In table 993130 records.
> The file is located in a directory on the server.
>
> Why aren't all the lines from the .csv file loaded?

How are you counting the lines in the file? Are any of your newline
characters escaped by being quoted?

select count(*) from test_csv where addr text like E'%\n%' or houseid like
E'%\n%';

Cheers,

Jeff