Re: Upcoming PG re-releases

Bruce Momjian wrote:
> Tom Lane wrote:
> > Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > > I have added your suggestions to the 8.1.X release notes.
> >
> > Did you read the followup discussion? Recommending -c without a large
> > warning seems a very bad idea.
>
> Well, I said it would remove invalid sequences. What else should we
> say?
>
> This will remove invalid character sequences.
>
> I saw no clear solution that allowed sequences to be corrected.

The release note text is:

Some users are having problems loading <literal>UTF8</> data into 8.1.X.
This is because previous versions allowed invalid <literal>UTF8</>
sequences to be entered into the database, and this release properly
accepts only valid <literal>UTF8</> sequences. One way to correct a
dumpfile is to use <command>iconv -c -f utf-8 -t utf-8</>. This will
remove invalid character sequences. <command>iconv</> reads the entire
input file into memory so it might be necessary to <command>split</> the
dump into multiple smaller files for processing.

One nice solution would be if iconv would report the lines with errors
and you could correct them, but I see no way to do that. The only thing
you could do is to diff the old and new files to see the problems. Is
that helpful? Here is new text I have used:

Some users are having problems loading <literal>UTF8</> data into 8.1.X.
This is because previous versions allowed invalid <literal>UTF8</>
sequences to be entered into the database, and this release properly
accepts only valid <literal>UTF8</> sequences. One way to correct a
dumpfile is to use <command>iconv -c -f utf-8 -t utf-8 -o cleanfile.sql
dumpfile.sql</>. The <literal>-c</> option removes invalid character
sequences. A diff of the two files will show the sequences that are
invalid. <command>iconv</> reads the entire input file into memory so
it might be necessary to <command>split</> the dump into multiple
smaller files for processing.

It highlights the 'diff' idea.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073