How can I Improve performance when loading data by copy?

[id10t]

Hi!

To achieve parallelism you don't have to split uncompressed CSV file.

Using the Parallel Load Library

You can use the HP Vertica PloadDelimitedSource library with the COPY statement to parallel load delimited files. This section refers to the functionality as Pload.

The Pload feature is ideal for loading very large data files (at least 10 GB). The library divides file parsing tasks across each core on the server node where the data file resides, significantly reducing file load time. For example, given a 12GB file to load, performance could be 3 - 5 times faster than loading the file without Pload.

You can specify the file division size (in bytes) by supplying an integer value to the pload chunk_size parameter.

After installing the library, you can use the COPY statement's WITH SOURCE PloadDelimitedSource parameter. Using Pload with COPY LOCAL is not supported.

[muaythai_duan]

hi sKwa:
where can I download the library and the document?thanks!

[id10t]

Hi!

Did you read this (I provided a link in my prev post also)? You have to read this, because there are explanation how to use and how to INSTALL !!!

[muaythai_duan]

I will read this web site，and pay more attention to practice.thank you!

[nnani]

Thanks for that skwa,

Learnt something new.

[muaythai_duan]

hi,nnani

Code: Select all

\set t_pwd `pwd`
\set input_file '''':t_pwd'/*.csv'''
copy test from :input_file1 nodename delimiter ',' direct ;
copy test from :input_file1 nodename delimiter ',' direct ;
copy test from :input_file3 nodename delimiter ',' direct ;

if files on only one node,the command is executed in turn,is not parallel.I am sure.

[id10t]

Hi!

"vsql script" always executed as sequential, even if data will be on different nodes. VSQL can't execute task in background.