The Vertica Database Forums

Posted: **Thu Jul 17, 2014 2:34 pm**

Hi,
I'm Using Talend Tool to Load into Vertica table.I have 25,000 files to load. Each file size is less Then 5MB.
it's taking 35 minute to load 50 files!!!!
But same Input Files if i write it to Output file (instead of vertica) it's taking less time!!
I'm using insert to load from Tool. (if less then 10MB file insert will be suffient to load data so i'm using insert instead of copy).

What i'm missing here..!! i need to change any settings!!

Regards,
Akshath

Posted: **Fri Jul 18, 2014 3:09 pm**

If you mean that your loads are being executed as a series of individual INSERT statements, you don’t want to do that, because as you’ve observed it’s slow. Let Talend batch the loads so that they are executed using COPY. Much much faster.

—Sharon

Posted: **Mon Jul 21, 2014 6:17 am**

hi sharon,
Thank you for reply. These are query i tried in my vertica database. It's taking really more time to execute.

Code: Select all

vertic=> \timing
Timing is on.
vertic=>select count(*) from Myschema.Test_Table;
 count
----------
 36866469
(1 row)

Time: First fetch (1 row):[b] 231522.939 ms.[/b] All rows formatted: 231523.017 ms

vertic=>select count(*) from Myschema.Test_Table_2;
 count
-------
     0
(1 row)

Time: First fetch (1 row): 254744.995 ms. All rows formatted: 254745.081 ms

if i'm not wrong, In vertica these queries should give output within a seconds.

Regards,
Akshath

Posted: **Mon Jul 21, 2014 6:18 am**

Hi,
There is one more error i'm getting as follows :

Code: Select all

[Vertica][VJDBC](3587) ERROR: Insufficient resources to execute plan on pool general [Timedout waiting for resource request: Request exceeds limits: Memory(KB) Exceeded: Requested = 3198162, Free = 3195436 (Limit = 38398432, Used = 35202996)]

Regards,
Akshath

The Vertica Database Forums

Taking more time to load from File to Vertica table.

Taking more time to load from File to Vertica table.

Re: Taking more time to load from File to Vertica table.

Re: Taking more time to load from File to Vertica table.

Re: Taking more time to load from File to Vertica table.