Taking more time to load from File to Vertica table.

Moderator: NorbertKrupa

Post Reply
akshathakki
Newbie
Newbie
Posts: 4
Joined: Fri Oct 04, 2013 8:30 am

Taking more time to load from File to Vertica table.

Post by akshathakki » Thu Jul 17, 2014 2:34 pm

Hi,
I'm Using Talend Tool to Load into Vertica table.I have 25,000 files to load. Each file size is less Then 5MB.
it's taking 35 minute to load 50 files!!!!
But same Input Files if i write it to Output file (instead of vertica) it's taking less time!!
I'm using insert to load from Tool. (if less then 10MB file insert will be suffient to load data so i'm using insert instead of copy).

What i'm missing here..!! i need to change any settings!!

Regards,
Akshath

scutter
Master
Master
Posts: 302
Joined: Tue Aug 07, 2012 2:15 am

Re: Taking more time to load from File to Vertica table.

Post by scutter » Fri Jul 18, 2014 3:09 pm

If you mean that your loads are being executed as a series of individual INSERT statements, you don’t want to do that, because as you’ve observed it’s slow. Let Talend batch the loads so that they are executed using COPY. Much much faster.

—Sharon
Sharon Cutter
Vertica Consultant, Zazz Technologies LLC

akshathakki
Newbie
Newbie
Posts: 4
Joined: Fri Oct 04, 2013 8:30 am

Re: Taking more time to load from File to Vertica table.

Post by akshathakki » Mon Jul 21, 2014 6:17 am

hi sharon,
Thank you for reply. These are query i tried in my vertica database. It's taking really more time to execute.

Code: Select all

vertic=> \timing
Timing is on.
vertic=>select count(*) from Myschema.Test_Table;
 count
----------
 36866469
(1 row)

Time: First fetch (1 row):[b] 231522.939 ms.[/b] All rows formatted: 231523.017 ms

vertic=>select count(*) from Myschema.Test_Table_2;
 count
-------
     0
(1 row)

Time: First fetch (1 row): 254744.995 ms. All rows formatted: 254745.081 ms
if i'm not wrong, In vertica these queries should give output within a seconds.

Regards,
Akshath
Last edited by akshathakki on Mon Jul 21, 2014 6:20 am, edited 1 time in total.

akshathakki
Newbie
Newbie
Posts: 4
Joined: Fri Oct 04, 2013 8:30 am

Re: Taking more time to load from File to Vertica table.

Post by akshathakki » Mon Jul 21, 2014 6:18 am

Hi,
There is one more error i'm getting as follows :

Code: Select all

[Vertica][VJDBC](3587) ERROR: Insufficient resources to execute plan on pool general [Timedout waiting for resource request: Request exceeds limits: Memory(KB) Exceeded: Requested = 3198162, Free = 3195436 (Limit = 38398432, Used = 35202996)]
Regards,
Akshath

Post Reply

Return to “Vertica Data Load”