Hi,
I'm Using Talend Tool to Load into Vertica table.I have 25,000 files to load. Each file size is less Then 5MB.
it's taking 35 minute to load 50 files!!!!
But same Input Files if i write it to Output file (instead of vertica) it's taking less time!!
I'm using insert to load from Tool. (if less then 10MB file insert will be suffient to load data so i'm using insert instead of copy).
What i'm missing here..!! i need to change any settings!!
Regards,
Akshath
Taking more time to load from File to Vertica table.
Moderator: NorbertKrupa
-
- Newbie
- Posts: 4
- Joined: Fri Oct 04, 2013 8:30 am
Re: Taking more time to load from File to Vertica table.
If you mean that your loads are being executed as a series of individual INSERT statements, you don’t want to do that, because as you’ve observed it’s slow. Let Talend batch the loads so that they are executed using COPY. Much much faster.
—Sharon
—Sharon
Sharon Cutter
Vertica Consultant, Zazz Technologies LLC
Vertica Consultant, Zazz Technologies LLC
-
- Newbie
- Posts: 4
- Joined: Fri Oct 04, 2013 8:30 am
Re: Taking more time to load from File to Vertica table.
hi sharon,
Thank you for reply. These are query i tried in my vertica database. It's taking really more time to execute.
if i'm not wrong, In vertica these queries should give output within a seconds.
Regards,
Akshath
Thank you for reply. These are query i tried in my vertica database. It's taking really more time to execute.
Code: Select all
vertic=> \timing
Timing is on.
vertic=>select count(*) from Myschema.Test_Table;
count
----------
36866469
(1 row)
Time: First fetch (1 row):[b] 231522.939 ms.[/b] All rows formatted: 231523.017 ms
vertic=>select count(*) from Myschema.Test_Table_2;
count
-------
0
(1 row)
Time: First fetch (1 row): 254744.995 ms. All rows formatted: 254745.081 ms
Regards,
Akshath
Last edited by akshathakki on Mon Jul 21, 2014 6:20 am, edited 1 time in total.
-
- Newbie
- Posts: 4
- Joined: Fri Oct 04, 2013 8:30 am
Re: Taking more time to load from File to Vertica table.
Hi,
There is one more error i'm getting as follows :
Regards,
Akshath
There is one more error i'm getting as follows :
Code: Select all
[Vertica][VJDBC](3587) ERROR: Insufficient resources to execute plan on pool general [Timedout waiting for resource request: Request exceeds limits: Memory(KB) Exceeded: Requested = 3198162, Free = 3195436 (Limit = 38398432, Used = 35202996)]
Akshath