Store into vertica using pig
Posted: Thu Jul 19, 2012 2:58 pm
Hi,
we are having a look into the vertica hadoop connector using pig.
Use case is, that we load data from hdfs and then store it into vertica. Simple tests worked fine so far.
But now... we would like to insert only changed data and new data.
To be honest - I have currently no idea how to do that. Is there a way to store the hdfs data in in tmp table and then merge this somehow with the 'Master'-table, meaning insert the new data and update the changed data?
I would be happy about any suggestions.
Thanks
Herberth
we are having a look into the vertica hadoop connector using pig.
Use case is, that we load data from hdfs and then store it into vertica. Simple tests worked fine so far.
But now... we would like to insert only changed data and new data.
To be honest - I have currently no idea how to do that. Is there a way to store the hdfs data in in tmp table and then merge this somehow with the 'Master'-table, meaning insert the new data and update the changed data?
I would be happy about any suggestions.
Thanks
Herberth