Announcing Pyvertica

Moderator: NorbertKrupa

Post Reply
guivert
Newbie
Newbie
Posts: 4
Joined: Thu Aug 16, 2012 10:39 am

Announcing Pyvertica

Post by guivert » Thu Aug 16, 2012 10:48 am

Hello,

We at Spilgames are using a lot of python with Vertica.

Over time we developed a Python library which currently features a high speed loader and an extendable base class to write batch importers. Some helper scripts are shipped with as well for instance to easily import csv-like formatted files into Vertica in one command. Depending on used hardware and extra processing, it is easily possible to import 100.000 records / second.

We are now making this library open, under a BSD license. You can find the source on github at https://github.com/spilgames/pyvertica and python package at http://pypi.python.org/pypi/pyvertica with its documentation at http://packages.python.org/pyvertica/ on the pypi repository.

Using it is very easy, for instance:

Code: Select all

from pyvertica.batch import VerticaBatch

batch = VerticaBatch(
   dsn='VerticaDWH',
   table_name='schema.my_table',
   truncate=True,
   column_list=['column_1', 'column_2'],
)

row_list = [
   ['row_1_val_1', 'row_1_val_2'],
   ['row_2_val_1', 'row_2_val_2'],
   ...
]

for column_data_list in row_list:
   batch.insert_list(column_data_list)

error_bool, error_file_obj = batch.get_errors()

if error_bool:
   print error_file_obj.read()

batch.commit()
Any comments, feedback or contributions are more than welcome.

Guillaume

id10t
GURU
GURU
Posts: 732
Joined: Mon Apr 16, 2012 2:44 pm

Re: Announcing Pyvertica

Post by id10t » Thu Aug 16, 2012 11:36 am

Hi!

SpillGames do not have such repo - pyvertica, but you have. Can you correct link?

PS
How do you connect to Vertica (I don't see any validation of pyODBC, unixODBC or vertica.ini or odbc.ini existence)?

guivert
Newbie
Newbie
Posts: 4
Joined: Thu Aug 16, 2012 10:39 am

Re: Announcing Pyvertica

Post by guivert » Thu Aug 16, 2012 12:48 pm

Hi,

The issue has been resolved and the repository is public again.

Regarding the connection to vertica, pyvertica just uses the system odbc configuration, which can be overridden via the ODBCINI environment variable.
You can then just pass a DSN to pyvertica.

Pyvertica itself depends on pyodbc, which will be automatically installed when you install the pyvertica package.

Guillaume

Post Reply

Return to “Python”