Node Shutdown - Received fatal signal SIGSEGV

[Julie]

Hi all,

Today one of my nodes shut down in a three node cluster. The Vertica log for the failed node had this entry right before dying:

Code: Select all

2012-09-12 12:35:41.910 Init Session:0x2aaaadb57a60 [Main] <INFO> Handling signal: 11
2012-09-12 12:35:43.156 Init Session:0x2aaaadb57a60 [Main] <PANIC> Received fatal signal SIGSEGV.
2012-09-12 12:35:43.156 Init Session:0x2aaaadb57a60 [Main] <PANIC> Info: si_code: 2, si_pid: -1730877168, si_uid: 10924, si_addr: 0x2aac98d4e910
2012-09-12 12:35:43.157 Init Session:0x2aaaadb57a60 <WARNING> @v_inhouse01_node0001: 01000/5439: Vertica suggests allowing 1 open file per MB of memory, minimum of 65536; see 'ulimit -n'

If I run the ulimit command I have the minumum value set:

Code: Select all

[root@v01 v_inhouse01_node0001_catalog]# ulimit -n
65536

Should I increase this value?

[jpcavanaugh]

What is the limit for the user the vertica process is running as?

[JimKnicely]

Julie,

I have a similar issue... It's easily repeatable by doing this:

Code: Select all

dbadmin=> CREATE TABLE test (col1 INT);
CREATE TABLE
dbadmin=> SELECT CAST(col1 AS VARCHAR(6)),
dbadmin->        CAST(col1 AS VARCHAR(4))
dbadmin->   FROM test
dbadmin->  GROUP
dbadmin->     BY CAST(col1 AS VARCHAR(6)),
dbadmin->        CAST(col1 AS VARCHAR(4));
col1 | col1
------+------
(0 rows)

dbadmin=> INSERT INTO test VALUES (20120919);
OUTPUT
--------
1
(1 row)

dbadmin=> SELECT CAST(col1 AS VARCHAR(6)),
dbadmin->        CAST(col1 AS VARCHAR(4))
dbadmin->   FROM test
dbadmin->  GROUP
dbadmin->     BY CAST(col1 AS VARCHAR(6)),
dbadmin->        CAST(col1 AS VARCHAR(4));
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.

Doing the above causes the node to crash!

I have a case open with Vertica and will keep you updated on a solution...

PS. The query above was something similar to what a developer was writing and it should be re-written. However, no matter how bad a query is I don't think it should crash an entire node

[scutter]

These are likely too different issues. You can get the "signature" of the exception from the ErrorReport.txt in the catalog directory, to see if one crash has a similar call stack as another.

[JimKnicely]

Scutter,

No information is generated in the ErrorReport.txt file when I cause the node to crash using my example query

Thanks

UPDATE (09/14/2012): Vertica has created bug task VER-24443 for their development team to review per my issue ... viewtopic.php?f=5&t=439&p=1434#p1428