agentStdMsg.log errors from inotify.py open StorageInfoByMin

[thatsafunnyname]

Subject: agentStdMsg.log full of errors from inotify.py from trying to open DataCollector/StorageInfoByMinute_*.log files that do not exist.

While running vertica-6.1.1-0.x86_64 on RedHat EL6.3 2.6.32-279.19.1.el6.x86_64 I noticed the gentStdMsg.log is full of errors from inotify.py, for example:

Traceback (most recent call last):
File "/opt/vertica/agent/services/inotify.py", line 221, in process_IN_MODIFY
fptr = open(fname, 'r')
IOError: [Errno 2] No such file or directory: '/data/1/catalog/vmartdb/v_vmartdb_node0001_catalog/DataCollector/StorageInfoByMinute_425738709892126.log'

Traceback (most recent call last):
File "/opt/vertica/agent/services/inotify.py", line 176, in process_IN_CREATE
fptr = open(fname, 'r')
IOError: [Errno 2] No such file or directory: '/data/1/catalog/vmartdb/v_vmartdb_node0001_catalog/DataCollector/StorageInfoByMinute_425739243639302.log'

wc -l for agentStdMsg.log for today reports 720,600 lines and is 1.8G, max line length according to wc -L is 152, the start of the file is full of nulls:

hexdump -v -n 64 /opt/vertica/log/agentStdMsg.log
0000000 0000 0000 0000 0000 0000 0000 0000 0000
0000010 0000 0000 0000 0000 0000 0000 0000 0000
0000020 0000 0000 0000 0000 0000 0000 0000 0000
0000030 0000 0000 0000 0000 0000 0000 0000 0000
0000040

tail -n 700000 /opt/vertica/log/agentStdMsg.log | grep 'File' | sort | uniq -c
4198 File "/opt/vertica/agent/services/inotify.py", line 176, in process_IN_CREATE
170802 File "/opt/vertica/agent/services/inotify.py", line 221, in process_IN_MODIFY

The vertica process was likely consuming all the CPUs (32 x Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz) on the host (a hp proliant dl380p gen8) at the time.

I suspect I may need to increase the retention policy for StorageInfoByMinute to avoid this.
https://my.vertica.com/docs/6.1.x/HTML/ ... #16155.htm

Any thoughts on this much appreciated, thanks.

SELECT GET_DATA_COLLECTOR_POLICY('StorageInfoByMinute');
-[ RECORD 1 ]-------------+-----------------------------------------
GET_DATA_COLLECTOR_POLICY | 64KB kept in memory, 256KB kept on disk.

SELECT * from data_collector where component='StorageInfoByMinute' and node_name='v_vmartdb_node0001';
-[ RECORD 1 ]----------+------------------------------------------------------------------
node_name | v_vmartdb_node0001
component | StorageInfoByMinute
table_name | dc_storage_info_by_minute
description | Storage information (Used and Free space) (historical, by minute)
access_restricted | t
in_db_log | f
in_vertica_log | f
memory_buffer_size_kb | 64
disk_size_kb | 256
record_too_big_errors | 0
lost_buffers | 2701
lost_records | 142072
retired_files | 243693
retired_records | 26309816
current_memory_records | 0
current_disk_records | 406
current_memory_bytes | 0
current_disk_bytes | 245688
first_time | 2013-06-28 10:18:00.852526-04
last_time | 2013-06-28 10:18:00.869702-04
kb_per_day | 2311603.10757301