One of my nodes failed last night bring the whole cluster down. Below is the errors that I think started the problem. But the disk drive as 1.7 TB frees,so I don;t understand the No space on device error Anyone have any idea why I would get this?
2014-04-03 04:07:38.340 LowDiskSpaceCheck:0x7f14e400f8b0 [Util] <INFO> Task 'LowDiskSpaceCheck' enabled
2014-04-03 04:07:39.011 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: Success
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CatalogInfoBySecond_449827618001034.log: Too many open files
2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CpuAggregateBySecond_449826639001019.log: Too many open files
2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CpuAggregateBySecond_449826639001019.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/Errors_449780580040753.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/Errors_449780580040753.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/IoInfo_449827570334719.log: Too many open files
Data collector, unable to write data to disk: No space left
Moderator: NorbertKrupa
Re: Data collector, unable to write data to disk: No space left
Hi Brett,
I have the same problem.
2015-11-05 18:50:11.014 nameless:0x87cedd0 <WARNING> @v_officedw_node0001: 01000/2971: Data collector, unable to open /D/vertica/databases/catalog/officedw/v_officedw_node0001_catalog/DataCollector/BlockMemoryManagerStatistics_500064347001406.log: Too many open files
I verified the system configuration:
/sbin/sysctl fs.file-max
fs.file-max = 3068821
Did you solve your issue ?
Thank you,
Veronica
I have the same problem.
2015-11-05 18:50:11.014 nameless:0x87cedd0 <WARNING> @v_officedw_node0001: 01000/2971: Data collector, unable to open /D/vertica/databases/catalog/officedw/v_officedw_node0001_catalog/DataCollector/BlockMemoryManagerStatistics_500064347001406.log: Too many open files
I verified the system configuration:
/sbin/sysctl fs.file-max
fs.file-max = 3068821
Did you solve your issue ?
Thank you,
Veronica
- JimKnicely
- Site Admin
- Posts: 1825
- Joined: Sat Jan 21, 2012 4:58 am
- Contact:
Re: Data collector, unable to write data to disk: No space left
Try checking how many delete vectors you have (i.e.):
select node_name, node_name, projection_name deleted_row_count from delete_vectors order by deleted_row_count desc;
If you have a lot, consider purging the projections.
select node_name, node_name, projection_name deleted_row_count from delete_vectors order by deleted_row_count desc;
If you have a lot, consider purging the projections.
Jim Knicely
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.
Re: Data collector, unable to write data to disk: No space left
Thank you, Jim!
In fact, there was an other error behind this, related to the insufficient space.
In fact, there was an other error behind this, related to the insufficient space.