Data collector, unable to write data to disk: No space left

Moderator: NorbertKrupa

Post Reply
Brett
Beginner
Beginner
Posts: 49
Joined: Fri Oct 11, 2013 1:19 am

Data collector, unable to write data to disk: No space left

Post by Brett » Thu Apr 03, 2014 1:42 pm

One of my nodes failed last night bring the whole cluster down. Below is the errors that I think started the problem. But the disk drive as 1.7 TB frees,so I don;t understand the No space on device error :( Anyone have any idea why I would get this?

2014-04-03 04:07:38.340 LowDiskSpaceCheck:0x7f14e400f8b0 [Util] <INFO> Task 'LowDiskSpaceCheck' enabled
2014-04-03 04:07:39.011 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: Success
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CatalogInfoBySecond_449827618001034.log: Too many open files
2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CpuAggregateBySecond_449826639001019.log: Too many open files
2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CpuAggregateBySecond_449826639001019.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/Errors_449780580040753.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/Errors_449780580040753.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/IoInfo_449827570334719.log: Too many open files

vcarusi
Beginner
Beginner
Posts: 29
Joined: Mon Apr 20, 2015 11:03 am

Re: Data collector, unable to write data to disk: No space left

Post by vcarusi » Mon Nov 09, 2015 9:51 am

Hi Brett,
I have the same problem.

2015-11-05 18:50:11.014 nameless:0x87cedd0 <WARNING> @v_officedw_node0001: 01000/2971: Data collector, unable to open /D/vertica/databases/catalog/officedw/v_officedw_node0001_catalog/DataCollector/BlockMemoryManagerStatistics_500064347001406.log: Too many open files

I verified the system configuration:
/sbin/sysctl fs.file-max
fs.file-max = 3068821

Did you solve your issue ?
Thank you,
Veronica

User avatar
JimKnicely
Site Admin
Site Admin
Posts: 1825
Joined: Sat Jan 21, 2012 4:58 am
Contact:

Re: Data collector, unable to write data to disk: No space left

Post by JimKnicely » Wed Nov 11, 2015 2:23 pm

Try checking how many delete vectors you have (i.e.):

select node_name, node_name, projection_name deleted_row_count from delete_vectors order by deleted_row_count desc;

If you have a lot, consider purging the projections.
Jim Knicely

Image

Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.

vcarusi
Beginner
Beginner
Posts: 29
Joined: Mon Apr 20, 2015 11:03 am

Re: Data collector, unable to write data to disk: No space left

Post by vcarusi » Wed Nov 18, 2015 3:20 pm

Thank you, Jim!
In fact, there was an other error behind this, related to the insufficient space.

Post Reply

Return to “New to Vertica Database Administration”