Data collector, unable to write data to disk: No space left
Posted: Thu Apr 03, 2014 1:42 pm
One of my nodes failed last night bring the whole cluster down. Below is the errors that I think started the problem. But the disk drive as 1.7 TB frees,so I don;t understand the No space on device error Anyone have any idea why I would get this?
2014-04-03 04:07:38.340 LowDiskSpaceCheck:0x7f14e400f8b0 [Util] <INFO> Task 'LowDiskSpaceCheck' enabled
2014-04-03 04:07:39.011 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: Success
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CatalogInfoBySecond_449827618001034.log: Too many open files
2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CpuAggregateBySecond_449826639001019.log: Too many open files
2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CpuAggregateBySecond_449826639001019.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/Errors_449780580040753.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/Errors_449780580040753.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/IoInfo_449827570334719.log: Too many open files
2014-04-03 04:07:38.340 LowDiskSpaceCheck:0x7f14e400f8b0 [Util] <INFO> Task 'LowDiskSpaceCheck' enabled
2014-04-03 04:07:39.011 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: Success
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.012 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2972: Data collector, unable to write data to disk: No space left on device
2014-04-03 04:07:39.2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CatalogInfoBySecond_449827618001034.log: Too many open files
2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CpuAggregateBySecond_449826639001019.log: Too many open files
2014-04-03 04:07:42.824 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/CpuAggregateBySecond_449826639001019.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/Errors_449780580040753.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/Errors_449780580040753.log: Too many open files
2014-04-03 04:07:42.825 nameless:0x6f470f0 <WARNING> @v_test_database_node0001: 01000/2971: Data collector, unable to open /home/dbadmin/test_database/v_test_database_node0001_catalog/DataCollector/IoInfo_449827570334719.log: Too many open files