Log analyser with Big data
Posted: Thu May 12, 2016 3:24 pm
Hi,
So, I have a task - setup server for store and search log from all our servers.But have one problem - some from those logs grow up to 1.5 GB per one day.Frist - I tried use MySQL DB. Then - save logs in files and use direct access to it from LogAnaliser. Both metod have same problem - when search in it from LogAnaliser it takes to much time. For example now MySQL DB about 10 GB, file - 7 GB. But we must save logs about 3 month. Can please anybody give any idiea what metod will be better for so big data? May be MongoDB? Or something else? Or some magic with MySQL or log-files?
Thanks in advance
So, I have a task - setup server for store and search log from all our servers.But have one problem - some from those logs grow up to 1.5 GB per one day.Frist - I tried use MySQL DB. Then - save logs in files and use direct access to it from LogAnaliser. Both metod have same problem - when search in it from LogAnaliser it takes to much time. For example now MySQL DB about 10 GB, file - 7 GB. But we must save logs about 3 month. Can please anybody give any idiea what metod will be better for so big data? May be MongoDB? Or something else? Or some magic with MySQL or log-files?
Thanks in advance