ELK Adapter enhancement to send logs to different ELK endpoints based on Org/Space

Just released, Log Analysis w logDNA [https://www.ibm.com/blogs/bluemix/2018/11/increase-observability-ibm-log-analysis-with-logdna-is-available-today-on-ibm-cloud/] supports multiple resource groups to support logging isolation in the account. Up to 32 logDNA instances can be support per account.

Hello,

We currently support enterprise application logging with ELK and have both production and non-production ELK environments to support that. The two environments are mirror images of each other and located in different data centers so that the non-prod environment can provide disaster recovery for the prod environment. Given ELK is our Bluemix logging solution we need the ability to log to non-prod ELK from our non-prod spaces and to production ELK from our production spaces. This is because logging in non-prod typically occurs at higher levels than prod, and, more importantly, production log data will have different retention requirements than non-prod log data.

With the ELK adaptor’s inability to log to multiple end points we are forced to drop another infrastructure layer in between Bluemix and ELK to provide the filtering/routing necessary to ensure that log data goes to the right ELK environment. There is obviously more cost involved with the additional infrastructure, not to mention the fact that it introduces another point of failure.

Ideally the ELK adaptor would be somewhat configurable and, at a minimum, able to route based on space name.

Please let me know if you have any additional questions.

Thank you!

David E. Spotts | Manager, Technology Services | BNSF Railway | 817-352-3838

Another consideration is that non-prod environment is used as a load testing environment by teams resulting in abnormally large volumes of logs. So, we cannot drain the non-prod logs to Prod BNSF ELK stack as that can be risk slowing down or worst case bringing down the Production ELK stack.

Ananda, We do not have the separation at this time. We have been draining all Bluemix logs (all spaces, dev, non-prod and prod) to our non-prod BNSF ELK stack. However, this is a huge risk as the Production applications in Bluemix do not have the same guarantees of availability and message loss from BNSF ELK stack as other non-Bluemix applications who are using the Production ELK stack. The Production BNSF ELK stack is highly available and the non-prod BNSF ELK stack is not guaranteed to be highly available. Also, we have limitations where security zones (separate security zone for Prod vs non-Prod) do not allow Production application components to connect to non-prod components. Thanks, Runu

Hi Ananda, we have different ELK environments (Development, Trial, Production) that corresponds to different spaces in our Bluemix environment.  Therefore, we would like to send logs from each of the Bluemix space to the corresponding ELK environments.  At this time, Bluemix logs from all spaces are being sent to only one ELK destination/environment.  We need help to configure logs from each space to sent to the correct corresponding ELK environment.

Thanks,

David Nguyen 

Hello. Thanks for adding this idea to the portal.

Would you please add some details about why you need the log destinations emanating from a single private Bluemix environment to go to multiple destinations and configured at the space level? How do you achieve this separation today? What difficulty do you run into without this ability?