请选择 进入手机版 | 继续访问电脑版

技术控

    今日:4| 主题:61818
收藏本版 (1)
最新软件应用技术尽在掌握

[其他] Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

[复制链接]
甩的就是男人 发表于 2016-11-30 19:50:53
369 2

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-1-技术控-describe,features,provides,required,specific

   One of the most popular features Logz.io provides its users with is the built-in alerting mechanism. Coupled with the rich indexing and querying capabilities that are part and parcel ofElasticsearch andKibana, Logz.io Alerts is a powerful tool to have on your side in day-to-day operations.
  This article will introduce you to some of the latest updates to the feature applied over the past few weeks, namely — the ability to trigger alerts on field aggregations and the new integration with PagerDuty.
  Integrating with PagerDuty

   For starters, I’m going to describe how to integrate Logz.io with PagerDuty. The result of the steps described below is an ability to get alerted when specific conditions that you define are triggered in yourELK Stack environment via PagerDuty.
   Retrieving a PagerDuty Service Key

  We will start with a crucial element required for integrating Logz.io with PagerDuty — the service key. PagerDuty service keys are basically integration API keys that are required for integrating with PagerDuty services.
   To retrieve a service key, first log into PagerDuty and go to Configuration | Services .
  You now have the choice of either using an existing service or creating a new one. As a best practice and for the sake of order and segregation of services, I recommend creating a new service for integrating with Logz.io.
   To do this, click Add New Service (if you want to add the Logz.io integration to an existing service, select it from the list of services, go to Integrations , and click New Integration ).
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-2-技术控-describe,features,provides,required,specific

   Name the new service, and in the Integration Settings section select the Use our API directly option.
  There are some additional settings to services, such as escalation policy and incident timeouts, but for now you can do with the default settings.
   To create the service, name the new integration, and hit the Add Service button at the bottom of the page.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-3-技术控-describe,features,provides,required,specific

  The service is added, and under Integrations you will be able to see the service key needed to integrate with Logz.io.
   Creating a New Endpoint

   Logz.io allows hooking into any paging or messaging app that uses webhooks. Inthis blog post we described how to hook into Slack to get notified on events. Now, you can easily integrate with PagerDuty as well.
  You can create new endpoints, and edit existing endpoints, from defined alerts, but we will start fresh by creating a new endpoint from the Alert Endpoints page under Alerts.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-4-技术控-describe,features,provides,required,specific

  Clicking the Create Endpoint link opens the Create Endpoint dialog.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-5-技术控-describe,features,provides,required,specific

  Here, you have the choice to select what kind of endpoint you want to add. As mentioned already, you can create a new custom endpoint in which all you have to do is define the REST API and webhook used, but in our case we’re going to select the built-in integration with PagerDuty.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-6-技术控-describe,features,provides,required,specific

  Name the new endpoint and give it a description (optional). Then, enter the PagerDuty service key we created in the first step above.
   When done, click Save . The new PagerDuty endpoint is added.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-7-技术控-describe,features,provides,required,specific

  Creating a New Alert

  It now starts to get interesting. Let’s take a closer look at how to create a new alert and the different alert types we can use in Logz.io.
  In the example below, I’m analyzing Apache access logs, and have used a filtered query to have Kibana show only Apache log messages containing an error response starting from ‘400’ onwards, and only for requests from the United States.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-8-技术控-describe,features,provides,required,specific

   To create an alert for this specific query, all I have to do is hit the Create Alert button next to the query field.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-9-技术控-describe,features,provides,required,specific

  As you can see, the exact query, together with the filter I used, is defined as the query. If we wanted, we could edit this query by editing the syntax within the Query box. If our syntax includes an error, we will be notified so there’s no way of making any mistakes. A best practice we recommend is to double-check the exact query you’re using for the alert — either in the Discover tab or even by trying the query for building a visualization.
  Now, you can decide what type of aggregation to use as the alert condition. A simple example is to use a count aggregation — meaning that an alert will be triggered should the query be found a defined number of times in Elasticsearch.
  You can then configure the exact condition and threshold to use. In the example above, we’re instructing Logz.io to trigger an alert should the query be found more than 5 times and within a time period of the last hour.
  On the next page of the wizard, enter a name and description for the alert, and select a severity.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-10-技术控-describe,features,provides,required,specific

  On the final page of the wizard, you can suppress alerts for a specific time period. This acts as a kind of “snoozing” mechanism, to make sure you don’t get bombarded by a large quantity of alerts once the first alert is triggered.
  This is also the place where we decide how we want to be notified. Since we went to all that trouble of configuring a new PagerDuty endpoint, I’m going to select the drop-down menu and select it from the list of available endpoints (as mentioned, you could also create a new endpoint directly from this wizard as well).
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-11-技术控-describe,features,provides,required,specific

   Hitting Create Alert , the new alert is then added to the other alerts I have defined in the system.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-12-技术控-describe,features,provides,required,specific

  On this page, you can view your alerts, edit them and disable them if you like. In any case. From this point onwards, should the conditions you defined be fulfilled, Logz.io will trigger an alert. In PagerDuty, these are called incidents, and they will appear under the Incidents tab, on the service page.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-13-技术控-describe,features,provides,required,specific

  Using aggregations for alerts

   In Elasticsearch, metric aggregations help track and calculate metrics across fields in log messages. For example, say you’d like to monitor memory usage for your servers. Using the max aggregation, you can easily track the maximum relevant value in your logs across all the values extracted from the aggregated documents.
   Logz.io now supports creating alerts using aggregations: max , min , avg , and sum . Let’s see how this works.
   In the example below, I’m monitoring server metrics using Metricbeat. Continuing the theme used above, I’m going to create an alert using an average aggregation on the ‘ system.memory.total ’ field.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-14-技术控-describe,features,provides,required,specific

   I’m asking the Logz.io alerting engine to trigger an alert should the aggregated average value exceed ‘1000000’ during the last 12 hours (Logz.io checks if conditions are met once every 60 seconds).
  This is what the incident looks like in PagerDuty.
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-15-技术控-describe,features,provides,required,specific

  The details of the alert specify the conditions that triggered the alert in JSON format. In this case:
  
       
  • The field upon which the aggregation was performed   
  • The result aggregation value   
  • Severity level   
  • Alert description   
  • Link to the alert in the Logz.io UI  
  Grouping aggregations in alerts

  Another interesting option when creating an alert is to group together aggregations according to a specific field. Under certain circumstances, when the conditions for triggering an alert are similar across a specific field, grouping can save you the time configuring multiple alerts.
  Say, for example, you’re monitoring the average amount of used memory in your environment.
   In the example below, we’re going to define an alert that will only trigger if the aggregated average for the ‘ system.memory.used.bytes’ field equals or surpasses ‘ 2000000000 ’ for the last hour (free tip: a good way to decide what specific threshold to use is to first build a visualization for monitoring expected behavior).
   In addition, we’re asking Logz.io to group together the aggregations per host (using the ‘ beat.hostname ’ field).
  

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-16-技术控-describe,features,provides,required,specific

  Taking a look this time at the alert details in PagerDuty, we can see that the aggregations are calculated and grouped according to the IP of the server.
  1. [{
  2.   "beat.hostname" : "ip-172-31-25-148",
  3.   "system.memory.used.bytes" : "8030429184.00"
  4. }
  5. {
  6.   "beat.hostname" : "ip-172-31-21-11",
  7.   "system.memory.used.bytes" : "4636509696.00"
  8. }]
复制代码
So, instead of configuring two different alerts using a different query for each host, we grouped the aggregations together in the same alert. Again, this feature will be especially useful when the conditions for triggering an alert are identical for multiple values of a specific field.
  A Summary

   Since being introduced last year , and following both our growing understanding of how Alerts are being used and direct feedback received from our users, this feature has evolved to become an extremely powerful alerting mechanism for monitoring modern IT environments and troubleshooting events in real-time.
  We have some great stuff in the pipeline, including some new integrations for other alerting and messaging applications. Stay tuned for news!
   Logz.io is an AI-powered log analysis platform that offers the open source ELK Stack as a cloud service with machine learning technology and can be used for log analysis, IT infrastructure and application monitoring, business intelligence, and more.  Start your free trial today  !
   

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts

Integrating Logz.io with PagerDuty & Using Aggregations for Alerts-17-技术控-describe,features,provides,required,specific

   Daniel Berman
   Daniel Berman is Product Evangelist at Logz.io. He is passionate about log analytics, big data, cloud, and family and loves running, Liverpool FC, and writing about disruptive tech stuff.
ZM1220 发表于 2016-12-1 19:53:29
因为谦卑,所以高贵.因为懂得,所以慈悲;因为陌生,所以勇敢,因为距离,所以美丽.
回复 支持 反对

使用道具 举报

放肆的玫瑰 发表于 2016-12-3 22:34:03
很经典,收藏了!
回复 支持 反对

使用道具 举报

我要投稿

推荐阅读


回页顶回复上一篇下一篇回列表
手机版/c.CoLaBug.com ( 粤ICP备05003221号 | 文网文[2010]257号 | 粤公网安备 44010402000842号 )

© 2001-2017 Comsenz Inc.

返回顶部 返回列表