Thursday, August 11, 2016

Datadog - Cassandra Number of Metrics Limit

A little trick here, datadog agent has a upper limit how how many metircs you can send through. For example:

      - instance #cassandra-localhost-7199 [WARNING] collected 350 metrics
          Warning: Number of returned metrics is too high for instance: cassandra-localhost-7199. Please read or get in touch with Datadog Support for more details. Truncating to 350 metrics.
      - Collected 350 metrics, 0 events & 0 service checks

In our situation, we have a busy Cassandra cluster and hosting 8 differnt keyspaces, yeah I know, that sounds crazy, but it is true. So each of our Cassandra instance is sending 500+ metircs! To by pass the limite, all you need to do is in your datadog cassandra.yaml file, at intance leve, set "max_returned_metrics" to a value you want.

This will work for other application yaml file as well.

No comments: