There are some docs on this, but it took way to much trial and error on my part to figure it out, so hopefully this helps some future person (or future me!).
The gist of this is there are two ways to exclude traces: on the agent side and on the tracer side.
Agent Side Exclusion
Can be done by ignoring resources based on their name or filtering based on their tags. One can set the DD_APM_IGNORE_RESOURCES environment variable to a set of regular expressions to ignore resources on the agent side. LIke this:
DD_APM_IGNORE_RESOURCES="health.*" # ignore health checks
Theoretically it’s possible to do multiple patterns here as a comma separated list
DD_APM_IGNORE_RESOURCES="health.*, other.*"
I had success with this on a single pattern, but limited success with multiple.
Moreover, as the docs say in somewhat hard to parse wording: if you do any sort of sample on the tracer side, the tracer may choose to send traces based on sampling rate that are dropped by the agent. If you’re sending 100% of traces, then that’s not a concern, but anything with sampling involved should likely do tracer side exclusion.
Tracer Side Exclusion
This can be done with sampling rules and the DD_TRACE_SAMPLING_RULES environment variable. These sample rules seem to be glob patterns in most tracers if they are coming from an environment variable, though in the PHP tracer, one must set DD_TRACE_SAMPLING_RULES_FORMAT=glob.
The idea here is to match undesired resources and set the sample rate to zero. For example, I wanted to ignore health checks and OPTIONS requests on a web service, so this is the ECS task definition
{
name = "DD_TRACE_SAMPLING_RULES_FORMAT"
value = "glob"
},
{
name = "DD_TRACE_SAMPLING_RULES",
value = jsonencode([
// don't want to sample any health checks
{
resource = "health.*"
sample_rate = 0
},
// ignore OPTIONS requests from the frontend
{
resource = "OPTIONS *"
sample_rate = 0
}
])
}
The advantage here is that the tracer is deciding what to send and there’s nothing I want to exclude getting sent to the tracer and then dropped.
I could definitely see the advantage of configuring the agent to drop traces, since it can be done in a single place per host. But mixing that in with sample rates probably means that tracer side is a better bet.
Christopher Davis 