# Debugging etcd Diagnosing issues in a distributed application is hard. etcd will help as much as it can - just enable these debug features using the CLI flag `-trace=*` or the config option `trace=*`. ## Logging Log verbosity can be increased to the max using either the `-vvv` CLI flag or the `very_very_verbose=true` config option. The only supported logging mode is to stdout. ## Metrics etcd itself can generate a set of metrics. These metrics represent many different internal data points that can be helpful when debugging etcd servers. #### Metrics reference Each individual metric name is prefixed with `etcd.`, where \ is the configured name of the etcd server. * `timer.appendentries.handle`: amount of time a peer takes to process an AppendEntriesRequest from the POV of the peer itself * `timer.peer..heartbeat`: amount of time a peer heartbeat operation takes from the POV of the leader that initiated that operation for peer \ * `timer.command.`: amount of time a given command took to be processed through the local server's raft state machine. This does not include time waiting on locks. #### Fetching metrics over HTTP Once tracing has been enabled on a given etcd server, all metric data is available at the server's `/debug/metrics` HTTP endpoint (i.e. `http://127.0.0.1:4001/debug/metrics`). Executing a GET HTTP command against the metrics endpoint will yield the current state of all metrics in the etcd server. #### Sending metrics to Graphite etcd supports [Graphite's Carbon plaintext protocol](https://graphite.readthedocs.org/en/latest/feeding-carbon.html#the-plaintext-protocol) - a TCP wire protocol designed for shipping metric data to an aggregator. To send metrics to a Graphite endpoint using this protocol, use of the `-graphite-host` CLI flag or the `graphite_host` config option (i.e. `graphite_host=172.17.0.19:2003`). See an [example graphite deploy script](https://github.com/coreos/etcd/contrib/graphite). #### Generating additional metrics with Collectd [Collectd](http://collectd.org/documentation.shtml) gathers metrics from the host running etcd. While these aren't metrics generated by etcd itself, it can be invaluable to compare etcd's view of the world to that of a separate process running next to etcd. See an [example collectd deploy script](https://github.com/coreos/etcd/contrib/collectd). ## Profiling etcd exposes profiling information from the Go pprof package over HTTP. The basic browsable interface is served by etcd at the `/debug/pprof` HTTP endpoint (i.e. `http://127.0.0.1:4001/debug/pprof`). For more information on using profiling tools, see http://blog.golang.org/profiling-go-programs. **NOTE**: In the following examples you need to ensure that the `./bin/etcd` is identical to the `./bin/etcd` that you are targeting (same git hash, arch, platform, etc). #### Heap memory profile ``` go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/heap ``` #### CPU profile ``` go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/profile ``` #### Blocked goroutine profile ``` go tool pprof ./bin/etcd http://127.0.0.1:4001/debug/pprof/block ```