|
|
@@ -1,6 +1,50 @@
|
|
|
# Monitoring etcd
|
|
|
|
|
|
-Each etcd server exports metrics under the `/metrics` path on its client port.
|
|
|
+Each etcd server provides local monitoring information on its client port through http endpoints. The monitoring data is useful for both system health checking and cluster debugging.
|
|
|
+
|
|
|
+## Debug endpoint
|
|
|
+
|
|
|
+If `--debug` is set, the etcd server exports debugging information on its client port under the `/debug` path. Take care when setting `--debug`, since there will be degraded performance and verbose logging.
|
|
|
+
|
|
|
+The `/debug/pprof` endpoint is the standard go runtime profiling endpoint. This can be used to profile CPU, heap, mutex, and goroutine utilization. For example, here `go tool pprof` gets the top 10 functions where etcd spends its time:
|
|
|
+
|
|
|
+```sh
|
|
|
+$ go tool pprof http://localhost:2379/debug/pprof/profile
|
|
|
+Fetching profile from http://localhost:2379/debug/pprof/profile
|
|
|
+Please wait... (30s)
|
|
|
+Saved profile in /home/etcd/pprof/pprof.etcd.localhost:2379.samples.cpu.001.pb.gz
|
|
|
+Entering interactive mode (type "help" for commands)
|
|
|
+(pprof) top10
|
|
|
+310ms of 480ms total (64.58%)
|
|
|
+Showing top 10 nodes out of 157 (cum >= 10ms)
|
|
|
+ flat flat% sum% cum cum%
|
|
|
+ 130ms 27.08% 27.08% 130ms 27.08% runtime.futex
|
|
|
+ 70ms 14.58% 41.67% 70ms 14.58% syscall.Syscall
|
|
|
+ 20ms 4.17% 45.83% 20ms 4.17% github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2/hpack.huffmanDecode
|
|
|
+ 20ms 4.17% 50.00% 30ms 6.25% runtime.pcvalue
|
|
|
+ 20ms 4.17% 54.17% 50ms 10.42% runtime.schedule
|
|
|
+ 10ms 2.08% 56.25% 10ms 2.08% github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).AuthInfoFromCtx
|
|
|
+ 10ms 2.08% 58.33% 10ms 2.08% github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).Lead
|
|
|
+ 10ms 2.08% 60.42% 10ms 2.08% github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/wait.(*timeList).Trigger
|
|
|
+ 10ms 2.08% 62.50% 10ms 2.08% github.com/coreos/etcd/cmd/vendor/github.com/prometheus/client_golang/prometheus.(*MetricVec).hashLabelValues
|
|
|
+ 10ms 2.08% 64.58% 10ms 2.08% github.com/coreos/etcd/cmd/vendor/golang.org/x/net/http2.(*Framer).WriteHeaders
|
|
|
+```
|
|
|
+
|
|
|
+The `/debug/requests` endpoint gives gRPC traces and performance statistics through a web browser. For example, here is a `Range` request for the key `abc`:
|
|
|
+
|
|
|
+```
|
|
|
+When Elapsed (s)
|
|
|
+2017/08/18 17:34:51.999317 0.000244 /etcdserverpb.KV/Range
|
|
|
+17:34:51.999382 . 65 ... RPC: from 127.0.0.1:47204 deadline:4.999377747s
|
|
|
+17:34:51.999395 . 13 ... recv: key:"abc"
|
|
|
+17:34:51.999499 . 104 ... OK
|
|
|
+17:34:51.999535 . 36 ... sent: header:<cluster_id:14841639068965178418 member_id:10276657743932975437 revision:15 raft_term:17 > kvs:<key:"abc" create_revision:6 mod_revision:14 version:9 value:"asda" > count:1
|
|
|
+```
|
|
|
+
|
|
|
+## Metrics endpoint
|
|
|
+
|
|
|
+Each etcd server exports metrics under the `/metrics` path on its client port and optionally on interfaces given by `--listen-metrics-urls`.
|
|
|
+>>>>>>> 607d0762e... Documentation/op-guide: remove grafana demo link
|
|
|
|
|
|
The metrics can be fetched with `curl`:
|
|
|
|
|
|
@@ -75,8 +119,6 @@ Access: proxy
|
|
|
|
|
|
Then import the default [etcd dashboard template][template] and customize. For instance, if Prometheus data source name is `my-etcd`, the `datasource` field values in JSON also need to be `my-etcd`.
|
|
|
|
|
|
-See the [demo][demo].
|
|
|
-
|
|
|
Sample dashboard:
|
|
|
|
|
|

|
|
|
@@ -85,4 +127,3 @@ Sample dashboard:
|
|
|
[prometheus]: https://prometheus.io/
|
|
|
[grafana]: http://grafana.org/
|
|
|
[template]: ./grafana.json
|
|
|
-[demo]: http://dash.etcd.io/dashboard/db/test-etcd
|