Merge pull request #3051 from yichengq/doc-rafthttp-metrics

docs: doc metrics used in rafthttp package

Documentation/metrics.md

@@ -46,3 +46,21 @@ Abnormally high fsync duration (`fsync_durations_microseconds`) indicates disk i
 | snapshot_save_total_durations_microseconds | The total latency distributions of save called by snapshot | Summary |
 
 Abnormally high snapshot duration (`snapshot_save_total_durations_microseconds`) indicates disk issues and might cause the cluster to be unstable.
+
+### rafthttp
+
+| Name                              | Description                                | Type    | Labels                     |
+|-----------------------------------|--------------------------------------------|---------|----------------------------|
+| message_sent_latency_microseconds | The latency distributions of messages sent | Summary | channel, msgType, remoteID |
+| message_sent_failed_total         | The total number of failed messages sent   | Summary | channel, msgType, remoteID |
+
+
+Abnormally high message duration (`message_sent_latency_microseconds`) indicates network issues and might cause the cluster to be unstable.
+
+An increase in message failures (`message_sent_failed_total`) indicates more severe network issues and might cause the cluster to be unstable.
+
+Label `channel` is the channel to send message. `message`, `msgapp` and `msgappv2` channels use HTTP streaming, while `pipeline` does HTTP request for each message.
+
+Label `msgType` is the type of raft message. `MsgApp` is log replication message; `MsgSnap` is snapshot install message; `MsgProp` is proposal forward message; the others are used to maintain raft internal status. If you have a large snapshot, you would expect a long msgSnap sending latency. For other types of messages, you would expect low latency, which is comparable to your ping latency if you have enough network bandwidth.
+
+Label `remoteID` is the member ID of the message destination.