Merge pull request #10949 from gyuho/docs

Documentation: move design docs to "Documentation", remove "docs"

CHANGELOG-3.4.md

@@ -19,7 +19,7 @@ See [code changes](https://github.com/etcd-io/etcd/compare/v3.3.0...v3.4.0) and
 - Add Raft learner: [#10725](https://github.com/etcd-io/etcd/pull/10725), [#10727](https://github.com/etcd-io/etcd/pull/10727), [#10730](https://github.com/etcd-io/etcd/pull/10730).
   - User guide: [runtime-configuration document](https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#add-a-new-member-as-learner).
   - API change: [API reference document](https://github.com/etcd-io/etcd/blob/master/Documentation/dev-guide/api_reference_v3.md).
-  - More details on implementation: [learner design document](https://github.com/etcd-io/etcd/blob/master/docs/server-learner.rst) and [implementation task list](https://github.com/etcd-io/etcd/issues/10537).
+  - More details on implementation: [learner design document](https://github.com/etcd-io/etcd/blob/master/Documentation/learning/design-learner.md) and [implementation task list](https://github.com/etcd-io/etcd/issues/10537).
 - Rewrite [client balancer](https://github.com/etcd-io/etcd/pull/9860) with [new gRPC balancer interface](https://github.com/etcd-io/etcd/issues/9106).
   - Upgrade [gRPC to v1.22.0](https://github.com/etcd-io/etcd/pull/10911).
   - Improve [client balancer failover against secure endpoints](https://github.com/etcd-io/etcd/pull/10911).

Documentation/docs.md

@@ -68,8 +68,10 @@ To learn more about the concepts and internals behind etcd, read the following p
  - [Understand data model][data_model]
  - [Understand APIs][understand_apis]
  - [Glossary][glossary]
- - Internals
-   - [Auth subsystem][auth_design]
+ - Design
+   - [Auth subsystem][design-auth-v3]
+   - [Client][design-client]
+   - [Learner][design-learner]
 
 [api_ref]: dev-guide/api_reference_v3.md
 [api_concurrency_ref]: dev-guide/api_concurrency_reference_v3.md
@@ -109,6 +111,8 @@ To learn more about the concepts and internals behind etcd, read the following p
 [aws_platform]: platforms/aws.md
 [experimental]: dev-guide/experimental_apis.md
 [authentication]: op-guide/authentication.md
-[auth_design]: learning/auth_design.md
+[design-auth-v3]: learning/design-auth-v3.md
+[design-client]: learning/design-client.md
+[design-learner]: learning/design-learner.md
 [tuning]: tuning.md
 [upgrading]: upgrades/upgrading-etcd.md

Documentation/learning/design-client.md

@@ -0,0 +1,138 @@
+---
+title: etcd client design
+---
+
+etcd Client Design
+==================
+
+*Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)*
+
+
+Introduction
+============
+
+etcd server has proven its robustness with years of failure injection testing. Most complex application logic is already handled by etcd server and its data stores (e.g. cluster membership is transparent to clients, with Raft-layer forwarding proposals to leader). Although server components are correct, its composition with client requires a different set of intricate protocols to guarantee its correctness and high availability under faulty conditions. Ideally, etcd server provides one logical cluster view of many physical machines, and client implements automatic failover between replicas. This documents client architectural decisions and its implementation details.
+
+
+Glossary
+========
+
+*clientv3*: etcd Official Go client for etcd v3 API.
+
+*clientv3-grpc1.0*: Official client implementation, with [`grpc-go v1.0.x`](https://github.com/grpc/grpc-go/releases/tag/v1.0.0), which is used in latest etcd v3.1.
+
+*clientv3-grpc1.7*: Official client implementation, with [`grpc-go v1.7.x`](https://github.com/grpc/grpc-go/releases/tag/v1.7.0), which is used in latest etcd v3.2 and v3.3.
+
+*clientv3-grpc1.23*: Official client implementation, with [`grpc-go v1.23.x`](https://github.com/grpc/grpc-go/releases/tag/v1.23.0), which is used in latest etcd v3.4.
+
+*Balancer*: etcd client load balancer that implements retry and failover mechanism. etcd client should automatically balance loads between multiple endpoints.
+
+*Endpoints*: A list of etcd server endpoints that clients can connect to. Typically, 3 or 5 client URLs of an etcd cluster.
+
+*Pinned endpoint*: When configured with multiple endpoints, <= v3.3 client balancer chooses only one endpoint to establish a TCP connection, in order to conserve total open connections to etcd cluster. In v3.4, balancer round-robins pinned endpoints for every request, thus distributing loads more evenly.
+
+*Client Connection*: TCP connection that has been established to an etcd server, via gRPC Dial.
+
+*Sub Connection*: gRPC SubConn interface. Each sub-connection contains a list of addresses. Balancer creates a SubConn from a list of resolved addresses. gRPC ClientConn can map to multiple SubConn (e.g. example.com resolves to `10.10.10.1` and `10.10.10.2` of two sub-connections). etcd v3.4 balancer employs internal resolver to establish one sub-connection for each endpoint.
+
+*Transient disconnect*: When gRPC server returns a status error of [`code Unavailable`](https://godoc.org/google.golang.org/grpc/codes#Code).
+
+
+Client Requirements
+===================
+
+*Correctness*. Requests may fail in the presence of server faults. However, it never violates consistency guarantees: global ordering properties, never write corrupted data, at-most once semantics for mutable operations, watch never observes partial events, and so on.
+
+*Liveness*. Servers may fail or disconnect briefly. Clients should make progress in either way. Clients should [never deadlock](https://github.com/etcd-io/etcd/issues/8980) waiting for a server to come back from offline, unless configured to do so. Ideally, clients detect unavailable servers with HTTP/2 ping and failover to other nodes with clear error messages.
+
+*Effectiveness*. Clients should operate effectively with minimum resources: previous TCP connections should be [gracefully closed](https://github.com/etcd-io/etcd/issues/9212) after endpoint switch. Failover mechanism should effectively predict the next replica to connect, without wastefully retrying on failed nodes.
+
+*Portability*. Official client should be clearly documented and its implementation be applicable to other language bindings. Error handling between different language bindings should be consistent. Since etcd is fully committed to gRPC, implementation should be closely aligned with gRPC long-term design goals (e.g. pluggable retry policy should be compatible with [gRPC retry](https://github.com/grpc/proposal/blob/master/A6-client-retries.md)). Upgrades between two client versions should be non-disruptive.
+
+
+Client Overview
+===============
+
+etcd client implements the following components:
+
+* balancer that establishes gRPC connections to an etcd cluster,
+* API client that sends RPCs to an etcd server, and
+* error handler that decides whether to retry a failed request or switch endpoints.
+
+Languages may differ in how to establish an initial connection (e.g. configure TLS), how to encode and send Protocol Buffer messages to server, how to handle stream RPCs, and so on. However, errors returned from etcd server will be the same. So should be error handling and retry policy.
+
+For example, etcd server may return `"rpc error: code = Unavailable desc = etcdserver: request timed out"`, which is transient error that expects retries. Or return `rpc error: code = InvalidArgument desc = etcdserver: key is not provided`, which means request was invalid and should not be retried. Go client can parse errors with `google.golang.org/grpc/status.FromError`, and Java client with `io.grpc.Status.fromThrowable`.
+
+
+clientv3-grpc1.0: Balancer Overview
+-----------------------------------
+
+`clientv3-grpc1.0` maintains multiple TCP connections when configured with multiple etcd endpoints. Then pick one address and use it to send all client requests. The pinned address is maintained until the client object is closed (see *Figure 1*). When the client receives an error, it randomly picks another and retries.
+
+![client-balancer-figure-01.png](img/client-balancer-figure-01.png)
+
+
+clientv3-grpc1.0: Balancer Limitation
+-------------------------------------
+
+`clientv3-grpc1.0` opening multiple TCP connections may provide faster balancer failover but requires more resources. The balancer does not understand node’s health status or cluster membership. So, it is possible that balancer gets stuck with one failed or partitioned node.
+
+
+clientv3-grpc1.7: Balancer Overview
+------------------------------------
+
+`clientv3-grpc1.7` maintains only one TCP connection to a chosen etcd server. When given multiple cluster endpoints, a client first tries to connect to them all. As soon as one connection is up, balancer pins the address, closing others (see *Figure 2*). The pinned address is to be maintained until the client object is closed. An error, from server or client network fault, is sent to client error handler (see *Figure 3*).
+
+![client-balancer-figure-02.png](img/client-balancer-figure-02.png)
+
+![client-balancer-figure-03.png](img/client-balancer-figure-03.png)
+
+The client error handler takes an error from gRPC server, and decides whether to retry on the same endpoint, or to switch to other addresses, based on the error code and message (see *Figure 4* and *Figure 5*).
+
+![client-balancer-figure-04.png](img/client-balancer-figure-04.png)
+
+![client-balancer-figure-05.png](img/client-balancer-figure-05.png)
+
+Stream RPCs, such as Watch and KeepAlive, are often requested with no timeouts. Instead, client can send periodic HTTP/2 pings to check the status of a pinned endpoint; if the server does not respond to the ping, balancer switches to other endpoints (see *Figure 6*).
+
+![client-balancer-figure-06.png](img/client-balancer-figure-06.png)
+
+
+clientv3-grpc1.7: Balancer Limitation
+-------------------------------------
+
+`clientv3-grpc1.7` balancer sends HTTP/2 keepalives to detect disconnects from streaming requests. It is a simple gRPC server ping mechanism and does not reason about cluster membership, thus unable to detect network partitions. Since partitioned gRPC server can still respond to client pings, balancer may get stuck with a partitioned node. Ideally, keepalive ping detects partition and triggers endpoint switch, before request time-out (see [etcd#8673](https://github.com/etcd-io/etcd/issues/8673) and *Figure 7*).
+
+![client-balancer-figure-07.png](img/client-balancer-figure-07.png)
+
+`clientv3-grpc1.7` balancer maintains a list of unhealthy endpoints. Disconnected addresses are added to “unhealthy” list, and considered unavailable until after wait duration, which is hard coded as dial timeout with default value 5-second. Balancer can have false positives on which endpoints are unhealthy. For instance, endpoint A may come back right after being blacklisted, but still unusable for next 5 seconds (see *Figure 8*).
+
+`clientv3-grpc1.0` suffered the same problems above.
+
+![client-balancer-figure-08.png](img/client-balancer-figure-08.png)
+
+Upstream gRPC Go had already migrated to new balancer interface. For example, `clientv3-grpc1.7` underlying balancer implementation uses new gRPC balancer and tries to be consistent with old balancer behaviors. While its compatibility has been maintained reasonably well, etcd client still [suffered from subtle breaking changes](https://github.com/grpc/grpc-go/issues/1649). Furthermore, gRPC maintainer recommends to [not rely on the old balancer interface](https://github.com/grpc/grpc-go/issues/1942#issuecomment-375368665). In general, to get better support from upstream, it is best to be in sync with latest gRPC releases. And new features, such as retry policy, may not be backported to gRPC 1.7 branch. Thus, both etcd server and client must migrate to latest gRPC versions.
+
+
+clientv3-grpc1.23: Balancer Overview
+------------------------------------
+
+`clientv3-grpc1.7` is so tightly coupled with old gRPC interface, that every single gRPC dependency upgrade broke client behavior. Majority of development and debugging efforts were devoted to fixing those client behavior changes. As a result, its implementation has become overly complicated with bad assumptions on server connectivities.
+
+The primary goal of `clientv3-grpc1.23` is to simplify balancer failover logic; rather than maintaining a list of unhealthy endpoints, which may be stale, simply roundrobin to the next endpoint whenever client gets disconnected from the current endpoint. It does not assume endpoint status. Thus, no more complicated status tracking is needed (see *Figure 8* and above). Upgrading to `clientv3-grpc1.23` should be no issue; all changes were internal while keeping all the backward compatibilities.
+
+Internally, when given multiple endpoints, `clientv3-grpc1.23` creates multiple sub-connections (one sub-connection per each endpoint), while `clientv3-grpc1.7` creates only one connection to a pinned endpoint (see *Figure 9*). For instance, in 5-node cluster, `clientv3-grpc1.23` balancer would require 5 TCP connections, while `clientv3-grpc1.7` only requires one. By preserving the pool of TCP connections, `clientv3-grpc1.23` may consume more resources but provide more flexible load balancer with better failover performance. The default balancing policy is round robin but can be easily extended to support other types of balancers (e.g. power of two, pick leader, etc.). `clientv3-grpc1.23` uses gRPC resolver group and implements balancer picker policy, in order to delegate complex balancing work to upstream gRPC. On the other hand, `clientv3-grpc1.7` manually handles each gRPC connection and balancer failover, which complicates the implementation. `clientv3-grpc1.23` implements retry in the gRPC interceptor chain that automatically handles gRPC internal errors and enables more advanced retry policies like backoff, while `clientv3-grpc1.7` manually interprets gRPC errors for retries.
+
+![client-balancer-figure-09.png](img/client-balancer-figure-09.png)
+
+
+clientv3-grpc1.23: Balancer Limitation
+--------------------------------------
+
+Improvements can be made by caching the status of each endpoint. For instance, balancer can ping each server in advance to maintain a list of healthy candidates, and use this information when doing round-robin. Or when disconnected, balancer can prioritize healthy endpoints. This may complicate the balancer implementation, thus can be addressed in later versions.
+
+Client-side keepalive ping still does not reason about network partitions. Streaming request may get stuck with a partitioned node. Advanced health checking service need to be implemented to understand the cluster membership (see [etcd#8673](https://github.com/etcd-io/etcd/issues/8673) for more detail).
+
+![client-balancer-figure-07.png](img/client-balancer-figure-07.png)
+
+Currently, retry logic is handled manually as an interceptor. This may be simplified via [official gRPC retries](https://github.com/grpc/proposal/blob/master/A6-client-retries.md).

docs/server-learner.rst → Documentation/learning/design-learner.md

@@ -1,15 +1,11 @@
-.. _server-learner:
+---
+title: etcd learner design
+---
 
+etcd Learner
+============
 
-Learner
-#######
-
-
-:Authors:
-    Gyuho Lee (github.com/gyuho, *Amazon.com*),
-    Joe Betz (github.com/jpbetz, *Google Inc.*)
-
-:Version: September 7, 2018 (last update October 28, 2018)
+*Gyuho Lee (github.com/gyuho, Amazon Web Services, Inc.), Joe Betz (github.com/jpbetz, Google Inc.)*
 
 
 Background
@@ -19,93 +15,67 @@ Membership reconfiguration has been one of the biggest operational challenges. L
 
 A newly joined etcd member starts with no data, thus demanding more updates from leader until it catches up with leader’s logs. Then leader’s network is more likely to be overloaded, blocking or dropping leader heartbeats to followers. In such case, a follower may election-timeout to start a new leader election. That is, a cluster with a new member is more vulnerable to leader election. Both leader election and the subsequent update propagation to the new member are prone to causing periods of cluster unavailability (see *Figure 1*).
 
-.. image:: img/server-learner-figure-01.png
-    :align: center
-    :alt: server-learner-figure-01
+![server-learner-figure-01](img/server-learner-figure-01.png)
 
 What if network partition happens? It depends on leader partition. If the leader still maintains the active quorum, the cluster would continue to operate (see *Figure 2*).
 
-.. image:: img/server-learner-figure-02.png
-    :align: center
-    :alt: server-learner-figure-02
+![server-learner-figure-02](img/server-learner-figure-02.png)
 
 What if the leader becomes isolated from the rest of the cluster? Leader monitors progress of each follower. When leader loses connectivity from the quorum, it reverts back to follower which will affect the cluster availability (see *Figure 3*).
 
-.. image:: img/server-learner-figure-03.png
-    :align: center
-    :alt: server-learner-figure-03
+![server-learner-figure-03](img/server-learner-figure-03.png)
 
 When a new node is added to 3 node cluster, the cluster size becomes 4 and the quorum size becomes 3. What if a new node had joined the cluster, and then network partition happens? It depends on which partition the new member gets located after partition. If the new node happens to be located in the same partition as leader’s, the leader still maintains the active quorum of 3. No leadership election happens, and no cluster availability gets affected (see *Figure 4*).
 
-.. image:: img/server-learner-figure-04.png
-    :align: center
-    :alt: server-learner-figure-04
+![server-learner-figure-04](img/server-learner-figure-04.png)
 
 If the cluster is 2-and-2 partitioned, then neither of partition maintains the quorum of 3. In this case, leadership election happens (see *Figure 5*).
 
-.. image:: img/server-learner-figure-05.png
-    :align: center
-    :alt: server-learner-figure-05
+![server-learner-figure-05](img/server-learner-figure-05.png)
 
 What if network partition happens first, and then a new member gets added? A partitioned 3-node cluster already has one disconnected follower. When a new member is added, the quorum changes from 2 to 3. Now, this cluster has only 2 active nodes out 4, thus losing quorum and starting a new leadership election (see *Figure 6*).
 
-.. image:: img/server-learner-figure-06.png
-    :align: center
-    :alt: server-learner-figure-06
+![server-learner-figure-06](img/server-learner-figure-06.png)
 
 Since member add operation can change the size of quorum, it is always recommended to “member remove” first to replace an unhealthy node.
 
 Adding a new member to a 1-node cluster changes the quorum size to 2, immediately causing a leader election when the previous leader finds out quorum is not active. This is because “member add” operation is a 2-step process where user needs to apply “member add” command first, and then starts the new node process (see *Figure 7*).
 
-.. image:: img/server-learner-figure-07.png
-    :align: center
-    :alt: server-learner-figure-07
+![server-learner-figure-07](img/server-learner-figure-07.png)
 
 An even worse case is when an added member is misconfigured. Membership reconfiguration is a two-step process: “etcdctl member add” and starting an etcd server process with the given peer URL. That is, “member add” command is applied regardless of URL, even when the URL value is invalid. If the first step is applied with invalid URLs, the second step cannot even start the new etcd. Once the cluster loses quorum, there is no way to revert the membership change (see *Figure 8*).
 
-.. image:: img/server-learner-figure-08.png
-    :align: center
-    :alt: server-learner-figure-08
+![server-learner-figure-08](img/server-learner-figure-08.png)
 
 Same applies to a multi-node cluster. For example, the cluster has two members down (one is failed, the other is misconfigured) and two members up, but now it requires at least 3 votes to change the cluster membership (see *Figure 9*).
 
-.. image:: img/server-learner-figure-09.png
-    :align: center
-    :alt: server-learner-figure-09
+![server-learner-figure-09](img/server-learner-figure-09.png)
 
-As seen above, a simple misconfiguration can fail the whole cluster into an inoperative state. In such case, an operator need manually recreate the cluster with ``etcd --force-new-cluster`` flag. As etcd has become a mission-critical service for Kubernetes, even the slightest outage may have significant impact on users. What can we better to make etcd such operations easier? Among other things, leader election is most critical to cluster availability: Can we make membership reconfiguration less disruptive by not changing the size of quorum? Can a new node be idle, only requesting the minimum updates from leader, until it catches up? Can membership misconfiguration be always reversible and handled in a more secure way (wrong member add command run should never fail the cluster)? Should an user worry about network topology when adding a new member? Can member add API work regardless of the location of nodes and ongoing network partitions?
+As seen above, a simple misconfiguration can fail the whole cluster into an inoperative state. In such case, an operator need manually recreate the cluster with `etcd --force-new-cluster` flag. As etcd has become a mission-critical service for Kubernetes, even the slightest outage may have significant impact on users. What can we better to make etcd such operations easier? Among other things, leader election is most critical to cluster availability: Can we make membership reconfiguration less disruptive by not changing the size of quorum? Can a new node be idle, only requesting the minimum updates from leader, until it catches up? Can membership misconfiguration be always reversible and handled in a more secure way (wrong member add command run should never fail the cluster)? Should an user worry about network topology when adding a new member? Can member add API work regardless of the location of nodes and ongoing network partitions?
 
 Raft Learner
 ============
 
-In order to mitigate such availability gaps in the previous section, `Raft §4.2.1 <https://github.com/ongardie/dissertation/blob/master/stanford.pdf>`_ introduces a new node state “Learner”, which joins the cluster as a **non-voting member** until it catches up to leader’s logs.
+In order to mitigate such availability gaps in the previous section, [Raft §4.2.1](https://github.com/ongardie/dissertation/blob/master/stanford.pdf) introduces a new node state “Learner”, which joins the cluster as a **non-voting member** until it catches up to leader’s logs.
 
 Features in v3.4
 ----------------
 
-An operator should do the minimum amount of work possible to add a new learner node. ``member add --learner`` command to add a new learner, which joins cluster as a non-voting member but still receives all data from leader (see *Figure 10*).
+An operator should do the minimum amount of work possible to add a new learner node. `member add --learner` command to add a new learner, which joins cluster as a non-voting member but still receives all data from leader (see *Figure 10*).
 
-.. image:: img/server-learner-figure-10.png
-    :align: center
-    :alt: server-learner-figure-10
+![server-learner-figure-10](img/server-learner-figure-10.png)
 
-When a learner has caught up with leader’s progress, the learner can be promoted to a voting member using ``member promote`` API, which then counts towards the quorum (see *Figure 11*).
+When a learner has caught up with leader’s progress, the learner can be promoted to a voting member using `member promote` API, which then counts towards the quorum (see *Figure 11*).
 
-.. image:: img/server-learner-figure-11.png
-    :align: center
-    :alt: server-learner-figure-11
+![server-learner-figure-11](img/server-learner-figure-11.png)
 
 etcd server validates promote request to ensure its operational safety. Only after its log has caught up to leader’s can learner be promoted to a voting member (see *Figure 12*).
 
-.. image:: img/server-learner-figure-12.png
-    :align: center
-    :alt: server-learner-figure-12
+![server-learner-figure-12](img/server-learner-figure-12.png)
 
 Learner only serves as a standby node until promoted: Leadership cannot be transferred to learner. Learner rejects client reads and writes (client balancer should not route requests to learner). Which means learner does not need issue Read Index requests to leader. Such limitation simplifies the initial learner implementation in v3.4 release (see *Figure 13*).
 
-.. image:: img/server-learner-figure-13.png
-    :align: center
-    :alt: server-learner-figure-13
+![server-learner-figure-13](img/server-learner-figure-13.png)
 
 In addition, etcd limits the total number of learners that a cluster can have, and avoids overloading the leader with log replication. Learner never promotes itself. While etcd provides learner status information and safety checks, cluster operator must make the final decision whether to promote learner or not.
 
@@ -130,20 +100,20 @@ Appendix: Learner Implementation in v3.4
 
 *Expose "Learner" node type to "MemberAdd" API.*
 
-etcd client adds a flag to “MemberAdd” API for learner node. And etcd server handler applies membership change entry with ``pb.ConfChangeAddLearnerNode`` type. Once the command has been applied, a server joins the cluster with ``etcd --initial-cluster-state=existing`` flag. This learner node can neither vote nor count as quorum.
+etcd client adds a flag to “MemberAdd” API for learner node. And etcd server handler applies membership change entry with `pb.ConfChangeAddLearnerNode` type. Once the command has been applied, a server joins the cluster with `etcd --initial-cluster-state=existing` flag. This learner node can neither vote nor count as quorum.
 
 etcd server must not transfer leadership to learner, since it may still lag behind and does not count as quorum. etcd server limits the number of learners that cluster can have to one: the more learners we have, the more data the leader has to propagate. Clients may talk to learner node, but learner rejects all requests other than serializable read and member status API. This is for simplicity of initial implementation. In the future, learner can be extended as a read-only server that continuously mirrors cluster data. Client balancer must provide helper function to exclude learner node endpoint. Otherwise, request sent to learner may fail. Client sync member call should factor into learner node type. So should client endpoints update call.
 
-``MemberList`` and ``MemberStatus`` responses should indicate which node is learner.
+`MemberList` and `MemberStatus` responses should indicate which node is learner.
 
 *Add "MemberPromote" API.*
 
-Internally in Raft, second ``MemberAdd`` call to learner node promotes it to a voting member. Leader maintains the progress of each follower and learner. If learner has not completed its snapshot message, reject promote request. Only accept promote request if and only if: The learner node is in a healthy state. The learner is in sync with leader or the delta is within the threshold (e.g. the number of entries to replicate to learner is less than 1/10 of snapshot count, which means it is less likely that even after promotion leader would not need send snapshot to the learner). All these logic are hard-coded in ``etcdserver`` package and not configurable.
+Internally in Raft, second `MemberAdd` call to learner node promotes it to a voting member. Leader maintains the progress of each follower and learner. If learner has not completed its snapshot message, reject promote request. Only accept promote request if and only if: The learner node is in a healthy state. The learner is in sync with leader or the delta is within the threshold (e.g. the number of entries to replicate to learner is less than 1/10 of snapshot count, which means it is less likely that even after promotion leader would not need send snapshot to the learner). All these logic are hard-coded in `etcdserver` package and not configurable.
 
 Reference
 =========
 
-- Original github issue: `issues#9161 <https://github.com/etcd-io/etcd/issues/9161>`_
-- Use case: `issues#3715 <https://github.com/etcd-io/etcd/issues/3715>`_
-- Use case: `issues#8888 <https://github.com/etcd-io/etcd/issues/8888>`_
-- Use case: `issues#10114 <https://github.com/etcd-io/etcd/issues/10114>`_
+- Original github issue: [etcd#9161](https://github.com/etcd-io/etcd/issues/9161)
+- Use case: [etcd#3715](https://github.com/etcd-io/etcd/issues/3715)
+- Use case: [etcd#8888](https://github.com/etcd-io/etcd/issues/8888)
+- Use case: [etcd#10114](https://github.com/etcd-io/etcd/issues/10114)

docs/.gitignore

@@ -1,2 +0,0 @@
-_build
-

docs/Makefile

@@ -1,20 +0,0 @@
-# Minimal makefile for Sphinx documentation
-#
-
-# You can set these variables from the command line.
-SPHINXOPTS    =
-SPHINXBUILD   = sphinx-build
-SPHINXPROJ    = etcd
-SOURCEDIR     = .
-BUILDDIR      = _build
-
-# Put it first so that "make" without argument is like "make help".
-help:
-	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
-
-.PHONY: help Makefile
-
-# Catch-all target: route all unknown targets to Sphinx using the new
-# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
-%: Makefile
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/README.md

@@ -1,7 +0,0 @@
-#### etcd Documentation
-
-Visit https://etcd.io for latest docs.
-
-#### Build Documentation
-
-See the website tooling repo https://github.com/etcd-io/website/blob/master/README.md

docs/client-architecture.rst

@@ -1,158 +0,0 @@
-.. _client-architecture:
-
-
-Client Architecture
-###################
-
-
-:Authors:
-    Gyuho Lee (github.com/gyuho, *Amazon.com*),
-    Joe Betz (github.com/jpbetz, *Google Inc.*)
-
-:Version: August 28, 2018
-
-
-Introduction
-============
-
-etcd server has proven its robustness with years of failure injection testing. Most complex application logic is already handled by etcd server and its data stores (e.g. cluster membership is transparent to clients, with Raft-layer forwarding proposals to leader). Although server components are correct, its composition with client requires a different set of intricate protocols to guarantee its correctness and high availability under faulty conditions. Ideally, etcd server provides one logical cluster view of many physical machines, and client implements automatic failover between replicas. This documents client architectural decisions and its implementation details.
-
-
-Glossary
-========
-
-*clientv3*: etcd Official Go client for etcd v3 API.
-
-*clientv3-grpc1.0*: Official client implementation, with `grpc-go v1.0.x <https://github.com/grpc/grpc-go/releases/tag/v1.0.0>`_, which is used in latest etcd v3.1.
-
-*clientv3-grpc1.7*: Official client implementation, with `grpc-go v1.7.x <https://github.com/grpc/grpc-go/releases/tag/v1.7.0>`_, which is used in latest etcd v3.2 and v3.3.
-
-*clientv3-grpc1.14*: Official client implementation, with `grpc-go v1.14.x <https://github.com/grpc/grpc-go/releases/tag/v1.14.0>`_, which is used in latest etcd v3.4.
-
-*Balancer*: etcd client load balancer that implements retry and failover mechanism. etcd client should automatically balance loads between multiple endpoints.
-
-*Endpoints*: A list of etcd server endpoints that clients can connect to. Typically, 3 or 5 client URLs of an etcd cluster.
-
-*Pinned endpoint*: When configured with multiple endpoints, <= v3.3 client balancer chooses only one endpoint to establish a TCP connection, in order to conserve total open connections to etcd cluster. In v3.4, balancer round-robins pinned endpoints for every request, thus distributing loads more evenly.
-
-*Client Connection*: TCP connection that has been established to an etcd server, via gRPC Dial.
-
-*Sub Connection*: gRPC SubConn interface. Each sub-connection contains a list of addresses. Balancer creates a SubConn from a list of resolved addresses. gRPC ClientConn can map to multiple SubConn (e.g. example.com resolves to ``10.10.10.1`` and ``10.10.10.2`` of two sub-connections). etcd v3.4 balancer employs internal resolver to establish one sub-connection for each endpoint.
-
-*Transient disconnect*: When gRPC server returns a status error of `code Unavailable <https://godoc.org/google.golang.org/grpc/codes#Code>`_.
-
-
-Client Requirements
-===================
-
-*Correctness*. Requests may fail in the presence of server faults. However, it never violates consistency guarantees: global ordering properties, never write corrupted data, at-most once semantics for mutable operations, watch never observes partial events, and so on.
-
-*Liveness*. Servers may fail or disconnect briefly. Clients should make progress in either way. Clients should `never deadlock <https://github.com/etcd-io/etcd/issues/8980>`_ waiting for a server to come back from offline, unless configured to do so. Ideally, clients detect unavailable servers with HTTP/2 ping and failover to other nodes with clear error messages.
-
-*Effectiveness*. Clients should operate effectively with minimum resources: previous TCP connections should be `gracefully closed <https://github.com/etcd-io/etcd/issues/9212>`_ after endpoint switch. Failover mechanism should effectively predict the next replica to connect, without wastefully retrying on failed nodes.
-
-*Portability*. Official client should be clearly documented and its implementation be applicable to other language bindings. Error handling between different language bindings should be consistent. Since etcd is fully committed to gRPC, implementation should be closely aligned with gRPC long-term design goals (e.g. pluggable retry policy should be compatible with `gRPC retry <https://github.com/grpc/proposal/blob/master/A6-client-retries.md>`_). Upgrades between two client versions should be non-disruptive.
-
-
-Client Overview
-===============
-
-etcd client implements the following components:
-
-* balancer that establishes gRPC connections to an etcd cluster,
-* API client that sends RPCs to an etcd server, and
-* error handler that decides whether to retry a failed request or switch endpoints.
-
-Languages may differ in how to establish an initial connection (e.g. configure TLS), how to encode and send Protocol Buffer messages to server, how to handle stream RPCs, and so on. However, errors returned from etcd server will be the same. So should be error handling and retry policy.
-
-For example, etcd server may return ``"rpc error: code = Unavailable desc = etcdserver: request timed out"``, which is transient error that expects retries. Or return `rpc error: code = InvalidArgument desc = etcdserver: key is not provided`, which means request was invalid and should not be retried. Go client can parse errors with ``google.golang.org/grpc/status.FromError``, and Java client with ``io.grpc.Status.fromThrowable``.
-
-
-clientv3-grpc1.0: Balancer Overview
------------------------------------
-
-``clientv3-grpc1.0`` maintains multiple TCP connections when configured with multiple etcd endpoints. Then pick one address and use it to send all client requests. The pinned address is maintained until the client object is closed (see *Figure 1*). When the client receives an error, it randomly picks another and retries.
-
-.. image:: img/client-architecture-balancer-figure-01.png
-    :align: center
-    :alt: client-architecture-balancer-figure-01
-
-
-clientv3-grpc1.0: Balancer Limitation
--------------------------------------
-
-``clientv3-grpc1.0`` opening multiple TCP connections may provide faster balancer failover but requires more resources. The balancer does not understand node’s health status or cluster membership. So, it is possible that balancer gets stuck with one failed or partitioned node.
-
-
-clientv3-grpc1.7: Balancer Overview
-------------------------------------
-
-``clientv3-grpc1.7`` maintains only one TCP connection to a chosen etcd server. When given multiple cluster endpoints, a client first tries to connect to them all. As soon as one connection is up, balancer pins the address, closing others (see *Figure 2*). The pinned address is to be maintained until the client object is closed. An error, from server or client network fault, is sent to client error handler (see *Figure 3*).
-
-.. image:: img/client-architecture-balancer-figure-02.png
-    :align: center
-    :alt: client-architecture-balancer-figure-02
-
-.. image:: img/client-architecture-balancer-figure-03.png
-    :align: center
-    :alt: client-architecture-balancer-figure-03
-
-The client error handler takes an error from gRPC server, and decides whether to retry on the same endpoint, or to switch to other addresses, based on the error code and message (see *Figure 4* and *Figure 5*).
-
-.. image:: img/client-architecture-balancer-figure-04.png
-    :align: center
-    :alt: client-architecture-balancer-figure-04
-
-.. image:: img/client-architecture-balancer-figure-05.png
-    :align: center
-    :alt: client-architecture-balancer-figure-05
-
-Stream RPCs, such as Watch and KeepAlive, are often requested with no timeouts. Instead, client can send periodic HTTP/2 pings to check the status of a pinned endpoint; if the server does not respond to the ping, balancer switches to other endpoints (see *Figure 6*).
-
-.. image:: img/client-architecture-balancer-figure-06.png
-    :align: center
-    :alt: client-architecture-balancer-figure-06
-
-
-clientv3-grpc1.7: Balancer Limitation
--------------------------------------
-
-``clientv3-grpc1.7`` balancer sends HTTP/2 keepalives to detect disconnects from streaming requests. It is a simple gRPC server ping mechanism and does not reason about cluster membership, thus unable to detect network partitions. Since partitioned gRPC server can still respond to client pings, balancer may get stuck with a partitioned node. Ideally, keepalive ping detects partition and triggers endpoint switch, before request time-out (see `issue#8673 <https://github.com/etcd-io/etcd/issues/8673>`_ and *Figure 7*).
-
-.. image:: img/client-architecture-balancer-figure-07.png
-    :align: center
-    :alt: client-architecture-balancer-figure-07
-
-``clientv3-grpc1.7`` balancer maintains a list of unhealthy endpoints. Disconnected addresses are added to “unhealthy” list, and considered unavailable until after wait duration, which is hard coded as dial timeout with default value 5-second. Balancer can have false positives on which endpoints are unhealthy. For instance, endpoint A may come back right after being blacklisted, but still unusable for next 5 seconds (see *Figure 8*).
-
-``clientv3-grpc1.0`` suffered the same problems above.
-
-.. image:: img/client-architecture-balancer-figure-08.png
-    :align: center
-    :alt: client-architecture-balancer-figure-08
-
-Upstream gRPC Go had already migrated to new balancer interface. For example, ``clientv3-grpc1.7`` underlying balancer implementation uses new gRPC balancer and tries to be consistent with old balancer behaviors. While its compatibility has been maintained reasonably well, etcd client still `suffered from subtle breaking changes <https://github.com/grpc/grpc-go/issues/1649>`_. Furthermore, gRPC maintainer recommends to `not rely on the old balancer interface <https://github.com/grpc/grpc-go/issues/1942#issuecomment-375368665>`_. In general, to get better support from upstream, it is best to be in sync with latest gRPC releases. And new features, such as retry policy, may not be backported to gRPC 1.7 branch. Thus, both etcd server and client must migrate to latest gRPC versions.
-
-
-clientv3-grpc1.14: Balancer Overview
-------------------------------------
-
-``clientv3-grpc1.7`` is so tightly coupled with old gRPC interface, that every single gRPC dependency upgrade broke client behavior. Majority of development and debugging efforts were devoted to fixing those client behavior changes. As a result, its implementation has become overly complicated with bad assumptions on server connectivities.
-
-The primary goal of ``clientv3-grpc1.14`` is to simplify balancer failover logic; rather than maintaining a list of unhealthy endpoints, which may be stale, simply roundrobin to the next endpoint whenever client gets disconnected from the current endpoint. It does not assume endpoint status. Thus, no more complicated status tracking is needed (see *Figure 8* and above). Upgrading to ``clientv3-grpc1.14`` should be no issue; all changes were internal while keeping all the backward compatibilities.
-
-Internally, when given multiple endpoints, ``clientv3-grpc1.14`` creates multiple sub-connections (one sub-connection per each endpoint), while ``clientv3-grpc1.7`` creates only one connection to a pinned endpoint (see *Figure 9*). For instance, in 5-node cluster, ``clientv3-grpc1.14`` balancer would require 5 TCP connections, while ``clientv3-grpc1.7`` only requires one. By preserving the pool of TCP connections, ``clientv3-grpc1.14`` may consume more resources but provide more flexible load balancer with better failover performance. The default balancing policy is round robin but can be easily extended to support other types of balancers (e.g. power of two, pick leader, etc.). ``clientv3-grpc1.14`` uses gRPC resolver group and implements balancer picker policy, in order to delegate complex balancing work to upstream gRPC. On the other hand, ``clientv3-grpc1.7`` manually handles each gRPC connection and balancer failover, which complicates the implementation. ``clientv3-grpc1.14`` implements retry in the gRPC interceptor chain that automatically handles gRPC internal errors and enables more advanced retry policies like backoff, while ``clientv3-grpc1.7`` manually interprets gRPC errors for retries.
-
-.. image:: img/client-architecture-balancer-figure-09.png
-    :align: center
-    :alt: client-architecture-balancer-figure-09
-
-
-clientv3-grpc1.14: Balancer Limitation
---------------------------------------
-
-Improvements can be made by caching the status of each endpoint. For instance, balancer can ping each server in advance to maintain a list of healthy candidates, and use this information when doing round-robin. Or when disconnected, balancer can prioritize healthy endpoints. This may complicate the balancer implementation, thus can be addressed in later versions.
-
-Client-side keepalive ping still does not reason about network partitions. Streaming request may get stuck with a partitioned node. Advanced health checking service need to be implemented to understand the cluster membership (see `issue#8673 <https://github.com/etcd-io/etcd/issues/8673>`_ for more detail).
-
-Currently, retry logic is handled manually as an interceptor. This may be simplified via `official gRPC retries <https://github.com/grpc/proposal/blob/master/A6-client-retries.md>`_.

docs/client-feature-matrix.rst

@@ -1,281 +0,0 @@
-.. _client-feature-matrix:
-
-
-Client Feature Matrix
-#####################
-
-
-Features
-========
-
-+----------------------------------------+-------------------------+-------------------+
-| Feature                                | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+========================================+=========================+===================+
-| Automatic retry                        | Yes                     | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-| Retry backoff                          | Yes                     | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-| Automatic failover                     | Yes                     | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-| Load balancer                          | Round-Robin             | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-| ``WithRequireLeader(context.Context)`` | Yes                     | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-| ``TLS``                                | Yes                     | Yes               |
-+----------------------------------------+-------------------------+-------------------+
-| ``SetEndpoints``                       | Yes                     | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-| ``Sync`` endpoints                     | Yes                     | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-| ``AutoSyncInterval``                   | Yes                     | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-| ``KeepAlive`` ping                     | Yes                     | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-| ``MaxCallSendMsgSize``                 | Yes                     | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-| ``MaxCallRecvMsgSize``                 | Yes                     | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-| ``RejectOldCluster``                   | Yes                     | ·                 |
-+----------------------------------------+-------------------------+-------------------+
-
-
-`KV <https://godoc.org/go.etcd.io/etcd/clientv3#KV>`_
-============================================================
-
-+---------------+-------------------------+-------------------+
-| Feature       | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+===============+=========================+===================+
-| ``Put``       | Yes                     | ·                 |
-+---------------+-------------------------+-------------------+
-| ``Get``       | Yes                     | ·                 |
-+---------------+-------------------------+-------------------+
-| ``Delete``    | Yes                     | ·                 |
-+---------------+-------------------------+-------------------+
-| ``Compact``   | Yes                     | ·                 |
-+---------------+-------------------------+-------------------+
-| ``Do(Op)``    | Yes                     | ·                 |
-+---------------+-------------------------+-------------------+
-| ``Txn``       | Yes                     | ·                 |
-+---------------+-------------------------+-------------------+
-
-
-`Lease <https://godoc.org/go.etcd.io/etcd/clientv3#Lease>`_
-==================================================================
-
-+-------------------+-------------------------+-------------------+
-| Feature           | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+===================+=========================+===================+
-| ``Grant``         | Yes                     | ·                 |
-+-------------------+-------------------------+-------------------+
-| ``Revoke``        | Yes                     | ·                 |
-+-------------------+-------------------------+-------------------+
-| ``TimeToLive``    | Yes                     | ·                 |
-+-------------------+-------------------------+-------------------+
-| ``Leases``        | Yes                     | ·                 |
-+-------------------+-------------------------+-------------------+
-| ``KeepAlive``     | Yes                     | ·                 |
-+-------------------+-------------------------+-------------------+
-| ``KeepAliveOnce`` | Yes                     | ·                 |
-+-------------------+-------------------------+-------------------+
-
-
-`Watcher <https://godoc.org/go.etcd.io/etcd/clientv3#Watcher>`_
-======================================================================
-
-+---------------------+-------------------------+-------------------+
-| Feature             | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+=====================+=========================+===================+
-| ``Watch``           | Yes                     | Yes               |
-+---------------------+-------------------------+-------------------+
-| ``RequestProgress`` | Yes                     | ·                 |
-+---------------------+-------------------------+-------------------+
-
-
-`Cluster <https://godoc.org/go.etcd.io/etcd/clientv3#Cluster>`_
-======================================================================
-
-+------------------+-------------------------+-------------------+
-| Feature          | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+==================+=========================+===================+
-| ``MemberList``   | Yes                     | Yes               |
-+------------------+-------------------------+-------------------+
-| ``MemberAdd``    | Yes                     | Yes               |
-+------------------+-------------------------+-------------------+
-| ``MemberRemove`` | Yes                     | Yes               |
-+------------------+-------------------------+-------------------+
-| ``MemberUpdate`` | Yes                     | Yes               |
-+------------------+-------------------------+-------------------+
-
-
-`Maintenance <https://godoc.org/go.etcd.io/etcd/clientv3#Maintenance>`_
-==============================================================================
-
-+-----------------+-------------------------+-------------------+
-| Feature         | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+=================+=========================+===================+
-| ``AlarmList``   | Yes                     | Yes               |
-+-----------------+-------------------------+-------------------+
-| ``AlarmDisarm`` | Yes                     | ·                 |
-+-----------------+-------------------------+-------------------+
-| ``Defragment``  | Yes                     | ·                 |
-+-----------------+-------------------------+-------------------+
-| ``Status``      | Yes                     | ·                 |
-+-----------------+-------------------------+-------------------+
-| ``HashKV``      | Yes                     | ·                 |
-+-----------------+-------------------------+-------------------+
-| ``Snapshot``    | Yes                     | ·                 |
-+-----------------+-------------------------+-------------------+
-| ``MoveLeader``  | Yes                     | ·                 |
-+-----------------+-------------------------+-------------------+
-
-
-`Auth <https://godoc.org/go.etcd.io/etcd/clientv3#Auth>`_
-================================================================
-
-+----------------------------+-------------------------+-------------------+
-| Feature                    | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+============================+=========================+===================+
-| ``AuthEnable``             | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``AuthDisable``            | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``UserAdd``                | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``UserDelete``             | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``UserChangePassword``     | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``UserGrantRole``          | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``UserGet``                | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``UserList``               | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``UserRevokeRole``         | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``RoleAdd``                | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``RoleGrantPermission``    | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``RoleGet``                | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``RoleList``               | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``RoleRevokePermission``   | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-| ``RoleDelete``             | Yes                     | ·                 |
-+----------------------------+-------------------------+-------------------+
-
-
-`clientv3util <https://godoc.org/go.etcd.io/etcd/clientv3/clientv3util>`_
-================================================================================
-
-+---------------------------+-------------------------+-------------------+
-| Feature                   | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+===========================+=========================+===================+
-| ``KeyExists``             | Yes                     | No                |
-+---------------------------+-------------------------+-------------------+
-| ``KeyMissing``            | Yes                     | No                |
-+---------------------------+-------------------------+-------------------+
-
-
-`concurrency <https://godoc.org/go.etcd.io/etcd/clientv3/concurrency>`_
-==============================================================================
-
-+----------------------------------------+-------------------------+-------------------+
-| Feature                                | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+========================================+=========================+===================+
-| ``Session``                            | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-| ``NewMutex(Session, prefix)``          | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-| ``NewElection(Session, prefix)``       | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-| ``NewLocker(Session, prefix)``         | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-| ``STM Isolation SerializableSnapshot`` | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-| ``STM Isolation Serializable``         | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-| ``STM Isolation RepeatableReads``      | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-| ``STM Isolation ReadCommitted``        | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-| ``STM Get``                            | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-| ``STM Put``                            | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-| ``STM Rev``                            | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-| ``STM Del``                            | Yes                     | No                |
-+----------------------------------------+-------------------------+-------------------+
-
-
-`leasing <https://godoc.org/go.etcd.io/etcd/clientv3/leasing>`_
-======================================================================
-
-+---------------------------+-------------------------+-------------------+
-| Feature                   | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+===========================+=========================+===================+
-| ``NewKV(Client, prefix)`` | Yes                     | No                |
-+---------------------------+-------------------------+-------------------+
-
-
-`mirror <https://godoc.org/go.etcd.io/etcd/clientv3/mirror>`_
-======================================================================
-
-+------------------+-------------------------+-------------------+
-| Feature          | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+==================+=========================+===================+
-| ``SyncBase``     | Yes                     | No                |
-+------------------+-------------------------+-------------------+
-| ``SyncUpdates``  | Yes                     | No                |
-+------------------+-------------------------+-------------------+
-
-
-`namespace <https://godoc.org/go.etcd.io/etcd/clientv3/namespace>`_
-==========================================================================
-
-+-------------+-------------------------+-------------------+
-| Feature     | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+=============+=========================+===================+
-| ``KV``      | Yes                     | No                |
-+-------------+-------------------------+-------------------+
-| ``Lease``   | Yes                     | No                |
-+-------------+-------------------------+-------------------+
-| ``Watcher`` | Yes                     | No                |
-+-------------+-------------------------+-------------------+
-
-
-`naming <https://godoc.org/go.etcd.io/etcd/clientv3/naming>`_
-====================================================================
-
-+--------------------+-------------------------+-------------------+
-| Feature            | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+====================+=========================+===================+
-| ``GRPCResolver``   | Yes                     | No                |
-+--------------------+-------------------------+-------------------+
-
-
-`ordering <https://godoc.org/go.etcd.io/etcd/clientv3/ordering>`_
-========================================================================
-
-+----------+-------------------------+-------------------+
-| Feature  | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+==========+=========================+===================+
-| ``KV``   | Yes                     | No                |
-+----------+-------------------------+-------------------+
-
-
-`snapshot <https://godoc.org/go.etcd.io/etcd/clientv3/snapshot>`_
-========================================================================
-
-+-------------+-------------------------+-------------------+
-| Feature     | ``clientv3-grpc1.14``   | ``jetcd v0.0.2``  |
-+=============+=========================+===================+
-| ``Save``    | Yes                     | No                |
-+-------------+-------------------------+-------------------+
-| ``Status``  | Yes                     | No                |
-+-------------+-------------------------+-------------------+
-| ``Restore`` | Yes                     | No                |
-+-------------+-------------------------+-------------------+

docs/conf.py

@@ -1,199 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# Configuration file for the Sphinx documentation builder.
-#
-# This file does only contain a selection of the most common options. For a
-# full list see the documentation:
-# http://www.sphinx-doc.org/en/master/config
-
-# -- Path setup --------------------------------------------------------------
-
-# If extensions (or modules to document with autodoc) are in another directory,
-# add these directories to sys.path here. If the directory is relative to the
-# documentation root, use os.path.abspath to make it absolute, like shown here.
-#
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
-
-
-# -- Project information -----------------------------------------------------
-
-project = u'etcd'
-copyright = u'2018, etcd'
-author = u'etcd'
-
-# The short X.Y version
-version = u''
-# The full version, including alpha/beta/rc tags
-release = u''
-
-
-# -- General configuration ---------------------------------------------------
-
-# If your documentation needs a minimal Sphinx version, state it here.
-#
-# needs_sphinx = '1.0'
-
-# Add any Sphinx extension module names here, as strings. They can be
-# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
-# ones.
-extensions = [
-    'sphinx.ext.autodoc',
-    'sphinx.ext.doctest',
-    'sphinx.ext.intersphinx',
-    'sphinx.ext.todo',
-    'sphinx.ext.coverage',
-    'sphinx.ext.mathjax',
-    'sphinx.ext.ifconfig',
-    'sphinx.ext.viewcode',
-    'sphinx.ext.githubpages',
-]
-
-# Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
-
-# The suffix(es) of source filenames.
-# You can specify multiple suffix as a list of string:
-#
-# source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
-
-# The master toctree document.
-master_doc = 'index'
-
-# The language for content autogenerated by Sphinx. Refer to documentation
-# for a list of supported languages.
-#
-# This is also used if you do content translation via gettext catalogs.
-# Usually you set "language" from the command line for these cases.
-language = None
-
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-# This pattern also affects html_static_path and html_extra_path .
-exclude_patterns = [u'_build', 'Thumbs.db', '.DS_Store']
-
-# The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
-
-
-# -- Options for HTML output -------------------------------------------------
-
-# The theme to use for HTML and HTML Help pages.  See the documentation for
-# a list of builtin themes.
-#
-html_theme = "sphinx_rtd_theme"
-html_theme_path = ["_themes", ]
-
-# Theme options are theme-specific and customize the look and feel of a theme
-# further.  For a list of options available for each theme, see the
-# documentation.
-#
-# html_theme_options = {}
-
-# Add any paths that contain custom static files (such as style sheets) here,
-# relative to this directory. They are copied after the builtin static files,
-# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static', 'img']
-
-# Custom sidebar templates, must be a dictionary that maps document names
-# to template names.
-#
-# The default sidebars (for documents that don't match any pattern) are
-# defined by theme itself.  Builtin themes are using these templates by
-# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
-# 'searchbox.html']``.
-#
-# html_sidebars = {}
-
-
-# -- Options for HTMLHelp output ---------------------------------------------
-
-# Output file base name for HTML help builder.
-htmlhelp_basename = 'etcddoc'
-
-
-# -- Options for LaTeX output ------------------------------------------------
-
-latex_elements = {
-    # The paper size ('letterpaper' or 'a4paper').
-    #
-    # 'papersize': 'letterpaper',
-
-    # The font size ('10pt', '11pt' or '12pt').
-    #
-    # 'pointsize': '10pt',
-
-    # Additional stuff for the LaTeX preamble.
-    #
-    # 'preamble': '',
-
-    # Latex figure (float) alignment
-    #
-    # 'figure_align': 'htbp',
-}
-
-# Grouping the document tree into LaTeX files. List of tuples
-# (source start file, target name, title,
-#  author, documentclass [howto, manual, or own class]).
-latex_documents = [
-    (master_doc, 'etcd.tex', u'etcd Documentation',
-     u'etcd', 'manual'),
-]
-
-
-# -- Options for manual page output ------------------------------------------
-
-# One entry per manual page. List of tuples
-# (source start file, name, description, authors, manual section).
-man_pages = [
-    (master_doc, 'etcd', u'etcd Documentation',
-     [author], 1)
-]
-
-
-# -- Options for Texinfo output ----------------------------------------------
-
-# Grouping the document tree into Texinfo files. List of tuples
-# (source start file, target name, title, author,
-#  dir menu entry, description, category)
-texinfo_documents = [
-    (master_doc, 'etcd', u'etcd Documentation',
-     author, 'etcd', 'One line description of project.',
-     'Miscellaneous'),
-]
-
-
-# -- Options for Epub output -------------------------------------------------
-
-# Bibliographic Dublin Core info.
-epub_title = project
-epub_author = author
-epub_publisher = author
-epub_copyright = copyright
-
-# The unique identifier of the text. This can be a ISBN number
-# or the project homepage.
-#
-# epub_identifier = ''
-
-# A unique identification for the text.
-#
-# epub_uid = ''
-
-# A list of files that should not be packed into the epub file.
-epub_exclude_files = ['search.html']
-
-
-# -- Extension configuration -------------------------------------------------
-
-# -- Options for intersphinx extension ---------------------------------------
-
-# Example configuration for intersphinx: refer to the Python standard library.
-intersphinx_mapping = {'https://docs.python.org/': None}
-
-# -- Options for todo extension ----------------------------------------------
-
-# If true, `todo` and `todoList` produce output, else they produce nothing.
-todo_include_todos = True

docs/faq.rst

@@ -1,41 +0,0 @@
-.. _faq:
-
-
-General
-#######
-
-
-What is etcd?
-=============
-
-etcd is a consistent distributed key-value store. Mainly used as a separate coordination service, in distributed systems. And designed to hold small amounts of data that can fit entirely in memory.
-
-
-How to pronounce etcd?
-======================
-
-etcd is pronounced ``/ˈɛtsiːdiː/``, and means distributed ``etc`` directory.
-
-
-Do clients have to send requests to the etcd leader?
-====================================================
-
-`Raft <https://raft.github.io/raft.pdf>`_ is leader-based; the leader handles all client requests which need cluster consensus. However, the client does not need to know which node is the leader. Any request that requires consensus sent to a follower is automatically forwarded to the leader. Requests that do not require consensus (e.g., serialized reads) can be processed by any cluster member.
-
-
-Configuration
-#############
-
-
-Difference between listen-<client,peer>-urls, advertise-client-urls or initial-advertise-peer-urls?
-===================================================================================================
-
-``--listen-client-urls`` and ``--listen-peer-urls`` specify the local addresses etcd server binds to for accepting incoming connections. To listen on a port for all interfaces, specify ``0.0.0.0`` as the listen IP address.
-
-``--advertise-client-urls`` and ``--initial-advertise-peer-urls`` specify the addresses etcd clients or other etcd members should use to contact the etcd server. The advertise addresses must be reachable from the remote machines. Do not advertise addresses like ``localhost`` or ``0.0.0.0`` for a production setup since these addresses are unreachable from remote machines.
-
-
-Changing "listen-peer-urls" or "initial-advertise-peer-urls" does not update advertised peer URLs in member list output?
-========================================================================================================================
-
-A member's advertised peer URLs come from ``--initial-advertise-peer-urls`` on initial cluster boot. Changing the listen peer URLs or the initial advertise peers after booting the member won't affect the exported advertise peer URLs (e.g. ``etcdctl member list`` output remains the same), since changes must go through quorum to avoid membership configuration split brain. Use `etcdctl member update` to update a member's peer URLs.