0doc.go 7.7 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195
  1. // Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
  2. // Use of this source code is governed by a MIT license found in the LICENSE file.
  3. /*
  4. High Performance, Feature-Rich Idiomatic Go 1.4+ codec/encoding library for
  5. binc, msgpack, cbor, json
  6. Supported Serialization formats are:
  7. - msgpack: https://github.com/msgpack/msgpack
  8. - binc: http://github.com/ugorji/binc
  9. - cbor: http://cbor.io http://tools.ietf.org/html/rfc7049
  10. - json: http://json.org http://tools.ietf.org/html/rfc7159
  11. - simple:
  12. To install:
  13. go get github.com/ugorji/go/codec
  14. This package will carefully use 'unsafe' for performance reasons in specific places.
  15. You can build without unsafe use by passing the safe or appengine tag
  16. i.e. 'go install -tags=safe ...'. Note that unsafe is only supported for the last 3
  17. go sdk versions e.g. current go release is go 1.9, so we support unsafe use only from
  18. go 1.7+ . This is because supporting unsafe requires knowledge of implementation details.
  19. For detailed usage information, read the primer at http://ugorji.net/blog/go-codec-primer .
  20. The idiomatic Go support is as seen in other encoding packages in
  21. the standard library (ie json, xml, gob, etc).
  22. Rich Feature Set includes:
  23. - Simple but extremely powerful and feature-rich API
  24. - Very High Performance.
  25. Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
  26. - Multiple conversions:
  27. Package coerces types where appropriate
  28. e.g. decode an int in the stream into a float, etc.
  29. - Corner Cases:
  30. Overflows, nil maps/slices, nil values in streams are handled correctly
  31. - Standard field renaming via tags
  32. - Support for omitting empty fields during an encoding
  33. - Encoding from any value and decoding into pointer to any value
  34. (struct, slice, map, primitives, pointers, interface{}, etc)
  35. - Extensions to support efficient encoding/decoding of any named types
  36. - Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
  37. - Decoding without a schema (into a interface{}).
  38. Includes Options to configure what specific map or slice type to use
  39. when decoding an encoded list or map into a nil interface{}
  40. - Encode a struct as an array, and decode struct from an array in the data stream
  41. - Comprehensive support for anonymous fields
  42. - Fast (no-reflection) encoding/decoding of common maps and slices
  43. - Code-generation for faster performance.
  44. - Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
  45. - Support indefinite-length formats to enable true streaming
  46. (for formats which support it e.g. json, cbor)
  47. - Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes.
  48. This mostly applies to maps, where iteration order is non-deterministic.
  49. - NIL in data stream decoded as zero value
  50. - Never silently skip data when decoding.
  51. User decides whether to return an error or silently skip data when keys or indexes
  52. in the data stream do not map to fields in the struct.
  53. - Detect and error when encoding a cyclic reference (instead of stack overflow shutdown)
  54. - Encode/Decode from/to chan types (for iterative streaming support)
  55. - Drop-in replacement for encoding/json. `json:` key in struct tag supported.
  56. - Provides a RPC Server and Client Codec for net/rpc communication protocol.
  57. - Handle unique idiosyncrasies of codecs e.g.
  58. - For messagepack, configure how ambiguities in handling raw bytes are resolved
  59. - For messagepack, provide rpc server/client codec to support
  60. msgpack-rpc protocol defined at:
  61. https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
  62. Extension Support
  63. Users can register a function to handle the encoding or decoding of
  64. their custom types.
  65. There are no restrictions on what the custom type can be. Some examples:
  66. type BisSet []int
  67. type BitSet64 uint64
  68. type UUID string
  69. type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
  70. type GifImage struct { ... }
  71. As an illustration, MyStructWithUnexportedFields would normally be
  72. encoded as an empty map because it has no exported fields, while UUID
  73. would be encoded as a string. However, with extension support, you can
  74. encode any of these however you like.
  75. RPC
  76. RPC Client and Server Codecs are implemented, so the codecs can be used
  77. with the standard net/rpc package.
  78. Usage
  79. The Handle is SAFE for concurrent READ, but NOT SAFE for concurrent modification.
  80. The Encoder and Decoder are NOT safe for concurrent use.
  81. Consequently, the usage model is basically:
  82. - Create and initialize the Handle before any use.
  83. Once created, DO NOT modify it.
  84. - Multiple Encoders or Decoders can now use the Handle concurrently.
  85. They only read information off the Handle (never write).
  86. - However, each Encoder or Decoder MUST not be used concurrently
  87. - To re-use an Encoder/Decoder, call Reset(...) on it first.
  88. This allows you use state maintained on the Encoder/Decoder.
  89. Sample usage model:
  90. // create and configure Handle
  91. var (
  92. bh codec.BincHandle
  93. mh codec.MsgpackHandle
  94. ch codec.CborHandle
  95. )
  96. mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
  97. // configure extensions
  98. // e.g. for msgpack, define functions and enable Time support for tag 1
  99. // mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)
  100. // create and use decoder/encoder
  101. var (
  102. r io.Reader
  103. w io.Writer
  104. b []byte
  105. h = &bh // or mh to use msgpack
  106. )
  107. dec = codec.NewDecoder(r, h)
  108. dec = codec.NewDecoderBytes(b, h)
  109. err = dec.Decode(&v)
  110. enc = codec.NewEncoder(w, h)
  111. enc = codec.NewEncoderBytes(&b, h)
  112. err = enc.Encode(v)
  113. //RPC Server
  114. go func() {
  115. for {
  116. conn, err := listener.Accept()
  117. rpcCodec := codec.GoRpc.ServerCodec(conn, h)
  118. //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
  119. rpc.ServeCodec(rpcCodec)
  120. }
  121. }()
  122. //RPC Communication (client side)
  123. conn, err = net.Dial("tcp", "localhost:5555")
  124. rpcCodec := codec.GoRpc.ClientCodec(conn, h)
  125. //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
  126. client := rpc.NewClientWithCodec(rpcCodec)
  127. */
  128. package codec
  129. // Benefits of go-codec:
  130. //
  131. // - encoding/json always reads whole file into memory first.
  132. // This makes it unsuitable for parsing very large files.
  133. // - encoding/xml cannot parse into a map[string]interface{}
  134. // I found this out on reading https://github.com/clbanning/mxj
  135. // TODO:
  136. //
  137. // - optimization for codecgen:
  138. // if len of entity is <= 3 words, then support a value receiver for encode.
  139. // - (En|De)coder should store an error when it occurs.
  140. // Until reset, subsequent calls return that error that was stored.
  141. // This means that free panics must go away.
  142. // All errors must be raised through errorf method.
  143. // - Decoding using a chan is good, but incurs concurrency costs.
  144. // This is because there's no fast way to use a channel without it
  145. // having to switch goroutines constantly.
  146. // Callback pattern is still the best. Maybe consider supporting something like:
  147. // type X struct {
  148. // Name string
  149. // Ys []Y
  150. // Ys chan <- Y
  151. // Ys func(Y) -> call this function for each entry
  152. // }
  153. // - Consider adding a isZeroer interface { isZero() bool }
  154. // It is used within isEmpty, for omitEmpty support.
  155. // - Consider making Handle used AS-IS within the encoding/decoding session.
  156. // This means that we don't cache Handle information within the (En|De)coder,
  157. // except we really need it at Reset(...)
  158. // - Consider adding math/big support
  159. // - Consider reducing the size of the generated functions:
  160. // Maybe use one loop, and put the conditionals in the loop.
  161. // for ... { if cLen > 0 { if j == cLen { break } } else if dd.CheckBreak() { break } }