codec: support cbor format; improve extension and streaming support.

The high level changes:
- cbor support
- Improved extension support
- Streaming support (indefinite length array/map/string/bytes) for cbor, etc

cbor format has gained traction as a binary format. It is specified by IETF,
and is used in constrained devices and many other places (see cbor.io).

To support cbor, we needed to redesign the way we handle Extensions and include
Streaming support, while ensuring that Encode/Decode can be called multiple times
on the same Encoder/Decoder.

The support for cbor is comprehensive.

codec/0doc.go

@@ -7,7 +7,9 @@ High Performance, Feature-Rich Idiomatic Go encoding library for msgpack and bin
 Supported Serialization formats are:
 
   - msgpack: [https://github.com/msgpack/msgpack]
-  - binc: [http://github.com/ugorji/binc]
+  - binc:    [http://github.com/ugorji/binc]
+  - cbor:    [http://cbor.io]
+  - simple: 
 
 To install:
 
@@ -21,41 +23,33 @@ Rich Feature Set includes:
   - Simple but extremely powerful and feature-rich API
   - Very High Performance.
     Our extensive benchmarks show us outperforming Gob, Json and Bson by 2-4X.
-    This was achieved by taking extreme care on:
-      - managing allocation
-      - function frame size (important due to Go's use of split stacks),
-      - reflection use (and by-passing reflection for common types)
-      - recursion implications
-      - zero-copy mode (encoding/decoding to byte slice without using temp buffers)
-  - Correct.
-    Care was taken to precisely handle corner cases like:
-      overflows, nil maps and slices, nil value in stream, etc.
-  - Efficient zero-copying into temporary byte buffers
-    when encoding into or decoding from a byte slice.
+    Achieved by extreme care on allocations, recursions, bypassing reflection, zero-copy, etc.
+  - Multiple conversions:
+    Package co-erces types where appropriate e.g. decode an int in the stream into a float, etc
+  - Corner Cases: Overflows, nil maps/slices, nil values in streams are handled correctly
   - Standard field renaming via tags
   - Encoding from any value
     (struct, slice, map, primitives, pointers, interface{}, etc)
-  - Decoding into pointer to any non-nil typed value
+  - Decoding into pointer to any value
     (struct, slice, map, int, float32, bool, string, reflect.Value, etc)
   - Supports extension functions to handle the encode/decode of custom types
   - Support Go 1.2 encoding.BinaryMarshaler/BinaryUnmarshaler
   - Schema-less decoding
-    (decode into a pointer to a nil interface{} as opposed to a typed non-nil value).
+    (decode into a pointer to a nil interface{} as opposed to a typed value).
     Includes Options to configure what specific map or slice type to use
     when decoding an encoded list or map into a nil interface{}
   - Provides a RPC Server and Client Codec for net/rpc communication protocol.
-  - Msgpack Specific:
-      - Provides extension functions to handle spec-defined extensions (binary, timestamp)
-      - Options to resolve ambiguities in handling raw bytes (as string or []byte)
-        during schema-less decoding (decoding into a nil interface{})
-      - RPC Server/Client Codec for msgpack-rpc protocol defined at:
-        https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
   - Fast Paths for some container types:
     For some container types, we circumvent reflection and its associated overhead
     and allocation costs, and encode/decode directly. These types are:
 	    Slice of all builtin types and interface{},
 	    map of all builtin types and interface{} to string, interface{}, int, int64, uint64
 	    symetrical maps of all builtin types and interface{}
+  - Msgpack Specific:
+      - Options to resolve ambiguities in handling raw bytes (as string or []byte)
+        during schema-less decoding (decoding into a nil interface{})
+      - RPC Server/Client Codec for msgpack-rpc protocol defined at:
+        https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
 
 Extension Support
 
@@ -88,13 +82,14 @@ Typical usage model:
     var (
       bh codec.BincHandle
       mh codec.MsgpackHandle
+      ch codec.CborHandle
     )
 
     mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
 
     // configure extensions
     // e.g. for msgpack, define functions and enable Time support for tag 1
-    // mh.AddExt(reflect.TypeOf(time.Time{}), 1, myMsgpackTimeEncodeExtFn, myMsgpackTimeDecodeExtFn)
+    // mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)
 
     // create and use decoder/encoder
     var (
@@ -138,3 +133,4 @@ see notes in ext_dep_test.go
 
 */
 package codec
+

codec/README.md

@@ -20,46 +20,35 @@ the standard library (ie json, xml, gob, etc).
 Rich Feature Set includes:
 
   - Simple but extremely powerful and feature-rich API
-  - Very High Performance.   
+  - Very High Performance.
     Our extensive benchmarks show us outperforming Gob, Json and Bson by 2-4X.
-    This was achieved by taking extreme care on:
-      - managing allocation
-      - function frame size (important due to Go's use of split stacks),
-      - reflection use (and by-passing reflection for common types)
-      - recursion implications
-      - zero-copy mode (encoding/decoding to byte slice without using temp buffers)
-  - Correct.  
-    Care was taken to precisely handle corner cases like: 
-      overflows, nil maps and slices, nil value in stream, etc.
-  - Efficient zero-copying into temporary byte buffers  
-    when encoding into or decoding from a byte slice.
+    Achieved by extreme care on allocations, recursions, bypassing reflection, zero-copy, etc.
+  - Multiple conversions:
+    Package co-erces types where appropriate e.g. decode an int in the stream into a float, etc
+  - Corner Cases: Overflows, nil maps/slices, nil values in streams are handled correctly
   - Standard field renaming via tags
-  - Encoding from any value  
+  - Encoding from any value
     (struct, slice, map, primitives, pointers, interface{}, etc)
-  - Decoding into pointer to any non-nil typed value  
+  - Decoding into pointer to any value
     (struct, slice, map, int, float32, bool, string, reflect.Value, etc)
   - Supports extension functions to handle the encode/decode of custom types
   - Support Go 1.2 encoding.BinaryMarshaler/BinaryUnmarshaler
-  - Schema-less decoding  
-    (decode into a pointer to a nil interface{} as opposed to a typed non-nil value).  
-    Includes Options to configure what specific map or slice type to use 
+  - Schema-less decoding
+    (decode into a pointer to a nil interface{} as opposed to a typed value).
+    Includes Options to configure what specific map or slice type to use
     when decoding an encoded list or map into a nil interface{}
   - Provides a RPC Server and Client Codec for net/rpc communication protocol.
+  - Fast Paths for some container types:
+    For some container types, we circumvent reflection and its associated overhead
+    and allocation costs, and encode/decode directly. These types are:
+	    Slice of all builtin types and interface{},
+	    map of all builtin types and interface{} to string, interface{}, int, int64, uint64
+	    symetrical maps of all builtin types and interface{}
   - Msgpack Specific:
-      - Provides extension functions to handle spec-defined extensions (binary, timestamp)
-      - Options to resolve ambiguities in handling raw bytes (as string or []byte)  
+      - Options to resolve ambiguities in handling raw bytes (as string or []byte)
         during schema-less decoding (decoding into a nil interface{})
-      - RPC Server/Client Codec for msgpack-rpc protocol defined at: 
+      - RPC Server/Client Codec for msgpack-rpc protocol defined at:
         https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
-  - Fast Paths for some container types:  
-    For some container types, we circumvent reflection and its associated overhead
-    and allocation costs, and encode/decode directly. These types are:  
-	    []interface{}
-	    []int
-	    []string
-	    map[interface{}]interface{}
-	    map[int]interface{}
-	    map[string]interface{}
 
 ## Extension Support
 
@@ -92,6 +81,7 @@ Typical usage model:
     var (
       bh codec.BincHandle
       mh codec.MsgpackHandle
+      ch codec.CborHandle
     )
 
     mh.MapType = reflect.TypeOf(map[string]interface{}(nil))

codec/bench_test.go

@@ -65,6 +65,7 @@ func benchInit() {
 		benchChecker{"binc-nosym", fnBincNoSymEncodeFn, fnBincNoSymDecodeFn},
 		benchChecker{"binc-sym", fnBincSymEncodeFn, fnBincSymDecodeFn},
 		benchChecker{"simple", fnSimpleEncodeFn, fnSimpleDecodeFn},
+		benchChecker{"cbor", fnCborEncodeFn, fnCborDecodeFn},
 		benchChecker{"gob", fnGobEncodeFn, fnGobDecodeFn},
 		benchChecker{"json", fnJsonEncodeFn, fnJsonDecodeFn},
 	)
@@ -76,7 +77,7 @@ func benchInit() {
 func runBenchInit() {
 	logT(nil, "..............................................")
 	logT(nil, "BENCHMARK INIT: %v", time.Now())
-	logT(nil, "To run full benchmark comparing encodings (MsgPack, Binc, Simple, JSON, GOB, etc), "+
+	logT(nil, "To run full benchmark comparing encodings (MsgPack, Binc, Simple, Cbor, JSON, GOB, etc), "+
 		"use: \"go test -bench=.\"")
 	logT(nil, "Benchmark: ")
 	logT(nil, "\tStruct recursive Depth:             %d", benchDepth)
@@ -252,6 +253,15 @@ func fnSimpleDecodeFn(buf []byte, ts interface{}) error {
 	return NewDecoderBytes(buf, testSimpleH).Decode(ts)
 }
 
+func fnCborEncodeFn(ts interface{}) (bs []byte, err error) {
+	err = NewEncoderBytes(&bs, testCborH).Encode(ts)
+	return
+}
+
+func fnCborDecodeFn(buf []byte, ts interface{}) error {
+	return NewDecoderBytes(buf, testCborH).Decode(ts)
+}
+
 func fnGobEncodeFn(ts interface{}) ([]byte, error) {
 	bbuf := new(bytes.Buffer)
 	err := gob.NewEncoder(bbuf).Encode(ts)
@@ -302,6 +312,14 @@ func Benchmark__Simple_____Decode(b *testing.B) {
 	fnBenchmarkDecode(b, "simple", benchTs, fnSimpleEncodeFn, fnSimpleDecodeFn, fnBenchNewTs)
 }
 
+func Benchmark__Cbor_______Encode(b *testing.B) {
+	fnBenchmarkEncode(b, "cbor", benchTs, fnCborEncodeFn)
+}
+
+func Benchmark__Cbor_______Decode(b *testing.B) {
+	fnBenchmarkDecode(b, "cbor", benchTs, fnCborEncodeFn, fnCborDecodeFn, fnBenchNewTs)
+}
+
 func Benchmark__Gob________Encode(b *testing.B) {
 	fnBenchmarkEncode(b, "gob", benchTs, fnGobEncodeFn)
 }

codec/binc.go

@@ -5,6 +5,7 @@ package codec
 
 import (
 	"math"
+	"reflect"
 	// "reflect"
 	// "sync/atomic"
 	"time"
@@ -172,6 +173,21 @@ func (e *bincEncDriver) encUint(bd byte, pos bool, v uint64) {
 	}
 }
 
+func (e *bincEncDriver) encodeExt(rv reflect.Value, xtag uint16, ext Ext, _ *Encoder) {
+	bs := ext.WriteExt(rv)
+	if bs == nil {
+		e.encodeNil()
+		return
+	}
+	e.encodeExtPreamble(uint8(xtag), len(bs))
+	e.w.writeb(bs)
+}
+
+func (e *bincEncDriver) encodeRawExt(re *RawExt, _ *Encoder) {
+	e.encodeExtPreamble(uint8(re.Tag), len(re.Data))
+	e.w.writeb(re.Data)
+}
+
 func (e *bincEncDriver) encodeExtPreamble(xtag byte, length int) {
 	e.encLen(bincVdCustomExt<<4, uint64(length))
 	e.w.writen1(xtag)
@@ -304,14 +320,16 @@ func (e *bincEncDriver) encLenNumber(bd byte, v uint64) {
 //------------------------------------
 
 type bincDecDriver struct {
+	h      *BincHandle
 	r      decReader
 	bdRead bool
 	bdType valueType
 	bd     byte
 	vd     byte
 	vs     byte
-	b      [8]byte
-	m      map[uint32]string // symbols (use uint32 as key, as map optimizes for it)
+	noStreamingCodec
+	b [8]byte
+	m map[uint32]string // symbols (use uint32 as key, as map optimizes for it)
 }
 
 func (d *bincDecDriver) initReadNext() {
@@ -337,16 +355,22 @@ func (d *bincDecDriver) currentEncodedType() valueType {
 			case bincSpNan, bincSpNegInf, bincSpPosInf, bincSpZeroFloat:
 				d.bdType = valueTypeFloat
 			case bincSpZero:
-				d.bdType = valueTypeUint
+				if d.h.SignedInteger {
+					d.bdType = valueTypeInt
+				} else {
+					d.bdType = valueTypeUint
+				}
 			case bincSpNegOne:
 				d.bdType = valueTypeInt
 			default:
 				decErr("currentEncodedType: Unrecognized special value 0x%x", d.vs)
 			}
-		case bincVdSmallInt:
-			d.bdType = valueTypeUint
-		case bincVdPosInt:
-			d.bdType = valueTypeUint
+		case bincVdSmallInt, bincVdPosInt:
+			if d.h.SignedInteger {
+				d.bdType = valueTypeInt
+			} else {
+				d.bdType = valueTypeUint
+			}
 		case bincVdNegInt:
 			d.bdType = valueTypeInt
 		case bincVdFloat:
@@ -664,7 +688,20 @@ func (d *bincDecDriver) decodeBytes(bs []byte) (bsOut []byte, changed bool) {
 	return
 }
 
-func (d *bincDecDriver) decodeExt(verifyTag bool, tag byte) (xtag byte, xbs []byte) {
+func (d *bincDecDriver) decodeExt(rv reflect.Value, xtag uint16, ext Ext, _ *Decoder) (realxtag uint16) {
+	realxtag1, xbs := d.decodeExtV(ext != nil, uint8(xtag))
+	realxtag = uint16(realxtag1)
+	if ext == nil {
+		re := rv.Interface().(*RawExt)
+		re.Tag = realxtag
+		re.Data = xbs
+	} else {
+		ext.ReadExt(rv, xbs)
+	}
+	return
+}
+
+func (d *bincDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs []byte) {
 	switch d.vd {
 	case bincVdCustomExt:
 		l := d.decLen()
@@ -682,7 +719,7 @@ func (d *bincDecDriver) decodeExt(verifyTag bool, tag byte) (xtag byte, xbs []by
 	return
 }
 
-func (d *bincDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
+func (d *bincDecDriver) decodeNaked(_ *Decoder) (v interface{}, vt valueType, decodeFurther bool) {
 	d.initReadNext()
 
 	switch d.vd {
@@ -710,7 +747,7 @@ func (d *bincDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurthe
 			v = float64(0)
 		case bincSpZero:
 			vt = valueTypeUint
-			v = int64(0) // int8(0)
+			v = uint64(0) // int8(0)
 		case bincSpNegOne:
 			vt = valueTypeInt
 			v = int64(-1) // int8(-1)
@@ -749,7 +786,7 @@ func (d *bincDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurthe
 		vt = valueTypeExt
 		l := d.decLen()
 		var re RawExt
-		re.Tag = d.r.readn1()
+		re.Tag = uint16(d.r.readn1())
 		re.Data = d.r.readn(l)
 		v = &re
 		vt = valueTypeExt
@@ -766,6 +803,10 @@ func (d *bincDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurthe
 	if !decodeFurther {
 		d.bdRead = false
 	}
+	if vt == valueTypeUint && d.h.SignedInteger {
+		d.bdType = valueTypeInt
+		v = int64(v.(uint64))
+	}
 	return
 }
 
@@ -781,6 +822,7 @@ func (d *bincDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurthe
 //    extended precision and decimal IEEE 754 floats are unsupported.
 //  - Only UTF-8 strings supported.
 //    Unicode_Other Binc types (UTF16, UTF32) are currently unsupported.
+//
 //Note that these EXCEPTIONS are temporary and full support is possible and may happen soon.
 type BincHandle struct {
 	BasicHandle
@@ -791,13 +833,8 @@ func (h *BincHandle) newEncDriver(w encWriter) encDriver {
 }
 
 func (h *BincHandle) newDecDriver(r decReader) decDriver {
-	return &bincDecDriver{r: r}
+	return &bincDecDriver{r: r, h: h}
 }
 
-func (_ *BincHandle) writeExt() bool {
-	return true
-}
-
-func (h *BincHandle) getBasicHandle() *BasicHandle {
-	return &h.BasicHandle
-}
+var _ decDriver = (*bincDecDriver)(nil)
+var _ encDriver = (*bincEncDriver)(nil)

codec/cbor.go

@@ -0,0 +1,555 @@
+// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved.
+// Use of this source code is governed by a BSD-style license found in the LICENSE file.
+
+package codec
+
+import (
+	"math"
+	"reflect"
+)
+
+const (
+	cborMajorUint byte = iota
+	cborMajorNegInt
+	cborMajorBytes
+	cborMajorText
+	cborMajorArray
+	cborMajorMap
+	cborMajorTag
+	cborMajorOther
+)
+
+const (
+	cborBdFalse byte = 0xf4 + iota
+	cborBdTrue
+	cborBdNil
+	cborBdUndefined
+	cborBdExt
+	cborBdFloat16
+	cborBdFloat32
+	cborBdFloat64
+)
+
+const (
+	cborBdIndefiniteBytes  byte = 0x5f
+	cborBdIndefiniteString      = 0x7f
+	cborBdIndefiniteArray       = 0x9f
+	cborBdIndefiniteMap         = 0xbf
+	cborBdBreak                 = 0xff
+)
+
+const (
+	CborStreamBytes  byte = 0x5f
+	CborStreamString      = 0x7f
+	CborStreamArray       = 0x9f
+	CborStreamMap         = 0xbf
+	cborStreamBreak       = 0xff
+)
+
+const (
+	cborBaseUint   byte = 0x00
+	cborBaseNegInt      = 0x20
+	cborBaseBytes       = 0x40
+	cborBaseString      = 0x60
+	cborBaseArray       = 0x80
+	cborBaseMap         = 0xa0
+	cborBaseTag         = 0xc0
+	cborBaseSimple      = 0xe0
+)
+
+// -------------------
+
+type cborEncDriver struct {
+	w encWriter
+	h *CborHandle
+	noBuiltInTypes
+}
+
+func (e *cborEncDriver) encodeNil() {
+	e.w.writen1(cborBdNil)
+}
+
+func (e *cborEncDriver) encodeBool(b bool) {
+	if b {
+		e.w.writen1(cborBdTrue)
+	} else {
+		e.w.writen1(cborBdFalse)
+	}
+}
+
+func (e *cborEncDriver) encodeFloat32(f float32) {
+	e.w.writen1(cborBdFloat32)
+	e.w.writeUint32(math.Float32bits(f))
+}
+
+func (e *cborEncDriver) encodeFloat64(f float64) {
+	e.w.writen1(cborBdFloat64)
+	e.w.writeUint64(math.Float64bits(f))
+}
+
+func (e *cborEncDriver) encUint(v uint64, bd byte) {
+	switch {
+	case v <= 0x17:
+		e.w.writen1(byte(v) + bd)
+	case v <= math.MaxUint8:
+		e.w.writen2(bd+0x18, uint8(v))
+	case v <= math.MaxUint16:
+		e.w.writen1(bd + 0x19)
+		e.w.writeUint16(uint16(v))
+	case v <= math.MaxUint32:
+		e.w.writen1(bd + 0x1a)
+		e.w.writeUint32(uint32(v))
+	case v <= math.MaxUint64:
+		e.w.writen1(bd + 0x1b)
+		e.w.writeUint64(v)
+	}
+}
+
+func (e *cborEncDriver) encodeInt(v int64) {
+	if v < 0 {
+		e.encUint(uint64(-1-v), cborBaseNegInt)
+	} else {
+		e.encUint(uint64(v), cborBaseUint)
+	}
+}
+
+func (e *cborEncDriver) encodeUint(v uint64) {
+	e.encUint(v, cborBaseUint)
+}
+
+func (e *cborEncDriver) encLen(bd byte, length int) {
+	e.encUint(uint64(length), bd)
+}
+
+func (e *cborEncDriver) encodeExt(rv reflect.Value, xtag uint16, ext Ext, en *Encoder) {
+	e.encUint(uint64(xtag), cborBaseTag)
+	if v := ext.ConvertExt(rv); v == nil {
+		e.encodeNil()
+	} else {
+		en.encode(v)
+	}
+}
+
+func (e *cborEncDriver) encodeRawExt(re *RawExt, en *Encoder) {
+	e.encUint(uint64(re.Tag), cborBaseTag)
+	if re.Data != nil {
+		en.encode(re.Data)
+	} else if re.Value == nil {
+		e.encodeNil()
+	} else {
+		en.encode(re.Value)
+	}
+}
+
+func (e *cborEncDriver) encodeArrayPreamble(length int) {
+	e.encLen(cborBaseArray, length)
+}
+
+func (e *cborEncDriver) encodeMapPreamble(length int) {
+	e.encLen(cborBaseMap, length)
+}
+
+func (e *cborEncDriver) encodeString(c charEncoding, v string) {
+	e.encLen(cborBaseString, len(v))
+	e.w.writestr(v)
+}
+
+func (e *cborEncDriver) encodeSymbol(v string) {
+	e.encodeString(c_UTF8, v)
+}
+
+func (e *cborEncDriver) encodeStringBytes(c charEncoding, v []byte) {
+	e.encLen(cborBaseBytes, len(v))
+	e.w.writeb(v)
+}
+
+// ----------------------
+
+type cborDecDriver struct {
+	h      *CborHandle
+	r      decReader
+	bdRead bool
+	bdType valueType
+	bd     byte
+	noBuiltInTypes
+}
+
+func (d *cborDecDriver) initReadNext() {
+	if d.bdRead {
+		return
+	}
+	d.bd = d.r.readn1()
+	d.bdRead = true
+	d.bdType = valueTypeUnset
+}
+
+func (d *cborDecDriver) currentEncodedType() valueType {
+	if d.bdType == valueTypeUnset {
+		switch d.bd {
+		case cborBdNil:
+			d.bdType = valueTypeNil
+		case cborBdFalse, cborBdTrue:
+			d.bdType = valueTypeBool
+		case cborBdFloat16, cborBdFloat32, cborBdFloat64:
+			d.bdType = valueTypeFloat
+		case cborBdIndefiniteBytes:
+			d.bdType = valueTypeBytes
+		case cborBdIndefiniteString:
+			d.bdType = valueTypeString
+		case cborBdIndefiniteArray:
+			d.bdType = valueTypeArray
+		case cborBdIndefiniteMap:
+			d.bdType = valueTypeMap
+		default:
+			switch {
+			case d.bd >= cborBaseUint && d.bd < cborBaseNegInt:
+				if d.h.SignedInteger {
+					d.bdType = valueTypeInt
+				} else {
+					d.bdType = valueTypeUint
+				}
+			case d.bd >= cborBaseNegInt && d.bd < cborBaseBytes:
+				d.bdType = valueTypeInt
+			case d.bd >= cborBaseBytes && d.bd < cborBaseString:
+				d.bdType = valueTypeBytes
+			case d.bd >= cborBaseString && d.bd < cborBaseArray:
+				d.bdType = valueTypeString
+			case d.bd >= cborBaseArray && d.bd < cborBaseMap:
+				d.bdType = valueTypeArray
+			case d.bd >= cborBaseMap && d.bd < cborBaseTag:
+				d.bdType = valueTypeMap
+			case d.bd >= cborBaseTag && d.bd < cborBaseSimple:
+				d.bdType = valueTypeExt
+			default:
+				decErr("currentEncodedType: Unrecognized d.bd: 0x%x", d.bd)
+			}
+		}
+	}
+	return d.bdType
+}
+
+func (d *cborDecDriver) tryDecodeAsNil() bool {
+	if d.bd == cborBdNil {
+		d.bdRead = false
+		return true
+	}
+	return false
+}
+
+func (d *cborDecDriver) checkBreak() bool {
+	d.initReadNext()
+	if d.bd == cborBdBreak {
+		d.bdRead = false
+		return true
+	}
+	return false
+}
+
+func (d *cborDecDriver) decUint() (ui uint64) {
+	v := d.bd & 0x1f
+	if v <= 0x17 {
+		ui = uint64(v)
+	} else {
+		switch v {
+		case 0x18:
+			ui = uint64(d.r.readn1())
+		case 0x19:
+			ui = uint64(d.r.readUint16())
+		case 0x1a:
+			ui = uint64(d.r.readUint32())
+		case 0x1b:
+			ui = uint64(d.r.readUint64())
+		default:
+			decErr("decIntAny: Invalid descriptor: %v", d.bd)
+		}
+	}
+	return
+}
+
+func (d *cborDecDriver) decIntAny() (ui uint64, i int64, neg bool) {
+	switch major := d.bd >> 5; major {
+	case cborMajorUint:
+	case cborMajorNegInt:
+		neg = true
+	default:
+		decErr("decIntAny: invalid major: %v (bd: %v)", major, d.bd)
+	}
+	ui = d.decUint()
+	if neg {
+		i = -int64(ui + 1)
+	} else {
+		i = int64(ui)
+	}
+	return
+}
+
+func (d *cborDecDriver) decodeInt(bitsize uint8) (i int64) {
+	_, i, _ = d.decIntAny()
+	checkOverflow(0, i, bitsize)
+	d.bdRead = false
+	return
+}
+
+func (d *cborDecDriver) decodeUint(bitsize uint8) (ui uint64) {
+	ui, i, neg := d.decIntAny()
+	if neg {
+		decErr("Assigning negative signed value: %v, to unsigned type", i)
+	}
+	checkOverflow(ui, 0, bitsize)
+	d.bdRead = false
+	return
+}
+
+func (d *cborDecDriver) decodeFloat(chkOverflow32 bool) (f float64) {
+	switch d.bd {
+	case cborBdFloat16:
+		f = math.Float64frombits(halfFloatToDoubleBits(d.r.readUint16()))
+	case cborBdFloat32:
+		f = float64(math.Float32frombits(d.r.readUint32()))
+	case cborBdFloat64:
+		f = math.Float64frombits(d.r.readUint64())
+	default:
+		if d.bd >= cborBaseUint && d.bd < cborBaseBytes {
+			_, i, _ := d.decIntAny()
+			f = float64(i)
+		} else {
+			decErr("Float only valid from float16/32/64: Invalid descriptor: %v", d.bd)
+		}
+	}
+	checkOverflowFloat32(f, chkOverflow32)
+	d.bdRead = false
+	return
+}
+
+// bool can be decoded from bool only (single byte).
+func (d *cborDecDriver) decodeBool() (b bool) {
+	switch d.bd {
+	case cborBdTrue:
+		b = true
+	case cborBdFalse:
+	default:
+		decErr("Invalid single-byte value for bool: %s: %x", msgBadDesc, d.bd)
+	}
+	d.bdRead = false
+	return
+}
+
+func (d *cborDecDriver) readMapLen() (length int) {
+	d.bdRead = false
+	if d.bd == cborBdIndefiniteMap {
+		return -1
+	}
+	return d.decLen()
+}
+
+func (d *cborDecDriver) readArrayLen() (length int) {
+	d.bdRead = false
+	if d.bd == cborBdIndefiniteArray {
+		return -1
+	}
+	return d.decLen()
+}
+
+func (d *cborDecDriver) decLen() int {
+	return int(d.decUint())
+}
+
+func (d *cborDecDriver) decIndefiniteBytes(bs []byte) []byte {
+	d.bdRead = false
+	for {
+		if d.checkBreak() {
+			break
+		}
+		if major := d.bd >> 5; major != cborMajorBytes && major != cborMajorText {
+			decErr("cbor: expect bytes or string major type in indefinite string/bytes; got: %v, byte: %v", major, d.bd)
+		}
+		bs = append(bs, d.r.readn(d.decLen())...)
+		d.bdRead = false
+	}
+	d.bdRead = false
+	return bs
+}
+
+func (d *cborDecDriver) decodeString() (s string) {
+	if d.bd == cborBdIndefiniteBytes || d.bd == cborBdIndefiniteString {
+		s = string(d.decIndefiniteBytes(nil))
+		return
+	}
+	s = string(d.r.readn(d.decLen()))
+	d.bdRead = false
+	return
+}
+
+func (d *cborDecDriver) decodeBytes(bs []byte) (bsOut []byte, changed bool) {
+	if d.bd == cborBdIndefiniteBytes || d.bd == cborBdIndefiniteString {
+		bsOut = d.decIndefiniteBytes(bs[:0])
+		if len(bsOut) != len(bs) {
+			changed = true
+		}
+		return
+	}
+	clen := d.decLen()
+	if clen > 0 {
+		// if no contents in stream, don't update the passed byteslice
+		if len(bs) != clen {
+			if len(bs) > clen {
+				bs = bs[:clen]
+			} else {
+				bs = make([]byte, clen)
+			}
+			bsOut = bs
+			changed = true
+		}
+		d.r.readb(bs)
+	}
+	d.bdRead = false
+	return
+}
+
+func (d *cborDecDriver) decodeExt(rv reflect.Value, xtag uint16, ext Ext, de *Decoder) (realxtag uint16) {
+	u := d.decUint()
+	d.bdRead = false
+	if u > math.MaxInt16 {
+		decErr("decodeExt: tag must be <= 0xffff; got: %v", u)
+	}
+	realxtag = uint16(u)
+	if ext == nil {
+		re := rv.Interface().(*RawExt)
+		re.Tag = realxtag
+		de.decode(&re.Value)
+	} else if xtag != realxtag {
+		decErr("Wrong extension tag. Got %b. Expecting: %v", realxtag, xtag)
+	} else {
+		var v interface{}
+		de.decode(&v)
+		ext.UpdateExt(rv, v)
+	}
+	d.bdRead = false
+	return
+}
+
+func (d *cborDecDriver) decodeNaked(de *Decoder) (v interface{}, vt valueType, decodeFurther bool) {
+	d.initReadNext()
+
+	switch d.bd {
+	case cborBdNil:
+		vt = valueTypeNil
+	case cborBdFalse:
+		vt = valueTypeBool
+		v = false
+	case cborBdTrue:
+		vt = valueTypeBool
+		v = true
+	case cborBdFloat16, cborBdFloat32:
+		vt = valueTypeFloat
+		v = d.decodeFloat(true)
+	case cborBdFloat64:
+		vt = valueTypeFloat
+		v = d.decodeFloat(false)
+	case cborBdIndefiniteBytes:
+		vt = valueTypeBytes
+		v, _ = d.decodeBytes(nil)
+	case cborBdIndefiniteString:
+		vt = valueTypeString
+		v = d.decodeString()
+	case cborBdIndefiniteArray:
+		vt = valueTypeArray
+		decodeFurther = true
+	case cborBdIndefiniteMap:
+		vt = valueTypeMap
+		decodeFurther = true
+	default:
+		switch {
+		case d.bd >= cborBaseUint && d.bd < cborBaseNegInt:
+			ui, i, _ := d.decIntAny()
+			if d.h.SignedInteger {
+				vt = valueTypeInt
+				v = i
+			} else {
+				vt = valueTypeUint
+				v = ui
+			}
+		case d.bd >= cborBaseNegInt && d.bd < cborBaseBytes:
+			vt = valueTypeInt
+			_, i, _ := d.decIntAny()
+			v = i
+		case d.bd >= cborBaseBytes && d.bd < cborBaseString:
+			vt = valueTypeBytes
+			v, _ = d.decodeBytes(nil)
+		case d.bd >= cborBaseString && d.bd < cborBaseArray:
+			vt = valueTypeString
+			v = d.decodeString()
+		case d.bd >= cborBaseArray && d.bd < cborBaseMap:
+			vt = valueTypeArray
+			decodeFurther = true
+		case d.bd >= cborBaseMap && d.bd < cborBaseTag:
+			vt = valueTypeMap
+			decodeFurther = true
+		case d.bd >= cborBaseTag && d.bd < cborBaseSimple:
+			vt = valueTypeExt
+			var re RawExt
+			ui := d.decUint()
+			d.bdRead = false
+			if ui > math.MaxInt16 {
+				decErr("decodeExt: tag must be <= 0xffff; got: %v", ui)
+			}
+			re.Tag = uint16(ui)
+			de.decode(&re.Value)
+			v = &re
+			// decodeFurther = true
+		default:
+			decErr("decodeNaked: Unrecognized d.bd: 0x%x", d.bd)
+		}
+	}
+
+	if !decodeFurther {
+		d.bdRead = false
+	}
+	return
+}
+
+// -------------------------
+
+// CborHandle is a Handle for the CBOR encoding format, defined at http://cbor.io .
+//
+// CBOR is comprehensively supported, including support for:
+//   - indefinite-length arrays/maps/bytes/strings
+//   - (extension) tags in range 0..0xffff (0 .. 65535)
+//   - half, single and double-precision floats
+//   - all numbers (1, 2, 4 and 8-byte signed and unsigned integers)
+//   - nil, true, false, ...
+//   - arrays and maps, bytes and text strings
+//
+// None of the optional extensions (with tags) defined in the spec are supported out-of-the-box.
+// Users can implement them as needed (using SetExt), including spec-documented ones:
+//   - timestamp, BigNum, BigFloat, Decimals, Encoded Text (e.g. URL, regexp, base64, MIME Message), etc.
+//
+// To encode with indefinite lengths (streaming), users will use
+// (Must)Write and (Must)Encode methods of *Encoder, along with CborStreamXXX constants.
+//
+// For example, to encode "one-byte" as an indefinite length string:
+//     var buf bytes.Buffer
+//     e := NewEncoder(&buf, new(CborHandle))
+//     e.MustWrite([]byte{CborStreamString})
+//     e.MustEncode("one-")
+//     e.MustEncode("byte")
+//     e.MustWrite([]byte{CborStreamBreak})
+//     encodedBytes := buf.Bytes()
+//     var vv interface{}
+//     NewDecoderBytes(buf.Bytes(), new(CborHandle)).MustDecode(&vv)
+//     // Now, vv contains the same string "one-byte"
+//
+type CborHandle struct {
+	BasicHandle
+}
+
+func (h *CborHandle) newEncDriver(w encWriter) encDriver {
+	return &cborEncDriver{w: w, h: h}
+}
+
+func (h *CborHandle) newDecDriver(r decReader) decDriver {
+	return &cborDecDriver{r: r, h: h}
+}
+
+var _ decDriver = (*cborDecDriver)(nil)
+var _ encDriver = (*cborEncDriver)(nil)

codec/codecs_test.go

@@ -16,6 +16,11 @@ package codec
 //
 // Taken together, the tests are pretty extensive.
 
+// The following manual tests must be done:
+//   - TestCodecUnderlyingType
+//   - Set fastpathEnabled to false and run tests (to ensure that regular reflection works).
+//     We don't want to use a variable there so that code is ellided.
+
 import (
 	"bytes"
 	"encoding/gob"
@@ -61,10 +66,11 @@ var (
 	// It's unnecessary, and makes it harder to do a reflect.DeepEqual.
 	// The Offset already tells what the offset should be, if not on UTC and unknown zone name.
 	timeLoc        = time.FixedZone("", -8*60*60) // UTC-08:00 //time.UTC-8
-	timeToCompare1 = time.Date(2012, 2, 2, 2, 2, 2, 2000, timeLoc)
-	timeToCompare2 = time.Date(1900, 2, 2, 2, 2, 2, 2000, timeLoc)
-	timeToCompare3 = time.Unix(0, 0).UTC()
-	timeToCompare4 = time.Time{}.UTC()
+	timeToCompare1 = time.Date(2012, 2, 2, 2, 2, 2, 2000, timeLoc).UTC()
+	timeToCompare2 = time.Date(1900, 2, 2, 2, 2, 2, 2000, timeLoc).UTC()
+	timeToCompare3 = time.Unix(0, 270).UTC() // use value that must be encoded as uint64 for nanoseconds (for cbor/msgpack comparison)
+	//timeToCompare4 = time.Time{}.UTC() // does not work well with simple cbor time encoding (overflow)
+	timeToCompare4 = time.Unix(-2013855848, 4223).UTC()
 
 	table              []interface{} // main items we encode
 	tableVerify        []interface{} // we verify encoded things against this after decode
@@ -75,6 +81,7 @@ var (
 	testMsgpackH = &MsgpackHandle{}
 	testBincH    = &BincHandle{}
 	testSimpleH  = &SimpleHandle{}
+	testCborH    = &CborHandle{}
 )
 
 func testInitFlags() {
@@ -151,31 +158,51 @@ func (r *TestRpcInt) Echo123(args []string, res *string) error {
 	return nil
 }
 
+type testCborTimeExt struct{}
+
+func (x testCborTimeExt) WriteExt(reflect.Value) []byte { panic("unsupported") }
+func (x testCborTimeExt) ReadExt(reflect.Value, []byte) { panic("unsupported") }
+func (x testCborTimeExt) ConvertExt(rv reflect.Value) interface{} {
+	return rv.Interface().(time.Time).UTC().UnixNano()
+}
+func (x testCborTimeExt) UpdateExt(rv reflect.Value, v interface{}) {
+	var tt time.Time
+	switch v2 := v.(type) {
+	case int64:
+		tt = time.Unix(0, v2).UTC()
+	case uint64:
+		tt = time.Unix(0, int64(v2)).UTC()
+	default:
+		panic(fmt.Sprintf("unsupported format for time conversion: expecting int64/uint64; got %T", v))
+	}
+	rv.Set(reflect.ValueOf(tt))
+}
+
 func testVerifyVal(v interface{}, arg testVerifyArg) (v2 interface{}) {
 	//for python msgpack,
 	//  - all positive integers are unsigned 64-bit ints
 	//  - all floats are float64
 	switch iv := v.(type) {
 	case int8:
-		if iv > 0 {
+		if iv >= 0 {
 			v2 = uint64(iv)
 		} else {
 			v2 = int64(iv)
 		}
 	case int16:
-		if iv > 0 {
+		if iv >= 0 {
 			v2 = uint64(iv)
 		} else {
 			v2 = int64(iv)
 		}
 	case int32:
-		if iv > 0 {
+		if iv >= 0 {
 			v2 = uint64(iv)
 		} else {
 			v2 = int64(iv)
 		}
 	case int64:
-		if iv > 0 {
+		if iv >= 0 {
 			v2 = uint64(iv)
 		} else {
 			v2 = int64(iv)
@@ -249,7 +276,7 @@ func testVerifyVal(v interface{}, arg testVerifyArg) (v2 interface{}) {
 	case time.Time:
 		switch arg {
 		case testVerifyForPython:
-			if iv2 := iv.UnixNano(); iv2 > 0 {
+			if iv2 := iv.UnixNano(); iv2 >= 0 {
 				v2 = uint64(iv2)
 			} else {
 				v2 = int64(iv2)
@@ -270,6 +297,8 @@ func testInit() {
 		fmt.Printf("====> depth: %v, ts: %#v\n", 2, ts0)
 	}
 
+	testCborH.StructToArray = testStructToArray
+	testSimpleH.StructToArray = testStructToArray
 	testBincH.StructToArray = testStructToArray
 	if testWriteNoSymbols {
 		testBincH.AsSymbols = AsSymbolNone
@@ -286,7 +315,9 @@ func testInit() {
 	timeDecExt := func(rv reflect.Value, bs []byte) error {
 		tt, err := decodeTime(bs)
 		if err == nil {
+			// fmt.Printf("+++++++++++++++++ decfn: before rv: %v\n", rv)
 			rv.Set(reflect.ValueOf(tt))
+			// fmt.Printf("+++++++++++++++++ decfn: after rv: %v\n", rv)
 		}
 		return err
 	}
@@ -294,6 +325,7 @@ func testInit() {
 	// add extensions for msgpack, simple for time.Time, so we can encode/decode same way.
 	testMsgpackH.AddExt(timeTyp, 1, timeEncExt, timeDecExt)
 	testSimpleH.AddExt(timeTyp, 1, timeEncExt, timeDecExt)
+	testCborH.SetExt(timeTyp, 1, &testCborTimeExt{})
 
 	primitives := []interface{}{
 		int8(-8),
@@ -407,21 +439,23 @@ func testInit() {
 	tablePythonVerify = tablePythonVerify[:24]
 }
 
-func testUnmarshal(v interface{}, data []byte, h Handle) error {
+func testUnmarshal(v interface{}, data []byte, h Handle) (err error) {
 	if testUseIoEncDec {
-		return NewDecoder(bytes.NewBuffer(data), h).Decode(v)
+		NewDecoder(bytes.NewBuffer(data), h).MustDecode(v)
+	} else {
+		NewDecoderBytes(data, h).MustDecode(v)
 	}
-	return NewDecoderBytes(data, h).Decode(v)
+	return
 }
 
 func testMarshal(v interface{}, h Handle) (bs []byte, err error) {
 	if testUseIoEncDec {
 		var buf bytes.Buffer
-		err = NewEncoder(&buf, h).Encode(v)
+		NewEncoder(&buf, h).MustEncode(v)
 		bs = buf.Bytes()
 		return
 	}
-	err = NewEncoderBytes(&bs, h).Encode(v)
+	NewEncoderBytes(&bs, h).MustEncode(v)
 	return
 }
 
@@ -556,8 +590,11 @@ func doTestCodecTableOne(t *testing.T, testNil bool, h Handle,
 		if err = deepEqual(v0check, v1); err == nil {
 			logT(t, "++++++++ Before and After marshal matched\n")
 		} else {
-			logT(t, "-------- Before and After marshal do not match: Error: %v"+
-				" ====> GOLDEN: (%T) %#v, DECODED: (%T) %#v\n", err, v0check, v0check, v1, v1)
+			// logT(t, "-------- Before and After marshal do not match: Error: %v"+
+			// 	" ====> GOLDEN: (%T) %#v, DECODED: (%T) %#v\n", err, v0check, v0check, v1, v1)
+			logT(t, "-------- Before and After marshal do not match: Error: %v", err)
+			logT(t, "    ....... GOLDEN:  (%T) %#v", v0check, v0check)
+			logT(t, "    ....... DECODED: (%T) %#v", v1, v1)
 			failT(t)
 		}
 	}
@@ -819,32 +856,37 @@ func doTestRpcOne(t *testing.T, rr Rpc, h Handle, doRequest bool, exitSleepMs ti
 	return
 }
 
-// Comprehensive testing that generates data encoded from python msgpack,
+// Comprehensive testing that generates data encoded from python handle (cbor, msgpack),
 // and validates that our code can read and write it out accordingly.
 // We keep this unexported here, and put actual test in ext_dep_test.go.
 // This way, it can be excluded by excluding file completely.
-func doTestMsgpackPythonGenStreams(t *testing.T) {
-	logT(t, "TestPythonGenStreams")
-	tmpdir, err := ioutil.TempDir("", "golang-msgpack-test")
+func doTestPythonGenStreams(t *testing.T, name string, h Handle) {
+	logT(t, "TestPythonGenStreams-%v", name)
+	tmpdir, err := ioutil.TempDir("", "golang-"+name+"-test")
 	if err != nil {
 		logT(t, "-------- Unable to create temp directory\n")
 		t.FailNow()
 	}
 	defer os.RemoveAll(tmpdir)
 	logT(t, "tmpdir: %v", tmpdir)
-	cmd := exec.Command("python", "msgpack_test.py", "testdata", tmpdir)
+	cmd := exec.Command("python", "test.py", "testdata", tmpdir)
 	//cmd.Stdin = strings.NewReader("some input")
 	//cmd.Stdout = &out
 	var cmdout []byte
 	if cmdout, err = cmd.CombinedOutput(); err != nil {
-		logT(t, "-------- Error running msgpack_test.py testdata. Err: %v", err)
+		logT(t, "-------- Error running test.py testdata. Err: %v", err)
 		logT(t, "         %v", string(cmdout))
 		t.FailNow()
 	}
 
-	oldMapType := testMsgpackH.MapType
+	bh := h.getBasicHandle()
+
+	oldMapType := bh.MapType
 	for i, v := range tablePythonVerify {
-		testMsgpackH.MapType = oldMapType
+		// if v == uint64(0) && h == testMsgpackH {
+		// 	v = int64(0)
+		// }
+		bh.MapType = oldMapType
 		//load up the golden file based on number
 		//decode it
 		//compare to in-mem object
@@ -853,16 +895,16 @@ func doTestMsgpackPythonGenStreams(t *testing.T) {
 		logT(t, "..............................................")
 		logT(t, "         Testing: #%d: %T, %#v\n", i, v, v)
 		var bss []byte
-		bss, err = ioutil.ReadFile(filepath.Join(tmpdir, strconv.Itoa(i)+".golden"))
+		bss, err = ioutil.ReadFile(filepath.Join(tmpdir, strconv.Itoa(i)+"."+name+".golden"))
 		if err != nil {
 			logT(t, "-------- Error reading golden file: %d. Err: %v", i, err)
 			failT(t)
 			continue
 		}
-		testMsgpackH.MapType = testMapStrIntfTyp
+		bh.MapType = testMapStrIntfTyp
 
 		var v1 interface{}
-		if err = testUnmarshal(&v1, bss, testMsgpackH); err != nil {
+		if err = testUnmarshal(&v1, bss, h); err != nil {
 			logT(t, "-------- Error decoding stream: %d: Err: %v", i, err)
 			failT(t)
 			continue
@@ -873,14 +915,15 @@ func doTestMsgpackPythonGenStreams(t *testing.T) {
 		//no need to indirect, because we pass a nil ptr, so we already have the value
 		//if v1 != nil { v1 = reflect.Indirect(reflect.ValueOf(v1)).Interface() }
 		if err = deepEqual(v, v1); err == nil {
-			logT(t, "++++++++ Objects match")
+			logT(t, "++++++++ Objects match: %T, %v", v, v)
 		} else {
 			logT(t, "-------- Objects do not match: %v. Source: %T. Decoded: %T", err, v, v1)
-			logT(t, "--------   AGAINST: %#v", v)
+			logT(t, "--------   GOLDEN: %#v", v)
+			// logT(t, "--------   DECODED: %#v <====> %#v", v1, reflect.Indirect(reflect.ValueOf(v1)).Interface())
 			logT(t, "--------   DECODED: %#v <====> %#v", v1, reflect.Indirect(reflect.ValueOf(v1)).Interface())
 			failT(t)
 		}
-		bsb, err := testMarshal(v1, testMsgpackH)
+		bsb, err := testMarshal(v1, h)
 		if err != nil {
 			logT(t, "Error encoding to stream: %d: Err: %v", i, err)
 			failT(t)
@@ -902,7 +945,7 @@ func doTestMsgpackPythonGenStreams(t *testing.T) {
 			logT(t, "%s     ENCODED: %4d] %v", xs, len(bsb), bsb)
 		}
 	}
-	testMsgpackH.MapType = oldMapType
+	bh.MapType = oldMapType
 }
 
 // To test MsgpackSpecRpc, we test 3 scenarios:
@@ -916,7 +959,7 @@ func doTestMsgpackPythonGenStreams(t *testing.T) {
 
 func doTestMsgpackRpcSpecGoClientToPythonSvc(t *testing.T) {
 	openPort := "6789"
-	cmd := exec.Command("python", "msgpack_test.py", "rpc-server", openPort, "2")
+	cmd := exec.Command("python", "test.py", "rpc-server", openPort, "2")
 	checkErrT(t, cmd.Start())
 	time.Sleep(100 * time.Millisecond) // time for python rpc server to start
 	bs, err2 := net.Dial("tcp", ":"+openPort)
@@ -935,11 +978,11 @@ func doTestMsgpackRpcSpecGoClientToPythonSvc(t *testing.T) {
 func doTestMsgpackRpcSpecPythonClientToGoSvc(t *testing.T) {
 	port := doTestRpcOne(t, MsgpackSpecRpc, testMsgpackH, false, 1*time.Second)
 	//time.Sleep(1000 * time.Millisecond)
-	cmd := exec.Command("python", "msgpack_test.py", "rpc-client-go-service", strconv.Itoa(port))
+	cmd := exec.Command("python", "test.py", "rpc-client-go-service", strconv.Itoa(port))
 	var cmdout []byte
 	var err error
 	if cmdout, err = cmd.CombinedOutput(); err != nil {
-		logT(t, "-------- Error running msgpack_test.py rpc-client-go-service. Err: %v", err)
+		logT(t, "-------- Error running test.py rpc-client-go-service. Err: %v", err)
 		logT(t, "         %v", string(cmdout))
 		t.FailNow()
 	}
@@ -983,6 +1026,18 @@ func TestMsgpackCodecsEmbeddedPointer(t *testing.T) {
 	testCodecEmbeddedPointer(t, testMsgpackH)
 }
 
+func TestCborCodecsTable(t *testing.T) {
+	testCodecTableOne(t, testCborH)
+}
+
+func TestCborCodecsMisc(t *testing.T) {
+	testCodecMiscOne(t, testCborH)
+}
+
+func TestCborCodecsEmbeddedPointer(t *testing.T) {
+	testCodecEmbeddedPointer(t, testCborH)
+}
+
 func TestBincRpcGo(t *testing.T) {
 	doTestRpcOne(t, GoRpc, testBincH, true, 0)
 }
@@ -995,6 +1050,10 @@ func TestMsgpackRpcGo(t *testing.T) {
 	doTestRpcOne(t, GoRpc, testMsgpackH, true, 0)
 }
 
+func TestCborRpcGo(t *testing.T) {
+	doTestRpcOne(t, GoRpc, testCborH, true, 0)
+}
+
 func TestMsgpackRpcSpec(t *testing.T) {
 	doTestRpcOne(t, MsgpackSpecRpc, testMsgpackH, true, 0)
 }
@@ -1006,20 +1065,97 @@ func TestCodecUnderlyingType(t *testing.T) {
 	type T1 map[string]string
 	v := T1{"1": "1s", "2": "2s"}
 	var bs []byte
-	err := NewEncoderBytes(&bs, testBincH).Encode(v)
+	var err error
+	NewEncoderBytes(&bs, testBincH).MustEncode(v)
 	if err != nil {
 		logT(t, "Error during encode: %v", err)
 		failT(t)
 	}
 	var v2 T1
-	err = NewDecoderBytes(bs, testBincH).Decode(&v2)
+	NewDecoderBytes(bs, testBincH).MustDecode(&v2)
 	if err != nil {
 		logT(t, "Error during decode: %v", err)
 		failT(t)
 	}
 }
 
+func TestCborIndefiniteLength(t *testing.T) {
+	oldMapType := testCborH.MapType
+	defer func() {
+		testCborH.MapType = oldMapType
+	}()
+	testCborH.MapType = testMapStrIntfTyp
+	// var (
+	// 	M1 map[string][]byte
+	// 	M2 map[uint64]bool
+	// 	L1 []interface{}
+	// 	S1 []string
+	// 	B1 []byte
+	// )
+	var v, vv interface{}
+	// define it (v), encode it using indefinite lengths, decode it (vv), compare v to vv
+	v = map[string]interface{}{
+		"one-byte-key":   []byte{1, 2, 3, 4, 5, 6},
+		"two-string-key": "two-value",
+		"three-list-key": []interface{}{true, false, uint64(1), int64(-1)},
+	}
+	var buf bytes.Buffer
+	// buf.Reset()
+	e := NewEncoder(&buf, testCborH)
+	buf.WriteByte(cborBdIndefiniteMap)
+	//----
+	buf.WriteByte(cborBdIndefiniteString)
+	e.MustEncode("one-")
+	e.MustEncode("byte-")
+	e.MustEncode("key")
+	buf.WriteByte(cborBdBreak)
+
+	buf.WriteByte(cborBdIndefiniteBytes)
+	e.MustEncode([]byte{1, 2, 3})
+	e.MustEncode([]byte{4, 5, 6})
+	buf.WriteByte(cborBdBreak)
+
+	//----
+	buf.WriteByte(cborBdIndefiniteString)
+	e.MustEncode("two-")
+	e.MustEncode("string-")
+	e.MustEncode("key")
+	buf.WriteByte(cborBdBreak)
+
+	buf.WriteByte(cborBdIndefiniteString)
+	e.MustEncode([]byte("two-")) // encode as bytes, to check robustness of code
+	e.MustEncode([]byte("value"))
+	buf.WriteByte(cborBdBreak)
+
+	//----
+	buf.WriteByte(cborBdIndefiniteString)
+	e.MustEncode("three-")
+	e.MustEncode("list-")
+	e.MustEncode("key")
+	buf.WriteByte(cborBdBreak)
+
+	buf.WriteByte(cborBdIndefiniteArray)
+	e.MustEncode(true)
+	e.MustEncode(false)
+	e.MustEncode(uint64(1))
+	e.MustEncode(int64(-1))
+	buf.WriteByte(cborBdBreak)
+
+	buf.WriteByte(cborBdBreak) // close map
+
+	NewDecoderBytes(buf.Bytes(), testCborH).MustDecode(&vv)
+	if err := deepEqual(v, vv); err != nil {
+		logT(t, "-------- Before and After marshal do not match: Error: %v", err)
+		logT(t, "    ....... GOLDEN:  (%T) %#v", v, v)
+		logT(t, "    ....... DECODED: (%T) %#v", vv, vv)
+		failT(t)
+	}
+}
+
 // TODO:
 //   Add Tests for:
 //   - decoding empty list/map in stream into a nil slice/map
 //   - binary(M|Unm)arsher support for time.Time
+//   - non fast-path scenarios e.g. map[string]uint16, []customStruct.
+//     Expand cbor to include indefinite length stuff for this non-fast-path types.
+//     This may not be necessary, since we have the manual tests (fastpathEnabled=false) to test/validate with.

codec/decode.go

@@ -16,7 +16,7 @@ const (
 	msgDecCannotExpandArr = "cannot expand go array from %v to stream length: %v"
 )
 
-// fastpathsEnc holds the rtid (reflect.Type Pointer) to fast decode function for a selected slice/map type.
+// fastpathsDec holds the rtid (reflect.Type Pointer) to fast decode function for a selected slice/map type.
 var fastpathsDec = make(map[uintptr]func(*decFnInfo, reflect.Value))
 
 // decReader abstracts the reading source, allowing implementations that can
@@ -32,12 +32,15 @@ type decReader interface {
 
 type decDriver interface {
 	initReadNext()
+	// this will call initReadNext implicitly, and then check if the next token is a break.
+	checkBreak() bool
 	tryDecodeAsNil() bool
 	currentEncodedType() valueType
 	isBuiltinType(rt uintptr) bool
 	decodeBuiltin(rt uintptr, v interface{})
 	//decodeNaked: Numbers are decoded as int64, uint64, float64 only (no smaller sized number types).
-	decodeNaked() (v interface{}, vt valueType, decodeFurther bool)
+	//for extensions, decodeNaked must completely decode them as a *RawExt.
+	decodeNaked(*Decoder) (v interface{}, vt valueType, decodeFurther bool)
 	decodeInt(bitsize uint8) (i int64)
 	decodeUint(bitsize uint8) (ui uint64)
 	decodeFloat(chkOverflow32 bool) (f float64)
@@ -45,7 +48,9 @@ type decDriver interface {
 	// decodeString can also decode symbols
 	decodeString() (s string)
 	decodeBytes(bs []byte) (bsOut []byte, changed bool)
-	decodeExt(verifyTag bool, tag byte) (xtag byte, xbs []byte)
+	// decodeExt will decode into a *RawExt or into an extension.
+	decodeExt(rv reflect.Value, xtag uint16, ext Ext, d *Decoder) (realxtag uint16)
+	// decodeExt(verifyTag bool, tag byte) (xtag byte, xbs []byte)
 	readMapLen() int
 	readArrayLen() int
 }
@@ -60,6 +65,8 @@ type DecodeOptions struct {
 	// ErrorIfNoField controls whether an error is returned when decoding a map
 	// from a codec stream into a struct, and no matching struct field is found.
 	ErrorIfNoField bool
+	// If true, use the int64 during schema-less decoding of unsigned values (not uint64).
+	SignedInteger bool
 }
 
 // ------------------------------------
@@ -183,8 +190,8 @@ type decFnInfo struct {
 	ti    *typeInfo
 	d     *Decoder
 	dd    decDriver
-	xfFn  func(reflect.Value, []byte) error
-	xfTag byte
+	xfFn  Ext
+	xfTag uint16
 	array bool
 }
 
@@ -200,16 +207,11 @@ func (f *decFnInfo) builtin(rv reflect.Value) {
 }
 
 func (f *decFnInfo) rawExt(rv reflect.Value) {
-	xtag, xbs := f.dd.decodeExt(false, 0)
-	rv.Field(0).SetUint(uint64(xtag))
-	rv.Field(1).SetBytes(xbs)
+	f.dd.decodeExt(rv, 0, nil, f.d)
 }
 
 func (f *decFnInfo) ext(rv reflect.Value) {
-	_, xbs := f.dd.decodeExt(true, f.xfTag)
-	if fnerr := f.xfFn(rv, xbs); fnerr != nil {
-		panic(fnerr)
-	}
+	f.dd.decodeExt(rv, f.xfTag, f.xfFn, f.d)
 }
 
 func (f *decFnInfo) binaryMarshal(rv reflect.Value) {
@@ -310,7 +312,7 @@ func (f *decFnInfo) kInterface(rv reflect.Value) {
 	// nil interface:
 	// use some hieristics to set the nil interface to an
 	// appropriate value based on the first byte read (byte descriptor bd)
-	v, vt, decodeFurther := f.dd.decodeNaked()
+	v, vt, decodeFurther := f.dd.decodeNaked(f.d)
 	if vt == valueTypeNil {
 		return
 	}
@@ -341,12 +343,16 @@ func (f *decFnInfo) kInterface(rv reflect.Value) {
 		}
 	case valueTypeExt:
 		re := v.(*RawExt)
-		var bfn func(reflect.Value, []byte) error
-		rvn, bfn = f.d.h.getDecodeExtForTag(re.Tag)
+		bfn := f.d.h.getExtForTag(re.Tag)
 		if bfn == nil {
 			rvn = reflect.ValueOf(*re)
-		} else if fnerr := bfn(rvn, re.Data); fnerr != nil {
-			panic(fnerr)
+		} else {
+			rvn = reflect.New(bfn.rt).Elem()
+			if re.Data != nil {
+				bfn.ext.ReadExt(rvn, re.Data)
+			} else {
+				bfn.ext.UpdateExt(rvn, re.Value)
+			}
 		}
 		rv.Set(rvn)
 		return
@@ -431,7 +437,6 @@ func (f *decFnInfo) kStruct(rv reflect.Value) {
 func (f *decFnInfo) kSlice(rv reflect.Value) {
 	// A slice can be set from a map or array in stream.
 	currEncodedType := f.dd.currentEncodedType()
-
 	switch currEncodedType {
 	case valueTypeBytes, valueTypeString:
 		if f.ti.rtid == uint8SliceTypId || f.ti.rt.Elem().Kind() == reflect.Uint8 {
@@ -447,34 +452,60 @@ func (f *decFnInfo) kSlice(rv reflect.Value) {
 	// an array can never return a nil slice. so no need to check f.array here.
 
 	if rv.IsNil() {
-		rv.Set(reflect.MakeSlice(f.ti.rt, containerLenS, containerLenS))
+		if containerLenS < 0 {
+			rv.Set(reflect.MakeSlice(f.ti.rt, 0, 0))
+		} else {
+			rv.Set(reflect.MakeSlice(f.ti.rt, containerLenS, containerLenS))
+		}
 	}
 
 	if containerLen == 0 {
 		return
 	}
 
+	rv0 := rv
+	rvChanged := false
+
 	if rvcap, rvlen := rv.Len(), rv.Cap(); containerLenS > rvcap {
 		if f.array { // !rv.CanSet()
 			decErr(msgDecCannotExpandArr, rvcap, containerLenS)
 		}
-		rvn := reflect.MakeSlice(f.ti.rt, containerLenS, containerLenS)
+		rv = reflect.MakeSlice(f.ti.rt, containerLenS, containerLenS)
 		if rvlen > 0 {
-			reflect.Copy(rvn, rv)
+			reflect.Copy(rv, rv0)
 		}
-		rv.Set(rvn)
+		rvChanged = true
 	} else if containerLenS > rvlen {
 		rv.SetLen(containerLenS)
 	}
 
-	rtelem := f.ti.rt.Elem()
+	rtelem0 := f.ti.rt.Elem()
+	rtelem := rtelem0
 	for rtelem.Kind() == reflect.Ptr {
 		rtelem = rtelem.Elem()
 	}
 	fn := f.d.getDecFn(rtelem)
-	for j := 0; j < containerLenS; j++ {
+	// for j := 0; j < containerLenS; j++ {
+
+	for j := 0; ; j++ {
+		if containerLenS >= 0 {
+			if j >= containerLenS {
+				break
+			}
+		} else if f.dd.checkBreak() {
+			break
+		}
+		// if indefinite, etc, then expand the slice if necessary
+		if j >= rv.Len() {
+			// TODO: Need to update this appropriately
+			rv = reflect.Append(rv, reflect.Zero(rtelem0))
+			rvChanged = true
+		}
 		f.d.decodeValue(rv.Index(j), fn)
 	}
+	if rvChanged {
+		rv0.Set(rv)
+	}
 }
 
 func (f *decFnInfo) kArray(rv reflect.Value) {
@@ -503,7 +534,15 @@ func (f *decFnInfo) kMap(rv reflect.Value) {
 	for xtyp = vtype; xtyp.Kind() == reflect.Ptr; xtyp = xtyp.Elem() {
 	}
 	valFn = f.d.getDecFn(xtyp)
-	for j := 0; j < containerLen; j++ {
+	// for j := 0; j < containerLen; j++ {
+	for j := 0; ; j++ {
+		if containerLen >= 0 {
+			if j >= containerLen {
+				break
+			}
+		} else if f.dd.checkBreak() {
+			break
+		}
 		rvk := reflect.New(ktype).Elem()
 		f.d.decodeValue(rvk, keyFn)
 
@@ -741,8 +780,8 @@ func (d *Decoder) getDecFn(rt reflect.Type) (fn decFn) {
 			fn.f = (*decFnInfo).rawExt
 		} else if d.d.isBuiltinType(rtid) {
 			fn.f = (*decFnInfo).builtin
-		} else if xfTag, xfFn := d.h.getDecodeExt(rtid); xfFn != nil {
-			fi.xfTag, fi.xfFn = xfTag, xfFn
+		} else if xfFn := d.h.getExt(rtid); xfFn != nil {
+			fi.xfTag, fi.xfFn = xfFn.tag, xfFn.ext
 			fn.f = (*decFnInfo).ext
 		} else if supportBinaryMarshal && fi.ti.unm {
 			fn.f = (*decFnInfo).binaryMarshal
@@ -865,9 +904,10 @@ func (d *Decoder) decEmbeddedField(rv reflect.Value, index []int) {
 // --------------------------------------------------
 
 func decContLens(dd decDriver, currEncodedType valueType) (containerLen, containerLenS int) {
-	if currEncodedType == valueTypeInvalid {
-		currEncodedType = dd.currentEncodedType()
-	}
+	// currEncodedType = dd.currentEncodedType()
+	// if currEncodedType == valueTypeInvalid {
+	// 	currEncodedType = dd.currentEncodedType()
+	// }
 	switch currEncodedType {
 	case valueTypeArray:
 		containerLen = dd.readArrayLen()

codec/encode.go

@@ -62,7 +62,9 @@ type encDriver interface {
 	encodeBool(b bool)
 	encodeFloat32(f float32)
 	encodeFloat64(f float64)
-	encodeExtPreamble(xtag byte, length int)
+	// encodeExtPreamble(xtag byte, length int)
+	encodeRawExt(re *RawExt, e *Encoder)
+	encodeExt(rv reflect.Value, xtag uint16, ext Ext, e *Encoder)
 	encodeArrayPreamble(length int)
 	encodeMapPreamble(length int)
 	encodeString(c charEncoding, v string)
@@ -275,8 +277,8 @@ type encFnInfo struct {
 	ti    *typeInfo
 	e     *Encoder
 	ee    encDriver
-	xfFn  func(reflect.Value) ([]byte, error)
-	xfTag byte
+	xfFn  Ext
+	xfTag uint16
 }
 
 func (f *encFnInfo) builtin(rv reflect.Value) {
@@ -284,25 +286,11 @@ func (f *encFnInfo) builtin(rv reflect.Value) {
 }
 
 func (f *encFnInfo) rawExt(rv reflect.Value) {
-	f.e.encRawExt(rv.Interface().(RawExt))
+	f.ee.encodeRawExt(rv.Interface().(*RawExt), f.e)
 }
 
 func (f *encFnInfo) ext(rv reflect.Value) {
-	bs, fnerr := f.xfFn(rv)
-	if fnerr != nil {
-		panic(fnerr)
-	}
-	if bs == nil {
-		f.ee.encodeNil()
-		return
-	}
-	if f.e.hh.writeExt() {
-		f.ee.encodeExtPreamble(f.xfTag, len(bs))
-		f.e.w.writeb(bs)
-	} else {
-		f.ee.encodeStringBytes(c_RAW, bs)
-	}
-
+	f.ee.encodeExt(rv, f.xfTag, f.xfFn, f.e)
 }
 
 func (f *encFnInfo) binaryMarshal(rv reflect.Value) {
@@ -699,6 +687,16 @@ func (e *Encoder) MustEncode(v interface{}) {
 	e.w.atEndOfEncode()
 }
 
+func (e *Encoder) Write(bs []byte) (err error) {
+	defer panicToErr(&err)
+	e.w.writeb(bs)
+	return
+}
+
+func (e *Encoder) MustWrite(bs []byte) {
+	e.w.writeb(bs)
+}
+
 func (e *Encoder) encode(iv interface{}) {
 	switch v := iv.(type) {
 	case nil:
@@ -817,8 +815,8 @@ func (e *Encoder) getEncFn(rt reflect.Type) (fn encFn) {
 			fn.f = (*encFnInfo).rawExt
 		} else if e.e.isBuiltinType(rtid) {
 			fn.f = (*encFnInfo).builtin
-		} else if xfTag, xfFn := e.h.getEncodeExt(rtid); xfFn != nil {
-			fi.xfTag, fi.xfFn = xfTag, xfFn
+		} else if xfFn := e.h.getExt(rtid); xfFn != nil {
+			fi.xfTag, fi.xfFn = xfFn.tag, xfFn.ext
 			fn.f = (*encFnInfo).ext
 		} else if supportBinaryMarshal && fi.ti.m {
 			fn.f = (*encFnInfo).binaryMarshal
@@ -889,19 +887,6 @@ func (e *Encoder) getEncFn(rt reflect.Type) (fn encFn) {
 	return
 }
 
-func (e *Encoder) encRawExt(re RawExt) {
-	if re.Data == nil {
-		e.e.encodeNil()
-		return
-	}
-	if e.hh.writeExt() {
-		e.e.encodeExtPreamble(re.Tag, len(re.Data))
-		e.w.writeb(re.Data)
-	} else {
-		e.e.encodeStringBytes(c_RAW, re.Data)
-	}
-}
-
 // ----------------------------------------
 
 func encErr(format string, params ...interface{}) {

codec/ext_dep_test.go

@@ -63,7 +63,11 @@ func Benchmark__VMsgpack___Decode(b *testing.B) {
 }
 
 func TestMsgpackPythonGenStreams(t *testing.T) {
-	doTestMsgpackPythonGenStreams(t)
+	doTestPythonGenStreams(t, "msgpack", testMsgpackH)
+}
+
+func TestCborPythonGenStreams(t *testing.T) {
+	doTestPythonGenStreams(t, "cbor", testCborH)
 }
 
 func TestMsgpackRpcSpecGoClientToPythonSvc(t *testing.T) {

codec/gen-fast-path.go

@@ -135,8 +135,15 @@ func (f *decFnInfo) {{ .MethodName false }}(rv reflect.Value) {
 	}
 
 	_, containerLenS := decContLens(f.dd, vtype)
+	if containerLenS == 0 {
+		return
+	}
 	if v == nil {
-		v = make([]{{ .Elem }}, containerLenS, containerLenS)
+		if containerLenS > 0 {
+			v = make([]{{ .Elem }}, containerLenS, containerLenS)
+		} else {
+			v = make([]{{ .Elem }}, 0, 4)
+		}
 	} else if containerLenS > cap(v) {
 		if f.array {
 			decErr(msgDecCannotExpandArr, cap(v), containerLenS)
@@ -147,7 +154,18 @@ func (f *decFnInfo) {{ .MethodName false }}(rv reflect.Value) {
 	} else if containerLenS > len(v) {
 		v = v[:containerLenS]
 	}
-	for j := 0; j < containerLenS; j++ {
+	// for j := 0; j < containerLenS; j++ {
+	for j := 0; ; j++ {
+		if containerLenS >= 0 {
+			if j >= containerLenS {
+				break
+			}
+		} else if f.dd.checkBreak() {
+			break
+		}
+		if j >= len(v) {
+			v = append(v, {{ zerocmd .Elem }})
+		}
 		{{ if eq .Elem "interface{}" }}f.d.decode(&v[j])
 		{{ else }}f.dd.initReadNext()
 		v[j] = {{ decmd .Elem }}
@@ -183,11 +201,26 @@ func (f *decFnInfo) {{ .MethodName false }}(rv reflect.Value) {
 	}
 
 	containerLen := f.dd.readMapLen()
+	if containerLen == 0 {
+		return
+	}
 	if xaddr && v == nil {
-		v = make(map[{{ .MapKey }}]{{ .Elem }}, containerLen)
+		if containerLen > 0 {
+			v = make(map[{{ .MapKey }}]{{ .Elem }}, containerLen)
+		} else {
+			v = make(map[{{ .MapKey }}]{{ .Elem }}) // supports indefinite-length, etc
+		}
 		*vp = v
 	}
-	for j := 0; j < containerLen; j++ {
+	// for j := 0; j < containerLen; j++ {
+	for j := 0; ; j++ {
+		if containerLen >= 0 {
+			if j >= containerLen {
+				break
+			}
+		} else if f.dd.checkBreak() {
+			break
+		}
 		{{ if eq .MapKey "interface{}" }}var mk interface{}
 		f.d.decode(&mk)
 		// special case if a byte array.
@@ -304,6 +337,19 @@ func titleCaseName(s string) string {
 	}
 }
 
+func ZeroValue(s string) string {
+	switch s {
+	case "interface{}":
+		return "nil"
+	case "bool":
+		return "false"
+	case "string":
+		return `""`
+	default:
+		return "0"
+	}
+}
+
 type genTmpl struct {
 	Values []genInfo
 }
@@ -360,6 +406,7 @@ func main() {
 	// funcs["haspfx"] = strings.HasPrefix
 	funcs["encmd"] = EncCommandAsString
 	funcs["decmd"] = DecCommandAsString
+	funcs["zerocmd"] = ZeroValue
 
 	t := template.New("")
 	t = t.Funcs(funcs)

codec/helper.go

@@ -73,7 +73,7 @@ const (
 	valueTypeTimestamp
 	valueTypeExt
 
-	valueTypeInvalid = 0xff
+	// valueTypeInvalid = 0xff
 )
 
 var (
@@ -134,45 +134,125 @@ type BasicHandle struct {
 	DecodeOptions
 }
 
+func (x *BasicHandle) getBasicHandle() *BasicHandle {
+	return x
+}
+
 // Handle is the interface for a specific encoding format.
 //
 // Typically, a Handle is pre-configured before first time use,
 // and not modified while in use. Such a pre-configured Handle
 // is safe for concurrent access.
 type Handle interface {
-	writeExt() bool
 	getBasicHandle() *BasicHandle
 	newEncDriver(w encWriter) encDriver
 	newDecDriver(r decReader) decDriver
 }
 
 // RawExt represents raw unprocessed extension data.
+// Some codecs will decode extension data as a RawExt if there is no registered extension for the tag.
+//
+// Only one of Data or Value is nil. If Data is nil, then the content of the RawExt is in the Value.
 type RawExt struct {
-	Tag  byte
+	Tag uint16
+	// Data is the []byte which represents the raw ext. If Data is nil, ext is exposed in Value.
+	// Data is used by codecs (e.g. binc, msgpack, simple) which do custom serialization of the types
 	Data []byte
+	// Value represents the extension, if Data is nil.
+	// Value is used by codecs (e.g. cbor) which use the format to do custom serialization of the types.
+	Value interface{}
 }
 
-type extTypeTagFn struct {
-	rtid  uintptr
-	rt    reflect.Type
-	tag   byte
+// Ext handles custom (de)serialization of custom types / extensions.
+type Ext interface {
+	// WriteExt converts a value to a []byte.
+	// It is used by codecs (e.g. binc, msgpack, simple) which do custom serialization of the types.
+	WriteExt(v reflect.Value) []byte
+
+	// ReadExt updates a value from a []byte.
+	// It is used by codecs (e.g. binc, msgpack, simple) which do custom serialization of the types.
+	ReadExt(v reflect.Value, src []byte)
+
+	// ConvertExt converts a value into a simpler interface for easy encoding e.g. convert time.Time to int64.
+	// It is used by codecs (e.g. cbor) which use the format to do custom serialization of the types.
+	ConvertExt(v reflect.Value) interface{}
+
+	// UpdateExt updates a value from a simpler interface for easy decoding e.g. convert int64 to time.Time.
+	// It is used by codecs (e.g. cbor) which use the format to do custom serialization of the types.
+	UpdateExt(v reflect.Value, src interface{})
+}
+
+// bytesExt is a wrapper implementation to support former AddExt exported method.
+type bytesExt struct {
 	encFn func(reflect.Value) ([]byte, error)
 	decFn func(reflect.Value, []byte) error
 }
 
+func (x bytesExt) WriteExt(rv reflect.Value) []byte {
+	bs, err := x.encFn(rv)
+	if err != nil {
+		panic(err)
+	}
+	return bs
+}
+
+func (x bytesExt) ReadExt(rv reflect.Value, bs []byte) {
+	if err := x.decFn(rv, bs); err != nil {
+		panic(err)
+	}
+}
+
+func (x bytesExt) ConvertExt(rv reflect.Value) interface{} {
+	return x.WriteExt(rv)
+}
+
+func (x bytesExt) UpdateExt(rv reflect.Value, v interface{}) {
+	x.ReadExt(rv, v.([]byte))
+}
+
+// noBuiltInTypes is embedded into many types which do not support builtins
+// e.g. msgpack, simple, cbor.
+type noBuiltInTypes struct{}
+
+func (_ noBuiltInTypes) isBuiltinType(rt uintptr) bool           { return false }
+func (_ noBuiltInTypes) encodeBuiltin(rt uintptr, v interface{}) {}
+func (_ noBuiltInTypes) decodeBuiltin(rt uintptr, v interface{}) {}
+
+type noStreamingCodec struct{}
+
+func (_ noStreamingCodec) checkBreak() bool { return false }
+
+type extTypeTagFn struct {
+	rtid uintptr
+	rt   reflect.Type
+	tag  uint16
+	ext  Ext
+}
+
 type extHandle []*extTypeTagFn
 
-// AddExt registers an encode and decode function for a reflect.Type.
-// Note that the type must be a named type, and specifically not
-// a pointer or Interface. An error is returned if that is not honored.
+// DEPRECATED: AddExt is deprecated in favor of SetExt. It exists for compatibility only.
 //
-// To Deregister an ext, call AddExt with 0 tag, nil encfn and nil decfn.
+// AddExt registes an encode and decode function for a reflect.Type.
+// AddExt internally calls SetExt.
+// To deregister an Ext, call AddExt with nil encfn and/or nil decfn.
 func (o *extHandle) AddExt(
-	rt reflect.Type,
-	tag byte,
-	encfn func(reflect.Value) ([]byte, error),
-	decfn func(reflect.Value, []byte) error,
+	rt reflect.Type, tag byte,
+	encfn func(reflect.Value) ([]byte, error), decfn func(reflect.Value, []byte) error,
 ) (err error) {
+	if encfn == nil || decfn == nil {
+		return o.SetExt(rt, uint16(tag), nil)
+	}
+	return o.SetExt(rt, uint16(tag), bytesExt{encfn, decfn})
+}
+
+// SetExt registers a tag and Ext for a reflect.Type.
+//
+// Note that the type must be a named type, and specifically not
+// a pointer or Interface. An error is returned if that is not honored.
+//
+// To Deregister an ext, call SetExt with nil Ext
+func (o *extHandle) SetExt(rt reflect.Type, tag uint16, ext Ext) (err error) {
 	// o is a pointer, because we may need to initialize it
 	if rt.PkgPath() == "" || rt.Kind() == reflect.Interface {
 		err = fmt.Errorf("codec.Handle.AddExt: Takes named type, especially not a pointer or interface: %T",
@@ -180,22 +260,15 @@ func (o *extHandle) AddExt(
 		return
 	}
 
-	// o cannot be nil, since it is always embedded in a Handle.
-	// if nil, let it panic.
-	// if o == nil {
-	// 	err = errors.New("codec.Handle.AddExt: extHandle cannot be a nil pointer.")
-	// 	return
-	// }
-
 	rtid := reflect.ValueOf(rt).Pointer()
 	for _, v := range *o {
 		if v.rtid == rtid {
-			v.tag, v.encFn, v.decFn = tag, encfn, decfn
+			v.tag, v.ext = tag, ext
 			return
 		}
 	}
 
-	*o = append(*o, &extTypeTagFn{rtid, rt, tag, encfn, decfn})
+	*o = append(*o, &extTypeTagFn{rtid, rt, tag, ext})
 	return
 }
 
@@ -208,7 +281,7 @@ func (o extHandle) getExt(rtid uintptr) *extTypeTagFn {
 	return nil
 }
 
-func (o extHandle) getExtForTag(tag byte) *extTypeTagFn {
+func (o extHandle) getExtForTag(tag uint16) *extTypeTagFn {
 	for _, v := range o {
 		if v.tag == tag {
 			return v
@@ -217,32 +290,6 @@ func (o extHandle) getExtForTag(tag byte) *extTypeTagFn {
 	return nil
 }
 
-func (o extHandle) getDecodeExtForTag(tag byte) (
-	rv reflect.Value, fn func(reflect.Value, []byte) error) {
-	if x := o.getExtForTag(tag); x != nil {
-		// ext is only registered for base
-		rv = reflect.New(x.rt).Elem()
-		fn = x.decFn
-	}
-	return
-}
-
-func (o extHandle) getDecodeExt(rtid uintptr) (tag byte, fn func(reflect.Value, []byte) error) {
-	if x := o.getExt(rtid); x != nil {
-		tag = x.tag
-		fn = x.decFn
-	}
-	return
-}
-
-func (o extHandle) getEncodeExt(rtid uintptr) (tag byte, fn func(reflect.Value) ([]byte, error)) {
-	if x := o.getExt(rtid); x != nil {
-		tag = x.tag
-		fn = x.encFn
-	}
-	return
-}
-
 type structFieldInfo struct {
 	encName string // encode name
 
@@ -252,11 +299,6 @@ type structFieldInfo struct {
 	i         int16 // field index in struct
 	omitEmpty bool
 	toArray   bool // if field is _struct, is the toArray set?
-
-	// tag       string   // tag
-	// name      string   // field name
-	// encNameBs []byte   // encoded name as byte stream
-	// ikind     int      // kind of the field as an int i.e. int(reflect.Kind)
 }
 
 func parseStructFieldInfo(fname string, stag string) *structFieldInfo {
@@ -264,9 +306,7 @@ func parseStructFieldInfo(fname string, stag string) *structFieldInfo {
 		panic("parseStructFieldInfo: No Field Name")
 	}
 	si := structFieldInfo{
-		// name: fname,
 		encName: fname,
-		// tag: stag,
 	}
 
 	if stag != "" {
@@ -414,18 +454,6 @@ func getTypeInfo(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
 		sfip := make([]*structFieldInfo, 0, rt.NumField())
 		rgetTypeInfo(rt, nil, make(map[string]bool), &sfip, siInfo)
 
-		// // try to put all si close together
-		// const tryToPutAllStructFieldInfoTogether = true
-		// if tryToPutAllStructFieldInfoTogether {
-		// 	sfip2 := make([]structFieldInfo, len(sfip))
-		// 	for i, si := range sfip {
-		// 		sfip2[i] = *si
-		// 	}
-		// 	for i := range sfip {
-		// 		sfip[i] = &sfip2[i]
-		// 	}
-		// }
-
 		ti.sfip = make([]*structFieldInfo, len(sfip))
 		ti.sfi = make([]*structFieldInfo, len(sfip))
 		copy(ti.sfip, sfip)
@@ -440,10 +468,6 @@ func getTypeInfo(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
 func rgetTypeInfo(rt reflect.Type, indexstack []int, fnameToHastag map[string]bool,
 	sfi *[]*structFieldInfo, siInfo *structFieldInfo,
 ) {
-	// for rt.Kind() == reflect.Ptr {
-	// 	// indexstack = append(indexstack, 0)
-	// 	rt = rt.Elem()
-	// }
 	for j := 0; j < rt.NumField(); j++ {
 		f := rt.Field(j)
 		stag := f.Tag.Get(structTagName)

codec/helper_internal.go

@@ -125,3 +125,36 @@ func implementsIntf(typ, iTyp reflect.Type) (success bool, indir int8) {
 	}
 	return false, 0
 }
+
+// validate that this function is correct ...
+// culled from OGRE (Object-Oriented Graphics Rendering Engine)
+// function: halfToFloatI (http://stderr.org/doc/ogre-doc/api/OgreBitwise_8h-source.html)
+func halfFloatToDoubleBits(yy uint16) (d uint64) {
+	y := uint64(yy)
+	s := (y >> 15) & 0x01
+	e := (y >> 10) & 0x1f
+	m := y & 0x03ff
+
+	if e == 0 {
+		if m == 0 { // plu or minus 0
+			return s << 31
+		} else { // Denormalized number -- renormalize it
+			for (m & 0x00000400) == 0 {
+				m <<= 1
+				e -= 1
+			}
+			e += 1
+			const zz uint64 = 0x0400
+			m &= ^zz
+		}
+	} else if e == 31 {
+		if m == 0 { // Inf
+			return (s << 31) | 0x7f800000
+		} else { // NaN
+			return (s << 31) | 0x7f800000 | (m << 31)
+		}
+	}
+	e = e + (127 - 15)
+	m = m << 13
+	return (s << 31) | (e << 23) | m
+}

codec/msgpack.go

@@ -24,6 +24,7 @@ import (
 	"io"
 	"math"
 	"net/rpc"
+	"reflect"
 )
 
 const (
@@ -104,15 +105,9 @@ var (
 type msgpackEncDriver struct {
 	w encWriter
 	h *MsgpackHandle
+	noBuiltInTypes
 }
 
-func (e *msgpackEncDriver) isBuiltinType(rt uintptr) bool {
-	//no builtin types. All encodings are based on kinds. Types supported as extensions.
-	return false
-}
-
-func (e *msgpackEncDriver) encodeBuiltin(rt uintptr, v interface{}) {}
-
 func (e *msgpackEncDriver) encodeNil() {
 	e.w.writen1(mpNil)
 }
@@ -174,6 +169,25 @@ func (e *msgpackEncDriver) encodeFloat64(f float64) {
 	e.w.writeUint64(math.Float64bits(f))
 }
 
+func (e *msgpackEncDriver) encodeExt(rv reflect.Value, xtag uint16, ext Ext, _ *Encoder) {
+	bs := ext.WriteExt(rv)
+	if bs == nil {
+		e.encodeNil()
+		return
+	}
+	if e.h.WriteExt {
+		e.encodeExtPreamble(uint8(xtag), len(bs))
+		e.w.writeb(bs)
+	} else {
+		e.encodeStringBytes(c_RAW, bs)
+	}
+}
+
+func (e *msgpackEncDriver) encodeRawExt(re *RawExt, _ *Encoder) {
+	e.encodeExtPreamble(uint8(re.Tag), len(re.Data))
+	e.w.writeb(re.Data)
+}
+
 func (e *msgpackEncDriver) encodeExtPreamble(xtag byte, l int) {
 	switch {
 	case l == 1:
@@ -257,21 +271,16 @@ type msgpackDecDriver struct {
 	bd     byte
 	bdRead bool
 	bdType valueType
+	noBuiltInTypes
+	noStreamingCodec
 }
 
-func (d *msgpackDecDriver) isBuiltinType(rt uintptr) bool {
-	//no builtin types. All encodings are based on kinds. Types supported as extensions.
-	return false
-}
-
-func (d *msgpackDecDriver) decodeBuiltin(rt uintptr, v interface{}) {}
-
 // Note: This returns either a primitive (int, bool, etc) for non-containers,
 // or a containerType, or a specific type denoting nil or extension.
 // It is called when a nil interface{} is passed, leaving it up to the DecDriver
 // to introspect the stream and decide how best to decode.
 // It deciphers the value by looking at the stream first.
-func (d *msgpackDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
+func (d *msgpackDecDriver) decodeNaked(_ *Decoder) (v interface{}, vt valueType, decodeFurther bool) {
 	d.initReadNext()
 	bd := d.bd
 
@@ -354,7 +363,7 @@ func (d *msgpackDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFur
 		case bd >= mpFixExt1 && bd <= mpFixExt16, bd >= mpExt8 && bd <= mpExt32:
 			clen := d.readExtLen()
 			var re RawExt
-			re.Tag = d.r.readn1()
+			re.Tag = uint16(d.r.readn1())
 			re.Data = d.r.readn(clen)
 			v = &re
 			vt = valueTypeExt
@@ -365,6 +374,10 @@ func (d *msgpackDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFur
 	if !decodeFurther {
 		d.bdRead = false
 	}
+	if vt == valueTypeUint && d.h.SignedInteger {
+		d.bdType = valueTypeInt
+		v = int64(v.(uint64))
+	}
 	return
 }
 
@@ -553,7 +566,11 @@ func (d *msgpackDecDriver) currentEncodedType() valueType {
 		case mpFloat, mpDouble:
 			d.bdType = valueTypeFloat
 		case mpUint8, mpUint16, mpUint32, mpUint64:
-			d.bdType = valueTypeUint
+			if d.h.SignedInteger {
+				d.bdType = valueTypeInt
+			} else {
+				d.bdType = valueTypeUint
+			}
 		case mpInt8, mpInt16, mpInt32, mpInt64:
 			d.bdType = valueTypeInt
 		default:
@@ -646,7 +663,20 @@ func (d *msgpackDecDriver) readExtLen() (clen int) {
 	return
 }
 
-func (d *msgpackDecDriver) decodeExt(verifyTag bool, tag byte) (xtag byte, xbs []byte) {
+func (d *msgpackDecDriver) decodeExt(rv reflect.Value, xtag uint16, ext Ext, _ *Decoder) (realxtag uint16) {
+	realxtag1, xbs := d.decodeExtV(ext != nil, uint8(xtag))
+	realxtag = uint16(realxtag1)
+	if ext == nil {
+		re := rv.Interface().(*RawExt)
+		re.Tag = realxtag
+		re.Data = xbs
+	} else {
+		ext.ReadExt(rv, xbs)
+	}
+	return
+}
+
+func (d *msgpackDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs []byte) {
 	xbd := d.bd
 	switch {
 	case xbd == mpBin8, xbd == mpBin16, xbd == mpBin32:
@@ -695,14 +725,6 @@ func (h *MsgpackHandle) newDecDriver(r decReader) decDriver {
 	return &msgpackDecDriver{r: r, h: h}
 }
 
-func (h *MsgpackHandle) writeExt() bool {
-	return h.WriteExt
-}
-
-func (h *MsgpackHandle) getBasicHandle() *BasicHandle {
-	return &h.BasicHandle
-}
-
 //--------------------------------------------------
 
 type msgpackSpecRpcCodec struct {

codec/simple.go

@@ -3,7 +3,10 @@
 
 package codec
 
-import "math"
+import (
+	"math"
+	"reflect"
+)
 
 const (
 	_               uint8 = iota
@@ -28,16 +31,10 @@ const (
 type simpleEncDriver struct {
 	h *SimpleHandle
 	w encWriter
+	noBuiltInTypes
 	//b [8]byte
 }
 
-func (e *simpleEncDriver) isBuiltinType(rt uintptr) bool {
-	return false
-}
-
-func (e *simpleEncDriver) encodeBuiltin(rt uintptr, v interface{}) {
-}
-
 func (e *simpleEncDriver) encodeNil() {
 	e.w.writen1(simpleVdNil)
 }
@@ -107,6 +104,21 @@ func (e *simpleEncDriver) encLen(bd byte, length int) {
 	}
 }
 
+func (e *simpleEncDriver) encodeExt(rv reflect.Value, xtag uint16, ext Ext, _ *Encoder) {
+	bs := ext.WriteExt(rv)
+	if bs == nil {
+		e.encodeNil()
+		return
+	}
+	e.encodeExtPreamble(uint8(xtag), len(bs))
+	e.w.writeb(bs)
+}
+
+func (e *simpleEncDriver) encodeRawExt(re *RawExt, _ *Encoder) {
+	e.encodeExtPreamble(uint8(re.Tag), len(re.Data))
+	e.w.writeb(re.Data)
+}
+
 func (e *simpleEncDriver) encodeExtPreamble(xtag byte, length int) {
 	e.encLen(simpleVdExt, length)
 	e.w.writen1(xtag)
@@ -142,6 +154,8 @@ type simpleDecDriver struct {
 	bdRead bool
 	bdType valueType
 	bd     byte
+	noBuiltInTypes
+	noStreamingCodec
 	//b      [8]byte
 }
 
@@ -162,7 +176,11 @@ func (d *simpleDecDriver) currentEncodedType() valueType {
 		case simpleVdTrue, simpleVdFalse:
 			d.bdType = valueTypeBool
 		case simpleVdPosInt, simpleVdPosInt + 1, simpleVdPosInt + 2, simpleVdPosInt + 3:
-			d.bdType = valueTypeUint
+			if d.h.SignedInteger {
+				d.bdType = valueTypeInt
+			} else {
+				d.bdType = valueTypeUint
+			}
 		case simpleVdNegInt, simpleVdNegInt + 1, simpleVdNegInt + 2, simpleVdNegInt + 3:
 			d.bdType = valueTypeInt
 		case simpleVdFloat32, simpleVdFloat64:
@@ -192,13 +210,6 @@ func (d *simpleDecDriver) tryDecodeAsNil() bool {
 	return false
 }
 
-func (d *simpleDecDriver) isBuiltinType(rt uintptr) bool {
-	return false
-}
-
-func (d *simpleDecDriver) decodeBuiltin(rt uintptr, v interface{}) {
-}
-
 func (d *simpleDecDriver) decIntAny() (ui uint64, i int64, neg bool) {
 	switch d.bd {
 	case simpleVdPosInt:
@@ -343,7 +354,20 @@ func (d *simpleDecDriver) decodeBytes(bs []byte) (bsOut []byte, changed bool) {
 	return
 }
 
-func (d *simpleDecDriver) decodeExt(verifyTag bool, tag byte) (xtag byte, xbs []byte) {
+func (d *simpleDecDriver) decodeExt(rv reflect.Value, xtag uint16, ext Ext, _ *Decoder) (realxtag uint16) {
+	realxtag1, xbs := d.decodeExtV(ext != nil, uint8(xtag))
+	realxtag = uint16(realxtag1)
+	if ext == nil {
+		re := rv.Interface().(*RawExt)
+		re.Tag = realxtag
+		re.Data = xbs
+	} else {
+		ext.ReadExt(rv, xbs)
+	}
+	return
+}
+
+func (d *simpleDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs []byte) {
 	switch d.bd {
 	case simpleVdExt, simpleVdExt + 1, simpleVdExt + 2, simpleVdExt + 3, simpleVdExt + 4:
 		l := d.decLen()
@@ -361,7 +385,7 @@ func (d *simpleDecDriver) decodeExt(verifyTag bool, tag byte) (xtag byte, xbs []
 	return
 }
 
-func (d *simpleDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
+func (d *simpleDecDriver) decodeNaked(_ *Decoder) (v interface{}, vt valueType, decodeFurther bool) {
 	d.initReadNext()
 
 	switch d.bd {
@@ -374,9 +398,14 @@ func (d *simpleDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurt
 		vt = valueTypeBool
 		v = true
 	case simpleVdPosInt, simpleVdPosInt + 1, simpleVdPosInt + 2, simpleVdPosInt + 3:
-		vt = valueTypeUint
-		ui, _, _ := d.decIntAny()
-		v = ui
+		ui, i, _ := d.decIntAny()
+		if d.h.SignedInteger {
+			vt = valueTypeInt
+			v = i
+		} else {
+			vt = valueTypeUint
+			v = ui
+		}
 	case simpleVdNegInt, simpleVdNegInt + 1, simpleVdNegInt + 2, simpleVdNegInt + 3:
 		vt = valueTypeInt
 		_, i, _ := d.decIntAny()
@@ -397,10 +426,9 @@ func (d *simpleDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurt
 		vt = valueTypeExt
 		l := d.decLen()
 		var re RawExt
-		re.Tag = d.r.readn1()
+		re.Tag = uint16(d.r.readn1())
 		re.Data = d.r.readn(l)
 		v = &re
-		vt = valueTypeExt
 	case simpleVdArray, simpleVdArray + 1, simpleVdArray + 2, simpleVdArray + 3, simpleVdArray + 4:
 		vt = valueTypeArray
 		decodeFurther = true
@@ -449,13 +477,5 @@ func (h *SimpleHandle) newDecDriver(r decReader) decDriver {
 	return &simpleDecDriver{r: r, h: h}
 }
 
-func (_ *SimpleHandle) writeExt() bool {
-	return true
-}
-
-func (h *SimpleHandle) getBasicHandle() *BasicHandle {
-	return &h.BasicHandle
-}
-
 var _ decDriver = (*simpleDecDriver)(nil)
 var _ encDriver = (*simpleEncDriver)(nil)

codec/test.py

@@ -0,0 +1,119 @@
+#!/usr/bin/env python
+
+# This will create golden files in a directory passed to it.
+# A Test calls this internally to create the golden files
+# So it can process them (so we don't have to checkin the files).
+
+# Ensure msgpack-python and cbor are installed first, using:
+#   pip install --user msgpack-python
+#   pip install --user cbor
+
+import cbor, msgpack, msgpackrpc, sys, os, threading
+
+def get_test_data_list():
+    # get list with all primitive types, and a combo type
+    l0 = [ 
+        -8,
+         -1616,
+         -32323232,
+         -6464646464646464,
+         192,
+         1616,
+         32323232,
+         6464646464646464,
+         192,
+         -3232.0,
+         -6464646464.0,
+         3232.0,
+         6464646464.0,
+         False,
+         True,
+         None,
+         u"someday",
+         u"",
+         u"bytestring",
+         1328176922000002000,
+         -2206187877999998000,
+         270,
+        -2013855847999995777,
+         #-6795364578871345152,
+         ]
+    l1 = [
+        { "true": True,
+          "false": False },
+        { "true": "True",
+          "false": False,
+          "uint16(1616)": 1616 },
+        { "list": [1616, 32323232, True, -3232.0, {"TRUE":True, "FALSE":False}, [True, False] ],
+          "int32":32323232, "bool": True, 
+          "LONG STRING": "123456789012345678901234567890123456789012345678901234567890",
+          "SHORT STRING": "1234567890" },	
+        { True: "true", 8: False, "false": 0 }
+        ]
+    
+    l = []
+    l.extend(l0)
+    l.append(l0)
+    l.extend(l1)
+    return l
+
+def build_test_data(destdir):
+    l = get_test_data_list()
+    for i in range(len(l)):
+        # packer = msgpack.Packer()
+        serialized = msgpack.dumps(l[i])
+        f = open(os.path.join(destdir, str(i) + '.msgpack.golden'), 'wb')
+        f.write(serialized)
+        f.close()
+        serialized = cbor.dumps(l[i])
+        f = open(os.path.join(destdir, str(i) + '.cbor.golden'), 'wb')
+        f.write(serialized)
+        f.close()
+
+def doRpcServer(port, stopTimeSec):
+    class EchoHandler(object):
+        def Echo123(self, msg1, msg2, msg3):
+            return ("1:%s 2:%s 3:%s" % (msg1, msg2, msg3))
+        def EchoStruct(self, msg):
+            return ("%s" % msg)
+    
+    addr = msgpackrpc.Address('localhost', port)
+    server = msgpackrpc.Server(EchoHandler())
+    server.listen(addr)
+    # run thread to stop it after stopTimeSec seconds if > 0
+    if stopTimeSec > 0:
+        def myStopRpcServer():
+            server.stop()
+        t = threading.Timer(stopTimeSec, myStopRpcServer)
+        t.start()
+    server.start()
+
+def doRpcClientToPythonSvc(port):
+    address = msgpackrpc.Address('localhost', port)
+    client = msgpackrpc.Client(address, unpack_encoding='utf-8')
+    print client.call("Echo123", "A1", "B2", "C3")
+    print client.call("EchoStruct", {"A" :"Aa", "B":"Bb", "C":"Cc"})
+   
+def doRpcClientToGoSvc(port):
+    # print ">>>> port: ", port, " <<<<<"
+    address = msgpackrpc.Address('localhost', port)
+    client = msgpackrpc.Client(address, unpack_encoding='utf-8')
+    print client.call("TestRpcInt.Echo123", ["A1", "B2", "C3"])
+    print client.call("TestRpcInt.EchoStruct", {"A" :"Aa", "B":"Bb", "C":"Cc"})
+
+def doMain(args):
+    if len(args) == 2 and args[0] == "testdata":
+        build_test_data(args[1])
+    elif len(args) == 3 and args[0] == "rpc-server":
+        doRpcServer(int(args[1]), int(args[2]))
+    elif len(args) == 2 and args[0] == "rpc-client-python-service":
+        doRpcClientToPythonSvc(int(args[1]))
+    elif len(args) == 2 and args[0] == "rpc-client-go-service":
+        doRpcClientToGoSvc(int(args[1]))
+    else:
+        print("Usage: test.py " + 
+              "[testdata|rpc-server|rpc-client-python-service|rpc-client-go-service] ...")
+    
+if __name__ == "__main__":
+    doMain(sys.argv[1:])
+