codec: Add Reset methods, performance improvements and bug fixes.

Highlights:
- ResetXXX methods for re-using Decoder/Encoders
- much faster json decoding
- performance via refactored state management
- reduced allocation
- cleaned up slice and map decoding (across reflection, fast-path and code-generation)
- notfastpath tag for faster codecgen and build/test
- test robustness

All enumerated below:

----
Add (En|De)coder.Reset(Bytes) method to allow re-use.

The state about what functions are mapped to given type ids does not
have to be recomputed. Also, slices which have been allocated for use
in the (En|De)coder can be re-used across other invocations.

This required adding a reset() method to (en|de)cDriver, so that it
can transitively be reset also.

----
Clean up test use of flags.

Now, benchmark tests depend on flags defined by codec_test.

This way, we can turn on any flag in 'go test' and get the designed
behaviour.

----
clean up msgpack DecodeNaked, so that only map/array need further decoding.

----
change DecodeNaked to use Decoder.decNaked to store state, reducing allocation.

Use a decNaked structure for maintaining DecodeNaked state.
This way, we do not have any allocation for primitives, as previously,
DecodeNaked would wrap primitives in an interface{} and return.

Also, use slices on decNaked to manage encoding/decoding into
interface{} or any of the known collection types we default to:
[]interface{}, map[interface{}]interface{}, map[string]interface{}

This causes a API change for DecodeNaked() to not return anything,
but just store state information in the decNaked field of Decoder.

Make changes to all Handles to leverage this new model.

Note: For RawExt, we will only read the tag and the data during decodeNaked.
If data is not read, then we know we have to read the value next.

In addition, we reduced the amount of indirection in json handle.
We set the internal pointer outside of the tight loop, so not much dereferencing again.
This gave some performance improvement.

----
bench: reduce unnecessary allocation by re-using a global var for newTestStruc.

----
add a timeExt for custom-designed time.go

----
update tests to test intern strings, and correctly test codecgen.

Previously, codecgen tests were not running because we didn't pass right flags.
So things passed, but codecgen wasn't actually working fully.

----
emit state when transitioning across map/array states.

let Encoder/decoder maintain state transitions and inform
Handle so that handle doesn't have to manage state.

To do this, we allow encDriver or decDriver to register for state
notifications. state notifications are
- (map|array)End, mapKey, mapValue, arrayElem
An (en|de)cDriver will implement the method sendContainerState
to be notified of the state transitions.

All code: reflection, fast-path and codecgen all support this now.

ReadEnd method is now removed, as its function is consumed by the state
notifications.

Since (Read|Encode)(Array|Map)Start get or return length information,
those methods already signal Start, so no extra Start signals are sent
(even though it is one of the state constants). (en|de)cDrivers can use
those methods as state notifications.

With this change, json is simpler as it doesn't have to also
maintain state using a stack. All the calls to sep() done before each
call are now removed. json just keeps track of the last state notification.
Based on the last one and the current one, json can easily determine what
separator to emit or consume i.e. one of {}[]:,

We are careful when implementing this to prevent extra function calls
and dereferencing. We inline: if cr != nil { cr.sendContainerState(...) }
and also cache cr within tight loops.

I also reduced potential allocation for simpleIoEncWriterWriter
by having that be a part of the Encoder itself.

----
convert IsContainerType(...) bool to ContainerType() valueType

This way, we don't keep asking same question multiple times,
but just get the answer and work with it.

----

initialize json's encDriver's bs scratch byte slice, from scratch array.

----
encode struct: account for anonyous fields being nil pointers

----
clean up slice and map decoding

Clean up/simplify decode.go logic for slices
- synchronize it across reflection, fast-path and codecgen
- clean up the logic

Use reflect.Zero during reflection's decoding of a map, to reduce amount of temporary allocation.

No need to set value of a map if it was a pointer which already existed in the map, since
we will be decoding into that pointer.

Ensure fastpathEnabled flag is always honored.

In addition, Fix up Msgpack RPC tests by using a randomly generated value for the python server
Also, add some sleep in a loop till you can get a connection, and explicitly kill process when done.

----
rearrange structures to reduce struct size

----
make json decoding faster

- keep track of token read after a white space
- skipWhitespace is only called if the tok value is "reset" to 0
- all methods that call skipWhitespace will how handle the character and may consume it
  (by setting tok to 0)
- expectChar goes away
- ContainerType method is simpler
- unreadn1 is basically now unused (except for decoding a number)

Performance improves because of:
- no more calling skipWhitespace all the time. Some function calls elided.
- simpler state management

----
add notfastpath tag support for faster test, build and codecgen execution.

fast-path generated code is very large (over 30K lines in the file).
Building codec takes a non-trivially longer time with it.

To mitigate that, we add the notfastpath tag, which uses a noop fast-path.not.go
instead of fast-path.generated.go. That file is much smaller resulting in faster builds
and tests when run with the '-tags=fastbuild'.

We also updated tests.sh to add ways to do a fast test execution (passing in the fastbuild tag)
or to have the fast-path mode elided during testing.

When running prebuild, we also pass in that fastbuild tag for faster execution.

----
codecgen: add notfastpath tag for faster codecgen execution.

----
test: msgpack rpc: use smaller time interval between attempts to connect to port.

----
decSliceHelper cleanup

- only ever initialized from decSliceHelperStart() method
- only a single bool is needed to track (since only 2 states exist)

-----
x_bench_test: update import paths for msgp, and add note on updating libs.

----
codec_test: for canonical test, only change Handle.Canonical if it is false

codec/0doc.go

@@ -98,7 +98,21 @@ with the standard net/rpc package.
 
 Usage
 
-Typical usage model:
+The Handle is SAFE for concurrent READ, but NOT SAFE for concurrent modification.
+
+The Encoder and Decoder are NOT safe for concurrent use.
+
+Consequently, the usage model is basically:
+
+    - Create and initialize the Handle before any use.
+      Once created, DO NOT modify it.
+    - Multiple Encoders or Decoders can now use the Handle concurrently.
+      They only read information off the Handle (never write).
+    - However, each Encoder or Decoder MUST not be used concurrently
+    - To re-use an Encoder/Decoder, call Reset(...) on it first.
+      This allows you use state maintained on the Encoder/Decoder.
+
+Sample usage model:
 
     // create and configure Handle
     var (
@@ -148,3 +162,32 @@ Typical usage model:
 */
 package codec
 
+// Benefits of go-codec:
+//
+//    - encoding/json always reads whole file into memory first.
+//      This makes it unsuitable for parsing very large files.
+//    - encoding/xml cannot parse into a map[string]interface{}
+//      I found this out on reading https://github.com/clbanning/mxj
+
+// TODO:
+//
+//   - (En|De)coder should store an error when it occurs.
+//     Until reset, subsequent calls return that error that was stored.
+//     This means that free panics must go away.
+//     All errors must be raised through errorf method.
+//   - Decoding using a chan is good, but incurs concurrency costs.
+//     This is because there's no fast way to use a channel without it
+//     having to switch goroutines constantly.
+//     Callback pattern is still the best. Maybe cnsider supporting something like:
+//        type X struct {
+//             Name string
+//             Ys []Y
+//             Ys chan <- Y
+//             Ys func(interface{}) -> call this interface for each entry in there.
+//        }
+//    - Consider adding a isZeroer interface { isZero() bool }
+//      It is used within isEmpty, for omitEmpty support.
+//    - Consider making Handle used AS-IS within the encoding/decoding session.
+//      This means that we don't cache Handle information within the (En|De)coder,
+//      except we really need it at Reset(...)
+//    - Handle recursive types during encoding/decoding?

codec/binc.go

@@ -59,8 +59,8 @@ type bincEncDriver struct {
 	e *Encoder
 	w encWriter
 	m map[string]uint16 // symbols
-	s uint16            // symbols sequencer
 	b [scratchByteArrayLen]byte
+	s uint16 // symbols sequencer
 	encNoSeparator
 }
 
@@ -318,9 +318,9 @@ func (e *bincEncDriver) encLenNumber(bd byte, v uint64) {
 //------------------------------------
 
 type bincDecSymbol struct {
-	i uint16
 	s string
 	b []byte
+	i uint16
 }
 
 type bincDecDriver struct {
@@ -329,7 +329,6 @@ type bincDecDriver struct {
 	r      decReader
 	br     bool // bytes reader
 	bdRead bool
-	bdType valueType
 	bd     byte
 	vd     byte
 	vs     byte
@@ -347,24 +346,23 @@ func (d *bincDecDriver) readNextBd() {
 	d.vd = d.bd >> 4
 	d.vs = d.bd & 0x0f
 	d.bdRead = true
-	d.bdType = valueTypeUnset
 }
 
-func (d *bincDecDriver) IsContainerType(vt valueType) (b bool) {
-	switch vt {
-	case valueTypeNil:
-		return d.vd == bincVdSpecial && d.vs == bincSpNil
-	case valueTypeBytes:
-		return d.vd == bincVdByteArray
-	case valueTypeString:
-		return d.vd == bincVdString
-	case valueTypeArray:
-		return d.vd == bincVdArray
-	case valueTypeMap:
-		return d.vd == bincVdMap
+func (d *bincDecDriver) ContainerType() (vt valueType) {
+	if d.vd == bincVdSpecial && d.vs == bincSpNil {
+		return valueTypeNil
+	} else if d.vd == bincVdByteArray {
+		return valueTypeBytes
+	} else if d.vd == bincVdString {
+		return valueTypeString
+	} else if d.vd == bincVdArray {
+		return valueTypeArray
+	} else if d.vd == bincVdMap {
+		return valueTypeMap
+	} else {
+		// d.d.errorf("isContainerType: unsupported parameter: %v", vt)
 	}
-	d.d.errorf("isContainerType: unsupported parameter: %v", vt)
-	return // "unreachable"
+	return valueTypeUnset
 }
 
 func (d *bincDecDriver) TryDecodeAsNil() bool {
@@ -695,7 +693,7 @@ func (d *bincDecDriver) decStringAndBytes(bs []byte, withString, zerocopy bool)
 			if withString {
 				s = string(bs2)
 			}
-			d.s = append(d.s, bincDecSymbol{symbol, s, bs2})
+			d.s = append(d.s, bincDecSymbol{i: symbol, s: s, b: bs2})
 		}
 	default:
 		d.d.errorf("Invalid d.vd. Expecting string:0x%x, bytearray:0x%x or symbol: 0x%x. Got: 0x%x",
@@ -784,97 +782,95 @@ func (d *bincDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs []b
 	return
 }
 
-func (d *bincDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
+func (d *bincDecDriver) DecodeNaked() {
 	if !d.bdRead {
 		d.readNextBd()
 	}
 
+	n := &d.d.n
+	var decodeFurther bool
+
 	switch d.vd {
 	case bincVdSpecial:
 		switch d.vs {
 		case bincSpNil:
-			vt = valueTypeNil
+			n.v = valueTypeNil
 		case bincSpFalse:
-			vt = valueTypeBool
-			v = false
+			n.v = valueTypeBool
+			n.b = false
 		case bincSpTrue:
-			vt = valueTypeBool
-			v = true
+			n.v = valueTypeBool
+			n.b = true
 		case bincSpNan:
-			vt = valueTypeFloat
-			v = math.NaN()
+			n.v = valueTypeFloat
+			n.f = math.NaN()
 		case bincSpPosInf:
-			vt = valueTypeFloat
-			v = math.Inf(1)
+			n.v = valueTypeFloat
+			n.f = math.Inf(1)
 		case bincSpNegInf:
-			vt = valueTypeFloat
-			v = math.Inf(-1)
+			n.v = valueTypeFloat
+			n.f = math.Inf(-1)
 		case bincSpZeroFloat:
-			vt = valueTypeFloat
-			v = float64(0)
+			n.v = valueTypeFloat
+			n.f = float64(0)
 		case bincSpZero:
-			vt = valueTypeUint
-			v = uint64(0) // int8(0)
+			n.v = valueTypeUint
+			n.u = uint64(0) // int8(0)
 		case bincSpNegOne:
-			vt = valueTypeInt
-			v = int64(-1) // int8(-1)
+			n.v = valueTypeInt
+			n.i = int64(-1) // int8(-1)
 		default:
 			d.d.errorf("decodeNaked: Unrecognized special value 0x%x", d.vs)
-			return
 		}
 	case bincVdSmallInt:
-		vt = valueTypeUint
-		v = uint64(int8(d.vs)) + 1 // int8(d.vs) + 1
+		n.v = valueTypeUint
+		n.u = uint64(int8(d.vs)) + 1 // int8(d.vs) + 1
 	case bincVdPosInt:
-		vt = valueTypeUint
-		v = d.decUint()
+		n.v = valueTypeUint
+		n.u = d.decUint()
 	case bincVdNegInt:
-		vt = valueTypeInt
-		v = -(int64(d.decUint()))
+		n.v = valueTypeInt
+		n.i = -(int64(d.decUint()))
 	case bincVdFloat:
-		vt = valueTypeFloat
-		v = d.decFloat()
+		n.v = valueTypeFloat
+		n.f = d.decFloat()
 	case bincVdSymbol:
-		vt = valueTypeSymbol
-		v = d.DecodeString()
+		n.v = valueTypeSymbol
+		n.s = d.DecodeString()
 	case bincVdString:
-		vt = valueTypeString
-		v = d.DecodeString()
+		n.v = valueTypeString
+		n.s = d.DecodeString()
 	case bincVdByteArray:
-		vt = valueTypeBytes
-		v = d.DecodeBytes(nil, false, false)
+		n.v = valueTypeBytes
+		n.l = d.DecodeBytes(nil, false, false)
 	case bincVdTimestamp:
-		vt = valueTypeTimestamp
+		n.v = valueTypeTimestamp
 		tt, err := decodeTime(d.r.readx(int(d.vs)))
 		if err != nil {
 			panic(err)
 		}
-		v = tt
+		n.t = tt
 	case bincVdCustomExt:
-		vt = valueTypeExt
+		n.v = valueTypeExt
 		l := d.decLen()
-		var re RawExt
-		re.Tag = uint64(d.r.readn1())
-		re.Data = d.r.readx(l)
-		v = &re
-		vt = valueTypeExt
+		n.u = uint64(d.r.readn1())
+		n.l = d.r.readx(l)
 	case bincVdArray:
-		vt = valueTypeArray
+		n.v = valueTypeArray
 		decodeFurther = true
 	case bincVdMap:
-		vt = valueTypeMap
+		n.v = valueTypeMap
 		decodeFurther = true
 	default:
 		d.d.errorf("decodeNaked: Unrecognized d.vd: 0x%x", d.vd)
-		return
 	}
 
 	if !decodeFurther {
 		d.bdRead = false
 	}
-	if vt == valueTypeUint && d.h.SignedInteger {
-		d.bdType = valueTypeInt
-		v = int64(v.(uint64))
+	if n.v == valueTypeUint && d.h.SignedInteger {
+		n.v = valueTypeInt
+		n.i = int64(n.u)
 	}
 	return
 }
@@ -898,6 +894,10 @@ type BincHandle struct {
 	binaryEncodingType
 }
 
+func (h *BincHandle) SetBytesExt(rt reflect.Type, tag uint64, ext BytesExt) (err error) {
+	return h.SetExt(rt, tag, &setExtWrapper{b: ext})
+}
+
 func (h *BincHandle) newEncDriver(e *Encoder) encDriver {
 	return &bincEncDriver{e: e, w: e.w}
 }
@@ -906,8 +906,12 @@ func (h *BincHandle) newDecDriver(d *Decoder) decDriver {
 	return &bincDecDriver{d: d, r: d.r, h: h, br: d.bytes}
 }
 
-func (h *BincHandle) SetBytesExt(rt reflect.Type, tag uint64, ext BytesExt) (err error) {
-	return h.SetExt(rt, tag, &setExtWrapper{b: ext})
+func (e *bincEncDriver) reset() {
+	e.w = e.e.w
+}
+
+func (d *bincDecDriver) reset() {
+	d.r = d.d.r
 }
 
 var _ decDriver = (*bincDecDriver)(nil)

codec/cbor.go

@@ -60,11 +60,11 @@ const (
 // -------------------
 
 type cborEncDriver struct {
+	noBuiltInTypes
+	encNoSeparator
 	e *Encoder
 	w encWriter
 	h *CborHandle
-	noBuiltInTypes
-	encNoSeparator
 	x [8]byte
 }
 
@@ -175,11 +175,10 @@ type cborDecDriver struct {
 	d      *Decoder
 	h      *CborHandle
 	r      decReader
+	b      [scratchByteArrayLen]byte
 	br     bool // bytes reader
 	bdRead bool
-	bdType valueType
 	bd     byte
-	b      [scratchByteArrayLen]byte
 	noBuiltInTypes
 	decNoSeparator
 }
@@ -187,24 +186,23 @@ type cborDecDriver struct {
 func (d *cborDecDriver) readNextBd() {
 	d.bd = d.r.readn1()
 	d.bdRead = true
-	d.bdType = valueTypeUnset
 }
 
-func (d *cborDecDriver) IsContainerType(vt valueType) (bv bool) {
-	switch vt {
-	case valueTypeNil:
-		return d.bd == cborBdNil
-	case valueTypeBytes:
-		return d.bd == cborBdIndefiniteBytes || (d.bd >= cborBaseBytes && d.bd < cborBaseString)
-	case valueTypeString:
-		return d.bd == cborBdIndefiniteString || (d.bd >= cborBaseString && d.bd < cborBaseArray)
-	case valueTypeArray:
-		return d.bd == cborBdIndefiniteArray || (d.bd >= cborBaseArray && d.bd < cborBaseMap)
-	case valueTypeMap:
-		return d.bd == cborBdIndefiniteMap || (d.bd >= cborBaseMap && d.bd < cborBaseTag)
+func (d *cborDecDriver) ContainerType() (vt valueType) {
+	if d.bd == cborBdNil {
+		return valueTypeNil
+	} else if d.bd == cborBdIndefiniteBytes || (d.bd >= cborBaseBytes && d.bd < cborBaseString) {
+		return valueTypeBytes
+	} else if d.bd == cborBdIndefiniteString || (d.bd >= cborBaseString && d.bd < cborBaseArray) {
+		return valueTypeString
+	} else if d.bd == cborBdIndefiniteArray || (d.bd >= cborBaseArray && d.bd < cborBaseMap) {
+		return valueTypeArray
+	} else if d.bd == cborBdIndefiniteMap || (d.bd >= cborBaseMap && d.bd < cborBaseTag) {
+		return valueTypeMap
+	} else {
+		// d.d.errorf("isContainerType: unsupported parameter: %v", vt)
 	}
-	d.d.errorf("isContainerType: unsupported parameter: %v", vt)
-	return // "unreachable"
+	return valueTypeUnset
 }
 
 func (d *cborDecDriver) TryDecodeAsNil() bool {
@@ -446,71 +444,72 @@ func (d *cborDecDriver) DecodeExt(rv interface{}, xtag uint64, ext Ext) (realxta
 	return
 }
 
-func (d *cborDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
+func (d *cborDecDriver) DecodeNaked() {
 	if !d.bdRead {
 		d.readNextBd()
 	}
 
+	n := &d.d.n
+	var decodeFurther bool
+
 	switch d.bd {
 	case cborBdNil:
-		vt = valueTypeNil
+		n.v = valueTypeNil
 	case cborBdFalse:
-		vt = valueTypeBool
-		v = false
+		n.v = valueTypeBool
+		n.b = false
 	case cborBdTrue:
-		vt = valueTypeBool
-		v = true
+		n.v = valueTypeBool
+		n.b = true
 	case cborBdFloat16, cborBdFloat32:
-		vt = valueTypeFloat
-		v = d.DecodeFloat(true)
+		n.v = valueTypeFloat
+		n.f = d.DecodeFloat(true)
 	case cborBdFloat64:
-		vt = valueTypeFloat
-		v = d.DecodeFloat(false)
+		n.v = valueTypeFloat
+		n.f = d.DecodeFloat(false)
 	case cborBdIndefiniteBytes:
-		vt = valueTypeBytes
-		v = d.DecodeBytes(nil, false, false)
+		n.v = valueTypeBytes
+		n.l = d.DecodeBytes(nil, false, false)
 	case cborBdIndefiniteString:
-		vt = valueTypeString
-		v = d.DecodeString()
+		n.v = valueTypeString
+		n.s = d.DecodeString()
 	case cborBdIndefiniteArray:
-		vt = valueTypeArray
+		n.v = valueTypeArray
 		decodeFurther = true
 	case cborBdIndefiniteMap:
-		vt = valueTypeMap
+		n.v = valueTypeMap
 		decodeFurther = true
 	default:
 		switch {
 		case d.bd >= cborBaseUint && d.bd < cborBaseNegInt:
 			if d.h.SignedInteger {
-				vt = valueTypeInt
-				v = d.DecodeInt(64)
+				n.v = valueTypeInt
+				n.i = d.DecodeInt(64)
 			} else {
-				vt = valueTypeUint
-				v = d.DecodeUint(64)
+				n.v = valueTypeUint
+				n.u = d.DecodeUint(64)
 			}
 		case d.bd >= cborBaseNegInt && d.bd < cborBaseBytes:
-			vt = valueTypeInt
-			v = d.DecodeInt(64)
+			n.v = valueTypeInt
+			n.i = d.DecodeInt(64)
 		case d.bd >= cborBaseBytes && d.bd < cborBaseString:
-			vt = valueTypeBytes
-			v = d.DecodeBytes(nil, false, false)
+			n.v = valueTypeBytes
+			n.l = d.DecodeBytes(nil, false, false)
 		case d.bd >= cborBaseString && d.bd < cborBaseArray:
-			vt = valueTypeString
-			v = d.DecodeString()
+			n.v = valueTypeString
+			n.s = d.DecodeString()
 		case d.bd >= cborBaseArray && d.bd < cborBaseMap:
-			vt = valueTypeArray
+			n.v = valueTypeArray
 			decodeFurther = true
 		case d.bd >= cborBaseMap && d.bd < cborBaseTag:
-			vt = valueTypeMap
+			n.v = valueTypeMap
 			decodeFurther = true
 		case d.bd >= cborBaseTag && d.bd < cborBaseSimple:
-			vt = valueTypeExt
-			var re RawExt
-			ui := d.decUint()
+			n.v = valueTypeExt
+			n.u = d.decUint()
+			n.l = nil
 			d.bdRead = false
-			re.Tag = ui
-			d.d.decode(&re.Value)
-			v = &re
+			// d.d.decode(&re.Value) // handled by decode itself.
 			// decodeFurther = true
 		default:
 			d.d.errorf("decodeNaked: Unrecognized d.bd: 0x%x", d.bd)
@@ -557,8 +556,12 @@ func (d *cborDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurthe
 //     // Now, vv contains the same string "one-byte"
 //
 type CborHandle struct {
-	BasicHandle
 	binaryEncodingType
+	BasicHandle
+}
+
+func (h *CborHandle) SetInterfaceExt(rt reflect.Type, tag uint64, ext InterfaceExt) (err error) {
+	return h.SetExt(rt, tag, &setExtWrapper{i: ext})
 }
 
 func (h *CborHandle) newEncDriver(e *Encoder) encDriver {
@@ -569,8 +572,12 @@ func (h *CborHandle) newDecDriver(d *Decoder) decDriver {
 	return &cborDecDriver{d: d, r: d.r, h: h, br: d.bytes}
 }
 
-func (h *CborHandle) SetInterfaceExt(rt reflect.Type, tag uint64, ext InterfaceExt) (err error) {
-	return h.SetExt(rt, tag, &setExtWrapper{i: ext})
+func (e *cborEncDriver) reset() {
+	e.w = e.e.w
+}
+
+func (d *cborDecDriver) reset() {
+	d.r = d.d.r
 }
 
 var _ decDriver = (*cborDecDriver)(nil)

codec/codec_test.go

@@ -28,6 +28,7 @@ import (
 	"fmt"
 	"io/ioutil"
 	"math"
+	"math/rand"
 	"net"
 	"net/rpc"
 	"os"
@@ -64,8 +65,11 @@ var (
 	testUseIoEncDec    bool
 	testStructToArray  bool
 	testCanonical      bool
+	testUseReset       bool
 	testWriteNoSymbols bool
 	testSkipIntf       bool
+	testInternStr      bool
+	testUseMust        bool
 
 	skipVerifyVal interface{} = &(struct{}{})
 
@@ -97,7 +101,14 @@ func testInitFlags() {
 	flag.BoolVar(&testStructToArray, "ts", false, "Set StructToArray option")
 	flag.BoolVar(&testWriteNoSymbols, "tn", false, "Set NoSymbols option")
 	flag.BoolVar(&testCanonical, "tc", false, "Set Canonical option")
+	flag.BoolVar(&testInternStr, "te", false, "Set InternStr option")
 	flag.BoolVar(&testSkipIntf, "tf", false, "Skip Interfaces")
+	flag.BoolVar(&testUseReset, "tr", false, "Use Reset")
+	flag.BoolVar(&testUseMust, "tm", true, "Use Must(En|De)code")
+}
+
+func testByteBuf(in []byte) *bytes.Buffer {
+	return bytes.NewBuffer(in)
 }
 
 type TestABC struct {
@@ -120,28 +131,36 @@ func (r *TestRpcInt) Echo123(args []string, res *string) error {
 	return nil
 }
 
-type testUnixNanoTimeExt struct{}
+type testUnixNanoTimeExt struct {
+	// keep timestamp here, so that do not incur interface-conversion costs
+	ts int64
+}
 
-func (x testUnixNanoTimeExt) WriteExt(interface{}) []byte { panic("unsupported") }
-func (x testUnixNanoTimeExt) ReadExt(interface{}, []byte) { panic("unsupported") }
-func (x testUnixNanoTimeExt) ConvertExt(v interface{}) interface{} {
+// func (x *testUnixNanoTimeExt) WriteExt(interface{}) []byte { panic("unsupported") }
+// func (x *testUnixNanoTimeExt) ReadExt(interface{}, []byte) { panic("unsupported") }
+func (x *testUnixNanoTimeExt) ConvertExt(v interface{}) interface{} {
 	switch v2 := v.(type) {
 	case time.Time:
-		return v2.UTC().UnixNano()
+		x.ts = v2.UTC().UnixNano()
 	case *time.Time:
-		return v2.UTC().UnixNano()
+		x.ts = v2.UTC().UnixNano()
 	default:
 		panic(fmt.Sprintf("unsupported format for time conversion: expecting time.Time; got %T", v))
 	}
+	return &x.ts
 }
-func (x testUnixNanoTimeExt) UpdateExt(dest interface{}, v interface{}) {
+func (x *testUnixNanoTimeExt) UpdateExt(dest interface{}, v interface{}) {
 	// fmt.Printf("testUnixNanoTimeExt.UpdateExt: v: %v\n", v)
 	tt := dest.(*time.Time)
 	switch v2 := v.(type) {
 	case int64:
 		*tt = time.Unix(0, v2).UTC()
+	case *int64:
+		*tt = time.Unix(0, *v2).UTC()
 	case uint64:
 		*tt = time.Unix(0, int64(v2)).UTC()
+	case *uint64:
+		*tt = time.Unix(0, int64(*v2)).UTC()
 	//case float64:
 	//case string:
 	default:
@@ -269,52 +288,40 @@ func testInit() {
 		fmt.Printf("====> depth: %v, ts: %#v\n", 2, ts0)
 	}
 
-	testJsonH.Canonical = testCanonical
-	testCborH.Canonical = testCanonical
-	testSimpleH.Canonical = testCanonical
-	testBincH.Canonical = testCanonical
-	testMsgpackH.Canonical = testCanonical
-
-	testJsonH.StructToArray = testStructToArray
-	testCborH.StructToArray = testStructToArray
-	testSimpleH.StructToArray = testStructToArray
-	testBincH.StructToArray = testStructToArray
-	testMsgpackH.StructToArray = testStructToArray
+	for _, v := range testHandles {
+		bh := v.getBasicHandle()
+		bh.InternString = testInternStr
+		bh.Canonical = testCanonical
+		bh.StructToArray = testStructToArray
+		// mostly doing this for binc
+		if testWriteNoSymbols {
+			bh.AsSymbols = AsSymbolNone
+		} else {
+			bh.AsSymbols = AsSymbolAll
+		}
+	}
 
 	testMsgpackH.RawToString = true
 
-	if testWriteNoSymbols {
-		testBincH.AsSymbols = AsSymbolNone
-	} else {
-		testBincH.AsSymbols = AsSymbolAll
-	}
-
 	// testMsgpackH.AddExt(byteSliceTyp, 0, testMsgpackH.BinaryEncodeExt, testMsgpackH.BinaryDecodeExt)
 	// testMsgpackH.AddExt(timeTyp, 1, testMsgpackH.TimeEncodeExt, testMsgpackH.TimeDecodeExt)
 	timeEncExt := func(rv reflect.Value) (bs []byte, err error) {
-		switch v2 := rv.Interface().(type) {
-		case time.Time:
-			bs = encodeTime(v2)
-		case *time.Time:
-			bs = encodeTime(*v2)
-		default:
-			err = fmt.Errorf("unsupported format for time conversion: expecting time.Time; got %T", v2)
-		}
+		defer panicToErr(&err)
+		bs = timeExt{}.WriteExt(rv.Interface())
 		return
 	}
 	timeDecExt := func(rv reflect.Value, bs []byte) (err error) {
-		tt, err := decodeTime(bs)
-		if err == nil {
-			*(rv.Interface().(*time.Time)) = tt
-		}
+		defer panicToErr(&err)
+		timeExt{}.ReadExt(rv.Interface(), bs)
 		return
 	}
 
 	// add extensions for msgpack, simple for time.Time, so we can encode/decode same way.
-	testMsgpackH.AddExt(timeTyp, 1, timeEncExt, timeDecExt)
+	// use different flavors of XXXExt calls, including deprecated ones.
 	testSimpleH.AddExt(timeTyp, 1, timeEncExt, timeDecExt)
-	testCborH.SetExt(timeTyp, 1, &testUnixNanoTimeExt{})
-	testJsonH.SetExt(timeTyp, 1, &testUnixNanoTimeExt{})
+	testMsgpackH.SetBytesExt(timeTyp, 1, timeExt{})
+	testCborH.SetInterfaceExt(timeTyp, 1, &testUnixNanoTimeExt{})
+	testJsonH.SetInterfaceExt(timeTyp, 1, &testUnixNanoTimeExt{})
 
 	primitives := []interface{}{
 		int8(-8),
@@ -429,23 +436,11 @@ func testInit() {
 }
 
 func testUnmarshal(v interface{}, data []byte, h Handle) (err error) {
-	if testUseIoEncDec {
-		NewDecoder(bytes.NewBuffer(data), h).MustDecode(v)
-	} else {
-		NewDecoderBytes(data, h).MustDecode(v)
-	}
-	return
+	return testCodecDecode(data, v, h)
 }
 
 func testMarshal(v interface{}, h Handle) (bs []byte, err error) {
-	if testUseIoEncDec {
-		var buf bytes.Buffer
-		NewEncoder(&buf, h).MustEncode(v)
-		bs = buf.Bytes()
-		return
-	}
-	NewEncoderBytes(&bs, h).MustEncode(v)
-	return
+	return testCodecEncode(v, nil, testByteBuf, h)
 }
 
 func testMarshalErr(v interface{}, h Handle, t *testing.T, name string) (bs []byte, err error) {
@@ -663,6 +658,13 @@ func testCodecMiscOne(t *testing.T, h Handle) {
 	// test both pointer and non-pointer (value)
 	for _, tarr1 := range []interface{}{tarr0, &tarr0} {
 		bs, err = testMarshalErr(tarr1, h, t, "tarr1")
+		if err != nil {
+			logT(t, "Error marshalling: %v", err)
+			t.FailNow()
+		}
+		if _, ok := h.(*JsonHandle); ok {
+			logT(t, "Marshal as: %s", bs)
+		}
 		var tarr2 tarr
 		testUnmarshalErr(&tarr2, bs, h, t, "tarr2")
 		checkEqualT(t, tarr0, tarr2, "tarr0=tarr2")
@@ -898,9 +900,10 @@ func doTestMapEncodeForCanonical(t *testing.T, name string, h Handle) {
 	// encode v1 into b1, decode b1 into v2, encode v2 into b2, compare b1 and b2
 
 	bh := h.getBasicHandle()
-	canonical0 := bh.Canonical
-	bh.Canonical = true
-	defer func() { bh.Canonical = canonical0 }()
+	if !bh.Canonical {
+		bh.Canonical = true
+		defer func() { bh.Canonical = false }()
+	}
 
 	e1 := NewEncoderBytes(&b1, h)
 	e1.MustEncode(v1)
@@ -1019,11 +1022,17 @@ func doTestMsgpackRpcSpecGoClientToPythonSvc(t *testing.T) {
 	if testSkipRPCTests {
 		return
 	}
-	openPort := "6789"
-	cmd := exec.Command("python", "test.py", "rpc-server", openPort, "2")
+	// openPorts are between 6700 and 6800
+	r := rand.New(rand.NewSource(time.Now().UnixNano()))
+	openPort := strconv.FormatInt(6700+r.Int63n(99), 10)
+	// openPort := "6792"
+	cmd := exec.Command("python", "test.py", "rpc-server", openPort, "4")
 	checkErrT(t, cmd.Start())
-	time.Sleep(100 * time.Millisecond) // time for python rpc server to start
 	bs, err2 := net.Dial("tcp", ":"+openPort)
+	for i := 0; i < 10 && err2 != nil; i++ {
+		time.Sleep(50 * time.Millisecond) // time for python rpc server to start
+		bs, err2 = net.Dial("tcp", ":"+openPort)
+	}
 	checkErrT(t, err2)
 	cc := MsgpackSpecRpc.ClientCodec(bs, testMsgpackH)
 	cl := rpc.NewClientWithCodec(cc)
@@ -1034,6 +1043,7 @@ func doTestMsgpackRpcSpecGoClientToPythonSvc(t *testing.T) {
 	var mArgs MsgpackSpecRpcMultiArgs = []interface{}{"A1", "B2", "C3"}
 	checkErrT(t, cl.Call("Echo123", mArgs, &rstr))
 	checkEqualT(t, rstr, "1:A1 2:B2 3:C3", "rstr=")
+	cmd.Process.Kill()
 }
 
 func doTestMsgpackRpcSpecPythonClientToGoSvc(t *testing.T) {

codec/codecgen/gen.go

@@ -220,7 +220,7 @@ func Generate(outfile, buildTag, codecPkgPath string, uid int64, useUnsafe bool,
 	os.Remove(outfile)
 
 	// execute go run frun
-	cmd := exec.Command("go", "run", "-tags="+goRunTag, frunMainName) //, frunPkg.Name())
+	cmd := exec.Command("go", "run", "-tags=notfastpath "+goRunTag, frunMainName) //, frunPkg.Name())
 	var buf bytes.Buffer
 	cmd.Stdout = &buf
 	cmd.Stderr = &buf

codec/codecgen_test.go

@@ -7,8 +7,9 @@ import (
 	"testing"
 )
 
-func TestCodecgenJson1(t *testing.T) {
+func _TestCodecgenJson1(t *testing.T) {
 	// This is just a simplistic test for codecgen.
+	// It is typically disabled. We only enable it for debugging purposes.
 	const callCodecgenDirect bool = true
 	v := newTestStruc(2, false, !testSkipIntf, false)
 	var bs []byte

codec/encode.go

@@ -62,13 +62,14 @@ type encDriver interface {
 	EncodeExt(v interface{}, xtag uint64, ext Ext, e *Encoder)
 	EncodeArrayStart(length int)
 	EncodeMapStart(length int)
-	EncodeEnd()
 	EncodeString(c charEncoding, v string)
 	EncodeSymbol(v string)
 	EncodeStringBytes(c charEncoding, v []byte)
 	//TODO
 	//encBignum(f *big.Int)
 	//encStringRunes(c charEncoding, v []rune)
+
+	reset()
 }
 
 type encDriverAsis interface {
@@ -158,6 +159,7 @@ func (o *simpleIoEncWriterWriter) Write(p []byte) (n int, err error) {
 // ioEncWriter implements encWriter and can write to an io.Writer implementation
 type ioEncWriter struct {
 	w ioEncWriterWriter
+	s simpleIoEncWriterWriter
 	// x [8]byte // temp byte array re-used internally for efficiency
 }
 
@@ -382,30 +384,32 @@ func (f *encFnInfo) kSlice(rv reflect.Value) {
 	//   (don't call rv.Bytes, rv.Slice, etc).
 	// E.g. type struct S{B [2]byte};
 	//   Encode(S{}) will bomb on "panic: slice of unaddressable array".
+	e := f.e
 	if f.seq != seqTypeArray {
 		if rv.IsNil() {
-			f.e.e.EncodeNil()
+			e.e.EncodeNil()
 			return
 		}
 		// If in this method, then there was no extension function defined.
 		// So it's okay to treat as []byte.
 		if ti.rtid == uint8SliceTypId {
-			f.e.e.EncodeStringBytes(c_RAW, rv.Bytes())
+			e.e.EncodeStringBytes(c_RAW, rv.Bytes())
 			return
 		}
 	}
+	cr := e.cr
 	rtelem := ti.rt.Elem()
 	l := rv.Len()
-	if rtelem.Kind() == reflect.Uint8 {
+	if ti.rtid == uint8SliceTypId || rtelem.Kind() == reflect.Uint8 {
 		switch f.seq {
 		case seqTypeArray:
-			// if l == 0 { f.e.e.encodeStringBytes(c_RAW, nil) } else
+			// if l == 0 { e.e.encodeStringBytes(c_RAW, nil) } else
 			if rv.CanAddr() {
-				f.e.e.EncodeStringBytes(c_RAW, rv.Slice(0, l).Bytes())
+				e.e.EncodeStringBytes(c_RAW, rv.Slice(0, l).Bytes())
 			} else {
 				var bs []byte
-				if l <= cap(f.e.b) {
-					bs = f.e.b[:l]
+				if l <= cap(e.b) {
+					bs = e.b[:l]
 				} else {
 					bs = make([]byte, l)
 				}
@@ -414,12 +418,12 @@ func (f *encFnInfo) kSlice(rv reflect.Value) {
 				// for i := 0; i < l; i++ {
 				// 	bs[i] = byte(rv.Index(i).Uint())
 				// }
-				f.e.e.EncodeStringBytes(c_RAW, bs)
+				e.e.EncodeStringBytes(c_RAW, bs)
 			}
 		case seqTypeSlice:
-			f.e.e.EncodeStringBytes(c_RAW, rv.Bytes())
+			e.e.EncodeStringBytes(c_RAW, rv.Bytes())
 		case seqTypeChan:
-			bs := f.e.b[:0]
+			bs := e.b[:0]
 			// do not use range, so that the number of elements encoded
 			// does not change, and encoding does not hang waiting on someone to close chan.
 			// for b := range rv.Interface().(<-chan byte) {
@@ -429,22 +433,21 @@ func (f *encFnInfo) kSlice(rv reflect.Value) {
 			for i := 0; i < l; i++ {
 				bs = append(bs, <-ch)
 			}
-			f.e.e.EncodeStringBytes(c_RAW, bs)
+			e.e.EncodeStringBytes(c_RAW, bs)
 		}
 		return
 	}
 
 	if ti.mbs {
 		if l%2 == 1 {
-			f.e.errorf("mapBySlice requires even slice length, but got %v", l)
+			e.errorf("mapBySlice requires even slice length, but got %v", l)
 			return
 		}
-		f.e.e.EncodeMapStart(l / 2)
+		e.e.EncodeMapStart(l / 2)
 	} else {
-		f.e.e.EncodeArrayStart(l)
+		e.e.EncodeArrayStart(l)
 	}
 
-	e := f.e
 	if l > 0 {
 		for rtelem.Kind() == reflect.Ptr {
 			rtelem = rtelem.Elem()
@@ -459,29 +462,48 @@ func (f *encFnInfo) kSlice(rv reflect.Value) {
 		}
 		// TODO: Consider perf implication of encoding odd index values as symbols if type is string
 		for j := 0; j < l; j++ {
+			if cr != nil {
+				if ti.mbs {
+					if l%2 == 0 {
+						cr.sendContainerState(containerMapKey)
+					} else {
+						cr.sendContainerState(containerMapValue)
+					}
+				} else {
+					cr.sendContainerState(containerArrayElem)
+				}
+			}
 			if f.seq == seqTypeChan {
 				if rv2, ok2 := rv.Recv(); ok2 {
 					e.encodeValue(rv2, fn)
+				} else {
+					e.encode(nil) // WE HAVE TO DO SOMETHING, so nil if nothing received.
 				}
 			} else {
 				e.encodeValue(rv.Index(j), fn)
 			}
 		}
-
 	}
 
-	f.e.e.EncodeEnd()
+	if cr != nil {
+		if ti.mbs {
+			cr.sendContainerState(containerMapEnd)
+		} else {
+			cr.sendContainerState(containerArrayEnd)
+		}
+	}
 }
 
 func (f *encFnInfo) kStruct(rv reflect.Value) {
 	fti := f.ti
 	e := f.e
+	cr := e.cr
 	tisfi := fti.sfip
 	toMap := !(fti.toArray || e.h.StructToArray)
 	newlen := len(fti.sfi)
 
 	// Use sync.Pool to reduce allocating slices unnecessarily.
-	// The cost of the occasional locking is less than the cost of locking.
+	// The cost of the occasional locking is less than the cost of new allocation.
 	pool, poolv, fkvs := encStructPoolGet(newlen)
 
 	// if toMap, use the sorted array. If toArray, use unsorted array (to match sequence in struct)
@@ -519,7 +541,7 @@ func (f *encFnInfo) kStruct(rv reflect.Value) {
 
 	// debugf(">>>> kStruct: newlen: %v", newlen)
 	// sep := !e.be
-	ee := f.e.e //don't dereference everytime
+	ee := e.e //don't dereference everytime
 
 	if toMap {
 		ee.EncodeMapStart(newlen)
@@ -527,21 +549,35 @@ func (f *encFnInfo) kStruct(rv reflect.Value) {
 		asSymbols := e.h.AsSymbols == AsSymbolDefault || e.h.AsSymbols&AsSymbolStructFieldNameFlag != 0
 		for j := 0; j < newlen; j++ {
 			kv = fkvs[j]
+			if cr != nil {
+				cr.sendContainerState(containerMapKey)
+			}
 			if asSymbols {
 				ee.EncodeSymbol(kv.v)
 			} else {
 				ee.EncodeString(c_UTF8, kv.v)
 			}
+			if cr != nil {
+				cr.sendContainerState(containerMapValue)
+			}
 			e.encodeValue(kv.r, nil)
 		}
+		if cr != nil {
+			cr.sendContainerState(containerMapEnd)
+		}
 	} else {
 		ee.EncodeArrayStart(newlen)
 		for j := 0; j < newlen; j++ {
 			kv = fkvs[j]
+			if cr != nil {
+				cr.sendContainerState(containerArrayElem)
+			}
 			e.encodeValue(kv.r, nil)
 		}
+		if cr != nil {
+			cr.sendContainerState(containerArrayEnd)
+		}
 	}
-	ee.EncodeEnd()
 
 	// do not use defer. Instead, use explicit pool return at end of function.
 	// defer has a cost we are trying to avoid.
@@ -578,8 +614,11 @@ func (f *encFnInfo) kMap(rv reflect.Value) {
 	l := rv.Len()
 	ee.EncodeMapStart(l)
 	e := f.e
+	cr := e.cr
 	if l == 0 {
-		ee.EncodeEnd()
+		if cr != nil {
+			cr.sendContainerState(containerMapEnd)
+		}
 		return
 	}
 	var asSymbols bool
@@ -622,6 +661,9 @@ func (f *encFnInfo) kMap(rv reflect.Value) {
 		e.kMapCanonical(rtkeyid, rtkey, rv, mks, valFn, asSymbols)
 	} else {
 		for j := range mks {
+			if cr != nil {
+				cr.sendContainerState(containerMapKey)
+			}
 			if keyTypeIsString {
 				if asSymbols {
 					ee.EncodeSymbol(mks[j].String())
@@ -631,15 +673,20 @@ func (f *encFnInfo) kMap(rv reflect.Value) {
 			} else {
 				e.encodeValue(mks[j], keyFn)
 			}
+			if cr != nil {
+				cr.sendContainerState(containerMapValue)
+			}
 			e.encodeValue(rv.MapIndex(mks[j]), valFn)
 		}
 	}
-
-	ee.EncodeEnd()
+	if cr != nil {
+		cr.sendContainerState(containerMapEnd)
+	}
 }
 
 func (e *Encoder) kMapCanonical(rtkeyid uintptr, rtkey reflect.Type, rv reflect.Value, mks []reflect.Value, valFn *encFn, asSymbols bool) {
 	ee := e.e
+	cr := e.cr
 	// we previously did out-of-band if an extension was registered.
 	// This is not necessary, as the natural kind is sufficient for ordering.
 
@@ -652,7 +699,13 @@ func (e *Encoder) kMapCanonical(rtkeyid uintptr, rtkey reflect.Type, rv reflect.
 		}
 		sort.Sort(bytesRvSlice(mksv))
 		for i := range mksv {
+			if cr != nil {
+				cr.sendContainerState(containerMapKey)
+			}
 			ee.EncodeStringBytes(c_RAW, mksv[i].v)
+			if cr != nil {
+				cr.sendContainerState(containerMapValue)
+			}
 			e.encodeValue(rv.MapIndex(mksv[i].r), valFn)
 		}
 	} else {
@@ -666,7 +719,13 @@ func (e *Encoder) kMapCanonical(rtkeyid uintptr, rtkey reflect.Type, rv reflect.
 			}
 			sort.Sort(boolRvSlice(mksv))
 			for i := range mksv {
+				if cr != nil {
+					cr.sendContainerState(containerMapKey)
+				}
 				ee.EncodeBool(mksv[i].v)
+				if cr != nil {
+					cr.sendContainerState(containerMapValue)
+				}
 				e.encodeValue(rv.MapIndex(mksv[i].r), valFn)
 			}
 		case reflect.String:
@@ -678,11 +737,17 @@ func (e *Encoder) kMapCanonical(rtkeyid uintptr, rtkey reflect.Type, rv reflect.
 			}
 			sort.Sort(stringRvSlice(mksv))
 			for i := range mksv {
+				if cr != nil {
+					cr.sendContainerState(containerMapKey)
+				}
 				if asSymbols {
 					ee.EncodeSymbol(mksv[i].v)
 				} else {
 					ee.EncodeString(c_UTF8, mksv[i].v)
 				}
+				if cr != nil {
+					cr.sendContainerState(containerMapValue)
+				}
 				e.encodeValue(rv.MapIndex(mksv[i].r), valFn)
 			}
 		case reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uint, reflect.Uintptr:
@@ -694,7 +759,13 @@ func (e *Encoder) kMapCanonical(rtkeyid uintptr, rtkey reflect.Type, rv reflect.
 			}
 			sort.Sort(uintRvSlice(mksv))
 			for i := range mksv {
+				if cr != nil {
+					cr.sendContainerState(containerMapKey)
+				}
 				ee.EncodeUint(mksv[i].v)
+				if cr != nil {
+					cr.sendContainerState(containerMapValue)
+				}
 				e.encodeValue(rv.MapIndex(mksv[i].r), valFn)
 			}
 		case reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Int:
@@ -706,7 +777,13 @@ func (e *Encoder) kMapCanonical(rtkeyid uintptr, rtkey reflect.Type, rv reflect.
 			}
 			sort.Sort(intRvSlice(mksv))
 			for i := range mksv {
+				if cr != nil {
+					cr.sendContainerState(containerMapKey)
+				}
 				ee.EncodeInt(mksv[i].v)
+				if cr != nil {
+					cr.sendContainerState(containerMapValue)
+				}
 				e.encodeValue(rv.MapIndex(mksv[i].r), valFn)
 			}
 		case reflect.Float32:
@@ -718,7 +795,13 @@ func (e *Encoder) kMapCanonical(rtkeyid uintptr, rtkey reflect.Type, rv reflect.
 			}
 			sort.Sort(floatRvSlice(mksv))
 			for i := range mksv {
+				if cr != nil {
+					cr.sendContainerState(containerMapKey)
+				}
 				ee.EncodeFloat32(float32(mksv[i].v))
+				if cr != nil {
+					cr.sendContainerState(containerMapValue)
+				}
 				e.encodeValue(rv.MapIndex(mksv[i].r), valFn)
 			}
 		case reflect.Float64:
@@ -730,7 +813,13 @@ func (e *Encoder) kMapCanonical(rtkeyid uintptr, rtkey reflect.Type, rv reflect.
 			}
 			sort.Sort(floatRvSlice(mksv))
 			for i := range mksv {
+				if cr != nil {
+					cr.sendContainerState(containerMapKey)
+				}
 				ee.EncodeFloat64(mksv[i].v)
+				if cr != nil {
+					cr.sendContainerState(containerMapValue)
+				}
 				e.encodeValue(rv.MapIndex(mksv[i].r), valFn)
 			}
 		default:
@@ -749,7 +838,13 @@ func (e *Encoder) kMapCanonical(rtkeyid uintptr, rtkey reflect.Type, rv reflect.
 			}
 			sort.Sort(bytesRvSlice(mksbv))
 			for j := range mksbv {
+				if cr != nil {
+					cr.sendContainerState(containerMapKey)
+				}
 				e.asis(mksbv[j].v)
+				if cr != nil {
+					cr.sendContainerState(containerMapValue)
+				}
 				e.encodeValue(rv.MapIndex(mksbv[j].r), valFn)
 			}
 		}
@@ -787,12 +882,15 @@ type Encoder struct {
 
 	wi ioEncWriter
 	wb bytesEncWriter
+
 	h  *BasicHandle
+	hh Handle
 
+	cr containerStateRecv
 	as encDriverAsis
-	hh Handle
-	f  map[uintptr]*encFn
-	b  [scratchByteArrayLen]byte
+
+	f map[uintptr]*encFn
+	b [scratchByteArrayLen]byte
 }
 
 // NewEncoder returns an Encoder for encoding into an io.Writer.
@@ -800,20 +898,8 @@ type Encoder struct {
 // For efficiency, Users are encouraged to pass in a memory buffered writer
 // (eg bufio.Writer, bytes.Buffer).
 func NewEncoder(w io.Writer, h Handle) *Encoder {
-	e := &Encoder{hh: h, h: h.getBasicHandle(), be: h.isBinary()}
-	ww, ok := w.(ioEncWriterWriter)
-	if !ok {
-		sww := simpleIoEncWriterWriter{w: w}
-		sww.bw, _ = w.(io.ByteWriter)
-		sww.sw, _ = w.(ioEncStringWriter)
-		ww = &sww
-		//ww = bufio.NewWriterSize(w, defEncByteBufSize)
-	}
-	e.wi.w = ww
-	e.w = &e.wi
-	_, e.js = h.(*JsonHandle)
-	e.e = h.newEncDriver(e)
-	e.as, _ = e.e.(encDriverAsis)
+	e := newEncoder(h)
+	e.Reset(w)
 	return e
 }
 
@@ -823,19 +909,56 @@ func NewEncoder(w io.Writer, h Handle) *Encoder {
 // It will potentially replace the output byte slice pointed to.
 // After encoding, the out parameter contains the encoded contents.
 func NewEncoderBytes(out *[]byte, h Handle) *Encoder {
+	e := newEncoder(h)
+	e.ResetBytes(out)
+	return e
+}
+
+func newEncoder(h Handle) *Encoder {
 	e := &Encoder{hh: h, h: h.getBasicHandle(), be: h.isBinary()}
+	_, e.js = h.(*JsonHandle)
+	e.e = h.newEncDriver(e)
+	e.as, _ = e.e.(encDriverAsis)
+	e.cr, _ = e.e.(containerStateRecv)
+	return e
+}
+
+// Reset the Encoder with a new output stream.
+//
+// This accomodates using the state of the Encoder,
+// where it has "cached" information about sub-engines.
+func (e *Encoder) Reset(w io.Writer) {
+	ww, ok := w.(ioEncWriterWriter)
+	if ok {
+		e.wi.w = ww
+	} else {
+		sww := &e.wi.s
+		sww.w = w
+		sww.bw, _ = w.(io.ByteWriter)
+		sww.sw, _ = w.(ioEncStringWriter)
+		e.wi.w = sww
+		//ww = bufio.NewWriterSize(w, defEncByteBufSize)
+	}
+	e.w = &e.wi
+	e.e.reset()
+}
+
+func (e *Encoder) ResetBytes(out *[]byte) {
 	in := *out
 	if in == nil {
 		in = make([]byte, defEncByteBufSize)
 	}
-	e.wb.b, e.wb.out = in, out
+	e.wb.b, e.wb.out, e.wb.c = in, out, 0
 	e.w = &e.wb
-	_, e.js = h.(*JsonHandle)
-	e.e = h.newEncDriver(e)
-	e.as, _ = e.e.(encDriverAsis)
-	return e
+	e.e.reset()
 }
 
+// func (e *Encoder) sendContainerState(c containerState) {
+// 	if e.cr != nil {
+// 		e.cr.sendContainerState(c)
+// 	}
+// }
+
 // Encode writes an object into a stream.
 //
 // Encoding can be configured via the struct tag for the fields.
@@ -1020,26 +1143,24 @@ func (e *Encoder) encodeI(iv interface{}, checkFastpath, checkCodecSelfer bool)
 }
 
 func (e *Encoder) preEncodeValue(rv reflect.Value) (rv2 reflect.Value, proceed bool) {
-LOOP:
-	for {
-		switch rv.Kind() {
-		case reflect.Ptr, reflect.Interface:
-			if rv.IsNil() {
-				e.e.EncodeNil()
-				return
-			}
-			rv = rv.Elem()
-			continue LOOP
-		case reflect.Slice, reflect.Map:
-			if rv.IsNil() {
-				e.e.EncodeNil()
-				return
-			}
-		case reflect.Invalid, reflect.Func:
+	// use a goto statement instead of a recursive function for ptr/interface.
+TOP:
+	switch rv.Kind() {
+	case reflect.Ptr, reflect.Interface:
+		if rv.IsNil() {
 			e.e.EncodeNil()
 			return
 		}
-		break
+		rv = rv.Elem()
+		goto TOP
+	case reflect.Slice, reflect.Map:
+		if rv.IsNil() {
+			e.e.EncodeNil()
+			return
+		}
+	case reflect.Invalid, reflect.Func:
+		e.e.EncodeNil()
+		return
 	}
 
 	return rv, true

codec/fast-path.go.tmpl

@@ -1,4 +1,4 @@
-// //+build ignore
+// +build !notfastpath
 
 // Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
 // Use of this source code is governed by a MIT license found in the LICENSE file.
@@ -106,6 +106,9 @@ func init() {
 
 // -- -- fast path type switch
 func fastpathEncodeTypeSwitch(iv interface{}, e *Encoder) bool {
+	if !fastpathEnabled {
+		return false
+	}
 	switch v := iv.(type) {
 {{range .Values}}{{if not .Primitive}}{{if not .MapKey }}
 	case []{{ .Elem }}:{{else}}
@@ -116,13 +119,16 @@ func fastpathEncodeTypeSwitch(iv interface{}, e *Encoder) bool {
 		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, fastpathCheckNilTrue, e)
 {{end}}{{end}}
 	default:
-		_ = v // TODO: workaround https://github.com/golang/go/issues/12927 (remove after go 1.6 release)
+        _ = v // TODO: workaround https://github.com/golang/go/issues/12927 (remove after go 1.6 release)
 		return false
 	}
 	return true
 }
 
 func fastpathEncodeTypeSwitchSlice(iv interface{}, e *Encoder) bool {
+	if !fastpathEnabled {
+		return false
+	}
 	switch v := iv.(type) {
 {{range .Values}}{{if not .Primitive}}{{if not .MapKey }}
 	case []{{ .Elem }}:
@@ -131,12 +137,16 @@ func fastpathEncodeTypeSwitchSlice(iv interface{}, e *Encoder) bool {
 		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, fastpathCheckNilTrue, e)
 {{end}}{{end}}{{end}}
 	default:
+        _ = v // TODO: workaround https://github.com/golang/go/issues/12927 (remove after go 1.6 release)
 		return false
 	}
 	return true
 }
 
 func fastpathEncodeTypeSwitchMap(iv interface{}, e *Encoder) bool {
+	if !fastpathEnabled {
+		return false
+	}
 	switch v := iv.(type) {
 {{range .Values}}{{if not .Primitive}}{{if .MapKey }}
 	case map[{{ .MapKey }}]{{ .Elem }}:
@@ -145,6 +155,7 @@ func fastpathEncodeTypeSwitchMap(iv interface{}, e *Encoder) bool {
 		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, fastpathCheckNilTrue, e)
 {{end}}{{end}}{{end}}
 	default:
+        _ = v // TODO: workaround https://github.com/golang/go/issues/12927 (remove after go 1.6 release)
 		return false
 	}
 	return true
@@ -157,16 +168,18 @@ func (f *encFnInfo) {{ .MethodNamePfx "fastpathEnc" false }}R(rv reflect.Value)
 	fastpathTV.{{ .MethodNamePfx "Enc" false }}V(rv.Interface().([]{{ .Elem }}), fastpathCheckNilFalse, f.e)
 }
 func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v []{{ .Elem }}, checkNil bool, e *Encoder) {
-	ee := e.e 
+	ee := e.e
+	cr := e.cr
 	if checkNil && v == nil {
 		ee.EncodeNil()
 		return
 	}
 	ee.EncodeArrayStart(len(v))
 	for _, v2 := range v {
+		if cr != nil { cr.sendContainerState(containerArrayElem) }
 		{{ encmd .Elem "v2"}}
 	}
-	ee.EncodeEnd()
+	if cr != nil { cr.sendContainerState(containerArrayEnd) }{{/* ee.EncodeEnd() */}}
 }
 
 {{end}}{{end}}{{end}}
@@ -178,6 +191,7 @@ func (f *encFnInfo) {{ .MethodNamePfx "fastpathEnc" false }}R(rv reflect.Value)
 }
 func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v map[{{ .MapKey }}]{{ .Elem }}, checkNil bool, e *Encoder) {
 	ee := e.e
+	cr := e.cr
 	if checkNil && v == nil {
 		ee.EncodeNil()
 		return
@@ -201,7 +215,9 @@ func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v map[{{ .MapKey }}]{{ .Ele
 		}
 		sort.Sort(bytesISlice(v2))
 		for j := range v2 {
+			if cr != nil { cr.sendContainerState(containerMapKey) }
 			e.asis(v2[j].v)
+			if cr != nil { cr.sendContainerState(containerMapValue) }
 			e.encode(v[v2[j].i])
 		} {{else}}{{ $x := sorttype .MapKey true}}v2 := make([]{{ $x }}, len(v))
 		var i int 
@@ -211,24 +227,28 @@ func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v map[{{ .MapKey }}]{{ .Ele
 		}
 		sort.Sort({{ sorttype .MapKey false}}(v2))
 		for _, k2 := range v2 {
+			if cr != nil { cr.sendContainerState(containerMapKey) }
 			{{if eq .MapKey "string"}}if asSymbols {
 				ee.EncodeSymbol(k2)
 			} else {
 				ee.EncodeString(c_UTF8, k2)
 			}{{else}}{{ $y := printf "%s(k2)" .MapKey }}{{ encmd .MapKey $y }}{{end}}
+			if cr != nil { cr.sendContainerState(containerMapValue) }
 			{{ $y := printf "v[%s(k2)]" .MapKey }}{{ encmd .Elem $y }}
 		} {{end}}
 	} else {
 		for k2, v2 := range v {
+			if cr != nil { cr.sendContainerState(containerMapKey) }
 			{{if eq .MapKey "string"}}if asSymbols {
 				ee.EncodeSymbol(k2)
 			} else {
 				ee.EncodeString(c_UTF8, k2)
 			}{{else}}{{ encmd .MapKey "k2"}}{{end}}
+			if cr != nil { cr.sendContainerState(containerMapValue) }
 			{{ encmd .Elem "v2"}}
 		}
 	}
-	ee.EncodeEnd()
+	if cr != nil { cr.sendContainerState(containerMapEnd) }{{/* ee.EncodeEnd() */}}
 }
 
 {{end}}{{end}}{{end}}
@@ -237,6 +257,9 @@ func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v map[{{ .MapKey }}]{{ .Ele
 
 // -- -- fast path type switch
 func fastpathDecodeTypeSwitch(iv interface{}, d *Decoder) bool {
+	if !fastpathEnabled {
+		return false
+	}
 	switch v := iv.(type) {
 {{range .Values}}{{if not .Primitive}}{{if not .MapKey }}
 	case []{{ .Elem }}:{{else}}
@@ -250,6 +273,7 @@ func fastpathDecodeTypeSwitch(iv interface{}, d *Decoder) bool {
 		}
 {{end}}{{end}}
 	default:
+        _ = v // TODO: workaround https://github.com/golang/go/issues/12927 (remove after go 1.6 release)
 		return false
 	}
 	return true
@@ -283,8 +307,7 @@ func (f fastpathT) {{ .MethodNamePfx "Dec" false }}X(vp *[]{{ .Elem }}, checkNil
 		*vp = v 
 	}
 }
-func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v []{{ .Elem }}, checkNil bool, canChange bool, 
-	d *Decoder) (_ []{{ .Elem }}, changed bool) {
+func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v []{{ .Elem }}, checkNil bool, canChange bool, d *Decoder) (_ []{{ .Elem }}, changed bool) {
 	dd := d.d
 	{{/* // if dd.isContainerType(valueTypeNil) { dd.TryDecodeAsNil() */}}
 	if checkNil && dd.TryDecodeAsNil() {
@@ -295,28 +318,22 @@ func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v []{{ .Elem }}, checkNil b
 	}
 
 	slh, containerLenS := d.decSliceHelperStart()
-    x2read := containerLenS
-    var xtrunc bool 
-	if canChange && v == nil {
-		var xlen int 
-		if xlen, xtrunc = decInferLen(containerLenS, d.h.MaxInitLen, {{ .Size }}); xtrunc {
-			x2read = xlen
-		}
-		v = make([]{{ .Elem }}, xlen)
-		changed = true
-	} 
 	if containerLenS == 0 {
-		if canChange && len(v) != 0 {
-			v = v[:0]
-			changed = true 
-		}{{/*
-			// slh.End() // dd.ReadArrayEnd()
-		*/}}
-		return v, changed 
+		if canChange {
+			if v == nil {
+				v = []{{ .Elem }}{}
+			} else if len(v) != 0 {
+				v = v[:0]
+			}
+			changed = true
+		}
+		slh.End()
+		return
 	}
 	
-	{{/* // for j := 0; j < containerLenS; j++ { */}}
 	if containerLenS > 0 {
+		x2read := containerLenS
+		var xtrunc bool 
 		if containerLenS > cap(v) {
 			if canChange { {{/*
 				// fast-path is for "basic" immutable types, so no need to copy them over
@@ -324,37 +341,64 @@ func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v []{{ .Elem }}, checkNil b
 				// copy(s, v[:cap(v)])
 				// v = s */}}
 				var xlen int 
-                if xlen, xtrunc = decInferLen(containerLenS, d.h.MaxInitLen, {{ .Size }}); xtrunc {
-					x2read = xlen
+                xlen, xtrunc = decInferLen(containerLenS, d.h.MaxInitLen, {{ .Size }})
+				if xtrunc {
+					if xlen <= cap(v) {
+						v = v[:xlen]
+					} else {
+						v = make([]{{ .Elem }}, xlen)
+					}
+				} else {
+					v = make([]{{ .Elem }}, xlen)
 				}
-                v = make([]{{ .Elem }}, xlen)
 				changed = true
 			} else {
 				d.arrayCannotExpand(len(v), containerLenS)
-				x2read = len(v)
 			}
+			x2read = len(v)
 		} else if containerLenS != len(v) {
-			v = v[:containerLenS]
-			changed = true
-		}
-		{{/* // all checks done. cannot go past len. */}}
+			if canChange {
+				v = v[:containerLenS]
+				changed = true
+			}
+		} {{/* // all checks done. cannot go past len. */}}
 		j := 0
-		for ; j < x2read; j++ { 
+		for ; j < x2read; j++ {
+			slh.ElemContainerState(j)
 			{{ if eq .Elem "interface{}" }}d.decode(&v[j]){{ else }}v[j] = {{ decmd .Elem }}{{ end }}
 		}
 		if xtrunc { {{/* // means canChange=true, changed=true already. */}}
 			for ; j < containerLenS; j++ {
 				v = append(v, {{ zerocmd .Elem }})
+				slh.ElemContainerState(j)
 				{{ if eq .Elem "interface{}" }}d.decode(&v[j]){{ else }}v[j] = {{ decmd .Elem }}{{ end }}
 			}
 		} else if !canChange {
-			for ; j < containerLenS; j++ { 
+			for ; j < containerLenS; j++ {
+				slh.ElemContainerState(j)
 				d.swallow()
 			}
 		}
 	} else {
-		j := 0
-		for ; !dd.CheckBreak(); j++ {
+		breakFound := dd.CheckBreak() {{/* check break first, so we can initialize v with a capacity of 4 if necessary */}}
+		if breakFound {
+			if canChange {
+				if v == nil {
+					v = []{{ .Elem }}{}
+				} else if len(v) != 0 {
+					v = v[:0]
+				}
+				changed = true
+			}
+			slh.End()
+			return
+		}
+		if cap(v) == 0 {
+			v = make([]{{ .Elem }}, 1, 4)
+			changed = true
+		}
+		j := 0	
+		for ; !breakFound; j++ {
 			if j >= len(v) { 
 				if canChange {
 					v = append(v, {{ zerocmd .Elem }})
@@ -362,16 +406,22 @@ func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v []{{ .Elem }}, checkNil b
 				} else {
 					d.arrayCannotExpand(len(v), j+1)
 				}
-			} 
+			}
+			slh.ElemContainerState(j)
 			if j < len(v) { {{/* // all checks done. cannot go past len. */}}
 				{{ if eq .Elem "interface{}" }}d.decode(&v[j])
 				{{ else }}v[j] = {{ decmd .Elem }}{{ end }}
 			} else {
 				d.swallow()
 			}
+			breakFound = dd.CheckBreak()
+		}
+		if canChange && j < len(v) {
+			v = v[:j]
+			changed = true
 		}
-		slh.End() 
 	}
+	slh.End() 
 	return v, changed 
 }
 
@@ -405,6 +455,7 @@ func (f fastpathT) {{ .MethodNamePfx "Dec" false }}X(vp *map[{{ .MapKey }}]{{ .E
 func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v map[{{ .MapKey }}]{{ .Elem }}, checkNil bool, canChange bool, 
 	d *Decoder) (_ map[{{ .MapKey }}]{{ .Elem }}, changed bool) {
 	dd := d.d
+	cr := d.cr
 	{{/* // if dd.isContainerType(valueTypeNil) {dd.TryDecodeAsNil() */}}
 	if checkNil && dd.TryDecodeAsNil() {
 		if v != nil {
@@ -424,11 +475,13 @@ func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v map[{{ .MapKey }}]{{ .Ele
 	var mv {{ .Elem }}
 	if containerLen > 0 {
 		for j := 0; j < containerLen; j++ {
+			if cr != nil { cr.sendContainerState(containerMapKey) }
 			{{ if eq .MapKey "interface{}" }}mk = nil 
 			d.decode(&mk)
 			if bv, bok := mk.([]byte); bok {
 				mk = d.string(bv) {{/* // maps cannot have []byte as key. switch to string. */}}
 			}{{ else }}mk = {{ decmd .MapKey }}{{ end }}
+			if cr != nil { cr.sendContainerState(containerMapValue) }
 			{{ if eq .Elem "interface{}" }}if mapGet { mv = v[mk] } else { mv = nil }
 			d.decode(&mv){{ else }}mv = {{ decmd .Elem }}{{ end }}
 			if v != nil {
@@ -437,19 +490,21 @@ func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v map[{{ .MapKey }}]{{ .Ele
 		}
 	} else if containerLen < 0 {
 		for j := 0; !dd.CheckBreak(); j++ {
+			if cr != nil { cr.sendContainerState(containerMapKey) }
 			{{ if eq .MapKey "interface{}" }}mk = nil 
 			d.decode(&mk)
 			if bv, bok := mk.([]byte); bok {
 				mk = d.string(bv) {{/* // maps cannot have []byte as key. switch to string. */}}
 			}{{ else }}mk = {{ decmd .MapKey }}{{ end }}
+			if cr != nil { cr.sendContainerState(containerMapValue) }
 			{{ if eq .Elem "interface{}" }}if mapGet { mv = v[mk] } else { mv = nil }
 			d.decode(&mv){{ else }}mv = {{ decmd .Elem }}{{ end }}
 			if v != nil {
 				v[mk] = mv
 			}
 		}
-		dd.ReadEnd()
 	}
+	if cr != nil { cr.sendContainerState(containerMapEnd) }
 	return v, changed
 }
 

codec/fast-path.not.go

@@ -0,0 +1,32 @@
+// +build notfastpath
+
+package codec
+
+import "reflect"
+
+// The generated fast-path code is very large, and adds a few seconds to the build time.
+// This causes test execution, execution of small tools which use codec, etc
+// to take a long time.
+//
+// To mitigate, we now support the notfastpath tag.
+// This tag disables fastpath during build, allowing for faster build, test execution,
+// short-program runs, etc.
+
+func fastpathDecodeTypeSwitch(iv interface{}, d *Decoder) bool      { return false }
+func fastpathEncodeTypeSwitch(iv interface{}, e *Encoder) bool      { return false }
+func fastpathEncodeTypeSwitchSlice(iv interface{}, e *Encoder) bool { return false }
+func fastpathEncodeTypeSwitchMap(iv interface{}, e *Encoder) bool   { return false }
+
+type fastpathT struct{}
+type fastpathE struct {
+	rtid  uintptr
+	rt    reflect.Type
+	encfn func(*encFnInfo, reflect.Value)
+	decfn func(*decFnInfo, reflect.Value)
+}
+type fastpathA [0]fastpathE
+
+func (x fastpathA) index(rtid uintptr) int { return -1 }
+
+var fastpathAV fastpathA
+var fastpathTV fastpathT

codec/gen-dec-array.go.tmpl

@@ -1,86 +1,101 @@
-{{var "v"}} := {{ if not isArray}}*{{ end }}{{ .Varname }}
+{{var "v"}} := {{if not isArray}}*{{end}}{{ .Varname }}
 {{var "h"}}, {{var "l"}} := z.DecSliceHelperStart() {{/* // helper, containerLenS */}}
-
-var {{var "rr"}}, {{var "rl"}} int {{/* // num2read, length of slice/array/chan */}}
-var {{var "c"}}, {{var "rt"}} bool {{/* // changed, truncated */}}
-_, _, _ = {{var "c"}}, {{var "rt"}}, {{var "rl"}}
-{{var "rr"}} = {{var "l"}}
-{{/* rl is NOT used. Only used for getting DecInferLen. len(r) used directly in code */}}
-
-{{ if not isArray }}if {{var "v"}} == nil {
-	if {{var "rl"}}, {{var "rt"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }}); {{var "rt"}} {
-		{{var "rr"}} = {{var "rl"}}
-	}
-	{{var "v"}} = make({{ .CTyp }}, {{var "rl"}})
-	{{var "c"}} = true 
-} 
-{{ end }}
-if {{var "l"}} == 0 { {{ if isSlice }}
-	if len({{var "v"}}) != 0 { 
-		{{var "v"}} = {{var "v"}}[:0] 
-		{{var "c"}} = true 
-	} {{ end }}
+var {{var "c"}} bool {{/* // changed */}}
+if {{var "l"}} == 0 {
+	{{if isSlice }}if {{var "v"}} == nil {
+		{{var "v"}} = []{{ .Typ }}{}
+		{{var "c"}} = true
+	} else if len({{var "v"}}) != 0 {
+		{{var "v"}} = {{var "v"}}[:0]
+		{{var "c"}} = true
+	} {{end}} {{if isChan }}if {{var "v"}} == nil {
+		{{var "v"}} = make({{ .CTyp }}, 0)
+		{{var "c"}} = true
+	} {{end}}
 } else if {{var "l"}} > 0 {
-	{{ if isChan }}
+	{{if isChan }}if {{var "v"}} == nil {
+		{{var "rl"}}, _ = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
+		{{var "v"}} = make({{ .CTyp }}, {{var "rl"}})
+		{{var "c"}} = true
+	}
 	for {{var "r"}} := 0; {{var "r"}} < {{var "l"}}; {{var "r"}}++ {
+		{{var "h"}}.ElemContainerState({{var "r"}})
 		var {{var "t"}} {{ .Typ }}
 		{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
-		{{var "v"}} <- {{var "t"}} 
-	{{ else }} 
+		{{var "v"}} <- {{var "t"}}
+	}
+	{{ else }}	var {{var "rr"}}, {{var "rl"}} int {{/* // num2read, length of slice/array/chan */}}
+	var {{var "rt"}} bool {{/* truncated */}}
 	if {{var "l"}} > cap({{var "v"}}) {
-		{{ if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "l"}})
-		{{ else }}{{var "rl"}}, {{var "rt"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
-		{{ if .Immutable }}
-		{{var "v2"}} := {{var "v"}}
-		{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
-		if len({{var "v"}}) > 0 {
-			copy({{var "v"}}, {{var "v2"}}[:cap({{var "v2"}})])
+		{{if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "l"}})
+		{{ else }}{{if not .Immutable }}
+		{{var "rg"}} := len({{var "v"}}) > 0
+		{{var "v2"}} := {{var "v"}} {{end}}
+		{{var "rl"}}, {{var "rt"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
+		if {{var "rt"}} {
+			if {{var "rl"}} <= cap({{var "v"}}) {
+				{{var "v"}} = {{var "v"}}[:{{var "rl"}}]
+			} else {
+				{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
+			}
+		} else {
+			{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
 		}
-		{{ else }}{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
-		{{ end }}{{var "c"}} = true 
-		{{ end }}
-		{{var "rr"}} = len({{var "v"}})
-	} else if {{var "l"}} != len({{var "v"}}) {
-		{{ if isSlice }}{{var "v"}} = {{var "v"}}[:{{var "l"}}]
-		{{var "c"}} = true {{ end }}
-	}
+		{{var "c"}} = true
+		{{var "rr"}} = len({{var "v"}}) {{if not .Immutable }}
+			if {{var "rg"}} { copy({{var "v"}}, {{var "v2"}}) } {{end}} {{end}}{{/* end not Immutable, isArray */}}
+	} {{if isSlice }} else if {{var "l"}} != len({{var "v"}}) {
+		{{var "v"}} = {{var "v"}}[:{{var "l"}}]
+		{{var "c"}} = true
+	} {{end}}	{{/* end isSlice:47 */}} 
 	{{var "j"}} := 0
 	for ; {{var "j"}} < {{var "rr"}} ; {{var "j"}}++ {
+		{{var "h"}}.ElemContainerState({{var "j"}})
 		{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
 	}
-	{{ if isArray }}for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
+	{{if isArray }}for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
+		{{var "h"}}.ElemContainerState({{var "j"}})
 		z.DecSwallow()
 	}
-	{{ else }}if {{var "rt"}} { {{/* means that it is mutable and slice */}}
+	{{ else }}if {{var "rt"}} {
 		for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
 			{{var "v"}} = append({{var "v"}}, {{ zero}})
+			{{var "h"}}.ElemContainerState({{var "j"}})
 			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
 		}
-	}
-	{{ end }}
-	{{ end }}{{/* closing 'if not chan' */}}
-} else {
-	for {{var "j"}} := 0; !r.CheckBreak(); {{var "j"}}++ {
-		if {{var "j"}} >= len({{var "v"}}) {
-			{{ if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "j"}}+1)
-			{{ else if isSlice}}{{var "v"}} = append({{var "v"}}, {{zero}})// var {{var "z"}} {{ .Typ }}
-			{{var "c"}} = true {{ end }}
-		}
-		{{ if isChan}}
+	} {{end}} {{/* end isArray:56 */}}
+	{{end}} {{/* end isChan:16 */}}
+} else { {{/* len < 0 */}}
+	{{var "j"}} := 0
+	for ; !r.CheckBreak(); {{var "j"}}++ {
+		{{if isChan }}
+		{{var "h"}}.ElemContainerState({{var "j"}})
 		var {{var "t"}} {{ .Typ }}
 		{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
 		{{var "v"}} <- {{var "t"}} 
 		{{ else }}
+		if {{var "j"}} >= len({{var "v"}}) {
+			{{if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "j"}}+1)
+			{{ else }}{{var "v"}} = append({{var "v"}}, {{zero}})// var {{var "z"}} {{ .Typ }}
+			{{var "c"}} = true {{end}}
+		}
+		{{var "h"}}.ElemContainerState({{var "j"}})
 		if {{var "j"}} < len({{var "v"}}) {
 			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
 		} else {
 			z.DecSwallow()
 		}
-		{{ end }}
+		{{end}}
 	}
-	{{var "h"}}.End()
+	{{if isSlice }}if {{var "j"}} < len({{var "v"}}) {
+		{{var "v"}} = {{var "v"}}[:{{var "j"}}]
+		{{var "c"}} = true
+	} else if {{var "j"}} == 0 && {{var "v"}} == nil {
+		{{var "v"}} = []{{ .Typ }}{}
+		{{var "c"}} = true
+	}{{end}}
 }
-{{ if not isArray }}if {{var "c"}} { 
+{{var "h"}}.End()
+{{if not isArray }}if {{var "c"}} { 
 	*{{ .Varname }} = {{var "v"}}
-}{{ end }}
-
+}{{end}}

codec/gen-dec-map.go.tmpl

@@ -8,7 +8,7 @@ if {{var "v"}} == nil {
 }
 var {{var "mk"}} {{ .KTyp }}
 var {{var "mv"}} {{ .Typ }}
-var {{var "mg"}} bool
+var {{var "mg"}} {{if decElemKindPtr}}, {{var "ms"}}, {{var "mok"}}{{end}} bool
 if {{var "bh"}}.MapValueReset {
 	{{if decElemKindPtr}}{{var "mg"}} = true
 	{{else if decElemKindIntf}}if !{{var "bh"}}.InterfaceReset { {{var "mg"}} = true }
@@ -16,31 +16,43 @@ if {{var "bh"}}.MapValueReset {
 	{{end}} }
 if {{var "l"}} > 0  {
 for {{var "j"}} := 0; {{var "j"}} < {{var "l"}}; {{var "j"}}++ {
+	z.DecSendContainerState(codecSelfer_containerMapKey{{ .Sfx }})
 	{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
 {{ if eq .KTyp "interface{}" }}{{/* // special case if a byte array. */}}if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
 		{{var "mk"}} = string({{var "bv"}})
-	}{{ end }}
+	}{{ end }}{{if decElemKindPtr}}
+	{{var "ms"}} = true{{end}}
 	if {{var "mg"}} {
-		{{var "mv"}} = {{var "v"}}[{{var "mk"}}]
+		{{if decElemKindPtr}}{{var "mv"}}, {{var "mok"}} = {{var "v"}}[{{var "mk"}}] 
+		if {{var "mok"}} {
+			{{var "ms"}} = false
+		} {{else}}{{var "mv"}} = {{var "v"}}[{{var "mk"}}] {{end}}
 	} {{if not decElemKindImmutable}}else { {{var "mv"}} = {{decElemZero}} }{{end}}
+	z.DecSendContainerState(codecSelfer_containerMapValue{{ .Sfx }})
 	{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
-	if {{var "v"}} != nil {
+	if {{if decElemKindPtr}} {{var "ms"}} && {{end}} {{var "v"}} != nil {
 		{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
 	}
 }
 } else if {{var "l"}} < 0  {
 for {{var "j"}} := 0; !r.CheckBreak(); {{var "j"}}++ {
+	z.DecSendContainerState(codecSelfer_containerMapKey{{ .Sfx }})
 	{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
 {{ if eq .KTyp "interface{}" }}{{/* // special case if a byte array. */}}if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
 		{{var "mk"}} = string({{var "bv"}})
-	}{{ end }}
+	}{{ end }}{{if decElemKindPtr}}
+	{{var "ms"}} = true {{ end }}
 	if {{var "mg"}} {
-		{{var "mv"}} = {{var "v"}}[{{var "mk"}}]
+		{{if decElemKindPtr}}{{var "mv"}}, {{var "mok"}} = {{var "v"}}[{{var "mk"}}] 
+		if {{var "mok"}} {
+			{{var "ms"}} = false
+		} {{else}}{{var "mv"}} = {{var "v"}}[{{var "mk"}}] {{end}}
 	} {{if not decElemKindImmutable}}else { {{var "mv"}} = {{decElemZero}} }{{end}}
+	z.DecSendContainerState(codecSelfer_containerMapValue{{ .Sfx }})
 	{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
-	if {{var "v"}} != nil {
+	if {{if decElemKindPtr}} {{var "ms"}} && {{end}} {{var "v"}} != nil {
 		{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
 	}
 }
-r.ReadEnd()
 } // else len==0: TODO: Should we clear map entries?
+z.DecSendContainerState(codecSelfer_containerMapEnd{{ .Sfx }})

codec/gen-helper.generated.go

@@ -115,6 +115,15 @@ func (f genHelperEncoder) EncExt(v interface{}) (r bool) {
 	return false
 }
 
+// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
+func (f genHelperEncoder) EncSendContainerState(c containerState) {
+	if f.e.cr != nil {
+		f.e.cr.sendContainerState(c)
+	}
+}
+
+// ---------------- DECODER FOLLOWS -----------------
+
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecBasicHandle() *BasicHandle {
 	return f.d.h
@@ -218,3 +227,10 @@ func (f genHelperDecoder) DecExt(v interface{}) (r bool) {
 func (f genHelperDecoder) DecInferLen(clen, maxlen, unit int) (rvlen int, truncated bool) {
 	return decInferLen(clen, maxlen, unit)
 }
+
+// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
+func (f genHelperDecoder) DecSendContainerState(c containerState) {
+	if f.d.cr != nil {
+		f.d.cr.sendContainerState(c)
+	}
+}

codec/gen-helper.go.tmpl

@@ -106,6 +106,14 @@ func (f genHelperEncoder) EncExt(v interface{}) (r bool) {
 	}
 	return false 
 }
+// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
+func (f genHelperEncoder) EncSendContainerState(c containerState) {
+	if f.e.cr != nil {
+		f.e.cr.sendContainerState(c)
+	}
+}
+
+// ---------------- DECODER FOLLOWS -----------------
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecBasicHandle() *BasicHandle {
@@ -195,6 +203,12 @@ func (f genHelperDecoder) DecExt(v interface{}) (r bool) {
 func (f genHelperDecoder) DecInferLen(clen, maxlen, unit int) (rvlen int, truncated bool) {
 	return decInferLen(clen, maxlen, unit)
 }
+// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
+func (f genHelperDecoder) DecSendContainerState(c containerState) {
+	if f.d.cr != nil {
+		f.d.cr.sendContainerState(c)
+	}
+}
 
 {{/*
 

codec/gen.generated.go

@@ -16,7 +16,7 @@ if {{var "v"}} == nil {
 }
 var {{var "mk"}} {{ .KTyp }}
 var {{var "mv"}} {{ .Typ }}
-var {{var "mg"}} bool
+var {{var "mg"}} {{if decElemKindPtr}}, {{var "ms"}}, {{var "mok"}}{{end}} bool
 if {{var "bh"}}.MapValueReset {
 	{{if decElemKindPtr}}{{var "mg"}} = true
 	{{else if decElemKindIntf}}if !{{var "bh"}}.InterfaceReset { {{var "mg"}} = true }
@@ -24,122 +24,149 @@ if {{var "bh"}}.MapValueReset {
 	{{end}} }
 if {{var "l"}} > 0  {
 for {{var "j"}} := 0; {{var "j"}} < {{var "l"}}; {{var "j"}}++ {
+	z.DecSendContainerState(codecSelfer_containerMapKey{{ .Sfx }})
 	{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
 {{ if eq .KTyp "interface{}" }}{{/* // special case if a byte array. */}}if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
 		{{var "mk"}} = string({{var "bv"}})
-	}{{ end }}
+	}{{ end }}{{if decElemKindPtr}}
+	{{var "ms"}} = true{{end}}
 	if {{var "mg"}} {
-		{{var "mv"}} = {{var "v"}}[{{var "mk"}}]
+		{{if decElemKindPtr}}{{var "mv"}}, {{var "mok"}} = {{var "v"}}[{{var "mk"}}] 
+		if {{var "mok"}} {
+			{{var "ms"}} = false
+		} {{else}}{{var "mv"}} = {{var "v"}}[{{var "mk"}}] {{end}}
 	} {{if not decElemKindImmutable}}else { {{var "mv"}} = {{decElemZero}} }{{end}}
+	z.DecSendContainerState(codecSelfer_containerMapValue{{ .Sfx }})
 	{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
-	if {{var "v"}} != nil {
+	if {{if decElemKindPtr}} {{var "ms"}} && {{end}} {{var "v"}} != nil {
 		{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
 	}
 }
 } else if {{var "l"}} < 0  {
 for {{var "j"}} := 0; !r.CheckBreak(); {{var "j"}}++ {
+	z.DecSendContainerState(codecSelfer_containerMapKey{{ .Sfx }})
 	{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
 {{ if eq .KTyp "interface{}" }}{{/* // special case if a byte array. */}}if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
 		{{var "mk"}} = string({{var "bv"}})
-	}{{ end }}
+	}{{ end }}{{if decElemKindPtr}}
+	{{var "ms"}} = true {{ end }}
 	if {{var "mg"}} {
-		{{var "mv"}} = {{var "v"}}[{{var "mk"}}]
+		{{if decElemKindPtr}}{{var "mv"}}, {{var "mok"}} = {{var "v"}}[{{var "mk"}}] 
+		if {{var "mok"}} {
+			{{var "ms"}} = false
+		} {{else}}{{var "mv"}} = {{var "v"}}[{{var "mk"}}] {{end}}
 	} {{if not decElemKindImmutable}}else { {{var "mv"}} = {{decElemZero}} }{{end}}
+	z.DecSendContainerState(codecSelfer_containerMapValue{{ .Sfx }})
 	{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
-	if {{var "v"}} != nil {
+	if {{if decElemKindPtr}} {{var "ms"}} && {{end}} {{var "v"}} != nil {
 		{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
 	}
 }
-r.ReadEnd()
 } // else len==0: TODO: Should we clear map entries?
+z.DecSendContainerState(codecSelfer_containerMapEnd{{ .Sfx }})
 `
 
 const genDecListTmpl = `
-{{var "v"}} := {{ if not isArray}}*{{ end }}{{ .Varname }}
+{{var "v"}} := {{if not isArray}}*{{end}}{{ .Varname }}
 {{var "h"}}, {{var "l"}} := z.DecSliceHelperStart() {{/* // helper, containerLenS */}}
-
-var {{var "rr"}}, {{var "rl"}} int {{/* // num2read, length of slice/array/chan */}}
-var {{var "c"}}, {{var "rt"}} bool {{/* // changed, truncated */}}
-_, _, _ = {{var "c"}}, {{var "rt"}}, {{var "rl"}}
-{{var "rr"}} = {{var "l"}}
-{{/* rl is NOT used. Only used for getting DecInferLen. len(r) used directly in code */}}
-
-{{ if not isArray }}if {{var "v"}} == nil {
-	if {{var "rl"}}, {{var "rt"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }}); {{var "rt"}} {
-		{{var "rr"}} = {{var "rl"}}
-	}
-	{{var "v"}} = make({{ .CTyp }}, {{var "rl"}})
-	{{var "c"}} = true 
-} 
-{{ end }}
-if {{var "l"}} == 0 { {{ if isSlice }}
-	if len({{var "v"}}) != 0 { 
-		{{var "v"}} = {{var "v"}}[:0] 
-		{{var "c"}} = true 
-	} {{ end }}
+var {{var "c"}} bool {{/* // changed */}}
+if {{var "l"}} == 0 {
+	{{if isSlice }}if {{var "v"}} == nil {
+		{{var "v"}} = []{{ .Typ }}{}
+		{{var "c"}} = true
+	} else if len({{var "v"}}) != 0 {
+		{{var "v"}} = {{var "v"}}[:0]
+		{{var "c"}} = true
+	} {{end}} {{if isChan }}if {{var "v"}} == nil {
+		{{var "v"}} = make({{ .CTyp }}, 0)
+		{{var "c"}} = true
+	} {{end}}
 } else if {{var "l"}} > 0 {
-	{{ if isChan }}
+	{{if isChan }}if {{var "v"}} == nil {
+		{{var "rl"}}, _ = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
+		{{var "v"}} = make({{ .CTyp }}, {{var "rl"}})
+		{{var "c"}} = true
+	}
 	for {{var "r"}} := 0; {{var "r"}} < {{var "l"}}; {{var "r"}}++ {
+		{{var "h"}}.ElemContainerState({{var "r"}})
 		var {{var "t"}} {{ .Typ }}
 		{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
-		{{var "v"}} <- {{var "t"}} 
-	{{ else }} 
+		{{var "v"}} <- {{var "t"}}
+	}
+	{{ else }}	var {{var "rr"}}, {{var "rl"}} int {{/* // num2read, length of slice/array/chan */}}
+	var {{var "rt"}} bool {{/* truncated */}}
 	if {{var "l"}} > cap({{var "v"}}) {
-		{{ if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "l"}})
-		{{ else }}{{var "rl"}}, {{var "rt"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
-		{{ if .Immutable }}
-		{{var "v2"}} := {{var "v"}}
-		{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
-		if len({{var "v"}}) > 0 {
-			copy({{var "v"}}, {{var "v2"}}[:cap({{var "v2"}})])
+		{{if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "l"}})
+		{{ else }}{{if not .Immutable }}
+		{{var "rg"}} := len({{var "v"}}) > 0
+		{{var "v2"}} := {{var "v"}} {{end}}
+		{{var "rl"}}, {{var "rt"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
+		if {{var "rt"}} {
+			if {{var "rl"}} <= cap({{var "v"}}) {
+				{{var "v"}} = {{var "v"}}[:{{var "rl"}}]
+			} else {
+				{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
+			}
+		} else {
+			{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
 		}
-		{{ else }}{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
-		{{ end }}{{var "c"}} = true 
-		{{ end }}
-		{{var "rr"}} = len({{var "v"}})
-	} else if {{var "l"}} != len({{var "v"}}) {
-		{{ if isSlice }}{{var "v"}} = {{var "v"}}[:{{var "l"}}]
-		{{var "c"}} = true {{ end }}
-	}
+		{{var "c"}} = true
+		{{var "rr"}} = len({{var "v"}}) {{if not .Immutable }}
+			if {{var "rg"}} { copy({{var "v"}}, {{var "v2"}}) } {{end}} {{end}}{{/* end not Immutable, isArray */}}
+	} {{if isSlice }} else if {{var "l"}} != len({{var "v"}}) {
+		{{var "v"}} = {{var "v"}}[:{{var "l"}}]
+		{{var "c"}} = true
+	} {{end}}	{{/* end isSlice:47 */}} 
 	{{var "j"}} := 0
 	for ; {{var "j"}} < {{var "rr"}} ; {{var "j"}}++ {
+		{{var "h"}}.ElemContainerState({{var "j"}})
 		{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
 	}
-	{{ if isArray }}for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
+	{{if isArray }}for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
+		{{var "h"}}.ElemContainerState({{var "j"}})
 		z.DecSwallow()
 	}
-	{{ else }}if {{var "rt"}} { {{/* means that it is mutable and slice */}}
+	{{ else }}if {{var "rt"}} {
 		for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
 			{{var "v"}} = append({{var "v"}}, {{ zero}})
+			{{var "h"}}.ElemContainerState({{var "j"}})
 			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
 		}
-	}
-	{{ end }}
-	{{ end }}{{/* closing 'if not chan' */}}
-} else {
-	for {{var "j"}} := 0; !r.CheckBreak(); {{var "j"}}++ {
-		if {{var "j"}} >= len({{var "v"}}) {
-			{{ if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "j"}}+1)
-			{{ else if isSlice}}{{var "v"}} = append({{var "v"}}, {{zero}})// var {{var "z"}} {{ .Typ }}
-			{{var "c"}} = true {{ end }}
-		}
-		{{ if isChan}}
+	} {{end}} {{/* end isArray:56 */}}
+	{{end}} {{/* end isChan:16 */}}
+} else { {{/* len < 0 */}}
+	{{var "j"}} := 0
+	for ; !r.CheckBreak(); {{var "j"}}++ {
+		{{if isChan }}
+		{{var "h"}}.ElemContainerState({{var "j"}})
 		var {{var "t"}} {{ .Typ }}
 		{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
 		{{var "v"}} <- {{var "t"}} 
 		{{ else }}
+		if {{var "j"}} >= len({{var "v"}}) {
+			{{if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "j"}}+1)
+			{{ else }}{{var "v"}} = append({{var "v"}}, {{zero}})// var {{var "z"}} {{ .Typ }}
+			{{var "c"}} = true {{end}}
+		}
+		{{var "h"}}.ElemContainerState({{var "j"}})
 		if {{var "j"}} < len({{var "v"}}) {
 			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
 		} else {
 			z.DecSwallow()
 		}
-		{{ end }}
+		{{end}}
 	}
-	{{var "h"}}.End()
+	{{if isSlice }}if {{var "j"}} < len({{var "v"}}) {
+		{{var "v"}} = {{var "v"}}[:{{var "j"}}]
+		{{var "c"}} = true
+	} else if {{var "j"}} == 0 && {{var "v"}} == nil {
+		{{var "v"}} = []{{ .Typ }}{}
+		{{var "c"}} = true
+	}{{end}}
 }
-{{ if not isArray }}if {{var "c"}} { 
+{{var "h"}}.End()
+{{if not isArray }}if {{var "c"}} { 
 	*{{ .Varname }} = {{var "v"}}
-}{{ end }}
-
+}{{end}}
 `
 

codec/gen.go

@@ -91,7 +91,8 @@ import (
 // v3: Changes for Kubernetes:
 //     changes in signature of some unpublished helper methods and codecgen cmdline arguments.
 // v4: Removed separator support from (en|de)cDriver, and refactored codec(gen)
-const GenVersion = 4
+// v5: changes to support faster json decoding. Let encoder/decoder maintain state of collections.
+const GenVersion = 5
 
 const (
 	genCodecPkg        = "codec1978"
@@ -110,6 +111,14 @@ const (
 	genUseOneFunctionForDecStructMap = true
 )
 
+type genStructMapStyle uint8
+
+const (
+	genStructMapStyleConsolidated genStructMapStyle = iota
+	genStructMapStyleLenPrefix
+	genStructMapStyleCheckBreak
+)
+
 var (
 	genAllTypesSamePkgErr  = errors.New("All types must be in the same package")
 	genExpectArrayOrMapErr = errors.New("unexpected type. Expecting array/map/slice")
@@ -230,10 +239,18 @@ func Gen(w io.Writer, buildTags, pkgName, uid string, useUnsafe bool, ti *TypeIn
 	x.line("")
 
 	x.line("const (")
+	x.linef("// ----- content types ----")
 	x.linef("codecSelferC_UTF8%s = %v", x.xs, int64(c_UTF8))
 	x.linef("codecSelferC_RAW%s = %v", x.xs, int64(c_RAW))
+	x.linef("// ----- value types used ----")
 	x.linef("codecSelferValueTypeArray%s = %v", x.xs, int64(valueTypeArray))
 	x.linef("codecSelferValueTypeMap%s = %v", x.xs, int64(valueTypeMap))
+	x.linef("// ----- containerStateValues ----")
+	x.linef("codecSelfer_containerMapKey%s = %v", x.xs, int64(containerMapKey))
+	x.linef("codecSelfer_containerMapValue%s = %v", x.xs, int64(containerMapValue))
+	x.linef("codecSelfer_containerMapEnd%s = %v", x.xs, int64(containerMapEnd))
+	x.linef("codecSelfer_containerArrayElem%s = %v", x.xs, int64(containerArrayElem))
+	x.linef("codecSelfer_containerArrayEnd%s = %v", x.xs, int64(containerArrayEnd))
 	x.line(")")
 	x.line("var (")
 	x.line("codecSelferBitsize" + x.xs + " = uint8(reflect.TypeOf(uint(0)).Bits())")
@@ -255,8 +272,6 @@ func Gen(w io.Writer, buildTags, pkgName, uid string, useUnsafe bool, ti *TypeIn
 	x.line(`err := fmt.Errorf("codecgen version mismatch: current: %v, need %v. Re-generate file: %v", `)
 	x.linef(`%v, %sGenVersion, file)`, GenVersion, x.cpfx)
 	x.line("panic(err)")
-	// x.linef(`panic(fmt.Errorf("Re-run codecgen due to version mismatch: `+
-	// 	`current: %%v, need %%v, file: %%v", %v, %sGenVersion, file))`, GenVersion, x.cpfx)
 	x.linef("}")
 	x.line("if false { // reference the types, but skip this branch at build/run time")
 	var n int
@@ -515,21 +530,21 @@ func (x *genRunner) selfer(encode bool) {
 		x.out(fnSigPfx)
 		x.line(") codecDecodeSelfFromMap(l int, d *" + x.cpfx + "Decoder) {")
 		x.genRequiredMethodVars(false)
-		x.decStructMap(genTopLevelVarName, "l", reflect.ValueOf(t0).Pointer(), t0, 0)
+		x.decStructMap(genTopLevelVarName, "l", reflect.ValueOf(t0).Pointer(), t0, genStructMapStyleConsolidated)
 		x.line("}")
 		x.line("")
 	} else {
 		x.out(fnSigPfx)
 		x.line(") codecDecodeSelfFromMapLenPrefix(l int, d *" + x.cpfx + "Decoder) {")
 		x.genRequiredMethodVars(false)
-		x.decStructMap(genTopLevelVarName, "l", reflect.ValueOf(t0).Pointer(), t0, 1)
+		x.decStructMap(genTopLevelVarName, "l", reflect.ValueOf(t0).Pointer(), t0, genStructMapStyleLenPrefix)
 		x.line("}")
 		x.line("")
 
 		x.out(fnSigPfx)
 		x.line(") codecDecodeSelfFromMapCheckBreak(l int, d *" + x.cpfx + "Decoder) {")
 		x.genRequiredMethodVars(false)
-		x.decStructMap(genTopLevelVarName, "l", reflect.ValueOf(t0).Pointer(), t0, 2)
+		x.decStructMap(genTopLevelVarName, "l", reflect.ValueOf(t0).Pointer(), t0, genStructMapStyleCheckBreak)
 		x.line("}")
 		x.line("")
 	}
@@ -548,10 +563,8 @@ func (x *genRunner) selfer(encode bool) {
 func (x *genRunner) xtraSM(varname string, encode bool, t reflect.Type) {
 	if encode {
 		x.linef("h.enc%s((%s%s)(%s), e)", x.genMethodNameT(t), x.arr2str(t, "*"), x.genTypeName(t), varname)
-		// x.line("h.enc" + x.genMethodNameT(t) + "(" + x.genTypeName(t) + "(" + varname + "), e)")
 	} else {
 		x.linef("h.dec%s((*%s)(%s), d)", x.genMethodNameT(t), x.genTypeName(t), varname)
-		// x.line("h.dec" + x.genMethodNameT(t) + "((*" + x.genTypeName(t) + ")(" + varname + "), d)")
 	}
 	if _, ok := x.tm[t]; !ok {
 		x.tm[t] = struct{}{}
@@ -815,12 +828,14 @@ func (x *genRunner) encStruct(varname string, rtid uintptr, t reflect.Type) {
 		}
 		x.linef("%s[%v] = %s", numfieldsvar, j, omitline)
 	}
+	x.linef("var %snn%s int", genTempVarPfx, i)
 	x.linef("if %s || %s {", ti2arrayvar, struct2arrvar) // if ti.toArray {
 	x.line("r.EncodeArrayStart(" + strconv.FormatInt(int64(len(tisfi)), 10) + ")")
 	x.linef("} else {") // if not ti.toArray
-	x.linef("var %snn%s int = %v", genTempVarPfx, i, nn)
+	x.linef("%snn%s = %v", genTempVarPfx, i, nn)
 	x.linef("for _, b := range %s { if b { %snn%s++ } }", numfieldsvar, genTempVarPfx, i)
 	x.linef("r.EncodeMapStart(%snn%s)", genTempVarPfx, i)
+	x.linef("%snn%s = %v", genTempVarPfx, i, 0)
 	// x.line("r.EncodeMapStart(" + strconv.FormatInt(int64(len(tisfi)), 10) + ")")
 	x.line("}") // close if not StructToArray
 
@@ -864,11 +879,9 @@ func (x *genRunner) encStruct(varname string, rtid uintptr, t reflect.Type) {
 		if labelUsed {
 			x.line("if " + isNilVarName + " { r.EncodeNil() } else { ")
 		}
+		x.linef("z.EncSendContainerState(codecSelfer_containerArrayElem%s)", x.xs)
 		if si.omitEmpty {
 			x.linef("if %s[%v] {", numfieldsvar, j)
-			// omitEmptyVarNameX := genTempVarPfx + "ov" + i
-			// x.line("var " + omitEmptyVarNameX + " " + x.genTypeName(t2.Type))
-			// x.encVar(omitEmptyVarNameX, t2.Type)
 		}
 		x.encVar(varname+"."+t2.Name, t2.Type)
 		if si.omitEmpty {
@@ -879,21 +892,15 @@ func (x *genRunner) encStruct(varname string, rtid uintptr, t reflect.Type) {
 		if labelUsed {
 			x.line("}")
 		}
+
 		x.linef("} else {") // if not ti.toArray
-		// omitEmptyVar := genTempVarPfx + "x" + i + t2.Name
-		// x.line("const " + omitEmptyVar + " bool = " + strconv.FormatBool(si.omitEmpty))
-		// doOmitEmpty := si.omitEmpty && t2.Type.Kind() != reflect.Struct
+
 		if si.omitEmpty {
 			x.linef("if %s[%v] {", numfieldsvar, j)
-			// x.linef(`println("Encoding field: %v")`, j)
-			// x.out("if ")
-			// if labelUsed {
-			// 	x.out("!" + isNilVarName + " && ")
-			// }
-			// x.line(varname + "." + t2.Name + " != " + genZeroValueR(t2.Type, x.tc) + " {")
 		}
-		// x.line("r.EncodeString(codecSelferC_UTF8" + x.xs + ", string(\"" + t2.Name + "\"))")
+		x.linef("z.EncSendContainerState(codecSelfer_containerMapKey%s)", x.xs)
 		x.line("r.EncodeString(codecSelferC_UTF8" + x.xs + ", string(\"" + si.encName + "\"))")
+		x.linef("z.EncSendContainerState(codecSelfer_containerMapValue%s)", x.xs)
 		if labelUsed {
 			x.line("if " + isNilVarName + " { r.EncodeNil() } else { ")
 			x.encVar(varname+"."+t2.Name, t2.Type)
@@ -906,9 +913,12 @@ func (x *genRunner) encStruct(varname string, rtid uintptr, t reflect.Type) {
 		}
 		x.linef("} ") // end if/else ti.toArray
 	}
-	x.line("if " + sepVarname + " {")
-	x.line("r.EncodeEnd()")
+	x.linef("if %s || %s {", ti2arrayvar, struct2arrvar) // if ti.toArray {
+	x.linef("z.EncSendContainerState(codecSelfer_containerArrayEnd%s)", x.xs)
+	x.line("} else {")
+	x.linef("z.EncSendContainerState(codecSelfer_containerMapEnd%s)", x.xs)
 	x.line("}")
+
 }
 
 func (x *genRunner) encListFallback(varname string, t reflect.Type) {
@@ -917,14 +927,16 @@ func (x *genRunner) encListFallback(varname string, t reflect.Type) {
 	x.line("r.EncodeArrayStart(len(" + varname + "))")
 	if t.Kind() == reflect.Chan {
 		x.linef("for %si%s, %si2%s := 0, len(%s); %si%s < %si2%s; %si%s++ {", g, i, g, i, varname, g, i, g, i, g, i)
+		x.linef("z.EncSendContainerState(codecSelfer_containerArrayElem%s)", x.xs)
 		x.linef("%sv%s := <-%s", g, i, varname)
 	} else {
 		// x.linef("for %si%s, %sv%s := range %s {", genTempVarPfx, i, genTempVarPfx, i, varname)
 		x.linef("for _, %sv%s := range %s {", genTempVarPfx, i, varname)
+		x.linef("z.EncSendContainerState(codecSelfer_containerArrayElem%s)", x.xs)
 	}
 	x.encVar(genTempVarPfx+"v"+i, t.Elem())
 	x.line("}")
-	x.line("r.EncodeEnd()")
+	x.linef("z.EncSendContainerState(codecSelfer_containerArrayEnd%s)", x.xs)
 }
 
 func (x *genRunner) encMapFallback(varname string, t reflect.Type) {
@@ -933,10 +945,12 @@ func (x *genRunner) encMapFallback(varname string, t reflect.Type) {
 	x.line("r.EncodeMapStart(len(" + varname + "))")
 	x.linef("for %sk%s, %sv%s := range %s {", genTempVarPfx, i, genTempVarPfx, i, varname)
 	// x.line("for " + genTempVarPfx + "k" + i + ", " + genTempVarPfx + "v" + i + " := range " + varname + " {")
+	x.linef("z.EncSendContainerState(codecSelfer_containerMapKey%s)", x.xs)
 	x.encVar(genTempVarPfx+"k"+i, t.Key())
+	x.linef("z.EncSendContainerState(codecSelfer_containerMapValue%s)", x.xs)
 	x.encVar(genTempVarPfx+"v"+i, t.Elem())
 	x.line("}")
-	x.line("r.EncodeEnd()")
+	x.linef("z.EncSendContainerState(codecSelfer_containerMapEnd%s)", x.xs)
 }
 
 func (x *genRunner) decVar(varname string, t reflect.Type, canBeNil bool) {
@@ -954,8 +968,6 @@ func (x *genRunner) decVar(varname string, t reflect.Type, canBeNil bool) {
 		x.line("if r.TryDecodeAsNil() {")
 		if t.Kind() == reflect.Ptr {
 			x.line("if " + varname + " != nil { ")
-			// x.line("var " + genTempVarPfx + i + " " + x.genTypeName(t.Elem()))
-			// x.line("*" + varname + " = " + genTempVarPfx + i)
 
 			// if varname is a field of a struct (has a dot in it),
 			// then just set it to nil
@@ -964,12 +976,8 @@ func (x *genRunner) decVar(varname string, t reflect.Type, canBeNil bool) {
 			} else {
 				x.line("*" + varname + " = " + x.genZeroValueR(t.Elem()))
 			}
-			// x.line("*" + varname + " = nil")
 			x.line("}")
-
 		} else {
-			// x.line("var " + genTempVarPfx + i + " " + x.genTypeName(t))
-			// x.line(varname + " = " + genTempVarPfx + i)
 			x.line(varname + " = " + x.genZeroValueR(t))
 		}
 		x.line("} else {")
@@ -1149,8 +1157,6 @@ func (x *genRunner) dec(varname string, t reflect.Type) {
 		} else if fastpathAV.index(rtid) != -1 {
 			g := x.newGenV(t)
 			x.line("z.F." + g.MethodNamePfx("Dec", false) + "X(" + varname + ", false, d)")
-			// x.line("z." + g.MethodNamePfx("Dec", false) + "(" + varname + ")")
-			// x.line(g.FastpathName(false) + "(" + varname + ", d)")
 		} else {
 			x.xtraSM(varname, false, t)
 			// x.decListFallback(varname, rtid, false, t)
@@ -1163,8 +1169,6 @@ func (x *genRunner) dec(varname string, t reflect.Type) {
 		if fastpathAV.index(rtid) != -1 {
 			g := x.newGenV(t)
 			x.line("z.F." + g.MethodNamePfx("Dec", false) + "X(" + varname + ", false, d)")
-			// x.line("z." + g.MethodNamePfx("Dec", false) + "(" + varname + ")")
-			// x.line(g.FastpathName(false) + "(" + varname + ", d)")
 		} else {
 			x.xtraSM(varname, false, t)
 			// x.decMapFallback(varname, rtid, t)
@@ -1294,6 +1298,7 @@ func (x *genRunner) decListFallback(varname string, rtid uintptr, t reflect.Type
 func (x *genRunner) decMapFallback(varname string, rtid uintptr, t reflect.Type) {
 	type tstruc struct {
 		TempVar string
+		Sfx     string
 		Rand    string
 		Varname string
 		KTyp    string
@@ -1303,7 +1308,7 @@ func (x *genRunner) decMapFallback(varname string, rtid uintptr, t reflect.Type)
 	telem := t.Elem()
 	tkey := t.Key()
 	ts := tstruc{
-		genTempVarPfx, x.varsfx(), varname, x.genTypeName(tkey),
+		genTempVarPfx, x.xs, x.varsfx(), varname, x.genTypeName(tkey),
 		x.genTypeName(telem), int(telem.Size() + tkey.Size()),
 	}
 
@@ -1359,6 +1364,7 @@ func (x *genRunner) decStructMapSwitch(kName string, varname string, rtid uintpt
 		if si.i != -1 {
 			t2 = t.Field(int(si.i))
 		} else {
+			//we must accomodate anonymous fields, where the embedded field is a nil pointer in the value.
 			// t2 = t.FieldByIndex(si.is)
 			t2typ := t
 			varname3 := varname
@@ -1370,8 +1376,7 @@ func (x *genRunner) decStructMapSwitch(kName string, varname string, rtid uintpt
 				t2typ = t2.Type
 				varname3 = varname3 + "." + t2.Name
 				if t2typ.Kind() == reflect.Ptr {
-					x.line("if " + varname3 + " == nil {" +
-						varname3 + " = new(" + x.genTypeName(t2typ.Elem()) + ") }")
+					x.linef("if %s == nil { %s = new(%s) }", varname3, varname3, x.genTypeName(t2typ.Elem()))
 				}
 			}
 		}
@@ -1380,11 +1385,10 @@ func (x *genRunner) decStructMapSwitch(kName string, varname string, rtid uintpt
 	x.line("default:")
 	// pass the slice here, so that the string will not escape, and maybe save allocation
 	x.line("z.DecStructFieldNotFound(-1, " + kName + ")")
-	// x.line("z.DecStructFieldNotFoundB(" + kName + "Slc)")
 	x.line("} // end switch " + kName)
 }
 
-func (x *genRunner) decStructMap(varname, lenvarname string, rtid uintptr, t reflect.Type, style uint8) {
+func (x *genRunner) decStructMap(varname, lenvarname string, rtid uintptr, t reflect.Type, style genStructMapStyle) {
 	tpfx := genTempVarPfx
 	i := x.varsfx()
 	kName := tpfx + "s" + i
@@ -1406,14 +1410,11 @@ func (x *genRunner) decStructMap(varname, lenvarname string, rtid uintptr, t ref
 
 	x.line("var " + kName + "Slc = z.DecScratchBuffer() // default slice to decode into")
 
-	// x.line("var " + kName + " string // default string to decode into")
-	// x.line("_ = " + kName)
 	x.line("_ = " + kName + "Slc")
-	// x.linef("var %sb%s bool", tpfx, i)                        // break
 	switch style {
-	case 1:
+	case genStructMapStyleLenPrefix:
 		x.linef("for %sj%s := 0; %sj%s < %s; %sj%s++ {", tpfx, i, tpfx, i, lenvarname, tpfx, i)
-	case 2:
+	case genStructMapStyleCheckBreak:
 		x.linef("for %sj%s := 0; !r.CheckBreak(); %sj%s++ {", tpfx, i, tpfx, i)
 	default: // 0, otherwise.
 		x.linef("var %shl%s bool = %s >= 0", tpfx, i, lenvarname) // has length
@@ -1421,11 +1422,9 @@ func (x *genRunner) decStructMap(varname, lenvarname string, rtid uintptr, t ref
 		x.linef("if %shl%s { if %sj%s >= %s { break }", tpfx, i, tpfx, i, lenvarname)
 		x.line("} else { if r.CheckBreak() { break }; }")
 	}
-	// x.line(kName + " = z.ReadStringAsBytes(" + kName + ")")
-	// x.line(kName + " = z.ReadString()")
+	x.linef("z.DecSendContainerState(codecSelfer_containerMapKey%s)", x.xs)
 	x.line(kName + "Slc = r.DecodeBytes(" + kName + "Slc, true, true)")
 	// let string be scoped to this loop alone, so it doesn't escape.
-	// x.line(kName + " := " + x.cpfx + "GenBytesToStringRO(" + kName + "Slc)")
 	if x.unsafe {
 		x.line(kName + "SlcHdr := codecSelferUnsafeString" + x.xs + "{uintptr(unsafe.Pointer(&" +
 			kName + "Slc[0])), len(" + kName + "Slc)}")
@@ -1433,16 +1432,11 @@ func (x *genRunner) decStructMap(varname, lenvarname string, rtid uintptr, t ref
 	} else {
 		x.line(kName + " := string(" + kName + "Slc)")
 	}
+	x.linef("z.DecSendContainerState(codecSelfer_containerMapValue%s)", x.xs)
 	x.decStructMapSwitch(kName, varname, rtid, t)
 
 	x.line("} // end for " + tpfx + "j" + i)
-	switch style {
-	case 1:
-	case 2:
-		x.line("r.ReadEnd()")
-	default:
-		x.linef("if !%shl%s { r.ReadEnd() }", tpfx, i)
-	}
+	x.linef("z.DecSendContainerState(codecSelfer_containerMapEnd%s)", x.xs)
 }
 
 func (x *genRunner) decStructArray(varname, lenvarname, breakString string, rtid uintptr, t reflect.Type) {
@@ -1451,25 +1445,37 @@ func (x *genRunner) decStructArray(varname, lenvarname, breakString string, rtid
 	ti := x.ti.get(rtid, t)
 	tisfi := ti.sfip // always use sequence from file. decStruct expects same thing.
 	x.linef("var %sj%s int", tpfx, i)
-	x.linef("var %sb%s bool", tpfx, i) // break
-	// x.linef("var %sl%s := r.ReadArrayStart()", tpfx, i)
+	x.linef("var %sb%s bool", tpfx, i)                        // break
 	x.linef("var %shl%s bool = %s >= 0", tpfx, i, lenvarname) // has length
 	for _, si := range tisfi {
 		var t2 reflect.StructField
 		if si.i != -1 {
 			t2 = t.Field(int(si.i))
 		} else {
-			t2 = t.FieldByIndex(si.is)
+			//we must accomodate anonymous fields, where the embedded field is a nil pointer in the value.
+			// t2 = t.FieldByIndex(si.is)
+			t2typ := t
+			varname3 := varname
+			for _, ix := range si.is {
+				for t2typ.Kind() == reflect.Ptr {
+					t2typ = t2typ.Elem()
+				}
+				t2 = t2typ.Field(ix)
+				t2typ = t2.Type
+				varname3 = varname3 + "." + t2.Name
+				if t2typ.Kind() == reflect.Ptr {
+					x.linef("if %s == nil { %s = new(%s) }", varname3, varname3, x.genTypeName(t2typ.Elem()))
+				}
+			}
 		}
 
 		x.linef("%sj%s++; if %shl%s { %sb%s = %sj%s > %s } else { %sb%s = r.CheckBreak() }",
 			tpfx, i, tpfx, i, tpfx, i,
 			tpfx, i, lenvarname, tpfx, i)
-		// x.line("if " + tpfx + "j" + i + "++; " + tpfx + "j" +
-		// i + " <=  " + tpfx + "l" + i + " {")
-		x.linef("if %sb%s { r.ReadEnd(); %s }", tpfx, i, breakString)
+		x.linef("if %sb%s { z.DecSendContainerState(codecSelfer_containerArrayEnd%s); %s }",
+			tpfx, i, x.xs, breakString)
+		x.linef("z.DecSendContainerState(codecSelfer_containerArrayElem%s)", x.xs)
 		x.decVar(varname+"."+t2.Name, t2.Type, true)
-		// x.line("} // end if " + tpfx + "j" + i + " <=  " + tpfx + "l" + i)
 	}
 	// read remaining values and throw away.
 	x.line("for {")
@@ -1477,19 +1483,20 @@ func (x *genRunner) decStructArray(varname, lenvarname, breakString string, rtid
 		tpfx, i, tpfx, i, tpfx, i,
 		tpfx, i, lenvarname, tpfx, i)
 	x.linef("if %sb%s { break }", tpfx, i)
+	x.linef("z.DecSendContainerState(codecSelfer_containerArrayElem%s)", x.xs)
 	x.linef(`z.DecStructFieldNotFound(%sj%s - 1, "")`, tpfx, i)
 	x.line("}")
-	x.line("r.ReadEnd()")
+	x.linef("z.DecSendContainerState(codecSelfer_containerArrayEnd%s)", x.xs)
 }
 
 func (x *genRunner) decStruct(varname string, rtid uintptr, t reflect.Type) {
 	// if container is map
-	// x.line("if z.DecContainerIsMap() { ")
 	i := x.varsfx()
-	x.line("if r.IsContainerType(codecSelferValueTypeMap" + x.xs + ") {")
+	x.linef("%sct%s := r.ContainerType()", genTempVarPfx, i)
+	x.linef("if %sct%s == codecSelferValueTypeMap%s {", genTempVarPfx, i, x.xs)
 	x.line(genTempVarPfx + "l" + i + " := r.ReadMapStart()")
 	x.linef("if %sl%s == 0 {", genTempVarPfx, i)
-	x.line("r.ReadEnd()")
+	x.linef("z.DecSendContainerState(codecSelfer_containerMapEnd%s)", x.xs)
 	if genUseOneFunctionForDecStructMap {
 		x.line("} else { ")
 		x.linef("x.codecDecodeSelfFromMap(%sl%s, d)", genTempVarPfx, i)
@@ -1502,18 +1509,16 @@ func (x *genRunner) decStruct(varname string, rtid uintptr, t reflect.Type) {
 	x.line("}")
 
 	// else if container is array
-	// x.line("} else if z.DecContainerIsArray() { ")
-	x.line("} else if r.IsContainerType(codecSelferValueTypeArray" + x.xs + ") {")
+	x.linef("} else if %sct%s == codecSelferValueTypeArray%s {", genTempVarPfx, i, x.xs)
 	x.line(genTempVarPfx + "l" + i + " := r.ReadArrayStart()")
 	x.linef("if %sl%s == 0 {", genTempVarPfx, i)
-	x.line("r.ReadEnd()")
+	x.linef("z.DecSendContainerState(codecSelfer_containerArrayEnd%s)", x.xs)
 	x.line("} else { ")
 	x.linef("x.codecDecodeSelfFromArray(%sl%s, d)", genTempVarPfx, i)
 	x.line("}")
 	// else panic
 	x.line("} else { ")
 	x.line("panic(codecSelferOnlyMapOrArrayEncodeToStructErr" + x.xs + ")")
-	// x.line("panic(`only encoded map or array can be decoded into a struct`)")
 	x.line("} ")
 }
 
@@ -1849,10 +1854,6 @@ func genInternalInit() {
 		"float64":     8,
 		"bool":        1,
 	}
-	// mapvaltypes2 := make(map[string]bool)
-	// for _, s := range mapvaltypes {
-	// 	mapvaltypes2[s] = true
-	// }
 	var gt genInternal
 
 	// For each slice or map type, there must be a (symetrical) Encode and Decode fast-path function

codec/helper.go

@@ -194,14 +194,6 @@ const (
 
 type seqType uint8
 
-// mirror json.Marshaler and json.Unmarshaler here, so we don't import the encoding/json package
-type jsonMarshaler interface {
-	MarshalJSON() ([]byte, error)
-}
-type jsonUnmarshaler interface {
-	UnmarshalJSON([]byte) error
-}
-
 const (
 	_ seqType = iota
 	seqTypeArray
@@ -209,13 +201,43 @@ const (
 	seqTypeChan
 )
 
+// note that containerMapStart and containerArraySend are not sent.
+// This is because the ReadXXXStart and EncodeXXXStart already does these.
+type containerState uint8
+
+const (
+	_ containerState = iota
+
+	containerMapStart // slot left open, since Driver method already covers it
+	containerMapKey
+	containerMapValue
+	containerMapEnd
+	containerArrayStart // slot left open, since Driver methods already cover it
+	containerArrayElem
+	containerArrayEnd
+)
+
+type containerStateRecv interface {
+	sendContainerState(containerState)
+}
+
+// mirror json.Marshaler and json.Unmarshaler here,
+// so we don't import the encoding/json package
+type jsonMarshaler interface {
+	MarshalJSON() ([]byte, error)
+}
+type jsonUnmarshaler interface {
+	UnmarshalJSON([]byte) error
+}
+
 var (
 	bigen               = binary.BigEndian
 	structInfoFieldName = "_struct"
 
-	// mapStrIntfTyp = reflect.TypeOf(map[string]interface{}(nil))
-	intfSliceTyp = reflect.TypeOf([]interface{}(nil))
-	intfTyp      = intfSliceTyp.Elem()
+	mapStrIntfTyp  = reflect.TypeOf(map[string]interface{}(nil))
+	mapIntfIntfTyp = reflect.TypeOf(map[interface{}]interface{}(nil))
+	intfSliceTyp   = reflect.TypeOf([]interface{}(nil))
+	intfTyp        = intfSliceTyp.Elem()
 
 	stringTyp     = reflect.TypeOf("")
 	timeTyp       = reflect.TypeOf(time.Time{})
@@ -241,6 +263,9 @@ var (
 	timeTypId       = reflect.ValueOf(timeTyp).Pointer()
 	stringTypId     = reflect.ValueOf(stringTyp).Pointer()
 
+	mapStrIntfTypId  = reflect.ValueOf(mapStrIntfTyp).Pointer()
+	mapIntfIntfTypId = reflect.ValueOf(mapIntfIntfTyp).Pointer()
+	intfSliceTypId   = reflect.ValueOf(intfSliceTyp).Pointer()
 	// mapBySliceTypId  = reflect.ValueOf(mapBySliceTyp).Pointer()
 
 	intBitsize  uint8 = uint8(reflect.TypeOf(int(0)).Bits())
@@ -283,7 +308,7 @@ type MapBySlice interface {
 type BasicHandle struct {
 	// TypeInfos is used to get the type info for any type.
 	//
-	// If not configure, the default TypeInfos is used, which uses struct tag keys: codec, json
+	// If not configured, the default TypeInfos is used, which uses struct tag keys: codec, json
 	TypeInfos *TypeInfos
 
 	extHandle
@@ -474,7 +499,7 @@ type extTypeTagFn struct {
 	ext  Ext
 }
 
-type extHandle []*extTypeTagFn
+type extHandle []extTypeTagFn
 
 // DEPRECATED: Use SetBytesExt or SetInterfaceExt on the Handle instead.
 //
@@ -513,12 +538,17 @@ func (o *extHandle) SetExt(rt reflect.Type, tag uint64, ext Ext) (err error) {
 		}
 	}
 
-	*o = append(*o, &extTypeTagFn{rtid, rt, tag, ext})
+	if *o == nil {
+		*o = make([]extTypeTagFn, 0, 4)
+	}
+	*o = append(*o, extTypeTagFn{rtid, rt, tag, ext})
 	return
 }
 
 func (o extHandle) getExt(rtid uintptr) *extTypeTagFn {
-	for _, v := range o {
+	var v *extTypeTagFn
+	for i := range o {
+		v = &o[i]
 		if v.rtid == rtid {
 			return v
 		}
@@ -527,7 +557,9 @@ func (o extHandle) getExt(rtid uintptr) *extTypeTagFn {
 }
 
 func (o extHandle) getExtForTag(tag uint64) *extTypeTagFn {
-	for _, v := range o {
+	var v *extTypeTagFn
+	for i := range o {
+		v = &o[i]
 		if v.tag == tag {
 			return v
 		}
@@ -650,6 +682,8 @@ type typeInfo struct {
 	rt   reflect.Type
 	rtid uintptr
 
+	numMeth uint16 // number of methods
+
 	// baseId gives pointer to the base reflect.Type, after deferencing
 	// the pointers. E.g. base type of ***time.Time is time.Time.
 	base      reflect.Type
@@ -746,14 +780,10 @@ func (x *TypeInfos) get(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
 		return
 	}
 
-	x.mu.Lock()
-	defer x.mu.Unlock()
-	if pti, ok = x.infos[rtid]; ok {
-		return
-	}
-
+	// do not hold lock while computing this.
+	// it may lead to duplication, but that's ok.
 	ti := typeInfo{rt: rt, rtid: rtid}
-	pti = &ti
+	ti.numMeth = uint16(rt.NumMethod())
 
 	var indir int8
 	if ok, indir = implementsIntf(rt, binaryMarshalerTyp); ok {
@@ -813,7 +843,13 @@ func (x *TypeInfos) get(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
 		copy(ti.sfi, sfip)
 	}
 	// sfi = sfip
-	x.infos[rtid] = pti
+
+	x.mu.Lock()
+	if pti, ok = x.infos[rtid]; !ok {
+		pti = &ti
+		x.infos[rtid] = pti
+	}
+	x.mu.Unlock()
 	return
 }
 

codec/helper_test.go

@@ -29,7 +29,22 @@ package codec
 // To fully test everything:
 //    go test -tags=x -benchtime=100ms -tv -bg -bi  -brw -bu -v -run=. -bench=.
 
+// Handling flags
+// codec_test.go will define a set of global flags for testing, including:
+//   - Use Reset
+//   - Use IO reader/writer (vs direct bytes)
+//   - Set Canonical
+//   - Set InternStrings
+//   - Use Symbols
+//
+// This way, we can test them all by running same set of tests with a different
+// set of flags.
+//
+// Following this, all the benchmarks will utilize flags set by codec_test.go
+// and will not redefine these "global" flags.
+
 import (
+	"bytes"
 	"errors"
 	"flag"
 	"fmt"
@@ -38,30 +53,45 @@ import (
 	"testing"
 )
 
-const (
-	testLogToT    = true
-	failNowOnFail = true
-)
+type testHED struct {
+	H Handle
+	E *Encoder
+	D *Decoder
+}
 
 var (
-	testNoopH      = NoopHandle(8)
-	testMsgpackH   = &MsgpackHandle{}
-	testBincH      = &BincHandle{}
-	testBincHNoSym = &BincHandle{}
-	testBincHSym   = &BincHandle{}
-	testSimpleH    = &SimpleHandle{}
-	testCborH      = &CborHandle{}
-	testJsonH      = &JsonHandle{}
-
+	testNoopH    = NoopHandle(8)
+	testMsgpackH = &MsgpackHandle{}
+	testBincH    = &BincHandle{}
+	testSimpleH  = &SimpleHandle{}
+	testCborH    = &CborHandle{}
+	testJsonH    = &JsonHandle{}
+
+	testHandles     []Handle
 	testPreInitFns  []func()
 	testPostInitFns []func()
 
 	testOnce sync.Once
+
+	testHEDs []testHED
 )
 
 func init() {
-	testBincHSym.AsSymbols = AsSymbolAll
-	testBincHNoSym.AsSymbols = AsSymbolNone
+	testHEDs = make([]testHED, 0, 32)
+	testHandles = append(testHandles,
+		testNoopH, testMsgpackH, testBincH, testSimpleH,
+		testCborH, testJsonH)
+}
+
+func testHEDGet(h Handle) *testHED {
+	for i := range testHEDs {
+		v := &testHEDs[i]
+		if v.H == h {
+			return v
+		}
+	}
+	testHEDs = append(testHEDs, testHED{h, NewEncoder(nil, h), NewDecoder(nil, h)})
+	return &testHEDs[len(testHEDs)-1]
 }
 
 func testInitAll() {
@@ -74,6 +104,95 @@ func testInitAll() {
 	}
 }
 
+func testCodecEncode(ts interface{}, bsIn []byte,
+	fn func([]byte) *bytes.Buffer, h Handle) (bs []byte, err error) {
+	// bs = make([]byte, 0, approxSize)
+	var e *Encoder
+	var buf *bytes.Buffer
+	if testUseReset {
+		e = testHEDGet(h).E
+	} else {
+		e = NewEncoder(nil, h)
+	}
+	if testUseIoEncDec {
+		buf = fn(bsIn)
+		e.Reset(buf)
+	} else {
+		bs = bsIn
+		e.ResetBytes(&bs)
+	}
+	if testUseMust {
+		e.MustEncode(ts)
+	} else {
+		err = e.Encode(ts)
+	}
+	if testUseIoEncDec {
+		bs = buf.Bytes()
+	}
+	return
+}
+
+func testCodecDecode(bs []byte, ts interface{}, h Handle) (err error) {
+	var d *Decoder
+	var buf *bytes.Reader
+	if testUseReset {
+		d = testHEDGet(h).D
+	} else {
+		d = NewDecoder(nil, h)
+	}
+	if testUseIoEncDec {
+		buf = bytes.NewReader(bs)
+		d.Reset(buf)
+	} else {
+		d.ResetBytes(bs)
+	}
+	if testUseMust {
+		d.MustDecode(ts)
+	} else {
+		err = d.Decode(ts)
+	}
+	return
+}
+
+// ----- functions below are used only by tests (not benchmarks)
+
+const (
+	testLogToT    = true
+	failNowOnFail = true
+)
+
+func checkErrT(t *testing.T, err error) {
+	if err != nil {
+		logT(t, err.Error())
+		failT(t)
+	}
+}
+
+func checkEqualT(t *testing.T, v1 interface{}, v2 interface{}, desc string) (err error) {
+	if err = deepEqual(v1, v2); err != nil {
+		logT(t, "Not Equal: %s: %v. v1: %v, v2: %v", desc, err, v1, v2)
+		failT(t)
+	}
+	return
+}
+
+func failT(t *testing.T) {
+	if failNowOnFail {
+		t.FailNow()
+	} else {
+		t.Fail()
+	}
+}
+
+// --- these functions are used by both benchmarks and tests
+
+func deepEqual(v1, v2 interface{}) (err error) {
+	if !reflect.DeepEqual(v1, v2) {
+		err = errors.New("Not Match")
+	}
+	return
+}
+
 func logT(x interface{}, format string, args ...interface{}) {
 	if t, ok := x.(*testing.T); ok && t != nil && testLogToT {
 		if testVerbose {
@@ -121,35 +240,3 @@ func approxDataSize(rv reflect.Value) (sum int) {
 	}
 	return
 }
-
-// ----- functions below are used only by tests (not benchmarks)
-
-func checkErrT(t *testing.T, err error) {
-	if err != nil {
-		logT(t, err.Error())
-		failT(t)
-	}
-}
-
-func checkEqualT(t *testing.T, v1 interface{}, v2 interface{}, desc string) (err error) {
-	if err = deepEqual(v1, v2); err != nil {
-		logT(t, "Not Equal: %s: %v. v1: %v, v2: %v", desc, err, v1, v2)
-		failT(t)
-	}
-	return
-}
-
-func failT(t *testing.T) {
-	if failNowOnFail {
-		t.FailNow()
-	} else {
-		t.Fail()
-	}
-}
-
-func deepEqual(v1, v2 interface{}) (err error) {
-	if !reflect.DeepEqual(v1, v2) {
-		err = errors.New("Not Match")
-	}
-	return
-}

codec/json.go

@@ -30,8 +30,6 @@ package codec
 
 // Top-level methods of json(End|Dec)Driver (which are implementations of (en|de)cDriver
 // MUST not call one-another.
-// They all must call sep(), and sep() MUST NOT be called more than once for each read.
-// If sep() is called and read is not done, you MUST call retryRead so separator wouldn't be read/written twice.
 
 import (
 	"bytes"
@@ -39,7 +37,6 @@ import (
 	"fmt"
 	"reflect"
 	"strconv"
-	"sync"
 	"unicode/utf16"
 	"unicode/utf8"
 )
@@ -60,12 +57,13 @@ var jsonUint64Pow10 = [...]uint64{
 }
 
 const (
-	// if jsonTrackSkipWhitespace, we track Whitespace and reduce the number of redundant checks.
-	// Make it a const flag, so that it can be elided during linking if false.
+	// jsonUnreadAfterDecNum controls whether we unread after decoding a number.
 	//
-	// It is not a clear win, because we continually set a flag behind a pointer
-	// and then check it each time, as opposed to just 4 conditionals on a stack variable.
-	jsonTrackSkipWhitespace = true
+	// instead of unreading, just update d.tok (iff it's not a whitespace char)
+	// However, doing this means that we may HOLD onto some data which belongs to another stream.
+	// Thus, it is safest to unread the data when done.
+	// keep behind a constant flag for now.
+	jsonUnreadAfterDecNum = true
 
 	// If !jsonValidateSymbols, decoding will be faster, by skipping some checks:
 	//   - If we see first character of null, false or true,
@@ -89,100 +87,6 @@ const (
 	// jsonNumDigitsUint64Largest = 19
 )
 
-// A stack is used to keep track of where we are in the tree.
-// This is necessary, as the Handle must know whether to consume or emit a separator.
-
-type jsonStackElem struct {
-	st byte // top of stack (either '}' or ']' or 0 for map, array or neither).
-	sf bool // NOT first time in that container at top of stack
-	so bool // stack ctr odd
-	sr bool // value has NOT been read, so do not re-send separator
-}
-
-func (x *jsonStackElem) retryRead() {
-	if x != nil && !x.sr {
-		x.sr = true
-	}
-}
-
-func (x *jsonStackElem) sep() (c byte) {
-	// do not use switch, so it's a candidate for inlining.
-	// to inline effectively, this must not be called from within another method.
-	// v := j.st
-	if x == nil || x.st == 0 {
-		return
-	}
-	if x.sr {
-		x.sr = false
-		return
-	}
-	// v == '}' OR ']'
-	if x.st == '}' {
-		// put , or : depending on if even or odd respectively
-		if x.so {
-			c = ':'
-			if !x.sf {
-				x.sf = true
-			}
-		} else if x.sf {
-			c = ','
-		}
-	} else {
-		if x.sf {
-			c = ','
-		} else {
-			x.sf = true
-		}
-	}
-	x.so = !x.so
-	// Note: Anything more, and this function doesn't inline. Keep it tight.
-	// if x.sr {
-	// 	x.sr = false
-	// }
-	return
-}
-
-const jsonStackPoolArrayLen = 32
-
-// pool used to prevent constant allocation of stacks.
-var jsonStackPool = sync.Pool{
-	New: func() interface{} {
-		return new([jsonStackPoolArrayLen]jsonStackElem)
-	},
-}
-
-// jsonStack contains the stack for tracking the state of the container (branch).
-// The same data structure is used during encode and decode, as it is similar functionality.
-type jsonStack struct {
-	s  []jsonStackElem // stack for map or array end tag. map=}, array=]
-	sc *jsonStackElem  // pointer to current (top) element on the stack.
-	sp *[jsonStackPoolArrayLen]jsonStackElem
-}
-
-func (j *jsonStack) start(c byte) {
-	if j.s == nil {
-		// j.s = make([]jsonStackElem, 0, 8)
-		j.sp = jsonStackPool.Get().(*[jsonStackPoolArrayLen]jsonStackElem)
-		j.s = j.sp[:0]
-	}
-	j.s = append(j.s, jsonStackElem{st: c})
-	j.sc = &(j.s[len(j.s)-1])
-}
-
-func (j *jsonStack) end() {
-	l := len(j.s) - 1 // length of new stack after pop'ing
-	if l == 0 {
-		jsonStackPool.Put(j.sp)
-		j.s = nil
-		j.sp = nil
-		j.sc = nil
-	} else {
-		j.s = j.s[:l]
-		j.sc = &(j.s[l-1])
-	}
-	//j.sc = &(j.s[len(j.s)-1])
-}
-
 type jsonEncDriver struct {
 	e  *Encoder
 	w  encWriter
@@ -190,21 +94,35 @@ type jsonEncDriver struct {
 	b  [64]byte // scratch
 	bs []byte   // scratch
 	se setExtWrapper
-	s  jsonStack
+	c  containerState
 	noBuiltInTypes
 }
 
-func (e *jsonEncDriver) EncodeNil() {
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
+func (e *jsonEncDriver) sendContainerState(c containerState) {
+	// determine whether to output separators
+	if c == containerMapKey {
+		if e.c != containerMapStart {
+			e.w.writen1(',')
+		}
+	} else if c == containerMapValue {
+		e.w.writen1(':')
+	} else if c == containerMapEnd {
+		e.w.writen1('}')
+	} else if c == containerArrayElem {
+		if e.c != containerArrayStart {
+			e.w.writen1(',')
+		}
+	} else if c == containerArrayEnd {
+		e.w.writen1(']')
 	}
+	e.c = c
+}
+
+func (e *jsonEncDriver) EncodeNil() {
 	e.w.writeb(jsonLiterals[9:13]) // null
 }
 
 func (e *jsonEncDriver) EncodeBool(b bool) {
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
 	if b {
 		e.w.writeb(jsonLiterals[0:4]) // true
 	} else {
@@ -213,94 +131,56 @@ func (e *jsonEncDriver) EncodeBool(b bool) {
 }
 
 func (e *jsonEncDriver) EncodeFloat32(f float32) {
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
 	e.w.writeb(strconv.AppendFloat(e.b[:0], float64(f), 'E', -1, 32))
 }
 
 func (e *jsonEncDriver) EncodeFloat64(f float64) {
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
 	// e.w.writestr(strconv.FormatFloat(f, 'E', -1, 64))
 	e.w.writeb(strconv.AppendFloat(e.b[:0], f, 'E', -1, 64))
 }
 
 func (e *jsonEncDriver) EncodeInt(v int64) {
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
 	e.w.writeb(strconv.AppendInt(e.b[:0], v, 10))
 }
 
 func (e *jsonEncDriver) EncodeUint(v uint64) {
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
 	e.w.writeb(strconv.AppendUint(e.b[:0], v, 10))
 }
 
 func (e *jsonEncDriver) EncodeExt(rv interface{}, xtag uint64, ext Ext, en *Encoder) {
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
 	if v := ext.ConvertExt(rv); v == nil {
 		e.w.writeb(jsonLiterals[9:13]) // null // e.EncodeNil()
 	} else {
-		e.s.sc.retryRead()
 		en.encode(v)
 	}
 }
 
 func (e *jsonEncDriver) EncodeRawExt(re *RawExt, en *Encoder) {
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
 	// only encodes re.Value (never re.Data)
 	if re.Value == nil {
 		e.w.writeb(jsonLiterals[9:13]) // null // e.EncodeNil()
 	} else {
-		e.s.sc.retryRead()
 		en.encode(re.Value)
 	}
 }
 
 func (e *jsonEncDriver) EncodeArrayStart(length int) {
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
-	e.s.start(']')
 	e.w.writen1('[')
+	e.c = containerArrayStart
 }
 
 func (e *jsonEncDriver) EncodeMapStart(length int) {
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
-	e.s.start('}')
 	e.w.writen1('{')
-}
-
-func (e *jsonEncDriver) EncodeEnd() {
-	b := e.s.sc.st
-	e.s.end()
-	e.w.writen1(b)
+	e.c = containerMapStart
 }
 
 func (e *jsonEncDriver) EncodeString(c charEncoding, v string) {
 	// e.w.writestr(strconv.Quote(v))
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
 	e.quoteStr(v)
 }
 
 func (e *jsonEncDriver) EncodeSymbol(v string) {
 	// e.EncodeString(c_UTF8, v)
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
 	e.quoteStr(v)
 }
 
@@ -310,14 +190,8 @@ func (e *jsonEncDriver) EncodeStringBytes(c charEncoding, v []byte) {
 		e.EncodeExt(v, 0, &e.se, e.e)
 		return
 	}
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
 	if c == c_RAW {
 		slen := base64.StdEncoding.EncodedLen(len(v))
-		if e.bs == nil {
-			e.bs = e.b[:]
-		}
 		if cap(e.bs) >= slen {
 			e.bs = e.bs[:slen]
 		} else {
@@ -334,9 +208,6 @@ func (e *jsonEncDriver) EncodeStringBytes(c charEncoding, v []byte) {
 }
 
 func (e *jsonEncDriver) EncodeAsis(v []byte) {
-	if c := e.s.sc.sep(); c != 0 {
-		e.w.writen1(c)
-	}
 	e.w.writeb(v)
 }
 
@@ -491,185 +362,199 @@ func (x *jsonNum) floatVal() (f float64, parseUsingStrConv bool) {
 }
 
 type jsonDecDriver struct {
-	d    *Decoder
-	h    *JsonHandle
-	r    decReader // *bytesDecReader decReader
-	ct   valueType // container type. one of unset, array or map.
-	bstr [8]byte   // scratch used for string \UXXX parsing
-	b    [64]byte  // scratch, used for parsing strings or numbers
-	b2   [64]byte  // scratch, used only for decodeBytes (after base64)
-	bs   []byte    // scratch. Initialized from b. Used for parsing strings or numbers.
+	noBuiltInTypes
+	d *Decoder
+	h *JsonHandle
+	r decReader
 
-	wsSkipped bool // whitespace skipped
+	c containerState
+	// tok is used to store the token read right after skipWhiteSpace.
+	tok uint8
 
-	se setExtWrapper
+	bstr [8]byte  // scratch used for string \UXXX parsing
+	b    [64]byte // scratch, used for parsing strings or numbers
+	b2   [64]byte // scratch, used only for decodeBytes (after base64)
+	bs   []byte   // scratch. Initialized from b. Used for parsing strings or numbers.
 
-	s jsonStack
+	se setExtWrapper
 
 	n jsonNum
-	noBuiltInTypes
+}
+
+func jsonIsWS(b byte) bool {
+	return b == ' ' || b == '\t' || b == '\r' || b == '\n'
 }
 
 // This will skip whitespace characters and return the next byte to read.
 // The next byte determines what the value will be one of.
-func (d *jsonDecDriver) skipWhitespace(unread bool) (b byte) {
-	// as initReadNext is not called all the time, we set ct to unSet whenever
-	// we skipwhitespace, as this is the signal that something new is about to be read.
-	d.ct = valueTypeUnset
-	b = d.r.readn1()
-	if !jsonTrackSkipWhitespace || !d.wsSkipped {
-		for ; b == ' ' || b == '\t' || b == '\r' || b == '\n'; b = d.r.readn1() {
+func (d *jsonDecDriver) skipWhitespace() {
+	// fast-path: do not enter loop. Just check first (in case no whitespace).
+	b := d.r.readn1()
+	if jsonIsWS(b) {
+		r := d.r
+		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
+		}
+	}
+	d.tok = b
+}
+
+// // To inline  if d.tok == 0 { d.skipWhitespace() }: USE:
+// func (d *jsonDecDriver) XXX() {
+// 	if d.tok == 0 {
+// 		b := d.r.readn1()
+// 		if jsonIsWS(b) {
+// 			r := d.r
+// 			for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
+// 			}
+// 		}
+// 		d.tok = b
+// 	}
+// 	// OR
+// 	if b, r := d.tok, d.r; b == 0 {
+// 		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
+// 		}
+// 		d.tok = b
+// 	}
+// }
+
+func (d *jsonDecDriver) sendContainerState(c containerState) {
+	if d.tok == 0 {
+		d.skipWhitespace()
+	}
+	var xc uint8 // char expected
+	if c == containerMapKey {
+		if d.c != containerMapStart {
+			xc = ','
 		}
-		if jsonTrackSkipWhitespace {
-			d.wsSkipped = true
+	} else if c == containerMapValue {
+		xc = ':'
+	} else if c == containerMapEnd {
+		xc = '}'
+	} else if c == containerArrayElem {
+		if d.c != containerArrayStart {
+			xc = ','
 		}
+	} else if c == containerArrayEnd {
+		xc = ']'
 	}
-	if unread {
-		d.r.unreadn1()
+	if xc != 0 {
+		if d.tok != xc {
+			d.d.errorf("json: expect char '%c' but got char '%c'", xc, d.tok)
+		}
+		d.tok = 0
 	}
-	return b
+	d.c = c
 }
 
 func (d *jsonDecDriver) CheckBreak() bool {
-	b := d.skipWhitespace(true)
-	return b == '}' || b == ']'
+	if d.tok == 0 {
+		d.skipWhitespace()
+	}
+	if d.tok == '}' || d.tok == ']' {
+		// d.tok = 0 // only checking, not consuming
+		return true
+	}
+	return false
 }
 
 func (d *jsonDecDriver) readStrIdx(fromIdx, toIdx uint8) {
 	bs := d.r.readx(int(toIdx - fromIdx))
+	d.tok = 0
 	if jsonValidateSymbols {
 		if !bytes.Equal(bs, jsonLiterals[fromIdx:toIdx]) {
 			d.d.errorf("json: expecting %s: got %s", jsonLiterals[fromIdx:toIdx], bs)
 			return
 		}
 	}
-	if jsonTrackSkipWhitespace {
-		d.wsSkipped = false
-	}
 }
 
 func (d *jsonDecDriver) TryDecodeAsNil() bool {
-	// we mustn't consume the state here, and end up trying to read separator twice.
-	// Instead, we keep track of the state and restore it if we couldn't decode as nil.
-
-	if c := d.s.sc.sep(); c != 0 {
-		d.expectChar(c)
+	if d.tok == 0 {
+		d.skipWhitespace()
 	}
-	b := d.skipWhitespace(false)
-	if b == 'n' {
+	if d.tok == 'n' {
 		d.readStrIdx(10, 13) // ull
-		d.ct = valueTypeNil
 		return true
 	}
-	d.r.unreadn1()
-	d.s.sc.retryRead()
 	return false
 }
 
 func (d *jsonDecDriver) DecodeBool() bool {
-	if c := d.s.sc.sep(); c != 0 {
-		d.expectChar(c)
+	if d.tok == 0 {
+		d.skipWhitespace()
 	}
-	b := d.skipWhitespace(false)
-	if b == 'f' {
+	if d.tok == 'f' {
 		d.readStrIdx(5, 9) // alse
 		return false
 	}
-	if b == 't' {
+	if d.tok == 't' {
 		d.readStrIdx(1, 4) // rue
 		return true
 	}
-	d.d.errorf("json: decode bool: got first char %c", b)
+	d.d.errorf("json: decode bool: got first char %c", d.tok)
 	return false // "unreachable"
 }
 
 func (d *jsonDecDriver) ReadMapStart() int {
-	if c := d.s.sc.sep(); c != 0 {
-		d.expectChar(c)
+	if d.tok == 0 {
+		d.skipWhitespace()
 	}
-	d.s.start('}')
-	d.expectChar('{')
-	d.ct = valueTypeMap
-	return -1
-}
-
-func (d *jsonDecDriver) ReadArrayStart() int {
-	if c := d.s.sc.sep(); c != 0 {
-		d.expectChar(c)
+	if d.tok != '{' {
+		d.d.errorf("json: expect char '%c' but got char '%c'", '{', d.tok)
 	}
-	d.s.start(']')
-	d.expectChar('[')
-	d.ct = valueTypeArray
+	d.tok = 0
+	d.c = containerMapStart
 	return -1
 }
 
-func (d *jsonDecDriver) ReadEnd() {
-	b := d.s.sc.st
-	d.s.end()
-	d.expectChar(b)
-}
-
-func (d *jsonDecDriver) expectChar(c uint8) {
-	b := d.skipWhitespace(false)
-	if b != c {
-		d.d.errorf("json: expect char '%c' but got char '%c'", c, b)
-		return
+func (d *jsonDecDriver) ReadArrayStart() int {
+	if d.tok == 0 {
+		d.skipWhitespace()
 	}
-	if jsonTrackSkipWhitespace {
-		d.wsSkipped = false
+	if d.tok != '[' {
+		d.d.errorf("json: expect char '%c' but got char '%c'", '[', d.tok)
 	}
+	d.tok = 0
+	d.c = containerArrayStart
+	return -1
 }
 
-// func (d *jsonDecDriver) maybeChar(c uint8) {
-// 	b := d.skipWhitespace(false)
-// 	if b != c {
-// 		d.r.unreadn1()
-// 		return
-// 	}
-// 	if jsonTrackSkipWhitespace {
-// 		d.wsSkipped = false
-// 	}
-// }
-
-func (d *jsonDecDriver) IsContainerType(vt valueType) bool {
+func (d *jsonDecDriver) ContainerType() (vt valueType) {
 	// check container type by checking the first char
-	if d.ct == valueTypeUnset {
-		b := d.skipWhitespace(true)
-		if b == '{' {
-			d.ct = valueTypeMap
-		} else if b == '[' {
-			d.ct = valueTypeArray
-		} else if b == 'n' {
-			d.ct = valueTypeNil
-		} else if b == '"' {
-			d.ct = valueTypeString
-		}
-	}
-	if vt == valueTypeNil || vt == valueTypeBytes || vt == valueTypeString ||
-		vt == valueTypeArray || vt == valueTypeMap {
-		return d.ct == vt
-	}
-	// ugorji: made switch into conditionals, so that IsContainerType can be inlined.
-	// switch vt {
-	// case valueTypeNil, valueTypeBytes, valueTypeString, valueTypeArray, valueTypeMap:
-	// 	return d.ct == vt
-	// }
-	d.d.errorf("isContainerType: unsupported parameter: %v", vt)
-	return false // "unreachable"
+	if d.tok == 0 {
+		d.skipWhitespace()
+	}
+	if b := d.tok; b == '{' {
+		return valueTypeMap
+	} else if b == '[' {
+		return valueTypeArray
+	} else if b == 'n' {
+		return valueTypeNil
+	} else if b == '"' {
+		return valueTypeString
+	}
+	return valueTypeUnset
+	// d.d.errorf("isContainerType: unsupported parameter: %v", vt)
+	// return false // "unreachable"
 }
 
 func (d *jsonDecDriver) decNum(storeBytes bool) {
 	// If it is has a . or an e|E, decode as a float; else decode as an int.
-	b := d.skipWhitespace(false)
+	if d.tok == 0 {
+		d.skipWhitespace()
+	}
+	b := d.tok
 	if !(b == '+' || b == '-' || b == '.' || (b >= '0' && b <= '9')) {
 		d.d.errorf("json: decNum: got first char '%c'", b)
 		return
 	}
+	d.tok = 0
 
 	const cutoff = (1<<64-1)/uint64(10) + 1 // cutoff64(base)
 	const jsonNumUintMaxVal = 1<<uint64(64) - 1
 
 	n := &d.n
+	r := d.r
 	n.reset()
 	d.bs = d.bs[:0]
 
@@ -690,7 +575,7 @@ LOOP:
 			case 0:
 				state = 2
 				// do not add sign to the slice ...
-				b, eof = d.r.readn1eof()
+				b, eof = r.readn1eof()
 				continue
 			case 6: // typ = jsonNumFloat
 				state = 7
@@ -703,7 +588,7 @@ LOOP:
 				state = 2
 				n.neg = true
 				// do not add sign to the slice ...
-				b, eof = d.r.readn1eof()
+				b, eof = r.readn1eof()
 				continue
 			case 6: // typ = jsonNumFloat
 				eNeg = true
@@ -769,7 +654,7 @@ LOOP:
 		if storeBytes {
 			d.bs = append(d.bs, b)
 		}
-		b, eof = d.r.readn1eof()
+		b, eof = r.readn1eof()
 	}
 
 	if jsonTruncateMantissa && n.mantissa != 0 {
@@ -790,10 +675,13 @@ LOOP:
 	// d.n = n
 
 	if !eof {
-		d.r.unreadn1()
-	}
-	if jsonTrackSkipWhitespace {
-		d.wsSkipped = false
+		if jsonUnreadAfterDecNum {
+			r.unreadn1()
+		} else {
+			if !jsonIsWS(b) {
+				d.tok = b
+			}
+		}
 	}
 	// fmt.Printf("1: n: bytes: %s, neg: %v, dot: %v, exponent: %v, mantissaEndIndex: %v\n",
 	// 	n.bytes, n.neg, n.dot, n.exponent, n.mantissaEndIndex)
@@ -801,9 +689,6 @@ LOOP:
 }
 
 func (d *jsonDecDriver) DecodeInt(bitsize uint8) (i int64) {
-	if c := d.s.sc.sep(); c != 0 {
-		d.expectChar(c)
-	}
 	d.decNum(false)
 	n := &d.n
 	if n.manOverflow {
@@ -851,9 +736,6 @@ func (d *jsonDecDriver) floatVal() (f float64) {
 }
 
 func (d *jsonDecDriver) DecodeUint(bitsize uint8) (u uint64) {
-	if c := d.s.sc.sep(); c != 0 {
-		d.expectChar(c)
-	}
 	d.decNum(false)
 	n := &d.n
 	if n.neg {
@@ -885,9 +767,6 @@ func (d *jsonDecDriver) DecodeUint(bitsize uint8) (u uint64) {
 }
 
 func (d *jsonDecDriver) DecodeFloat(chkOverflow32 bool) (f float64) {
-	if c := d.s.sc.sep(); c != 0 {
-		d.expectChar(c)
-	}
 	d.decNum(true)
 	f = d.floatVal()
 	if chkOverflow32 && chkOvf.Float32(f) {
@@ -898,10 +777,6 @@ func (d *jsonDecDriver) DecodeFloat(chkOverflow32 bool) (f float64) {
 }
 
 func (d *jsonDecDriver) DecodeExt(rv interface{}, xtag uint64, ext Ext) (realxtag uint64) {
-	// No need to call sep here, as d.d.decode() handles it
-	// if c := d.s.sc.sep(); c != 0 {
-	// 	d.expectChar(c)
-	// }
 	if ext == nil {
 		re := rv.(*RawExt)
 		re.Tag = xtag
@@ -921,9 +796,6 @@ func (d *jsonDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut [
 		d.DecodeExt(&bsOut, 0, &d.se)
 		return
 	}
-	if c := d.s.sc.sep(); c != 0 {
-		d.expectChar(c)
-	}
 	d.appendStringAsBytes()
 	// if isstring, then just return the bytes, even if it is using the scratch buffer.
 	// the bytes will be converted to a string as needed.
@@ -951,30 +823,32 @@ func (d *jsonDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut [
 }
 
 func (d *jsonDecDriver) DecodeString() (s string) {
-	if c := d.s.sc.sep(); c != 0 {
-		d.expectChar(c)
-	}
-	return d.decString()
-}
-
-func (d *jsonDecDriver) decString() (s string) {
 	d.appendStringAsBytes()
-	if x := d.s.sc; x != nil && x.st == '}' && x.so { // map key
+	// if x := d.s.sc; x != nil && x.so && x.st == '}' { // map key
+	if d.c == containerMapKey {
 		return d.d.string(d.bs)
 	}
 	return string(d.bs)
 }
 
 func (d *jsonDecDriver) appendStringAsBytes() {
-	d.expectChar('"')
+	if d.tok == 0 {
+		d.skipWhitespace()
+	}
+	if d.tok != '"' {
+		d.d.errorf("json: expect char '%c' but got char '%c'", '"', d.tok)
+	}
+	d.tok = 0
+
 	v := d.bs[:0]
 	var c uint8
+	r := d.r
 	for {
-		c = d.r.readn1()
+		c = r.readn1()
 		if c == '"' {
 			break
 		} else if c == '\\' {
-			c = d.r.readn1()
+			c = r.readn1()
 			switch c {
 			case '"', '\\', '/', '\'':
 				v = append(v, c)
@@ -1003,21 +877,19 @@ func (d *jsonDecDriver) appendStringAsBytes() {
 			v = append(v, c)
 		}
 	}
-	if jsonTrackSkipWhitespace {
-		d.wsSkipped = false
-	}
 	d.bs = v
 }
 
 func (d *jsonDecDriver) jsonU4(checkSlashU bool) rune {
-	if checkSlashU && !(d.r.readn1() == '\\' && d.r.readn1() == 'u') {
+	r := d.r
+	if checkSlashU && !(r.readn1() == '\\' && r.readn1() == 'u') {
 		d.d.errorf(`json: unquoteStr: invalid unicode sequence. Expecting \u`)
 		return 0
 	}
 	// u, _ := strconv.ParseUint(string(d.bstr[:4]), 16, 64)
 	var u uint32
 	for i := 0; i < 4; i++ {
-		v := d.r.readn1()
+		v := r.readn1()
 		if '0' <= v && v <= '9' {
 			v = v - '0'
 		} else if 'a' <= v && v <= 'z' {
@@ -1033,75 +905,79 @@ func (d *jsonDecDriver) jsonU4(checkSlashU bool) rune {
 	return rune(u)
 }
 
-func (d *jsonDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
-	if c := d.s.sc.sep(); c != 0 {
-		d.expectChar(c)
+func (d *jsonDecDriver) DecodeNaked() {
+	z := &d.d.n
+	// var decodeFurther bool
+
+	if d.tok == 0 {
+		d.skipWhitespace()
 	}
-	n := d.skipWhitespace(true)
-	switch n {
+	switch d.tok {
 	case 'n':
-		d.readStrIdx(9, 13) // null
-		vt = valueTypeNil
+		d.readStrIdx(10, 13) // ull
+		z.v = valueTypeNil
 	case 'f':
-		d.readStrIdx(4, 9) // false
-		vt = valueTypeBool
-		v = false
+		d.readStrIdx(5, 9) // alse
+		z.v = valueTypeBool
+		z.b = false
 	case 't':
-		d.readStrIdx(0, 4) // true
-		vt = valueTypeBool
-		v = true
+		d.readStrIdx(1, 4) // rue
+		z.v = valueTypeBool
+		z.b = true
 	case '{':
-		vt = valueTypeMap
-		decodeFurther = true
+		z.v = valueTypeMap
+		// d.tok = 0 // don't consume. kInterfaceNaked will call ReadMapStart
+		// decodeFurther = true
 	case '[':
-		vt = valueTypeArray
-		decodeFurther = true
+		z.v = valueTypeArray
+		// d.tok = 0 // don't consume. kInterfaceNaked will call ReadArrayStart
+		// decodeFurther = true
 	case '"':
-		vt = valueTypeString
-		v = d.decString() // same as d.DecodeString(), but skipping sep() call.
+		z.v = valueTypeString
+		z.s = d.DecodeString()
 	default: // number
 		d.decNum(true)
 		n := &d.n
 		// if the string had a any of [.eE], then decode as float.
 		switch {
 		case n.explicitExponent, n.dot, n.exponent < 0, n.manOverflow:
-			vt = valueTypeFloat
-			v = d.floatVal()
+			z.v = valueTypeFloat
+			z.f = d.floatVal()
 		case n.exponent == 0:
 			u := n.mantissa
 			switch {
 			case n.neg:
-				vt = valueTypeInt
-				v = -int64(u)
+				z.v = valueTypeInt
+				z.i = -int64(u)
 			case d.h.SignedInteger:
-				vt = valueTypeInt
-				v = int64(u)
+				z.v = valueTypeInt
+				z.i = int64(u)
 			default:
-				vt = valueTypeUint
-				v = u
+				z.v = valueTypeUint
+				z.u = u
 			}
 		default:
 			u, overflow := n.uintExp()
 			switch {
 			case overflow:
-				vt = valueTypeFloat
-				v = d.floatVal()
+				z.v = valueTypeFloat
+				z.f = d.floatVal()
 			case n.neg:
-				vt = valueTypeInt
-				v = -int64(u)
+				z.v = valueTypeInt
+				z.i = -int64(u)
 			case d.h.SignedInteger:
-				vt = valueTypeInt
-				v = int64(u)
+				z.v = valueTypeInt
+				z.i = int64(u)
 			default:
-				vt = valueTypeUint
-				v = u
+				z.v = valueTypeUint
+				z.u = u
 			}
 		}
 		// fmt.Printf("DecodeNaked: Number: %T, %v\n", v, v)
 	}
-	if decodeFurther {
-		d.s.sc.retryRead()
-	}
+	// if decodeFurther {
+	// 	d.s.sc.retryRead()
+	// }
 	return
 }
 
@@ -1124,15 +1000,20 @@ func (d *jsonDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurthe
 // For example, a user can read a json value, then a cbor value, then a msgpack value,
 // all from the same stream in sequence.
 type JsonHandle struct {
-	BasicHandle
 	textEncodingType
+	BasicHandle
 	// RawBytesExt, if configured, is used to encode and decode raw bytes in a custom way.
 	// If not configured, raw bytes are encoded to/from base64 text.
 	RawBytesExt InterfaceExt
 }
 
+func (h *JsonHandle) SetInterfaceExt(rt reflect.Type, tag uint64, ext InterfaceExt) (err error) {
+	return h.SetExt(rt, tag, &setExtWrapper{i: ext})
+}
+
 func (h *JsonHandle) newEncDriver(e *Encoder) encDriver {
 	hd := jsonEncDriver{e: e, w: e.w, h: h}
+	hd.bs = hd.b[:0]
 	hd.se.i = h.RawBytesExt
 	return &hd
 }
@@ -1145,8 +1026,12 @@ func (h *JsonHandle) newDecDriver(d *Decoder) decDriver {
 	return &hd
 }
 
-func (h *JsonHandle) SetInterfaceExt(rt reflect.Type, tag uint64, ext InterfaceExt) (err error) {
-	return h.SetExt(rt, tag, &setExtWrapper{i: ext})
+func (e *jsonEncDriver) reset() {
+	e.w = e.e.w
+}
+
+func (d *jsonDecDriver) reset() {
+	d.r = d.d.r
 }
 
 var jsonEncodeTerminate = []byte{' '}

codec/msgpack.go

@@ -103,11 +103,11 @@ var (
 //---------------------------------------------
 
 type msgpackEncDriver struct {
+	noBuiltInTypes
+	encNoSeparator
 	e *Encoder
 	w encWriter
 	h *MsgpackHandle
-	noBuiltInTypes
-	encNoSeparator
 	x [8]byte
 }
 
@@ -271,7 +271,6 @@ type msgpackDecDriver struct {
 	bd     byte
 	bdRead bool
 	br     bool // bytes reader
-	bdType valueType
 	noBuiltInTypes
 	noStreamingCodec
 	decNoSeparator
@@ -282,106 +281,100 @@ type msgpackDecDriver struct {
 // It is called when a nil interface{} is passed, leaving it up to the DecDriver
 // to introspect the stream and decide how best to decode.
 // It deciphers the value by looking at the stream first.
-func (d *msgpackDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
+func (d *msgpackDecDriver) DecodeNaked() {
 	if !d.bdRead {
 		d.readNextBd()
 	}
 	bd := d.bd
+	n := &d.d.n
+	var decodeFurther bool
 
 	switch bd {
 	case mpNil:
-		vt = valueTypeNil
+		n.v = valueTypeNil
 		d.bdRead = false
 	case mpFalse:
-		vt = valueTypeBool
-		v = false
+		n.v = valueTypeBool
+		n.b = false
 	case mpTrue:
-		vt = valueTypeBool
-		v = true
+		n.v = valueTypeBool
+		n.b = true
 
 	case mpFloat:
-		vt = valueTypeFloat
-		v = float64(math.Float32frombits(bigen.Uint32(d.r.readx(4))))
+		n.v = valueTypeFloat
+		n.f = float64(math.Float32frombits(bigen.Uint32(d.r.readx(4))))
 	case mpDouble:
-		vt = valueTypeFloat
-		v = math.Float64frombits(bigen.Uint64(d.r.readx(8)))
+		n.v = valueTypeFloat
+		n.f = math.Float64frombits(bigen.Uint64(d.r.readx(8)))
 
 	case mpUint8:
-		vt = valueTypeUint
-		v = uint64(d.r.readn1())
+		n.v = valueTypeUint
+		n.u = uint64(d.r.readn1())
 	case mpUint16:
-		vt = valueTypeUint
-		v = uint64(bigen.Uint16(d.r.readx(2)))
+		n.v = valueTypeUint
+		n.u = uint64(bigen.Uint16(d.r.readx(2)))
 	case mpUint32:
-		vt = valueTypeUint
-		v = uint64(bigen.Uint32(d.r.readx(4)))
+		n.v = valueTypeUint
+		n.u = uint64(bigen.Uint32(d.r.readx(4)))
 	case mpUint64:
-		vt = valueTypeUint
-		v = uint64(bigen.Uint64(d.r.readx(8)))
+		n.v = valueTypeUint
+		n.u = uint64(bigen.Uint64(d.r.readx(8)))
 
 	case mpInt8:
-		vt = valueTypeInt
-		v = int64(int8(d.r.readn1()))
+		n.v = valueTypeInt
+		n.i = int64(int8(d.r.readn1()))
 	case mpInt16:
-		vt = valueTypeInt
-		v = int64(int16(bigen.Uint16(d.r.readx(2))))
+		n.v = valueTypeInt
+		n.i = int64(int16(bigen.Uint16(d.r.readx(2))))
 	case mpInt32:
-		vt = valueTypeInt
-		v = int64(int32(bigen.Uint32(d.r.readx(4))))
+		n.v = valueTypeInt
+		n.i = int64(int32(bigen.Uint32(d.r.readx(4))))
 	case mpInt64:
-		vt = valueTypeInt
-		v = int64(int64(bigen.Uint64(d.r.readx(8))))
+		n.v = valueTypeInt
+		n.i = int64(int64(bigen.Uint64(d.r.readx(8))))
 
 	default:
 		switch {
 		case bd >= mpPosFixNumMin && bd <= mpPosFixNumMax:
 			// positive fixnum (always signed)
-			vt = valueTypeInt
-			v = int64(int8(bd))
+			n.v = valueTypeInt
+			n.i = int64(int8(bd))
 		case bd >= mpNegFixNumMin && bd <= mpNegFixNumMax:
 			// negative fixnum
-			vt = valueTypeInt
-			v = int64(int8(bd))
+			n.v = valueTypeInt
+			n.i = int64(int8(bd))
 		case bd == mpStr8, bd == mpStr16, bd == mpStr32, bd >= mpFixStrMin && bd <= mpFixStrMax:
 			if d.h.RawToString {
-				var rvm string
-				vt = valueTypeString
-				v = &rvm
+				n.v = valueTypeString
+				n.s = d.DecodeString()
 			} else {
-				var rvm = zeroByteSlice
-				vt = valueTypeBytes
-				v = &rvm
+				n.v = valueTypeBytes
+				n.l = d.DecodeBytes(nil, false, false)
 			}
-			decodeFurther = true
 		case bd == mpBin8, bd == mpBin16, bd == mpBin32:
-			var rvm = zeroByteSlice
-			vt = valueTypeBytes
-			v = &rvm
-			decodeFurther = true
+			n.v = valueTypeBytes
+			n.l = d.DecodeBytes(nil, false, false)
 		case bd == mpArray16, bd == mpArray32, bd >= mpFixArrayMin && bd <= mpFixArrayMax:
-			vt = valueTypeArray
+			n.v = valueTypeArray
 			decodeFurther = true
 		case bd == mpMap16, bd == mpMap32, bd >= mpFixMapMin && bd <= mpFixMapMax:
-			vt = valueTypeMap
+			n.v = valueTypeMap
 			decodeFurther = true
 		case bd >= mpFixExt1 && bd <= mpFixExt16, bd >= mpExt8 && bd <= mpExt32:
+			n.v = valueTypeExt
 			clen := d.readExtLen()
-			var re RawExt
-			re.Tag = uint64(d.r.readn1())
-			re.Data = d.r.readx(clen)
-			v = &re
-			vt = valueTypeExt
+			n.u = uint64(d.r.readn1())
+			n.l = d.r.readx(clen)
 		default:
 			d.d.errorf("Nil-Deciphered DecodeValue: %s: hex: %x, dec: %d", msgBadDesc, bd, bd)
-			return
 		}
 	}
 	if !decodeFurther {
 		d.bdRead = false
 	}
-	if vt == valueTypeUint && d.h.SignedInteger {
-		d.bdType = valueTypeInt
-		v = int64(v.(uint64))
+	if n.v == valueTypeUint && d.h.SignedInteger {
+		n.v = valueTypeInt
+		n.i = int64(n.v)
 	}
 	return
 }
@@ -566,28 +559,27 @@ func (d *msgpackDecDriver) DecodeString() (s string) {
 func (d *msgpackDecDriver) readNextBd() {
 	d.bd = d.r.readn1()
 	d.bdRead = true
-	d.bdType = valueTypeUnset
 }
 
-func (d *msgpackDecDriver) IsContainerType(vt valueType) bool {
+func (d *msgpackDecDriver) ContainerType() (vt valueType) {
 	bd := d.bd
-	switch vt {
-	case valueTypeNil:
-		return bd == mpNil
-	case valueTypeBytes:
-		return bd == mpBin8 || bd == mpBin16 || bd == mpBin32 ||
-			(!d.h.RawToString &&
-				(bd == mpStr8 || bd == mpStr16 || bd == mpStr32 || (bd >= mpFixStrMin && bd <= mpFixStrMax)))
-	case valueTypeString:
-		return d.h.RawToString &&
-			(bd == mpStr8 || bd == mpStr16 || bd == mpStr32 || (bd >= mpFixStrMin && bd <= mpFixStrMax))
-	case valueTypeArray:
-		return bd == mpArray16 || bd == mpArray32 || (bd >= mpFixArrayMin && bd <= mpFixArrayMax)
-	case valueTypeMap:
-		return bd == mpMap16 || bd == mpMap32 || (bd >= mpFixMapMin && bd <= mpFixMapMax)
-	}
-	d.d.errorf("isContainerType: unsupported parameter: %v", vt)
-	return false // "unreachable"
+	if bd == mpNil {
+		return valueTypeNil
+	} else if bd == mpBin8 || bd == mpBin16 || bd == mpBin32 ||
+		(!d.h.RawToString &&
+			(bd == mpStr8 || bd == mpStr16 || bd == mpStr32 || (bd >= mpFixStrMin && bd <= mpFixStrMax))) {
+		return valueTypeBytes
+	} else if d.h.RawToString &&
+		(bd == mpStr8 || bd == mpStr16 || bd == mpStr32 || (bd >= mpFixStrMin && bd <= mpFixStrMax)) {
+		return valueTypeString
+	} else if bd == mpArray16 || bd == mpArray32 || (bd >= mpFixArrayMin && bd <= mpFixArrayMax) {
+		return valueTypeArray
+	} else if bd == mpMap16 || bd == mpMap32 || (bd >= mpFixMapMin && bd <= mpFixMapMax) {
+		return valueTypeMap
+	} else {
+		// d.d.errorf("isContainerType: unsupported parameter: %v", vt)
+	}
+	return valueTypeUnset
 }
 
 func (d *msgpackDecDriver) TryDecodeAsNil() (v bool) {
@@ -701,7 +693,6 @@ func (d *msgpackDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs
 //MsgpackHandle is a Handle for the Msgpack Schema-Free Encoding Format.
 type MsgpackHandle struct {
 	BasicHandle
-	binaryEncodingType
 
 	// RawToString controls how raw bytes are decoded into a nil interface{}.
 	RawToString bool
@@ -717,6 +708,11 @@ type MsgpackHandle struct {
 	// type is provided (e.g. decoding into a nil interface{}), you get back
 	// a []byte or string based on the setting of RawToString.
 	WriteExt bool
+	binaryEncodingType
+}
+
+func (h *MsgpackHandle) SetBytesExt(rt reflect.Type, tag uint64, ext BytesExt) (err error) {
+	return h.SetExt(rt, tag, &setExtWrapper{b: ext})
 }
 
 func (h *MsgpackHandle) newEncDriver(e *Encoder) encDriver {
@@ -727,8 +723,12 @@ func (h *MsgpackHandle) newDecDriver(d *Decoder) decDriver {
 	return &msgpackDecDriver{d: d, r: d.r, h: h, br: d.bytes}
 }
 
-func (h *MsgpackHandle) SetBytesExt(rt reflect.Type, tag uint64, ext BytesExt) (err error) {
-	return h.SetExt(rt, tag, &setExtWrapper{b: ext})
+func (e *msgpackEncDriver) reset() {
+	e.w = e.e.w
+}
+
+func (d *msgpackDecDriver) reset() {
+	d.r = d.d.r
 }
 
 //--------------------------------------------------

codec/noop.go

@@ -38,21 +38,25 @@ type noopHandle struct {
 }
 
 type noopDrv struct {
+	d    *Decoder
+	e    *Encoder
 	i    int
 	S    []string
 	B    [][]byte
 	mks  []bool    // stack. if map (true), else if array (false)
 	mk   bool      // top of stack. what container are we on? map or array?
-	ct   valueType // last request for IsContainerType.
-	cb   bool      // last response for IsContainerType.
+	ct   valueType // last response for IsContainerType.
+	cb   int       // counter for ContainerType
 	rand *rand.Rand
 }
 
 func (h *noopDrv) r(v int) int { return h.rand.Intn(v) }
 func (h *noopDrv) m(v int) int { h.i++; return h.i % v }
 
-func (h *noopDrv) newEncDriver(_ *Encoder) encDriver { return h }
-func (h *noopDrv) newDecDriver(_ *Decoder) decDriver { return h }
+func (h *noopDrv) newEncDriver(e *Encoder) encDriver { h.e = e; return h }
+func (h *noopDrv) newDecDriver(d *Decoder) decDriver { h.d = d; return h }
+
+func (h *noopDrv) reset() {}
 
 // --- encDriver
 
@@ -111,18 +115,48 @@ func (h *noopDrv) ReadEnd() { h.end() }
 func (h *noopDrv) ReadMapStart() int   { h.start(true); return h.m(10) }
 func (h *noopDrv) ReadArrayStart() int { h.start(false); return h.m(10) }
 
-func (h *noopDrv) IsContainerType(vt valueType) bool {
+func (h *noopDrv) ContainerType() (vt valueType) {
 	// return h.m(2) == 0
-	// handle kStruct
-	if h.ct == valueTypeMap && vt == valueTypeArray || h.ct == valueTypeArray && vt == valueTypeMap {
-		h.cb = !h.cb
-		h.ct = vt
-		return h.cb
-	}
-	// go in a loop and check it.
-	h.ct = vt
-	h.cb = h.m(7) == 0
-	return h.cb
+	// handle kStruct, which will bomb is it calls this and doesn't get back a map or array.
+	// consequently, if the return value is not map or array, reset it to one of them based on h.m(7) % 2
+	// for kstruct: at least one out of every 2 times, return one of valueTypeMap or Array (else kstruct bombs)
+	// however, every 10th time it is called, we just return something else.
+	var vals = [...]valueType{valueTypeArray, valueTypeMap}
+	//  ------------ TAKE ------------
+	// if h.cb%2 == 0 {
+	// 	if h.ct == valueTypeMap || h.ct == valueTypeArray {
+	// 	} else {
+	// 		h.ct = vals[h.m(2)]
+	// 	}
+	// } else if h.cb%5 == 0 {
+	// 	h.ct = valueType(h.m(8))
+	// } else {
+	// 	h.ct = vals[h.m(2)]
+	// }
+	//  ------------ TAKE ------------
+	// if h.cb%16 == 0 {
+	// 	h.ct = valueType(h.cb % 8)
+	// } else {
+	// 	h.ct = vals[h.cb%2]
+	// }
+	h.ct = vals[h.cb%2]
+	h.cb++
+	return h.ct
+
+	// if h.ct == valueTypeNil || h.ct == valueTypeString || h.ct == valueTypeBytes {
+	// 	return h.ct
+	// }
+	// return valueTypeUnset
+	// TODO: may need to tweak this so it works.
+	// if h.ct == valueTypeMap && vt == valueTypeArray || h.ct == valueTypeArray && vt == valueTypeMap {
+	// 	h.cb = !h.cb
+	// 	h.ct = vt
+	// 	return h.cb
+	// }
+	// // go in a loop and check it.
+	// h.ct = vt
+	// h.cb = h.m(7) == 0
+	// return h.cb
 }
 func (h *noopDrv) TryDecodeAsNil() bool {
 	if h.mk {
@@ -135,7 +169,7 @@ func (h *noopDrv) DecodeExt(rv interface{}, xtag uint64, ext Ext) uint64 {
 	return 0
 }
 
-func (h *noopDrv) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
+func (h *noopDrv) DecodeNaked() {
 	// use h.r (random) not h.m() because h.m() could cause the same value to be given.
 	var sk int
 	if h.mk {
@@ -144,32 +178,35 @@ func (h *noopDrv) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool
 	} else {
 		sk = h.r(12)
 	}
+	n := &h.d.n
 	switch sk {
 	case 0:
-		vt = valueTypeNil
+		n.v = valueTypeNil
 	case 1:
-		vt, v = valueTypeBool, false
+		n.v, n.b = valueTypeBool, false
 	case 2:
-		vt, v = valueTypeBool, true
+		n.v, n.b = valueTypeBool, true
 	case 3:
-		vt, v = valueTypeInt, h.DecodeInt(64)
+		n.v, n.i = valueTypeInt, h.DecodeInt(64)
 	case 4:
-		vt, v = valueTypeUint, h.DecodeUint(64)
+		n.v, n.u = valueTypeUint, h.DecodeUint(64)
 	case 5:
-		vt, v = valueTypeFloat, h.DecodeFloat(true)
+		n.v, n.f = valueTypeFloat, h.DecodeFloat(true)
 	case 6:
-		vt, v = valueTypeFloat, h.DecodeFloat(false)
+		n.v, n.f = valueTypeFloat, h.DecodeFloat(false)
 	case 7:
-		vt, v = valueTypeString, h.DecodeString()
+		n.v, n.s = valueTypeString, h.DecodeString()
 	case 8:
-		vt, v = valueTypeBytes, h.B[h.m(len(h.B))]
+		n.v, n.l = valueTypeBytes, h.B[h.m(len(h.B))]
 	case 9:
-		vt, decodeFurther = valueTypeArray, true
+		n.v = valueTypeArray
 	case 10:
-		vt, decodeFurther = valueTypeMap, true
+		n.v = valueTypeMap
 	default:
-		vt, v = valueTypeExt, &RawExt{Tag: h.DecodeUint(64), Data: h.B[h.m(len(h.B))]}
+		n.v = valueTypeExt
+		n.u = h.DecodeUint(64)
+		n.l = h.B[h.m(len(h.B))]
 	}
-	h.ct = vt
+	h.ct = n.v
 	return
 }

codec/prebuild.sh

@@ -49,7 +49,8 @@ _build() {
         # [ -e "safe${_gg}" ] && mv safe${_gg} safe${_gg}__${_zts}.bak
         # [ -e "unsafe${_gg}" ] && mv unsafe${_gg} unsafe${_gg}__${_zts}.bak
     else 
-        rm -f fast-path.generated.go gen.generated.go gen-helper.generated.go *safe.generated.go *_generated_test.go *.generated_ffjson_expose.go
+        rm -f fast-path.generated.go gen.generated.go gen-helper.generated.go \
+           *safe.generated.go *_generated_test.go *.generated_ffjson_expose.go
     fi
 
     cat > gen.generated.go <<EOF
@@ -76,27 +77,6 @@ EOF
     cat >> gen.generated.go <<EOF
 \`
 
-EOF
-    # All functions, variables which must exist are put in this file.
-    # This way, build works before we generate the right things.
-    cat > fast-path.generated.go <<EOF
-package codec 
-import "reflect"
-// func GenBytesToStringRO(b []byte) string { return string(b) }
-func fastpathDecodeTypeSwitch(iv interface{}, d *Decoder) bool { return false }
-func fastpathEncodeTypeSwitch(iv interface{}, e *Encoder) bool { return false }
-func fastpathEncodeTypeSwitchSlice(iv interface{}, e *Encoder) bool { return false }
-func fastpathEncodeTypeSwitchMap(iv interface{}, e *Encoder) bool { return false }
-type fastpathE struct {
-	rtid uintptr
-	rt reflect.Type 
-	encfn func(*encFnInfo, reflect.Value)
-	decfn func(*decFnInfo, reflect.Value)
-}
-type fastpathA [0]fastpathE
-func (x fastpathA) index(rtid uintptr) int { return -1 }
-var fastpathAV fastpathA 
-
 EOF
 
     cat > gen-from-tmpl.codec.generated.go <<EOF
@@ -137,7 +117,7 @@ run("gen-helper.go.tmpl", "gen-helper.generated.go", false)
 }
 
 EOF
-    go run gen-from-tmpl.generated.go && \
+    go run -tags=notfastpath gen-from-tmpl.generated.go && \
         rm -f gen-from-tmpl.*generated.go 
 }
 
@@ -152,15 +132,18 @@ _codegenerators() {
         # Consequently, we should start msgp and ffjson first, and also put a small time latency before
         # starting codecgen.
         # Without this, ffjson chokes on one of the temporary files from codecgen.
-        echo "ffjson ... " && \
-            ffjson -w values_ffjson${zsfx} $zfin &
-        zzzIdFF=$!
-        echo "msgp ... " && \
-            msgp -tests=false -pkg=codec -o=values_msgp${zsfx} -file=$zfin &
-        zzzIdMsgp=$!
-
-        sleep 1 # give ffjson and msgp some buffer time. see note above.
-
+        if [[ $zexternal == "1" ]]
+        then 
+            echo "ffjson ... " && \
+                ffjson -w values_ffjson${zsfx} $zfin &
+            zzzIdFF=$!
+            echo "msgp ... " && \
+                msgp -tests=false -o=values_msgp${zsfx} -file=$zfin &
+            zzzIdMsgp=$!
+            
+            sleep 1 # give ffjson and msgp some buffer time. see note above.
+        fi
+        
         echo "codecgen - !unsafe ... " && \
             codecgen -rt codecgen -t 'x,codecgen,!unsafe' -o values_codecgen${zsfx} -d 19780 $zfin &
         zzzIdC=$!
@@ -169,8 +152,11 @@ _codegenerators() {
         zzzIdCU=$!
         wait $zzzIdC $zzzIdCU $zzzIdMsgp $zzzIdFF && \
             # remove (M|Unm)arshalJSON implementations, so they don't conflict with encoding/json bench \
-            sed -i 's+ MarshalJSON(+ _MarshalJSON(+g' values_ffjson${zsfx} && \
-            sed -i 's+ UnmarshalJSON(+ _UnmarshalJSON(+g' values_ffjson${zsfx} && \
+            if [[ $zexternal == "1" ]]
+            then
+                sed -i 's+ MarshalJSON(+ _MarshalJSON(+g' values_ffjson${zsfx} && \
+                    sed -i 's+ UnmarshalJSON(+ _UnmarshalJSON(+g' values_ffjson${zsfx}
+            fi && \
             echo "generators done!" && \
             true
     fi 
@@ -179,11 +165,12 @@ _codegenerators() {
 # _init reads the arguments and sets up the flags
 _init() {
 OPTIND=1
-while getopts "fb" flag
+while getopts "fbx" flag
 do
     case "x$flag" in 
         'xf') zforce=1;;
         'xb') zbak=1;;
+        'xx') zexternal=1;;
         *) echo "prebuild.sh accepts [-fb] only"; return 1;;
     esac
 done

codec/simple.go

@@ -29,12 +29,12 @@ const (
 )
 
 type simpleEncDriver struct {
+	noBuiltInTypes
+	encNoSeparator
 	e *Encoder
 	h *SimpleHandle
 	w encWriter
-	noBuiltInTypes
 	b [8]byte
-	encNoSeparator
 }
 
 func (e *simpleEncDriver) EncodeNil() {
@@ -153,7 +153,6 @@ type simpleDecDriver struct {
 	h      *SimpleHandle
 	r      decReader
 	bdRead bool
-	bdType valueType
 	bd     byte
 	br     bool // bytes reader
 	noBuiltInTypes
@@ -165,28 +164,27 @@ type simpleDecDriver struct {
 func (d *simpleDecDriver) readNextBd() {
 	d.bd = d.r.readn1()
 	d.bdRead = true
-	d.bdType = valueTypeUnset
-}
-
-func (d *simpleDecDriver) IsContainerType(vt valueType) bool {
-	switch vt {
-	case valueTypeNil:
-		return d.bd == simpleVdNil
-	case valueTypeBytes:
-		const x uint8 = simpleVdByteArray
-		return d.bd == x || d.bd == x+1 || d.bd == x+2 || d.bd == x+3 || d.bd == x+4
-	case valueTypeString:
-		const x uint8 = simpleVdString
-		return d.bd == x || d.bd == x+1 || d.bd == x+2 || d.bd == x+3 || d.bd == x+4
-	case valueTypeArray:
-		const x uint8 = simpleVdArray
-		return d.bd == x || d.bd == x+1 || d.bd == x+2 || d.bd == x+3 || d.bd == x+4
-	case valueTypeMap:
-		const x uint8 = simpleVdMap
-		return d.bd == x || d.bd == x+1 || d.bd == x+2 || d.bd == x+3 || d.bd == x+4
+}
+
+func (d *simpleDecDriver) ContainerType() (vt valueType) {
+	if d.bd == simpleVdNil {
+		return valueTypeNil
+	} else if d.bd == simpleVdByteArray || d.bd == simpleVdByteArray+1 ||
+		d.bd == simpleVdByteArray+2 || d.bd == simpleVdByteArray+3 || d.bd == simpleVdByteArray+4 {
+		return valueTypeBytes
+	} else if d.bd == simpleVdString || d.bd == simpleVdString+1 ||
+		d.bd == simpleVdString+2 || d.bd == simpleVdString+3 || d.bd == simpleVdString+4 {
+		return valueTypeString
+	} else if d.bd == simpleVdArray || d.bd == simpleVdArray+1 ||
+		d.bd == simpleVdArray+2 || d.bd == simpleVdArray+3 || d.bd == simpleVdArray+4 {
+		return valueTypeArray
+	} else if d.bd == simpleVdMap || d.bd == simpleVdMap+1 ||
+		d.bd == simpleVdMap+2 || d.bd == simpleVdMap+3 || d.bd == simpleVdMap+4 {
+		return valueTypeMap
+	} else {
+		// d.d.errorf("isContainerType: unsupported parameter: %v", vt)
 	}
-	d.d.errorf("isContainerType: unsupported parameter: %v", vt)
-	return false // "unreachable"
+	return valueTypeUnset
 }
 
 func (d *simpleDecDriver) TryDecodeAsNil() bool {
@@ -410,59 +408,59 @@ func (d *simpleDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs [
 	return
 }
 
-func (d *simpleDecDriver) DecodeNaked() (v interface{}, vt valueType, decodeFurther bool) {
+func (d *simpleDecDriver) DecodeNaked() {
 	if !d.bdRead {
 		d.readNextBd()
 	}
 
+	n := &d.d.n
+	var decodeFurther bool
+
 	switch d.bd {
 	case simpleVdNil:
-		vt = valueTypeNil
+		n.v = valueTypeNil
 	case simpleVdFalse:
-		vt = valueTypeBool
-		v = false
+		n.v = valueTypeBool
+		n.b = false
 	case simpleVdTrue:
-		vt = valueTypeBool
-		v = true
+		n.v = valueTypeBool
+		n.b = true
 	case simpleVdPosInt, simpleVdPosInt + 1, simpleVdPosInt + 2, simpleVdPosInt + 3:
 		if d.h.SignedInteger {
-			vt = valueTypeInt
-			v = d.DecodeInt(64)
+			n.v = valueTypeInt
+			n.i = d.DecodeInt(64)
 		} else {
-			vt = valueTypeUint
-			v = d.DecodeUint(64)
+			n.v = valueTypeUint
+			n.u = d.DecodeUint(64)
 		}
 	case simpleVdNegInt, simpleVdNegInt + 1, simpleVdNegInt + 2, simpleVdNegInt + 3:
-		vt = valueTypeInt
-		v = d.DecodeInt(64)
+		n.v = valueTypeInt
+		n.i = d.DecodeInt(64)
 	case simpleVdFloat32:
-		vt = valueTypeFloat
-		v = d.DecodeFloat(true)
+		n.v = valueTypeFloat
+		n.f = d.DecodeFloat(true)
 	case simpleVdFloat64:
-		vt = valueTypeFloat
-		v = d.DecodeFloat(false)
+		n.v = valueTypeFloat
+		n.f = d.DecodeFloat(false)
 	case simpleVdString, simpleVdString + 1, simpleVdString + 2, simpleVdString + 3, simpleVdString + 4:
-		vt = valueTypeString
-		v = d.DecodeString()
+		n.v = valueTypeString
+		n.s = d.DecodeString()
 	case simpleVdByteArray, simpleVdByteArray + 1, simpleVdByteArray + 2, simpleVdByteArray + 3, simpleVdByteArray + 4:
-		vt = valueTypeBytes
-		v = d.DecodeBytes(nil, false, false)
+		n.v = valueTypeBytes
+		n.l = d.DecodeBytes(nil, false, false)
 	case simpleVdExt, simpleVdExt + 1, simpleVdExt + 2, simpleVdExt + 3, simpleVdExt + 4:
-		vt = valueTypeExt
+		n.v = valueTypeExt
 		l := d.decLen()
-		var re RawExt
-		re.Tag = uint64(d.r.readn1())
-		re.Data = d.r.readx(l)
-		v = &re
+		n.u = uint64(d.r.readn1())
+		n.l = d.r.readx(l)
 	case simpleVdArray, simpleVdArray + 1, simpleVdArray + 2, simpleVdArray + 3, simpleVdArray + 4:
-		vt = valueTypeArray
+		n.v = valueTypeArray
 		decodeFurther = true
 	case simpleVdMap, simpleVdMap + 1, simpleVdMap + 2, simpleVdMap + 3, simpleVdMap + 4:
-		vt = valueTypeMap
+		n.v = valueTypeMap
 		decodeFurther = true
 	default:
 		d.d.errorf("decodeNaked: Unrecognized d.bd: 0x%x", d.bd)
-		return
 	}
 
 	if !decodeFurther {
@@ -496,6 +494,10 @@ type SimpleHandle struct {
 	binaryEncodingType
 }
 
+func (h *SimpleHandle) SetBytesExt(rt reflect.Type, tag uint64, ext BytesExt) (err error) {
+	return h.SetExt(rt, tag, &setExtWrapper{b: ext})
+}
+
 func (h *SimpleHandle) newEncDriver(e *Encoder) encDriver {
 	return &simpleEncDriver{e: e, w: e.w, h: h}
 }
@@ -504,8 +506,12 @@ func (h *SimpleHandle) newDecDriver(d *Decoder) decDriver {
 	return &simpleDecDriver{d: d, r: d.r, h: h, br: d.bytes}
 }
 
-func (h *SimpleHandle) SetBytesExt(rt reflect.Type, tag uint64, ext BytesExt) (err error) {
-	return h.SetExt(rt, tag, &setExtWrapper{b: ext})
+func (e *simpleEncDriver) reset() {
+	e.w = e.e.w
+}
+
+func (d *simpleDecDriver) reset() {
+	d.r = d.d.r
 }
 
 var _ decDriver = (*simpleDecDriver)(nil)

codec/tests.sh

@@ -4,22 +4,29 @@
 # This helps ensure that nothing gets broken.
 
 _run() {
-    # 1. VARIATIONS: regular (t), canonical (c), IO R/W (i), binc-nosymbols (n), struct2array (s)
-    # 2. MODE: reflection (r), codecgen (x), codecgen+unsafe (u)
+    # 1. VARIATIONS: regular (t), canonical (c), IO R/W (i),
+    #                binc-nosymbols (n), struct2array (s), intern string (e),
+    # 2. MODE: reflection (r), external (x), codecgen (g), unsafe (u), notfastpath (f)
+    # 3. OPTIONS: verbose (v), reset (z), must (m),
     # 
-    # Typically, you would run a combination of one value from a and b.
+    # Use combinations of mode to get exactly what you want,
+    # and then pass the variations you need.
 
     ztags=""
+    zargs=""
     local OPTIND 
     OPTIND=1
-    while getopts "xurtcinsvg" flag
+    while getopts "xurtcinsvgzmef" flag
     do
         case "x$flag" in 
             'xr')  ;;
+            'xf') ztags="$ztags notfastpath" ;;
             'xg') ztags="$ztags codecgen" ;;
             'xx') ztags="$ztags x" ;;
             'xu') ztags="$ztags unsafe" ;;
-            'xv') zverbose="-tv" ;; 
+            'xv') zargs="$zargs -tv" ;;
+            'xz') zargs="$zargs -tr" ;;
+            'xm') zargs="$zargs -tm" ;;
             *) ;;
         esac
     done
@@ -28,14 +35,15 @@ _run() {
     # echo ">>>>>>> TAGS: $ztags"
     
     OPTIND=1
-    while getopts "xurtcinsvg" flag
+    while getopts "xurtcinsvgzmef" flag
     do
         case "x$flag" in 
-            'xt') printf ">>>>>>> REGULAR    : "; go test "-tags=$ztags" "$zverbose" ; sleep 2 ;;
-            'xc') printf ">>>>>>> CANONICAL  : "; go test "-tags=$ztags" "$zverbose" -tc; sleep 2 ;;
-            'xi') printf ">>>>>>> I/O        : "; go test "-tags=$ztags" "$zverbose" -ti; sleep 2 ;;
-            'xn') printf ">>>>>>> NO_SYMBOLS : "; go test "-tags=$ztags" "$zverbose" -tn; sleep 2 ;;
-            'xs') printf ">>>>>>> TO_ARRAY   : "; go test "-tags=$ztags" "$zverbose" -ts; sleep 2 ;;
+            'xt') printf ">>>>>>> REGULAR    : "; go test "-tags=$ztags" $zargs ; sleep 2 ;;
+            'xc') printf ">>>>>>> CANONICAL  : "; go test "-tags=$ztags" $zargs -tc; sleep 2 ;;
+            'xi') printf ">>>>>>> I/O        : "; go test "-tags=$ztags" $zargs -ti; sleep 2 ;;
+            'xn') printf ">>>>>>> NO_SYMBOLS : "; go test "-tags=$ztags" $zargs -tn; sleep 2 ;;
+            'xs') printf ">>>>>>> TO_ARRAY   : "; go test "-tags=$ztags" $zargs -ts; sleep 2 ;;
+            'xe') printf ">>>>>>> INTERN     : "; go test "-tags=$ztags" $zargs -te; sleep 2 ;;
             *) ;;
         esac
     done
@@ -46,11 +54,21 @@ _run() {
 
 # echo ">>>>>>> RUNNING VARIATIONS OF TESTS"    
 if [[ "x$@" = "x" ]]; then
-    # r, x, g, gu
-    _run "-rtcins"
-    _run "-xtcins"
-    _run "-gtcins"
-    _run "-gutcins"
+    # All: r, x, g, gu
+    _run "-rtcinsm"  # regular
+    _run "-rtcinsmz" # regular with reset
+    _run "-rtcinsmf" # regular with no fastpath (notfastpath)
+    _run "-xtcinsm" # external
+    _run "-gxtcinsm" # codecgen: requires external
+    _run "-gxutcinsm" # codecgen + unsafe
+elif [[ "x$@" = "x-Z" ]]; then
+    # Regular
+    _run "-rtcinsm"  # regular
+    _run "-rtcinsmz" # regular with reset
+elif [[ "x$@" = "x-F" ]]; then
+    # regular with notfastpath
+    _run "-rtcinsmf"  # regular
+    _run "-rtcinsmzf" # regular with reset
 else
     _run "$@"
 fi

codec/time.go

@@ -4,6 +4,7 @@
 package codec
 
 import (
+	"fmt"
 	"time"
 )
 
@@ -11,6 +12,56 @@ var (
 	timeDigits = [...]byte{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'}
 )
 
+type timeExt struct{}
+
+func (x timeExt) WriteExt(v interface{}) (bs []byte) {
+	switch v2 := v.(type) {
+	case time.Time:
+		bs = encodeTime(v2)
+	case *time.Time:
+		bs = encodeTime(*v2)
+	default:
+		panic(fmt.Errorf("unsupported format for time conversion: expecting time.Time; got %T", v2))
+	}
+	return
+}
+func (x timeExt) ReadExt(v interface{}, bs []byte) {
+	tt, err := decodeTime(bs)
+	if err != nil {
+		panic(err)
+	}
+	*(v.(*time.Time)) = tt
+}
+
+func (x timeExt) ConvertExt(v interface{}) interface{} {
+	var bs []byte
+	switch v2 := v.(type) {
+	case time.Time:
+		bs = encodeTime(v2)
+	case *time.Time:
+		bs = encodeTime(*v2)
+	default:
+		panic(fmt.Errorf("unsupported format for time conversion: expecting time.Time; got %T", v2))
+	}
+	return bs
+}
+func (x timeExt) UpdateExt(v interface{}, src interface{}) {
+	var bs []byte
+	switch s2 := src.(type) {
+	case []byte:
+		bs = s2
+	case *[]byte:
+		bs = *s2
+	default:
+		panic(fmt.Errorf("unsupported format for time conversion: expecting []byte; got %T", s2))
+	}
+	tt, err := decodeTime(bs)
+	if err != nil {
+		panic(err)
+	}
+	*(v.(*time.Time)) = tt
+}
+
 // EncodeTime encodes a time.Time as a []byte, including
 // information on the instant in time and UTC offset.
 //