codec: Multiple fixes, enhancements and test expansion across the board.

Details below ...

Expanded Tests across the board, including but not limited to:
    - non-string keys
    - chan
    - swallow (decode into equivalent of /dev/null)
    - zero
    - raw ext
    - map keys that are structs
    - canonical
    - decoding nil map value
    - ...

Fixed issues:
    - expand set of options used in testing
    - use bitset(256|128) instead of a large array to test whether bits are set
    - allow decodeValue intelligently handle whether a function requires a pointer or not.
      This way, folks don't have to call Addr themselves.
    - use bounded arrays for the cache used during decodeNaked
    - decNaked: use an array of reflect.Value to pre-load all decNaked stuff
    - everyone calls into encodeValue or decodeValue (removing all cruft)
    - support recognizedRtid behind a flag
    - separate setting a value to its zero value, from decode.
    - remove many helpers (e.g. predecodevalue): now just a few methods do the right things
    - cleanup fast path: no tryNil argument needed, and the arg is either a pointer or a base value.
    - support arrays well in codecgen
    - full support for io.Reader passed to decode
    - msgpack: full support for strings read from a []byte

 codec: collection of correctness and simplification changes

    Move decNaked out of the Decoder struct. The Decoder struct was too big, and decNaked is only used
    if decoding into an empty interface{}. So make it an explicit request to grab a decNaked if doing DecodeNaked.

    Use a sync.Pool so that the decNaked's can be shared across multiple decodings.
    ----
    Add SliceElementReset as a decode option, to align with MapValueReset and InterfaceReset.

    This eliminates the resetSliceElemToZeroValue constant.
    ----
    Clean out comments and commented code blocks.
    ----
    Move the code for getting a codec function into a single place. Previously, there was much
    duplication of the same logic in encode.go and decode.go. Now, we put a single type "codecFner"
    that is a member element of Decoder and Encoder.

    This means that getEncFn and getDecFn go away, and fast-path looks different, because the definition
    of encfn and decfn have changed.

    This will allow us move the code to be a member of the Handle later on, which should be where it
    belongs.
    ----
    Use a single pool struct to consolidate how pools are used in the codebase,
    for decNaked, encStruct, etc.
    ----
    Reduce bounds-checking in json.go, by looking up an index in the string just once,
    and setting into a char variable. Helped shave much time from json decoding.
    ----

codec: add error field to (De|En)coder to flag such as unusable until reset

json: differentiate empty vs nil slice when encoding/decoding

clean up encoding of chans

codecgen: add support for NoExtensions

    Extensions support causes the receiver of CodecDecodeSelf to escape,
    which can have dramatic performance implications.

    Since extensions is a seldom used feature, it is not fair for everyone
    to pay the price.

add String method to valueType, for debugging

msgpack: support DecodeBytes where valueTypeBytes comes from a msgpack string container

cbor: fix extension encoding - only encode the Value component of RawExt

support new Decode Option: DeleteOnNilMapValue
    - DeleteOnNilMapValue controls how to decode a nil value in the stream.
    - add a test

Ensure T or ... *****T are recognized as Selfer during encode or decode
    - to do this, always pass checkSelfer=true when getting the function for encode/decode

separare values_test.go from values_flex_test.go

    Currently, we use the same file (values_test.go) for testing and benchmarks.

    However, our benchmarks compare codec against other (en|de)coders which are
    stricter and may support only maps with string keys.

    To support these 2 modes elegantly, we separate value_test.go, to be used by
    benchmarks, and values_flex_test.go to be used by codec tests.

    values_test.go will now be used for testing and benchmarks, but values_flex_test.go
    will only be used for testing.

Fix: ensure we encode separator before encoding nil in struct2array mode of encode struct

test: expand TestStruc to have fields of struct type with no omitempty.

    This makes the tests for omitempty more expansive, as the code takes 2 different
    paths if there are omitempty fields or not.

optimize structFieldInfo, struct field lookup and (en|de)coding with no omitempty

    - Use a fixed array for the structFieldInfo instead of a slice.
      It uses same amount of memory (roughly) while reducing the array bounds checking.
      Due to this, we no longer support embedding over 16 levels deep.
      In practice, this is not a concern.
    - Also, introduct StructFieldNode which allows us use a cache
      when iterating a Struct during encoding and decoding,
      so we don't lookup Value.Field, Value.Elem and Value.IsNil repeatedly.
    - Determine whether a typeInfo has any omitEmpty fields early, so we
      can take a fast-path if we no omitEmpty is not a concern.

add faster versions of readn3,4 and writen4,5 for use by json codec

kInterface MUST take a settable interface value. ensure kMap honors that.

decode: can decode into non-nil chan/slice/map/ptr or array, not just non-nil ptr

codec/0doc.go

@@ -31,8 +31,13 @@ the standard library (ie json, xml, gob, etc).
 Rich Feature Set includes:
 
   - Simple but extremely powerful and feature-rich API
+  - Support for go1.4 and above, while selectively using newer APIs for later releases
+  - Good code coverage ( > 70% )
   - Very High Performance.
     Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
+  - Careful selected use of 'unsafe' for targeted performance gains.
+    100% mode exists where 'unsafe' is not used at all.
+  - Lock-free (sans mutex) concurrency for scaling to 100's of cores
   - Multiple conversions:
     Package coerces types where appropriate
     e.g. decode an int in the stream into a float, etc.
@@ -156,40 +161,25 @@ Sample usage model:
     //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
     client := rpc.NewClientWithCodec(rpcCodec)
 
+Running Tests
+
+To run tests, use the following:
+
+    go test
+
+To run the full suite of tests, use the following:
+
+    go test -tags alltests -run Suite
+
+You can run the tag 'safe' to run tests or build in safe mode. e.g.
+
+    go test -tags safe -run Json
+    go test -tags "alltests safe" -run Suite
+
+Running Benchmarks
+
+Please see http://github.com/ugorji/go-codec-bench .
+
 */
 package codec
 
-// Benefits of go-codec:
-//
-//    - encoding/json always reads whole file into memory first.
-//      This makes it unsuitable for parsing very large files.
-//    - encoding/xml cannot parse into a map[string]interface{}
-//      I found this out on reading https://github.com/clbanning/mxj
-
-// TODO:
-//
-//   - optimization for codecgen:
-//     if len of entity is <= 3 words, then support a value receiver for encode.
-//   - (En|De)coder should store an error when it occurs.
-//     Until reset, subsequent calls return that error that was stored.
-//     This means that free panics must go away.
-//     All errors must be raised through errorf method.
-//   - Decoding using a chan is good, but incurs concurrency costs.
-//     This is because there's no fast way to use a channel without it
-//     having to switch goroutines constantly.
-//     Callback pattern is still the best. Maybe consider supporting something like:
-//        type X struct {
-//             Name string
-//             Ys []Y
-//             Ys chan <- Y
-//             Ys func(Y) -> call this function for each entry
-//        }
-//    - Consider adding a isZeroer interface { isZero() bool }
-//      It is used within isEmpty, for omitEmpty support.
-//    - Consider making Handle used AS-IS within the encoding/decoding session.
-//      This means that we don't cache Handle information within the (En|De)coder,
-//      except we really need it at Reset(...)
-//    - Consider adding math/big support
-//    - Consider reducing the size of the generated functions:
-//      Maybe use one loop, and put the conditionals in the loop.
-//      for ... { if cLen > 0 { if j == cLen { break } } else if dd.CheckBreak() { break } }

codec/binc.go

@@ -512,7 +512,7 @@ func (d *bincDecDriver) DecodeInt(bitsize uint8) (i int64) {
 		i = -i
 	}
 	if chkOvf.Int(i, bitsize) {
-		d.d.errorf("binc: overflow integer: %v", i)
+		d.d.errorf("binc: overflow integer: %v for num bits: %v", i, bitsize)
 		return
 	}
 	d.bdRead = false
@@ -804,7 +804,7 @@ func (d *bincDecDriver) DecodeNaked() {
 		d.readNextBd()
 	}
 
-	n := &d.d.n
+	n := d.d.n
 	var decodeFurther bool
 
 	switch d.vd {
@@ -923,6 +923,10 @@ func (h *BincHandle) newDecDriver(d *Decoder) decDriver {
 	return &bincDecDriver{d: d, h: h, r: d.r, br: d.bytes}
 }
 
+func (_ *BincHandle) IsBuiltinType(rt uintptr) bool {
+	return rt == timeTypId
+}
+
 func (e *bincEncDriver) reset() {
 	e.w = e.e.w
 	e.s = 0

codec/cbor.go

@@ -134,12 +134,12 @@ func (e *cborEncDriver) EncodeExt(rv interface{}, xtag uint64, ext Ext, en *Enco
 
 func (e *cborEncDriver) EncodeRawExt(re *RawExt, en *Encoder) {
 	e.encUint(uint64(re.Tag), cborBaseTag)
-	if re.Data != nil {
+	if false && re.Data != nil {
 		en.encode(re.Data)
-	} else if re.Value == nil {
-		e.EncodeNil()
-	} else {
+	} else if re.Value != nil {
 		en.encode(re.Value)
+	} else {
+		e.EncodeNil()
 	}
 }
 
@@ -469,7 +469,7 @@ func (d *cborDecDriver) DecodeNaked() {
 		d.readNextBd()
 	}
 
-	n := &d.d.n
+	n := d.d.n
 	var decodeFurther bool
 
 	switch d.bd {

codec/codec_test.go

@@ -42,6 +42,11 @@ import (
 
 func init() {
 	testPreInitFns = append(testPreInitFns, testInit)
+	fmt.Printf("sizeof: Decoder: %v, Encoder: %v, decNaked: %v\n",
+		reflect.TypeOf((*Decoder)(nil)).Elem().Size(),
+		reflect.TypeOf((*Encoder)(nil)).Elem().Size(),
+		reflect.TypeOf((*decNaked)(nil)).Elem().Size(),
+	)
 }
 
 // make this a mapbyslice
@@ -186,7 +191,6 @@ func (x *testUnixNanoTimeExt) ConvertExt(v interface{}) interface{} {
 	return &x.ts
 }
 func (x *testUnixNanoTimeExt) UpdateExt(dest interface{}, v interface{}) {
-	// fmt.Printf("testUnixNanoTimeExt.UpdateExt: v: %v\n", v)
 	tt := dest.(*time.Time)
 	switch v2 := v.(type) {
 	case int64:
@@ -202,7 +206,6 @@ func (x *testUnixNanoTimeExt) UpdateExt(dest interface{}, v interface{}) {
 	default:
 		panic(fmt.Sprintf("unsupported format for time conversion: expecting int64/uint64; got %T", v))
 	}
-	// fmt.Printf("testUnixNanoTimeExt.UpdateExt: v: %v, tt: %#v\n", v, tt)
 }
 
 func testVerifyVal(v interface{}, arg testVerifyArg) (v2 interface{}) {
@@ -324,14 +327,20 @@ func testVerifyVal(v interface{}, arg testVerifyArg) (v2 interface{}) {
 }
 
 func testInit() {
-	gob.Register(new(TestStruc))
+	gob.Register(new(TestStrucFlex))
 	if testInitDebug {
-		ts0 := newTestStruc(2, false, !testSkipIntf, false)
-		fmt.Printf("====> depth: %v, ts: %#v\n", 2, ts0)
+		ts0 := newTestStrucFlex(2, false, !testSkipIntf, false)
+		logT(nil, "====> depth: %v, ts: %#v\n", 2, ts0)
 	}
 
 	for _, v := range testHandles {
 		bh := v.getBasicHandle()
+		// pre-fill them first
+		bh.EncodeOptions = testEncodeOptions
+		bh.DecodeOptions = testDecodeOptions
+		// bh.InterfaceReset = true // TODO: remove
+		// bh.PreferArrayOverSlice = true // TODO: remove
+		// modify from flag'ish things
 		bh.InternString = testInternStr
 		bh.Canonical = testCanonical
 		bh.CheckCircularRef = testCheckCircRef
@@ -464,7 +473,7 @@ ugorji
 	table = append(table, primitives)
 	table = append(table, testMbsT(primitives))
 	table = append(table, maps...)
-	table = append(table, newTestStruc(0, false, !testSkipIntf, false))
+	table = append(table, newTestStrucFlex(0, false, !testSkipIntf, false))
 
 	tableVerify = make([]interface{}, len(table))
 	tableTestNilVerify = make([]interface{}, len(table))
@@ -561,10 +570,14 @@ func doTestCodecTableOne(t *testing.T, testNil bool, h Handle,
 		if err != nil {
 			continue
 		}
+		var b1 = b0
+		if len(b1) > 256 {
+			b1 = b1[:256]
+		}
 		if h.isBinary() {
-			logT(t, "         Encoded bytes: len: %v, %v\n", len(b0), b0)
+			logT(t, "         Encoded bytes: len: %v, %v\n", len(b0), b1)
 		} else {
-			logT(t, "         Encoded string: len: %v, %v\n", len(string(b0)), string(b0))
+			logT(t, "         Encoded string: len: %v, %v\n", len(b0), string(b1))
 			// println("########### encoded string: " + string(b0))
 		}
 		var v1 interface{}
@@ -574,10 +587,15 @@ func doTestCodecTableOne(t *testing.T, testNil bool, h Handle,
 		} else {
 			if v0 != nil {
 				v0rt := reflect.TypeOf(v0) // ptr
-				rv1 := reflect.New(v0rt)
-				err = testUnmarshal(rv1.Interface(), b0, h)
-				v1 = rv1.Elem().Interface()
-				// v1 = reflect.Indirect(reflect.ValueOf(v1)).Interface()
+				if v0rt.Kind() == reflect.Ptr {
+					err = testUnmarshal(v0, b0, h)
+					v1 = v0
+				} else {
+					rv1 := reflect.New(v0rt)
+					err = testUnmarshal(rv1.Interface(), b0, h)
+					v1 = rv1.Elem().Interface()
+					// v1 = reflect.Indirect(reflect.ValueOf(v1)).Interface()
+				}
 			}
 		}
 
@@ -603,7 +621,7 @@ func doTestCodecTableOne(t *testing.T, testNil bool, h Handle,
 		} else {
 			// logT(t, "-------- Before and After marshal do not match: Error: %v"+
 			// 	" ====> GOLDEN: (%T) %#v, DECODED: (%T) %#v\n", err, v0check, v0check, v1, v1)
-			logT(t, "-------- Before and After marshal do not match: Error: %v", err)
+			logT(t, "-------- FAIL: Before and After marshal do not match: Error: %v", err)
 			logT(t, "    ....... GOLDEN:  (%T) %#v", v0check, v0check)
 			logT(t, "    ....... DECODED: (%T) %#v", v1, v1)
 			failT(t)
@@ -624,6 +642,7 @@ func testCodecTableOne(t *testing.T, h Handle) {
 		var oldWriteExt, oldRawToString bool
 		oldWriteExt, v.WriteExt = v.WriteExt, true
 		oldRawToString, v.RawToString = v.RawToString, true
+		// defer func() { v.WriteExt, v.RawToString = oldWriteExt, oldRawToString }()
 		doTestCodecTableOne(t, false, h, table, tableVerify)
 		v.WriteExt, v.RawToString = oldWriteExt, oldRawToString
 	case *JsonHandle:
@@ -644,13 +663,11 @@ func testCodecTableOne(t *testing.T, h Handle) {
 	v := h.getBasicHandle()
 
 	oldMapType, v.MapType = v.MapType, testMapStrIntfTyp
-
+	// defer func() { v.MapType = oldMapType }()
 	//skip time.Time, []interface{} containing time.Time, last map, and newStruc
 	doTestCodecTableOne(t, true, h, table[:idxTime], tableTestNilVerify[:idxTime])
 	doTestCodecTableOne(t, true, h, table[idxMap:idxMap+numMap-1], tableTestNilVerify[idxMap:idxMap+numMap-1])
-
 	v.MapType = oldMapType
-
 	// func TestMsgpackNilIntf(t *testing.T) {
 
 	//do last map and newStruc
@@ -677,18 +694,22 @@ func testCodecMiscOne(t *testing.T, h Handle) {
 	}
 
 	// func TestMsgpackDecodePtr(t *testing.T) {
-	ts := newTestStruc(0, false, !testSkipIntf, false)
+	ts := newTestStrucFlex(testDepth, false, !testSkipIntf, false)
 	b, err = testMarshalErr(ts, h, t, "pointer-to-struct")
 	if len(b) < 40 {
 		logT(t, "------- Size must be > 40. Size: %d", len(b))
 		failT(t)
 	}
+	var b1 = b
+	if len(b1) > 256 {
+		b1 = b1[:256]
+	}
 	if h.isBinary() {
-		logT(t, "------- b: %v", b)
+		logT(t, "------- b: size: %v, value: %v", len(b), b1)
 	} else {
-		logT(t, "------- b: %s", b)
+		logT(t, "------- b: size: %v, value: %s", len(b), b1)
 	}
-	ts2 := new(TestStruc)
+	ts2 := new(TestStrucFlex)
 	err = testUnmarshalErr(ts2, b, h, t, "pointer-to-struct")
 	if ts2.I64 != math.MaxInt64*2/3 {
 		logT(t, "------- Unmarshal wrong. Expect I64 = 64. Got: %v", ts2.I64)
@@ -705,9 +726,10 @@ func testCodecMiscOne(t *testing.T, h Handle) {
 	err = testUnmarshalErr(&p2, bs, h, t, "&p2")
 
 	if m2["A"] != 2 || m2["B"] != 3 {
-		logT(t, "m2 not as expected: expecting: %v, got: %v", m, m2)
+		logT(t, "FAIL: m2 not as expected: expecting: %v, got: %v", m, m2)
 		failT(t)
 	}
+
 	// log("m: %v, m2: %v, p: %v, p2: %v", m, m2, p, p2)
 	checkEqualT(t, p, p2, "p=p2")
 	checkEqualT(t, m, m2, "m=m2")
@@ -759,7 +781,6 @@ func testCodecMiscOne(t *testing.T, h Handle) {
 		var tarr2 tarr
 		testUnmarshalErr(&tarr2, bs, h, t, "tarr2")
 		checkEqualT(t, tarr0, tarr2, "tarr0=tarr2")
-		// fmt.Printf(">>>> err: %v. tarr1: %v, tarr2: %v\n", err, tarr0, tarr2)
 	}
 
 	// test byte array, even if empty (msgpack only)
@@ -786,7 +807,6 @@ func testCodecEmbeddedPointer(t *testing.T, h Handle) {
 	var z Z = 4
 	x1 := &B{&z, &A{5}, 6}
 	bs, err := testMarshalErr(x1, h, t, "x1")
-	// fmt.Printf("buf: len(%v): %x\n", buf.Len(), buf.Bytes())
 	var x2 = new(B)
 	err = testUnmarshalErr(x2, bs, h, t, "x2")
 	err = checkEqualT(t, x1, x2, "x1=x2")
@@ -816,40 +836,115 @@ func testCodecUnderlyingType(t *testing.T, h Handle) {
 }
 
 func testCodecChan(t *testing.T, h Handle) {
+	testOnce.Do(testInitAll)
 	// - send a slice []*int64 (sl1) into an chan (ch1) with cap > len(s1)
 	// - encode ch1 as a stream array
 	// - decode a chan (ch2), with cap > len(s1) from the stream array
 	// - receive from ch2 into slice sl2
 	// - compare sl1 and sl2
 	// - do this for codecs: json, cbor (covers all types)
-	sl1 := make([]*int64, 4)
-	for i := range sl1 {
-		var j int64 = int64(i)
-		sl1[i] = &j
+
+	if true {
+		sl1 := make([]*int64, 4)
+		for i := range sl1 {
+			var j int64 = int64(i)
+			sl1[i] = &j
+		}
+		ch1 := make(chan *int64, 4)
+		for _, j := range sl1 {
+			ch1 <- j
+		}
+		var bs []byte
+		NewEncoderBytes(&bs, h).MustEncode(ch1)
+		ch2 := make(chan *int64, 8)
+		NewDecoderBytes(bs, h).MustDecode(&ch2)
+		close(ch2)
+		var sl2 []*int64
+		for j := range ch2 {
+			sl2 = append(sl2, j)
+		}
+		if err := deepEqual(sl1, sl2); err != nil {
+			logT(t, "FAIL: Not Match: %v; len: %v, %v", err, len(sl1), len(sl2))
+			failT(t)
+		}
+	}
+
+	if true {
+		type testBytesT []byte
+		sl1 := make([]testBytesT, 4)
+		for i := range sl1 {
+			var j = []byte(strings.Repeat(strconv.FormatInt(int64(i), 10), i))
+			sl1[i] = j
+		}
+		ch1 := make(chan testBytesT, 4)
+		for _, j := range sl1 {
+			ch1 <- j
+		}
+		var bs []byte
+		NewEncoderBytes(&bs, h).MustEncode(ch1)
+		ch2 := make(chan testBytesT, 8)
+		NewDecoderBytes(bs, h).MustDecode(&ch2)
+		close(ch2)
+		var sl2 []testBytesT
+		for j := range ch2 {
+			sl2 = append(sl2, j)
+		}
+		if err := deepEqual(sl1, sl2); err != nil {
+			logT(t, "FAIL: Not Match: %v; len: %v, %v", err, len(sl1), len(sl2))
+			failT(t)
+		}
 	}
-	ch1 := make(chan *int64, 4)
-	for _, j := range sl1 {
-		ch1 <- j
+	if true {
+		type testBytesT byte
+		sl1 := make([]testBytesT, 4)
+		for i := range sl1 {
+			var j = strconv.FormatInt(int64(i), 10)[0]
+			sl1[i] = testBytesT(j)
+		}
+		ch1 := make(chan testBytesT, 4)
+		for _, j := range sl1 {
+			ch1 <- j
+		}
+		var bs []byte
+		NewEncoderBytes(&bs, h).MustEncode(ch1)
+		ch2 := make(chan testBytesT, 8)
+		NewDecoderBytes(bs, h).MustDecode(&ch2)
+		close(ch2)
+		var sl2 []testBytesT
+		for j := range ch2 {
+			sl2 = append(sl2, j)
+		}
+		if err := deepEqual(sl1, sl2); err != nil {
+			logT(t, "FAIL: Not Match: %v; len: %v, %v", err, len(sl1), len(sl2))
+			failT(t)
+		}
 	}
-	var bs []byte
-	NewEncoderBytes(&bs, h).MustEncode(ch1)
-	// if !h.isBinary() {
-	// 	fmt.Printf("before: len(ch1): %v, bs: %s\n", len(ch1), bs)
-	// }
-	// var ch2 chan *int64 // this will block if json, etc.
-	ch2 := make(chan *int64, 8)
-	NewDecoderBytes(bs, h).MustDecode(&ch2)
-	// logT(t, "Len(ch2): %v", len(ch2))
-	// fmt.Printf("after:  len(ch2): %v, ch2: %v\n", len(ch2), ch2)
-	close(ch2)
-	var sl2 []*int64
-	for j := range ch2 {
-		sl2 = append(sl2, j)
-	}
-	if err := deepEqual(sl1, sl2); err != nil {
-		logT(t, "Not Match: %v; len: %v, %v", err, len(sl1), len(sl2))
-		failT(t)
+
+	if true {
+		sl1 := make([]byte, 4)
+		for i := range sl1 {
+			var j = strconv.FormatInt(int64(i), 10)[0]
+			sl1[i] = byte(j)
+		}
+		ch1 := make(chan byte, 4)
+		for _, j := range sl1 {
+			ch1 <- j
+		}
+		var bs []byte
+		NewEncoderBytes(&bs, h).MustEncode(ch1)
+		ch2 := make(chan byte, 8)
+		NewDecoderBytes(bs, h).MustDecode(&ch2)
+		close(ch2)
+		var sl2 []byte
+		for j := range ch2 {
+			sl2 = append(sl2, j)
+		}
+		if err := deepEqual(sl1, sl2); err != nil {
+			logT(t, "FAIL: Not Match: %v; len: %v, %v", err, len(sl1), len(sl2))
+			failT(t)
+		}
 	}
+
 }
 
 func testCodecRpcOne(t *testing.T, rr Rpc, h Handle, doRequest bool, exitSleepMs time.Duration,
@@ -918,7 +1013,6 @@ func testCodecRpcOne(t *testing.T, rr Rpc, h Handle, doRequest bool, exitSleepMs
 	connFn := func() (bs net.Conn) {
 		// log("calling f1")
 		bs, err2 := net.Dial(ln.Addr().Network(), ln.Addr().String())
-		//fmt.Printf("f1. bs: %v, err2: %v\n", bs, err2)
 		checkErrT(t, err2)
 		return
 	}
@@ -954,10 +1048,12 @@ func testCodecRpcOne(t *testing.T, rr Rpc, h Handle, doRequest bool, exitSleepMs
 }
 
 func doTestMapEncodeForCanonical(t *testing.T, name string, h Handle) {
-	v1 := map[string]interface{}{
-		"a": 1,
-		"b": "hello",
-		"c": map[string]interface{}{
+	testOnce.Do(testInitAll)
+	// println("doTestMapEncodeForCanonical")
+	v1 := map[stringUint64T]interface{}{
+		{"a", 1}: 1,
+		{"b", 2}: "hello",
+		{"c", 3}: map[string]interface{}{
 			"c/a": 1,
 			"c/b": "world",
 			"c/c": []int{1, 2, 3, 4},
@@ -985,7 +1081,7 @@ func doTestMapEncodeForCanonical(t *testing.T, name string, h Handle) {
 			},
 		},
 	}
-	var v2 map[string]interface{}
+	var v2 map[stringUint64T]interface{}
 	var b1, b2 []byte
 
 	// encode v1 into b1, decode b1 into v2, encode v2 into b2, compare b1 and b2
@@ -1009,6 +1105,7 @@ func doTestMapEncodeForCanonical(t *testing.T, name string, h Handle) {
 }
 
 func doTestStdEncIntf(t *testing.T, name string, h Handle) {
+	testOnce.Do(testInitAll)
 	args := [][2]interface{}{
 		{&TestABC{"A", "BB", "CCC"}, new(TestABC)},
 		{&TestABC2{"AAA", "BB", "C"}, new(TestABC2)},
@@ -1022,13 +1119,14 @@ func doTestStdEncIntf(t *testing.T, name string, h Handle) {
 		if err := deepEqual(a[0], a[1]); err == nil {
 			logT(t, "++++ Objects match")
 		} else {
-			logT(t, "---- Objects do not match: y1: %v, err: %v", a[1], err)
+			logT(t, "---- FAIL: Objects do not match: y1: %v, err: %v", a[1], err)
 			failT(t)
 		}
 	}
 }
 
 func doTestEncCircularRef(t *testing.T, name string, h Handle) {
+	testOnce.Do(testInitAll)
 	type T1 struct {
 		S string
 		B bool
@@ -1063,7 +1161,7 @@ func doTestEncCircularRef(t *testing.T, name string, h Handle) {
 	if x := err.Error(); strings.Contains(x, "circular") || strings.Contains(x, "cyclic") {
 		logT(t, "error detected as expected: %v", x)
 	} else {
-		logT(t, "error detected was not as expected: %v", x)
+		logT(t, "FAIL: error detected was not as expected: %v", x)
 		failT(t)
 	}
 }
@@ -1085,6 +1183,7 @@ type (
 )
 
 func doTestAnonCycle(t *testing.T, name string, h Handle) {
+	testOnce.Do(testInitAll)
 	var x TestAnonCycleT1
 	x.S = "hello"
 	x.TestAnonCycleT2.S2 = "hello.2"
@@ -1098,6 +1197,7 @@ func doTestAnonCycle(t *testing.T, name string, h Handle) {
 }
 
 func doTestJsonLargeInteger(t *testing.T, v interface{}, ias uint8) {
+	testOnce.Do(testInitAll)
 	logT(t, "Running doTestJsonLargeInteger: v: %#v, ias: %c", v, ias)
 	oldIAS := testJsonH.IntegerAsString
 	defer func() { testJsonH.IntegerAsString = oldIAS }()
@@ -1163,11 +1263,11 @@ func doTestJsonLargeInteger(t *testing.T, v interface{}, ias uint8) {
 			logT(t, "Expecting equal values from %s: got golden: %v, decoded: %v", b, v2, vu)
 			failT(t)
 		}
-		// fmt.Printf("%v: %s, decode: %d, bool: %v, equal_on_decode: %v\n", v, b, vu, vb, vu == v.(uint))
 	}
 }
 
 func doTestRawValue(t *testing.T, name string, h Handle) {
+	testOnce.Do(testInitAll)
 	bh := h.getBasicHandle()
 	if !bh.Raw {
 		bh.Raw = true
@@ -1208,6 +1308,7 @@ func doTestRawValue(t *testing.T, name string, h Handle) {
 // We keep this unexported here, and put actual test in ext_dep_test.go.
 // This way, it can be excluded by excluding file completely.
 func doTestPythonGenStreams(t *testing.T, name string, h Handle) {
+	testOnce.Do(testInitAll)
 	logT(t, "TestPythonGenStreams-%v", name)
 	tmpdir, err := ioutil.TempDir("", "golang-"+name+"-test")
 	if err != nil {
@@ -1264,7 +1365,7 @@ func doTestPythonGenStreams(t *testing.T, name string, h Handle) {
 		if err = deepEqual(v, v1); err == nil {
 			logT(t, "++++++++ Objects match: %T, %v", v, v)
 		} else {
-			logT(t, "-------- Objects do not match: %v. Source: %T. Decoded: %T", err, v, v1)
+			logT(t, "-------- FAIL: Objects do not match: %v. Source: %T. Decoded: %T", err, v, v1)
 			logT(t, "--------   GOLDEN: %#v", v)
 			// logT(t, "--------   DECODED: %#v <====> %#v", v1, reflect.Indirect(reflect.ValueOf(v1)).Interface())
 			logT(t, "--------   DECODED: %#v <====> %#v", v1, reflect.Indirect(reflect.ValueOf(v1)).Interface())
@@ -1279,7 +1380,7 @@ func doTestPythonGenStreams(t *testing.T, name string, h Handle) {
 		if err = deepEqual(bsb, bss); err == nil {
 			logT(t, "++++++++ Bytes match")
 		} else {
-			logT(t, "???????? Bytes do not match. %v.", err)
+			logT(t, "???????? FAIL: Bytes do not match. %v.", err)
 			xs := "--------"
 			if reflect.ValueOf(v).Kind() == reflect.Map {
 				xs = "        "
@@ -1308,6 +1409,7 @@ func doTestMsgpackRpcSpecGoClientToPythonSvc(t *testing.T) {
 	if testSkipRPCTests {
 		return
 	}
+	testOnce.Do(testInitAll)
 	// openPorts are between 6700 and 6800
 	r := rand.New(rand.NewSource(time.Now().UnixNano()))
 	openPort := strconv.FormatInt(6700+r.Int63n(99), 10)
@@ -1336,6 +1438,7 @@ func doTestMsgpackRpcSpecPythonClientToGoSvc(t *testing.T) {
 	if testSkipRPCTests {
 		return
 	}
+	testOnce.Do(testInitAll)
 	port := testCodecRpcOne(t, MsgpackSpecRpc, testMsgpackH, false, 1*time.Second)
 	//time.Sleep(1000 * time.Millisecond)
 	cmd := exec.Command("python", "test.py", "rpc-client-go-service", strconv.Itoa(port))
@@ -1350,7 +1453,126 @@ func doTestMsgpackRpcSpecPythonClientToGoSvc(t *testing.T) {
 		fmt.Sprintf("%#v\n%#v\n", []string{"A1", "B2", "C3"}, TestRpcABC{"Aa", "Bb", "Cc"}), "cmdout=")
 }
 
+func doTestSwallowAndZero(t *testing.T, h Handle) {
+	testOnce.Do(testInitAll)
+	v1 := newTestStrucFlex(testDepth, false, false, false)
+	var b1 []byte
+
+	e1 := NewEncoderBytes(&b1, h)
+	e1.MustEncode(v1)
+	d1 := NewDecoderBytes(b1, h)
+	d1.swallow()
+	if d1.r.numread() != len(b1) {
+		logT(t, "swallow didn't consume all encoded bytes: %v out of %v", d1.r.numread(), len(b1))
+		failT(t)
+	}
+	d1.setZero(v1)
+	testDeepEqualErr(v1, &TestStrucFlex{}, t, "filled-and-zeroed")
+}
+
+func doTestRawExt(t *testing.T, h Handle) {
+	testOnce.Do(testInitAll)
+	// return // TODO: need to fix this ...
+	var b []byte
+	var v interface{}
+	_, isJson := h.(*JsonHandle)
+	_, isCbor := h.(*CborHandle)
+	isValuer := isJson || isCbor
+	_ = isValuer
+	for _, r := range []RawExt{
+		{Tag: 99, Value: "9999", Data: []byte("9999")},
+	} {
+		e := NewEncoderBytes(&b, h)
+		e.MustEncode(&r)
+		d := NewDecoderBytes(b, h)
+		d.MustDecode(&v)
+		switch h.(type) {
+		case *JsonHandle:
+			testDeepEqualErr(r.Value, v, t, "rawext-json")
+		default:
+			r2 := r
+			if isValuer {
+				r2.Data = nil
+			} else {
+				r2.Value = nil
+			}
+			testDeepEqualErr(v, r2, t, "rawext-default")
+		}
+	}
+}
+
+func doTestMapStructKey(t *testing.T, h Handle) {
+	testOnce.Do(testInitAll)
+	var b []byte
+	var v interface{} // map[stringUint64T]wrapUint64Slice // interface{}
+	bh := h.getBasicHandle()
+	m := map[stringUint64T]wrapUint64Slice{
+		{"55555", 55555}: []wrapUint64{12345},
+		{"333", 333}:     []wrapUint64{123},
+	}
+	oldCanonical := bh.Canonical
+	oldMapType := bh.MapType
+	defer func() {
+		bh.Canonical = oldCanonical
+		bh.MapType = oldMapType
+	}()
+
+	bh.MapType = reflect.TypeOf((*map[stringUint64T]wrapUint64Slice)(nil)).Elem()
+	for _, bv := range [2]bool{true, false} {
+		b, v = nil, nil
+		bh.Canonical = bv
+		e := NewEncoderBytes(&b, h)
+		e.MustEncode(m)
+		d := NewDecoderBytes(b, h)
+		d.MustDecode(&v)
+		testDeepEqualErr(v, m, t, "map-structkey")
+	}
+}
+
+func doTestDecodeNilMapValue(t *testing.T, handle Handle) {
+	type Struct struct {
+		Field map[uint16]map[uint32]struct{}
+	}
+
+	bh := handle.getBasicHandle()
+	oldMapType := bh.MapType
+	oldDeleteOnNilMapValue := bh.DeleteOnNilMapValue
+	defer func() {
+		bh.MapType = oldMapType
+		bh.DeleteOnNilMapValue = oldDeleteOnNilMapValue
+	}()
+	bh.MapType = reflect.TypeOf(map[interface{}]interface{}(nil))
+	bh.DeleteOnNilMapValue = false
+
+	_, isJsonHandle := handle.(*JsonHandle)
+
+	toEncode := Struct{Field: map[uint16]map[uint32]struct{}{
+		1: nil,
+	}}
+
+	bs, err := testMarshal(toEncode, handle)
+	if err != nil {
+		logT(t, "Error encoding: %v, Err: %v", toEncode, err)
+		failT(t)
+	}
+	if isJsonHandle {
+		logT(t, "json encoded: %s\n", bs)
+	}
+
+	var decoded Struct
+	err = testUnmarshal(&decoded, bs, handle)
+	if err != nil {
+		logT(t, "Error decoding: %v", err)
+		failT(t)
+	}
+	if !reflect.DeepEqual(decoded, toEncode) {
+		logT(t, "Decoded value %#v != %#v", decoded, toEncode)
+		failT(t)
+	}
+}
+
 func testRandomFillRV(v reflect.Value) {
+	testOnce.Do(testInitAll)
 	fneg := func() int64 {
 		i := rand.Intn(1)
 		if i == 1 {
@@ -1428,7 +1650,6 @@ func testMammoth(t *testing.T, name string, h Handle) {
 	testRandomFillRV(reflect.ValueOf(&m).Elem())
 	b, _ := testMarshalErr(&m, h, t, "mammoth-"+name)
 	testUnmarshalErr(&m2, b, h, t, "mammoth-"+name)
-	// fmt.Printf("m2: %v", &m2)
 	testDeepEqualErr(&m, &m2, t, "mammoth-"+name)
 }
 
@@ -1603,6 +1824,86 @@ func TestBincUnderlyingType(t *testing.T) {
 	testCodecUnderlyingType(t, testBincH)
 }
 
+func TestJsonSwallowAndZero(t *testing.T) {
+	doTestSwallowAndZero(t, testJsonH)
+}
+
+func TestCborSwallowAndZero(t *testing.T) {
+	doTestSwallowAndZero(t, testCborH)
+}
+
+func TestMsgpackSwallowAndZero(t *testing.T) {
+	doTestSwallowAndZero(t, testMsgpackH)
+}
+
+func TestBincSwallowAndZero(t *testing.T) {
+	doTestSwallowAndZero(t, testBincH)
+}
+
+func TestSimpleSwallowAndZero(t *testing.T) {
+	doTestSwallowAndZero(t, testSimpleH)
+}
+
+func TestJsonRawExt(t *testing.T) {
+	doTestRawExt(t, testJsonH)
+}
+
+func TestCborRawExt(t *testing.T) {
+	doTestRawExt(t, testCborH)
+}
+
+func TestMsgpackRawExt(t *testing.T) {
+	doTestRawExt(t, testMsgpackH)
+}
+
+func TestBincRawExt(t *testing.T) {
+	doTestRawExt(t, testBincH)
+}
+
+func TestSimpleRawExt(t *testing.T) {
+	doTestRawExt(t, testSimpleH)
+}
+
+func TestJsonMapStructKey(t *testing.T) {
+	doTestMapStructKey(t, testJsonH)
+}
+
+func TestCborMapStructKey(t *testing.T) {
+	doTestMapStructKey(t, testCborH)
+}
+
+func TestMsgpackMapStructKey(t *testing.T) {
+	doTestMapStructKey(t, testMsgpackH)
+}
+
+func TestBincMapStructKey(t *testing.T) {
+	doTestMapStructKey(t, testBincH)
+}
+
+func TestSimpleMapStructKey(t *testing.T) {
+	doTestMapStructKey(t, testSimpleH)
+}
+
+func TestJsonDecodeNilMapValue(t *testing.T) {
+	doTestDecodeNilMapValue(t, testJsonH)
+}
+
+func TestCborDecodeNilMapValue(t *testing.T) {
+	doTestDecodeNilMapValue(t, testCborH)
+}
+
+func TestMsgpackDecodeNilMapValue(t *testing.T) {
+	doTestDecodeNilMapValue(t, testMsgpackH)
+}
+
+func TestBincDecodeNilMapValue(t *testing.T) {
+	doTestDecodeNilMapValue(t, testBincH)
+}
+
+func TestSimpleDecodeNilMapValue(t *testing.T) {
+	doTestDecodeNilMapValue(t, testSimpleH)
+}
+
 func TestJsonLargeInteger(t *testing.T) {
 	for _, i := range []uint8{'L', 'A', 0} {
 		for _, j := range []interface{}{

codec/codecgen/gen.go

@@ -62,7 +62,7 @@ func CodecGenTempWrite{{ .RandString }}() {
 	var t{{ $index }} {{ . }}
 	typs = append(typs, reflect.TypeOf(t{{ $index }}))
 {{ end }}
-	{{ if not .CodecPkgFiles }}{{ .CodecPkgName }}.{{ end }}Gen(&out, "{{ .BuildTag }}", "{{ .PackageName }}", "{{ .RandString }}", {{ if not .CodecPkgFiles }}{{ .CodecPkgName }}.{{ end }}NewTypeInfos(strings.Split("{{ .StructTags }}", ",")), typs...)
+	{{ if not .CodecPkgFiles }}{{ .CodecPkgName }}.{{ end }}Gen(&out, "{{ .BuildTag }}", "{{ .PackageName }}", "{{ .RandString }}", {{ .NoExtensions }}, {{ if not .CodecPkgFiles }}{{ .CodecPkgName }}.{{ end }}NewTypeInfos(strings.Split("{{ .StructTags }}", ",")), typs...)
 	bout, err := format.Source(out.Bytes())
 	if err != nil {
 		fout.Write(out.Bytes())
@@ -82,8 +82,12 @@ func CodecGenTempWrite{{ .RandString }}() {
 // Tool then executes: "go run __frun__" which creates fout.
 // fout contains Codec(En|De)codeSelf implementations for every type T.
 //
-func Generate(outfile, buildTag, codecPkgPath string, uid int64, goRunTag string,
-	st string, regexName *regexp.Regexp, notRegexName *regexp.Regexp, deleteTempFile bool, infiles ...string) (err error) {
+func Generate(outfile, buildTag, codecPkgPath string,
+	uid int64,
+	goRunTag string, st string,
+	regexName, notRegexName *regexp.Regexp,
+	deleteTempFile, noExtensions bool,
+	infiles ...string) (err error) {
 	// For each file, grab AST, find each type, and write a call to it.
 	if len(infiles) == 0 {
 		return
@@ -121,6 +125,7 @@ func Generate(outfile, buildTag, codecPkgPath string, uid int64, goRunTag string
 		StructTags      string
 		Types           []string
 		CodecPkgFiles   bool
+		NoExtensions    bool
 	}
 	tv := tmplT{
 		CodecPkgName:    genCodecPkg,
@@ -129,6 +134,7 @@ func Generate(outfile, buildTag, codecPkgPath string, uid int64, goRunTag string
 		BuildTag:        buildTag,
 		RandString:      strconv.FormatInt(uid, 10),
 		StructTags:      st,
+		NoExtensions:    noExtensions,
 	}
 	tv.ImportPath = pkg.ImportPath
 	if tv.ImportPath == tv.CodecImportPath {
@@ -313,9 +319,12 @@ func main() {
 	x := flag.Bool("x", false, "keep temp file")
 	_ = flag.Bool("u", false, "*IGNORED - kept for backwards compatibility*: Allow unsafe use")
 	d := flag.Int64("d", 0, "random identifier for use in generated code")
+	nx := flag.Bool("nx", false, "no extensions")
+
 	flag.Parse()
 	if err := Generate(*o, *t, *c, *d, *rt, *st,
-		regexp.MustCompile(*r), regexp.MustCompile(*nr), !*x, flag.Args()...); err != nil {
+		regexp.MustCompile(*r), regexp.MustCompile(*nr), !*x, *nx,
+		flag.Args()...); err != nil {
 		fmt.Fprintf(os.Stderr, "codecgen error: %v\n", err)
 		os.Exit(1)
 	}

codec/codecgen_test.go

@@ -1,24 +0,0 @@
-// +build x,codecgen
-
-package codec
-
-import (
-	"fmt"
-	"testing"
-)
-
-func _TestCodecgenJson1(t *testing.T) {
-	// This is just a simplistic test for codecgen.
-	// It is typically disabled. We only enable it for debugging purposes.
-	const callCodecgenDirect bool = true
-	v := newTestStruc(2, false, !testSkipIntf, false)
-	var bs []byte
-	e := NewEncoderBytes(&bs, testJsonH)
-	if callCodecgenDirect {
-		v.CodecEncodeSelf(e)
-		e.w.atEndOfEncode()
-	} else {
-		e.MustEncode(v)
-	}
-	fmt.Printf("%s\n", bs)
-}

codec/fast-path.go.tmpl

@@ -40,9 +40,6 @@ import (
 
 const fastpathEnabled = true
 
-const fastpathCheckNilFalse = false // for reflect
-const fastpathCheckNilTrue = true // for type switch
-
 type fastpathT struct {}
 
 var fastpathTV fastpathT
@@ -50,8 +47,8 @@ var fastpathTV fastpathT
 type fastpathE struct {
 	rtid uintptr
 	rt reflect.Type 
-	encfn func(*encFnInfo, reflect.Value)
-	decfn func(*decFnInfo, reflect.Value)
+	encfn func(*Encoder, *codecFnInfo, reflect.Value)
+	decfn func(*Decoder, *codecFnInfo, reflect.Value)
 }
 
 type fastpathA [{{ .FastpathLen }}]fastpathE
@@ -84,19 +81,25 @@ var fastpathAV fastpathA
 // due to possible initialization loop error, make fastpath in an init()
 func init() {
 	i := 0
-	fn := func(v interface{}, fe func(*encFnInfo, reflect.Value), fd func(*decFnInfo, reflect.Value)) (f fastpathE) {
+	fn := func(v interface{},
+		fe func(*Encoder, *codecFnInfo, reflect.Value),
+		fd func(*Decoder, *codecFnInfo, reflect.Value)) (f fastpathE) {
 		xrt := reflect.TypeOf(v)
 		xptr := rt2id(xrt)
+		if useLookupRecognizedTypes {
+			recognizedRtids = append(recognizedRtids, xptr)
+			recognizedRtidPtrs = append(recognizedRtidPtrs, rt2id(reflect.PtrTo(xrt)))
+		}
 		fastpathAV[i] = fastpathE{xptr, xrt, fe, fd}
 		i++
 		return
 	}
 	
 	{{range .Values}}{{if not .Primitive}}{{if not .MapKey }}
-	fn([]{{ .Elem }}(nil), (*encFnInfo).{{ .MethodNamePfx "fastpathEnc" false }}R, (*decFnInfo).{{ .MethodNamePfx "fastpathDec" false }}R){{end}}{{end}}{{end}}
+	fn([]{{ .Elem }}(nil), (*Encoder).{{ .MethodNamePfx "fastpathEnc" false }}R, (*Decoder).{{ .MethodNamePfx "fastpathDec" false }}R){{end}}{{end}}{{end}}
 	
 	{{range .Values}}{{if not .Primitive}}{{if .MapKey }}
-	fn(map[{{ .MapKey }}]{{ .Elem }}(nil), (*encFnInfo).{{ .MethodNamePfx "fastpathEnc" false }}R, (*decFnInfo).{{ .MethodNamePfx "fastpathDec" false }}R){{end}}{{end}}{{end}}
+	fn(map[{{ .MapKey }}]{{ .Elem }}(nil), (*Encoder).{{ .MethodNamePfx "fastpathEnc" false }}R, (*Decoder).{{ .MethodNamePfx "fastpathDec" false }}R){{end}}{{end}}{{end}}
 	
 	sort.Sort(fastpathAslice(fastpathAV[:]))
 }
@@ -109,10 +112,10 @@ func fastpathEncodeTypeSwitch(iv interface{}, e *Encoder) bool {
 {{range .Values}}{{if not .Primitive}}{{if not .MapKey }}
 	case []{{ .Elem }}:{{else}}
 	case map[{{ .MapKey }}]{{ .Elem }}:{{end}}
-		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(v, fastpathCheckNilTrue, e){{if not .MapKey }}
+		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(v, e){{if not .MapKey }}
 	case *[]{{ .Elem }}:{{else}}
 	case *map[{{ .MapKey }}]{{ .Elem }}:{{end}}
-		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, fastpathCheckNilTrue, e)
+		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, e)
 {{end}}{{end}}
 	default:
         _ = v // TODO: workaround https://github.com/golang/go/issues/12927 (remove after go 1.6 release)
@@ -125,9 +128,9 @@ func fastpathEncodeTypeSwitchSlice(iv interface{}, e *Encoder) bool {
 	switch v := iv.(type) {
 {{range .Values}}{{if not .Primitive}}{{if not .MapKey }}
 	case []{{ .Elem }}:
-		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(v, fastpathCheckNilTrue, e)
+		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(v, e)
 	case *[]{{ .Elem }}:
-		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, fastpathCheckNilTrue, e)
+		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, e)
 {{end}}{{end}}{{end}}
 	default:
         _ = v // TODO: workaround https://github.com/golang/go/issues/12927 (remove after go 1.6 release)
@@ -140,9 +143,9 @@ func fastpathEncodeTypeSwitchMap(iv interface{}, e *Encoder) bool {
 	switch v := iv.(type) {
 {{range .Values}}{{if not .Primitive}}{{if .MapKey }}
 	case map[{{ .MapKey }}]{{ .Elem }}:
-		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(v, fastpathCheckNilTrue, e)
+		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(v, e)
 	case *map[{{ .MapKey }}]{{ .Elem }}:
-		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, fastpathCheckNilTrue, e)
+		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(*v, e)
 {{end}}{{end}}{{end}}
 	default:
         _ = v // TODO: workaround https://github.com/golang/go/issues/12927 (remove after go 1.6 release)
@@ -154,20 +157,15 @@ func fastpathEncodeTypeSwitchMap(iv interface{}, e *Encoder) bool {
 // -- -- fast path functions
 {{range .Values}}{{if not .Primitive}}{{if not .MapKey }} 
 
-func (f *encFnInfo) {{ .MethodNamePfx "fastpathEnc" false }}R(rv reflect.Value) {
+func (e *Encoder) {{ .MethodNamePfx "fastpathEnc" false }}R(f *codecFnInfo, rv reflect.Value) {
 	if f.ti.mbs {
-		fastpathTV.{{ .MethodNamePfx "EncAsMap" false }}V(rv2i(rv).([]{{ .Elem }}), fastpathCheckNilFalse, f.e)
+		fastpathTV.{{ .MethodNamePfx "EncAsMap" false }}V(rv2i(rv).([]{{ .Elem }}), e)
 	} else {
-		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(rv2i(rv).([]{{ .Elem }}), fastpathCheckNilFalse, f.e)
+		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(rv2i(rv).([]{{ .Elem }}), e)
 	}
 }
-func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v []{{ .Elem }}, checkNil bool, e *Encoder) {
-	ee := e.e
-	cr := e.cr
-	if checkNil && v == nil {
-		ee.EncodeNil()
-		return
-	}
+func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v []{{ .Elem }}, e *Encoder) {
+	ee, cr := e.e, e.cr 
 	ee.EncodeArrayStart(len(v))
 	for _, v2 := range v {
 		if cr != nil { cr.sendContainerState(containerArrayElem) }
@@ -176,13 +174,8 @@ func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v []{{ .Elem }}, checkNil b
 	if cr != nil { cr.sendContainerState(containerArrayEnd) }{{/* ee.EncodeEnd() */}}
 }
 
-func (_ fastpathT) {{ .MethodNamePfx "EncAsMap" false }}V(v []{{ .Elem }}, checkNil bool, e *Encoder) {
-	ee := e.e
-	cr := e.cr
-	if checkNil && v == nil {
-		ee.EncodeNil()
-		return
-	}
+func (_ fastpathT) {{ .MethodNamePfx "EncAsMap" false }}V(v []{{ .Elem }}, e *Encoder) {
+	ee, cr := e.e, e.cr 
 	if len(v)%2 == 1 {
 		e.errorf("mapBySlice requires even slice length, but got %v", len(v))
 		return
@@ -205,16 +198,11 @@ func (_ fastpathT) {{ .MethodNamePfx "EncAsMap" false }}V(v []{{ .Elem }}, check
 
 {{range .Values}}{{if not .Primitive}}{{if .MapKey }}
 
-func (f *encFnInfo) {{ .MethodNamePfx "fastpathEnc" false }}R(rv reflect.Value) {
-	fastpathTV.{{ .MethodNamePfx "Enc" false }}V(rv2i(rv).(map[{{ .MapKey }}]{{ .Elem }}), fastpathCheckNilFalse, f.e)
+func (e *Encoder) {{ .MethodNamePfx "fastpathEnc" false }}R(f *codecFnInfo, rv reflect.Value) {
+	fastpathTV.{{ .MethodNamePfx "Enc" false }}V(rv2i(rv).(map[{{ .MapKey }}]{{ .Elem }}), e)
 }
-func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v map[{{ .MapKey }}]{{ .Elem }}, checkNil bool, e *Encoder) {
-	ee := e.e
-	cr := e.cr
-	if checkNil && v == nil {
-		ee.EncodeNil()
-		return
-	}
+func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v map[{{ .MapKey }}]{{ .Elem }}, e *Encoder) {
+	ee, cr := e.e, e.cr 
 	ee.EncodeMapStart(len(v))
 	{{if eq .MapKey "string"}}asSymbols := e.h.AsSymbols&AsSymbolMapStringKeysFlag != 0
 	{{end}}if e.h.Canonical {
@@ -280,11 +268,10 @@ func fastpathDecodeTypeSwitch(iv interface{}, d *Decoder) bool {
 {{range .Values}}{{if not .Primitive}}{{if not .MapKey }}
 	case []{{ .Elem }}:{{else}}
 	case map[{{ .MapKey }}]{{ .Elem }}:{{end}}
-		fastpathTV.{{ .MethodNamePfx "Dec" false }}V(v, fastpathCheckNilFalse, false, d){{if not .MapKey }}
-	case *[]{{ .Elem }}:{{else}}
-	case *map[{{ .MapKey }}]{{ .Elem }}:{{end}}
-		v2, changed2 := fastpathTV.{{ .MethodNamePfx "Dec" false }}V(*v, fastpathCheckNilFalse, true, d)
-		if changed2 {
+		fastpathTV.{{ .MethodNamePfx "Dec" false }}V(v, false, d){{if not .MapKey }}
+	case *[]{{ .Elem }}: {{else}}
+	case *map[{{ .MapKey }}]{{ .Elem }}: {{end}}
+		if v2, changed2 := fastpathTV.{{ .MethodNamePfx "Dec" false }}V(*v, true, d); changed2 {
 			*v = v2 
 		}
 {{end}}{{end}}
@@ -295,6 +282,20 @@ func fastpathDecodeTypeSwitch(iv interface{}, d *Decoder) bool {
 	return true
 }
 
+func fastpathDecodeSetZeroTypeSwitch(iv interface{}, d *Decoder) bool {
+	switch v := iv.(type) {
+{{range .Values}}{{if not .Primitive}}{{if not .MapKey }}
+	case *[]{{ .Elem }}: {{else}}
+	case *map[{{ .MapKey }}]{{ .Elem }}: {{end}}
+		*v = nil 
+{{end}}{{end}}
+	default:
+        _ = v // TODO: workaround https://github.com/golang/go/issues/12927 (remove after go 1.6 release)
+		return false
+	}
+	return true
+}
+
 // -- -- fast path functions
 {{range .Values}}{{if not .Primitive}}{{if not .MapKey }}
 {{/*
@@ -303,33 +304,25 @@ Slices can change if they
 - are addressable (from a ptr)
 - are settable (e.g. contained in an interface{})
 */}}
-func (f *decFnInfo) {{ .MethodNamePfx "fastpathDec" false }}R(rv reflect.Value) { 
-	array := f.seq == seqTypeArray
-	if !array && rv.CanAddr() { {{/* // CanSet => CanAddr + Exported */}}
-		vp := rv2i(rv.Addr()).(*[]{{ .Elem }})
-		v, changed := fastpathTV.{{ .MethodNamePfx "Dec" false }}V(*vp, fastpathCheckNilFalse, !array, f.d)
-		if changed {
-			*vp = v
+func (d *Decoder) {{ .MethodNamePfx "fastpathDec" false }}R(f *codecFnInfo, rv reflect.Value) { 
+	if array := f.seq == seqTypeArray; !array && rv.Kind() == reflect.Ptr {
+		var vp = rv2i(rv).(*[]{{ .Elem }})
+		if v, changed := fastpathTV.{{ .MethodNamePfx "Dec" false }}V(*vp, !array, d); changed {
+				*vp = v
 		}
 	} else {
-		v := rv2i(rv).([]{{ .Elem }})
-		fastpathTV.{{ .MethodNamePfx "Dec" false }}V(v, fastpathCheckNilFalse, false, f.d)
+		fastpathTV.{{ .MethodNamePfx "Dec" false }}V(rv2i(rv).([]{{ .Elem }}), !array, d)
 	}
 }
 
-func (f fastpathT) {{ .MethodNamePfx "Dec" false }}X(vp *[]{{ .Elem }}, checkNil bool, d *Decoder) {
-	v, changed := f.{{ .MethodNamePfx "Dec" false }}V(*vp, checkNil, true, d)
-	if changed {
+func (f fastpathT) {{ .MethodNamePfx "Dec" false }}X(vp *[]{{ .Elem }}, d *Decoder) {
+	if v, changed := f.{{ .MethodNamePfx "Dec" false }}V(*vp, true, d); changed {
 		*vp = v 
 	}
 }
-func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v []{{ .Elem }}, checkNil bool, canChange bool, d *Decoder) (_ []{{ .Elem }}, changed bool) {
+func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v []{{ .Elem }}, canChange bool, d *Decoder) (_ []{{ .Elem }}, changed bool) {
 	dd := d.d
 	{{/* // if dd.isContainerType(valueTypeNil) { dd.TryDecodeAsNil() */}}
-	if checkNil && dd.TryDecodeAsNil() {
-		if v != nil { changed = true }
-		return nil, changed 
-	}
 	slh, containerLenS := d.decSliceHelperStart()
 	if containerLenS == 0 {
 		if canChange {
@@ -411,33 +404,25 @@ Maps can change if they are
 - addressable (from a ptr)
 - settable (e.g. contained in an interface{})
 */}}
-func (f *decFnInfo) {{ .MethodNamePfx "fastpathDec" false }}R(rv reflect.Value) { 
-	if rv.CanAddr() {
-		vp := rv2i(rv.Addr()).(*map[{{ .MapKey }}]{{ .Elem }})
-		v, changed := fastpathTV.{{ .MethodNamePfx "Dec" false }}V(*vp, fastpathCheckNilFalse, true, f.d)
-		if changed {
+func (d *Decoder) {{ .MethodNamePfx "fastpathDec" false }}R(f *codecFnInfo, rv reflect.Value) {
+	if rv.Kind() == reflect.Ptr {
+		vp := rv2i(rv).(*map[{{ .MapKey }}]{{ .Elem }})
+		if v, changed := fastpathTV.{{ .MethodNamePfx "Dec" false }}V(*vp, true, d); changed {
 			*vp = v
 		}
-	} else {
-		v := rv2i(rv).(map[{{ .MapKey }}]{{ .Elem }})
-		fastpathTV.{{ .MethodNamePfx "Dec" false }}V(v, fastpathCheckNilFalse, false, f.d)
+		return
 	}
+	fastpathTV.{{ .MethodNamePfx "Dec" false }}V(rv2i(rv).(map[{{ .MapKey }}]{{ .Elem }}), false, d)
 }
-func (f fastpathT) {{ .MethodNamePfx "Dec" false }}X(vp *map[{{ .MapKey }}]{{ .Elem }}, checkNil bool, d *Decoder) {
-	v, changed := f.{{ .MethodNamePfx "Dec" false }}V(*vp, checkNil, true, d)
-	if changed {
+func (f fastpathT) {{ .MethodNamePfx "Dec" false }}X(vp *map[{{ .MapKey }}]{{ .Elem }}, d *Decoder) {
+	if v, changed := f.{{ .MethodNamePfx "Dec" false }}V(*vp, true, d); changed {
 		*vp = v 
 	}
 }
-func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v map[{{ .MapKey }}]{{ .Elem }}, checkNil bool, canChange bool, 
+func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v map[{{ .MapKey }}]{{ .Elem }}, canChange bool, 
 	d *Decoder) (_ map[{{ .MapKey }}]{{ .Elem }}, changed bool) {
-	dd := d.d
-	cr := d.cr
+	dd, cr := d.d, d.cr
 	{{/* // if dd.isContainerType(valueTypeNil) {dd.TryDecodeAsNil() */}}
-	if checkNil && dd.TryDecodeAsNil() {
-		if v != nil { changed = true } 
-		return nil, changed
-	}
 	containerLen := dd.ReadMapStart()
 	if canChange && v == nil {
 		xlen := decInferLen(containerLen, d.h.MaxInitLen, {{ .Size }})
@@ -460,6 +445,10 @@ func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v map[{{ .MapKey }}]{{ .Ele
 			mk = d.string(bv) {{/* // maps cannot have []byte as key. switch to string. */}}
 		}{{ else }}mk = {{ decmd .MapKey }}{{ end }}
 		if cr != nil { cr.sendContainerState(containerMapValue) }
+		if dd.TryDecodeAsNil() {
+			if d.h.DeleteOnNilMapValue { delete(v, mk) } else { v[mk] = {{ zerocmd .Elem }} }
+			continue 
+		}
 		{{ if eq .Elem "interface{}" }}if mapGet { mv = v[mk] } else { mv = nil }
 		d.decode(&mv){{ else }}mv = {{ decmd .Elem }}{{ end }}
 		if v != nil {

codec/fast-path.not.go

@@ -14,17 +14,18 @@ const fastpathEnabled = false
 // This tag disables fastpath during build, allowing for faster build, test execution,
 // short-program runs, etc.
 
-func fastpathDecodeTypeSwitch(iv interface{}, d *Decoder) bool      { return false }
-func fastpathEncodeTypeSwitch(iv interface{}, e *Encoder) bool      { return false }
-func fastpathEncodeTypeSwitchSlice(iv interface{}, e *Encoder) bool { return false }
-func fastpathEncodeTypeSwitchMap(iv interface{}, e *Encoder) bool   { return false }
+func fastpathDecodeTypeSwitch(iv interface{}, d *Decoder) bool        { return false }
+func fastpathEncodeTypeSwitch(iv interface{}, e *Encoder) bool        { return false }
+func fastpathEncodeTypeSwitchSlice(iv interface{}, e *Encoder) bool   { return false }
+func fastpathEncodeTypeSwitchMap(iv interface{}, e *Encoder) bool     { return false }
+func fastpathDecodeSetZeroTypeSwitch(iv interface{}, d *Decoder) bool { return false }
 
 type fastpathT struct{}
 type fastpathE struct {
 	rtid  uintptr
 	rt    reflect.Type
-	encfn func(*encFnInfo, reflect.Value)
-	decfn func(*decFnInfo, reflect.Value)
+	encfn func(*Encoder, *codecFnInfo, reflect.Value)
+	decfn func(*Decoder, *codecFnInfo, reflect.Value)
 }
 type fastpathA [0]fastpathE
 

codec/gen-dec-array.go.tmpl

@@ -15,7 +15,7 @@ if {{var "l"}} == 0 {
 	} {{end}}
 } else {
 	{{var "hl"}} := {{var "l"}} > 0
-	var {{var "rl"}} int 
+	var {{var "rl"}} int; _ =  {{var "rl"}}
 	{{if isSlice }} if {{var "hl"}} {
 	if {{var "l"}} > cap({{var "v"}}) {
 		{{var "rl"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
@@ -30,9 +30,10 @@ if {{var "l"}} == 0 {
 		{{var "c"}} = true
 	}
 	} {{end}}
-	{{var "j"}} := 0
+	var {{var "j"}} int 
+    // var {{var "dn"}} bool 
 	for ; ({{var "hl"}} && {{var "j"}} < {{var "l"}}) || !({{var "hl"}} || r.CheckBreak()); {{var "j"}}++ {
-		if {{var "j"}} == 0 && len({{var "v"}}) == 0 {
+		{{if not isArray}} if {{var "j"}} == 0 && len({{var "v"}}) == 0 {
 			if {{var "hl"}} {
 				{{var "rl"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
 			} else {
@@ -40,20 +41,26 @@ if {{var "l"}} == 0 {
 			}
 			{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
 			{{var "c"}} = true 
-		}
+		}{{end}}
+		{{var "h"}}.ElemContainerState({{var "j"}})
+        // {{var "dn"}} = r.TryDecodeAsNil()
+        {{if isChan}}{{ $x := printf "%[1]vv%[2]v" .TempVar .Rand }}var {{var $x}} {{ .Typ }}
+		{{ decLineVar $x }}
+		{{var "v"}} <- {{ $x }}
+        {{else}}
 		// if indefinite, etc, then expand the slice if necessary
 		var {{var "db"}} bool
 		if {{var "j"}} >= len({{var "v"}}) {
 			{{if isSlice }} {{var "v"}} = append({{var "v"}}, {{ zero }}); {{var "c"}} = true
-			{{end}} {{if isArray}} z.DecArrayCannotExpand(len(v), {{var "j"}}+1); {{var "db"}} = true
+			{{else}} z.DecArrayCannotExpand(len(v), {{var "j"}}+1); {{var "db"}} = true
 			{{end}}
 		}
-		{{var "h"}}.ElemContainerState({{var "j"}})
 		if {{var "db"}} {
 			z.DecSwallow()
 		} else {
 			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
 		}
+        {{end}}
 	}
 	{{if isSlice}} if {{var "j"}} < len({{var "v"}}) {
 		{{var "v"}} = {{var "v"}}[:{{var "j"}}]

codec/gen-dec-map.go.tmpl

@@ -8,7 +8,7 @@ if {{var "v"}} == nil {
 }
 var {{var "mk"}} {{ .KTyp }}
 var {{var "mv"}} {{ .Typ }}
-var {{var "mg"}} {{if decElemKindPtr}}, {{var "ms"}}, {{var "mok"}}{{end}} bool
+var {{var "mg"}}, {{var "mdn"}} {{if decElemKindPtr}}, {{var "ms"}}, {{var "mok"}}{{end}} bool
 if {{var "bh"}}.MapValueReset {
 	{{if decElemKindPtr}}{{var "mg"}} = true
 	{{else if decElemKindIntf}}if !{{var "bh"}}.InterfaceReset { {{var "mg"}} = true }
@@ -30,8 +30,11 @@ if {{var "l"}} != 0 {
 		} {{else}}{{var "mv"}} = {{var "v"}}[{{var "mk"}}] {{end}}
 	} {{if not decElemKindImmutable}}else { {{var "mv"}} = {{decElemZero}} }{{end}}
 	z.DecSendContainerState(codecSelfer_containerMapValue{{ .Sfx }})
-	{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
-	if {{if decElemKindPtr}} {{var "ms"}} && {{end}} {{var "v"}} != nil {
+	{{var "mdn"}} = false
+	{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ $y := printf "%vmdn%v" .TempVar .Rand }}{{ decLineVar $x $y }}
+	if {{var "mdn"}} {
+		if {{ var "bh" }}.DeleteOnNilMapValue { delete({{var "v"}}, {{var "mk"}}) } else { {{var "v"}}[{{var "mk"}}] = {{decElemZero}} }
+	} else if {{if decElemKindPtr}} {{var "ms"}} && {{end}} {{var "v"}} != nil {
 		{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
 	}
 }

codec/gen-helper.generated.go

@@ -61,13 +61,14 @@ func (f genHelperEncoder) EncBasicHandle() *BasicHandle {
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperEncoder) EncBinary() bool {
-	return f.e.be // f.e.hh.isBinaryEncoding()
+	return f.e.cf.be // f.e.hh.isBinaryEncoding()
 }
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperEncoder) EncFallback(iv interface{}) {
 	// println(">>>>>>>>> EncFallback")
-	f.e.encodeI(iv, false, false)
+	// f.e.encodeI(iv, false, false)
+	f.e.encodeValue(reflect.ValueOf(iv), nil, false)
 }
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
@@ -90,7 +91,7 @@ func (f genHelperEncoder) EncBinaryMarshal(iv encoding.BinaryMarshaler) {
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperEncoder) EncRaw(iv Raw) {
-	f.e.raw(iv)
+	f.e.rawBytes(iv)
 }
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
@@ -103,7 +104,7 @@ func (f genHelperEncoder) TimeRtidIfBinc() uintptr {
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperEncoder) IsJSONHandle() bool {
-	return f.e.js
+	return f.e.cf.js
 }
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
@@ -157,7 +158,12 @@ func (f genHelperDecoder) DecScratchBuffer() []byte {
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecFallback(iv interface{}, chkPtr bool) {
 	// println(">>>>>>>>> DecFallback")
-	f.d.decodeI(iv, chkPtr, false, false, false)
+	rv := reflect.ValueOf(iv)
+	if chkPtr {
+		rv = f.d.ensureDecodeable(rv)
+	}
+	f.d.decodeValue(rv, nil, false, false)
+	// f.d.decodeValueFallback(rv)
 }
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
@@ -203,7 +209,7 @@ func (f genHelperDecoder) DecBinaryUnmarshal(bm encoding.BinaryUnmarshaler) {
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecRaw() []byte {
-	return f.d.raw()
+	return f.d.rawBytes()
 }
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*

codec/gen-helper.go.tmpl

@@ -61,12 +61,13 @@ func (f genHelperEncoder) EncBasicHandle() *BasicHandle {
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperEncoder) EncBinary() bool {
-	return f.e.be // f.e.hh.isBinaryEncoding()
+	return f.e.cf.be // f.e.hh.isBinaryEncoding()
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperEncoder) EncFallback(iv interface{}) {
 	// println(">>>>>>>>> EncFallback")
-	f.e.encodeI(iv, false, false)
+	// f.e.encodeI(iv, false, false)
+	f.e.encodeValue(reflect.ValueOf(iv), nil, false)
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperEncoder) EncTextMarshal(iv encoding.TextMarshaler) {
@@ -85,7 +86,7 @@ func (f genHelperEncoder) EncBinaryMarshal(iv encoding.BinaryMarshaler) {
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperEncoder) EncRaw(iv Raw) {
-	f.e.raw(iv)
+	f.e.rawBytes(iv)
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperEncoder) TimeRtidIfBinc() uintptr {
@@ -96,7 +97,7 @@ func (f genHelperEncoder) TimeRtidIfBinc() uintptr {
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperEncoder) IsJSONHandle() bool {
-	return f.e.js
+	return f.e.cf.js
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperEncoder) HasExtensions() bool {
@@ -143,7 +144,12 @@ func (f genHelperDecoder) DecScratchBuffer() []byte {
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecFallback(iv interface{}, chkPtr bool) {
 	// println(">>>>>>>>> DecFallback")
-	f.d.decodeI(iv, chkPtr, false, false, false)
+	rv := reflect.ValueOf(iv)
+	if chkPtr {
+		rv = f.d.ensureDecodeable(rv)
+	}
+	f.d.decodeValue(rv, nil, false, false)
+	// f.d.decodeValueFallback(rv)
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecSliceHelperStart() (decSliceHelper, int) {
@@ -182,7 +188,7 @@ func (f genHelperDecoder) DecBinaryUnmarshal(bm encoding.BinaryUnmarshaler) {
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecRaw() []byte {
-	return f.d.raw()
+	return f.d.rawBytes()
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) TimeRtidIfBinc() uintptr {

codec/gen.generated.go

@@ -16,7 +16,7 @@ if {{var "v"}} == nil {
 }
 var {{var "mk"}} {{ .KTyp }}
 var {{var "mv"}} {{ .Typ }}
-var {{var "mg"}} {{if decElemKindPtr}}, {{var "ms"}}, {{var "mok"}}{{end}} bool
+var {{var "mg"}}, {{var "mdn"}} {{if decElemKindPtr}}, {{var "ms"}}, {{var "mok"}}{{end}} bool
 if {{var "bh"}}.MapValueReset {
 	{{if decElemKindPtr}}{{var "mg"}} = true
 	{{else if decElemKindIntf}}if !{{var "bh"}}.InterfaceReset { {{var "mg"}} = true }
@@ -38,8 +38,11 @@ if {{var "l"}} != 0 {
 		} {{else}}{{var "mv"}} = {{var "v"}}[{{var "mk"}}] {{end}}
 	} {{if not decElemKindImmutable}}else { {{var "mv"}} = {{decElemZero}} }{{end}}
 	z.DecSendContainerState(codecSelfer_containerMapValue{{ .Sfx }})
-	{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
-	if {{if decElemKindPtr}} {{var "ms"}} && {{end}} {{var "v"}} != nil {
+	{{var "mdn"}} = false
+	{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ $y := printf "%vmdn%v" .TempVar .Rand }}{{ decLineVar $x $y }}
+	if {{var "mdn"}} {
+		if {{ var "bh" }}.DeleteOnNilMapValue { delete({{var "v"}}, {{var "mk"}}) } else { {{var "v"}}[{{var "mk"}}] = {{decElemZero}} }
+	} else if {{if decElemKindPtr}} {{var "ms"}} && {{end}} {{var "v"}} != nil {
 		{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
 	}
 }
@@ -65,7 +68,7 @@ if {{var "l"}} == 0 {
 	} {{end}}
 } else {
 	{{var "hl"}} := {{var "l"}} > 0
-	var {{var "rl"}} int 
+	var {{var "rl"}} int; _ =  {{var "rl"}}
 	{{if isSlice }} if {{var "hl"}} {
 	if {{var "l"}} > cap({{var "v"}}) {
 		{{var "rl"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
@@ -80,9 +83,10 @@ if {{var "l"}} == 0 {
 		{{var "c"}} = true
 	}
 	} {{end}}
-	{{var "j"}} := 0
+	var {{var "j"}} int 
+    // var {{var "dn"}} bool 
 	for ; ({{var "hl"}} && {{var "j"}} < {{var "l"}}) || !({{var "hl"}} || r.CheckBreak()); {{var "j"}}++ {
-		if {{var "j"}} == 0 && len({{var "v"}}) == 0 {
+		{{if not isArray}} if {{var "j"}} == 0 && len({{var "v"}}) == 0 {
 			if {{var "hl"}} {
 				{{var "rl"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
 			} else {
@@ -90,20 +94,26 @@ if {{var "l"}} == 0 {
 			}
 			{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
 			{{var "c"}} = true 
-		}
+		}{{end}}
+		{{var "h"}}.ElemContainerState({{var "j"}})
+        // {{var "dn"}} = r.TryDecodeAsNil()
+        {{if isChan}}{{ $x := printf "%[1]vv%[2]v" .TempVar .Rand }}var {{var $x}} {{ .Typ }}
+		{{ decLineVar $x }}
+		{{var "v"}} <- {{ $x }}
+        {{else}}
 		// if indefinite, etc, then expand the slice if necessary
 		var {{var "db"}} bool
 		if {{var "j"}} >= len({{var "v"}}) {
 			{{if isSlice }} {{var "v"}} = append({{var "v"}}, {{ zero }}); {{var "c"}} = true
-			{{end}} {{if isArray}} z.DecArrayCannotExpand(len(v), {{var "j"}}+1); {{var "db"}} = true
+			{{else}} z.DecArrayCannotExpand(len(v), {{var "j"}}+1); {{var "db"}} = true
 			{{end}}
 		}
-		{{var "h"}}.ElemContainerState({{var "j"}})
 		if {{var "db"}} {
 			z.DecSwallow()
 		} else {
 			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
 		}
+        {{end}}
 	}
 	{{if isSlice}} if {{var "j"}} < len({{var "v"}}) {
 		{{var "v"}} = {{var "v"}}[:{{var "j"}}]

codec/gen.go

@@ -163,13 +163,16 @@ type genRunner struct {
 
 	ti *TypeInfos
 	// rr *rand.Rand // random generator for file-specific types
+
+	nx bool // no extensions
 }
 
 // Gen will write a complete go file containing Selfer implementations for each
 // type passed. All the types must be in the same package.
 //
 // Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.
-func Gen(w io.Writer, buildTags, pkgName, uid string, ti *TypeInfos, typ ...reflect.Type) {
+func Gen(w io.Writer, buildTags, pkgName, uid string, noExtensions bool,
+	ti *TypeInfos, typ ...reflect.Type) {
 	// All types passed to this method do not have a codec.Selfer method implemented directly.
 	// codecgen already checks the AST and skips any types that define the codec.Selfer methods.
 	// Consequently, there's no need to check and trim them if they implement codec.Selfer
@@ -190,6 +193,7 @@ func Gen(w io.Writer, buildTags, pkgName, uid string, ti *TypeInfos, typ ...refl
 		bp:  genImportPath(typ[0]),
 		xs:  uid,
 		ti:  ti,
+		nx:  noExtensions,
 	}
 	if x.ti == nil {
 		x.ti = defTypeInfos
@@ -713,7 +717,7 @@ func (x *genRunner) enc(varname string, t reflect.Type) {
 		x.linef("r.EncodeBuiltin(%s, %s)", vrtid, varname)
 	}
 	// only check for extensions if the type is named, and has a packagePath.
-	if genImportPath(t) != "" && t.Name() != "" {
+	if !x.nx && genImportPath(t) != "" && t.Name() != "" {
 		// first check if extensions are configued, before doing the interface conversion
 		x.linef("} else if z.HasExtensions() && z.EncExt(%s) {", varname)
 	}
@@ -773,7 +777,7 @@ func (x *genRunner) enc(varname string, t reflect.Type) {
 			x.line("r.EncodeStringBytes(codecSelferC_RAW" + x.xs + ", []byte(" + varname + "))")
 		} else if fastpathAV.index(rtid) != -1 {
 			g := x.newGenV(t)
-			x.line("z.F." + g.MethodNamePfx("Enc", false) + "V(" + varname + ", false, e)")
+			x.line("z.F." + g.MethodNamePfx("Enc", false) + "V(" + varname + ", e)")
 		} else {
 			x.xtraSM(varname, true, t)
 			// x.encListFallback(varname, rtid, t)
@@ -787,7 +791,7 @@ func (x *genRunner) enc(varname string, t reflect.Type) {
 		// x.line("if " + varname + " == nil { \nr.EncodeNil()\n } else { ")
 		if fastpathAV.index(rtid) != -1 {
 			g := x.newGenV(t)
-			x.line("z.F." + g.MethodNamePfx("Enc", false) + "V(" + varname + ", false, e)")
+			x.line("z.F." + g.MethodNamePfx("Enc", false) + "V(" + varname + ", e)")
 		} else {
 			x.xtraSM(varname, true, t)
 			// x.encMapFallback(varname, rtid, t)
@@ -845,55 +849,64 @@ func (x *genRunner) encStruct(varname string, rtid uintptr, t reflect.Type) {
 	// number of non-empty things we write out first.
 	// This is required as we need to pre-determine the size of the container,
 	// to support length-prefixing.
-	x.linef("var %s [%v]bool", numfieldsvar, len(tisfi))
-	x.linef("_, _, _ = %s, %s, %s", sepVarname, numfieldsvar, struct2arrvar)
+	if ti.anyOmitEmpty {
+		x.linef("var %s [%v]bool", numfieldsvar, len(tisfi))
+		x.linef("_ = %s", numfieldsvar)
+	}
+	x.linef("_, _ = %s, %s", sepVarname, struct2arrvar)
 	x.linef("const %s bool = %v", ti2arrayvar, ti.toArray)
-	nn := 0
-	for j, si := range tisfi {
-		if !si.omitEmpty {
-			nn++
-			continue
-		}
-		var t2 reflect.StructField
-		var omitline string
-		if si.i != -1 {
-			t2 = t.Field(int(si.i))
-		} else {
-			t2typ := t
-			varname3 := varname
-			for _, ix := range si.is {
-				for t2typ.Kind() == reflect.Ptr {
-					t2typ = t2typ.Elem()
-				}
-				t2 = t2typ.Field(ix)
-				t2typ = t2.Type
-				varname3 = varname3 + "." + t2.Name
-				if t2typ.Kind() == reflect.Ptr {
-					omitline += varname3 + " != nil && "
+	var nn int
+	if ti.anyOmitEmpty {
+		for j, si := range tisfi {
+			if !si.omitEmpty {
+				nn++
+				continue
+			}
+			var t2 reflect.StructField
+			var omitline string
+			{
+				t2typ := t
+				varname3 := varname
+				for ij, ix := range si.is {
+					if uint8(ij) == si.nis {
+						break
+					}
+					for t2typ.Kind() == reflect.Ptr {
+						t2typ = t2typ.Elem()
+					}
+					t2 = t2typ.Field(int(ix))
+					t2typ = t2.Type
+					varname3 = varname3 + "." + t2.Name
+					if t2typ.Kind() == reflect.Ptr {
+						omitline += varname3 + " != nil && "
+					}
 				}
 			}
+			// never check omitEmpty on a struct type, as it may contain uncomparable map/slice/etc.
+			// also, for maps/slices/arrays, check if len ! 0 (not if == zero value)
+			switch t2.Type.Kind() {
+			case reflect.Struct:
+				omitline += " true"
+			case reflect.Map, reflect.Slice, reflect.Array, reflect.Chan:
+				omitline += "len(" + varname + "." + t2.Name + ") != 0"
+			default:
+				omitline += varname + "." + t2.Name + " != " + x.genZeroValueR(t2.Type)
+			}
+			x.linef("%s[%v] = %s", numfieldsvar, j, omitline)
 		}
-		// never check omitEmpty on a struct type, as it may contain uncomparable map/slice/etc.
-		// also, for maps/slices/arrays, check if len ! 0 (not if == zero value)
-		switch t2.Type.Kind() {
-		case reflect.Struct:
-			omitline += " true"
-		case reflect.Map, reflect.Slice, reflect.Array, reflect.Chan:
-			omitline += "len(" + varname + "." + t2.Name + ") != 0"
-		default:
-			omitline += varname + "." + t2.Name + " != " + x.genZeroValueR(t2.Type)
-		}
-		x.linef("%s[%v] = %s", numfieldsvar, j, omitline)
 	}
-	x.linef("var %snn%s int", genTempVarPfx, i)
+	// x.linef("var %snn%s int", genTempVarPfx, i)
 	x.linef("if %s || %s {", ti2arrayvar, struct2arrvar) // if ti.toArray {
-	x.line("r.EncodeArrayStart(" + strconv.FormatInt(int64(len(tisfi)), 10) + ")")
+	x.linef("r.EncodeArrayStart(%d)", len(tisfi))
 	x.linef("} else {") // if not ti.toArray
-	x.linef("%snn%s = %v", genTempVarPfx, i, nn)
-	x.linef("for _, b := range %s { if b { %snn%s++ } }", numfieldsvar, genTempVarPfx, i)
-	x.linef("r.EncodeMapStart(%snn%s)", genTempVarPfx, i)
-	x.linef("%snn%s = %v", genTempVarPfx, i, 0)
-	// x.line("r.EncodeMapStart(" + strconv.FormatInt(int64(len(tisfi)), 10) + ")")
+	if ti.anyOmitEmpty {
+		x.linef("var %snn%s = %v", genTempVarPfx, i, nn)
+		x.linef("for _, b := range %s { if b { %snn%s++ } }", numfieldsvar, genTempVarPfx, i)
+		x.linef("r.EncodeMapStart(%snn%s)", genTempVarPfx, i)
+		x.linef("%snn%s = %v", genTempVarPfx, i, 0)
+	} else {
+		x.linef("r.EncodeMapStart(%d)", len(tisfi))
+	}
 	x.line("}") // close if not StructToArray
 
 	for j, si := range tisfi {
@@ -901,17 +914,18 @@ func (x *genRunner) encStruct(varname string, rtid uintptr, t reflect.Type) {
 		isNilVarName := genTempVarPfx + "n" + i
 		var labelUsed bool
 		var t2 reflect.StructField
-		if si.i != -1 {
-			t2 = t.Field(int(si.i))
-		} else {
+		{
 			t2typ := t
 			varname3 := varname
-			for _, ix := range si.is {
+			for ij, ix := range si.is {
+				if uint8(ij) == si.nis {
+					break
+				}
 				// fmt.Printf("%%%% %v, ix: %v\n", t2typ, ix)
 				for t2typ.Kind() == reflect.Ptr {
 					t2typ = t2typ.Elem()
 				}
-				t2 = t2typ.Field(ix)
+				t2 = t2typ.Field(int(ix))
 				t2typ = t2.Type
 				varname3 = varname3 + "." + t2.Name
 				if t2typ.Kind() == reflect.Ptr {
@@ -934,7 +948,7 @@ func (x *genRunner) encStruct(varname string, rtid uintptr, t reflect.Type) {
 
 		x.linef("if %s || %s {", ti2arrayvar, struct2arrvar) // if ti.toArray
 		if labelUsed {
-			x.line("if " + isNilVarName + " { r.EncodeNil() } else { ")
+			x.linef("if %s { z.EncSendContainerState(codecSelfer_containerArrayElem%s); r.EncodeNil() } else { ", isNilVarName, x.xs)
 		}
 		x.linef("z.EncSendContainerState(codecSelfer_containerArrayElem%s)", x.xs)
 		if si.omitEmpty {
@@ -1018,7 +1032,7 @@ func (x *genRunner) encMapFallback(varname string, t reflect.Type) {
 	x.linef("z.EncSendContainerState(codecSelfer_containerMapEnd%s)", x.xs)
 }
 
-func (x *genRunner) decVar(varname string, t reflect.Type, canBeNil bool) {
+func (x *genRunner) decVar(varname, decodedNilVarname string, t reflect.Type, canBeNil bool) {
 	// We only encode as nil if a nillable value.
 	// This removes some of the wasted checks for TryDecodeAsNil.
 	// We need to think about this more, to see what happens if omitempty, etc
@@ -1031,7 +1045,9 @@ func (x *genRunner) decVar(varname string, t reflect.Type, canBeNil bool) {
 	}
 	if canBeNil {
 		x.line("if r.TryDecodeAsNil() {")
-		if t.Kind() == reflect.Ptr {
+		if decodedNilVarname != "" {
+			x.line(decodedNilVarname + " = true")
+		} else if t.Kind() == reflect.Ptr {
 			x.line("if " + varname + " != nil { ")
 
 			// if varname is a field of a struct (has a dot in it),
@@ -1149,7 +1165,7 @@ func (x *genRunner) dec(varname string, t reflect.Type) {
 		x.linef("r.DecodeBuiltin(%s, %s)", vrtid, varname)
 	}
 	// only check for extensions if the type is named, and has a packagePath.
-	if genImportPath(t) != "" && t.Name() != "" {
+	if !x.nx && genImportPath(t) != "" && t.Name() != "" {
 		// first check if extensions are configued, before doing the interface conversion
 		x.linef("} else if z.HasExtensions() && z.DecExt(%s) {", varname)
 	}
@@ -1227,7 +1243,7 @@ func (x *genRunner) dec(varname string, t reflect.Type) {
 			x.line("*" + varname + " = r.DecodeBytes(*(*[]byte)(" + varname + "), false)")
 		} else if fastpathAV.index(rtid) != -1 {
 			g := x.newGenV(t)
-			x.line("z.F." + g.MethodNamePfx("Dec", false) + "X(" + varname + ", false, d)")
+			x.line("z.F." + g.MethodNamePfx("Dec", false) + "X(" + varname + ", d)")
 		} else {
 			x.xtraSM(varname, false, t)
 			// x.decListFallback(varname, rtid, false, t)
@@ -1239,7 +1255,7 @@ func (x *genRunner) dec(varname string, t reflect.Type) {
 		// - else call Encoder.encode(XXX) on it.
 		if fastpathAV.index(rtid) != -1 {
 			g := x.newGenV(t)
-			x.line("z.F." + g.MethodNamePfx("Dec", false) + "X(" + varname + ", false, d)")
+			x.line("z.F." + g.MethodNamePfx("Dec", false) + "X(" + varname + ", d)")
 		} else {
 			x.xtraSM(varname, false, t)
 			// x.decMapFallback(varname, rtid, t)
@@ -1333,13 +1349,13 @@ func (x *genRunner) decListFallback(varname string, rtid uintptr, t reflect.Type
 	funcs := make(template.FuncMap)
 
 	funcs["decLineVar"] = func(varname string) string {
-		x.decVar(varname, telem, false)
-		return ""
-	}
-	funcs["decLine"] = func(pfx string) string {
-		x.decVar(ts.TempVar+pfx+ts.Rand, reflect.PtrTo(telem), false)
+		x.decVar(varname, "", telem, false)
 		return ""
 	}
+	// funcs["decLine"] = func(pfx string) string {
+	// 	x.decVar(ts.TempVar+pfx+ts.Rand, "", reflect.PtrTo(telem), false)
+	// 	return ""
+	// }
 	funcs["var"] = func(s string) string {
 		return ts.TempVar + s + ts.Rand
 	}
@@ -1395,21 +1411,21 @@ func (x *genRunner) decMapFallback(varname string, rtid uintptr, t reflect.Type)
 		return telem.Kind() == reflect.Interface
 	}
 	funcs["decLineVarK"] = func(varname string) string {
-		x.decVar(varname, tkey, false)
+		x.decVar(varname, "", tkey, false)
 		return ""
 	}
-	funcs["decLineVar"] = func(varname string) string {
-		x.decVar(varname, telem, false)
-		return ""
-	}
-	funcs["decLineK"] = func(pfx string) string {
-		x.decVar(ts.TempVar+pfx+ts.Rand, reflect.PtrTo(tkey), false)
-		return ""
-	}
-	funcs["decLine"] = func(pfx string) string {
-		x.decVar(ts.TempVar+pfx+ts.Rand, reflect.PtrTo(telem), false)
+	funcs["decLineVar"] = func(varname, decodedNilVarname string) string {
+		x.decVar(varname, decodedNilVarname, telem, false)
 		return ""
 	}
+	// funcs["decLineK"] = func(pfx string) string {
+	// 	x.decVar(ts.TempVar+pfx+ts.Rand, reflect.PtrTo(tkey), false)
+	// 	return ""
+	// }
+	// funcs["decLine"] = func(pfx string) string {
+	// 	x.decVar(ts.TempVar+pfx+ts.Rand, reflect.PtrTo(telem), false)
+	// 	return ""
+	// }
 	funcs["var"] = func(s string) string {
 		return ts.TempVar + s + ts.Rand
 	}
@@ -1430,18 +1446,19 @@ func (x *genRunner) decStructMapSwitch(kName string, varname string, rtid uintpt
 	for _, si := range tisfi {
 		x.line("case \"" + si.encName + "\":")
 		var t2 reflect.StructField
-		if si.i != -1 {
-			t2 = t.Field(int(si.i))
-		} else {
+		{
 			//we must accommodate anonymous fields, where the embedded field is a nil pointer in the value.
 			// t2 = t.FieldByIndex(si.is)
 			t2typ := t
 			varname3 := varname
-			for _, ix := range si.is {
+			for ij, ix := range si.is {
+				if uint8(ij) == si.nis {
+					break
+				}
 				for t2typ.Kind() == reflect.Ptr {
 					t2typ = t2typ.Elem()
 				}
-				t2 = t2typ.Field(ix)
+				t2 = t2typ.Field(int(ix))
 				t2typ = t2.Type
 				varname3 = varname3 + "." + t2.Name
 				if t2typ.Kind() == reflect.Ptr {
@@ -1449,7 +1466,7 @@ func (x *genRunner) decStructMapSwitch(kName string, varname string, rtid uintpt
 				}
 			}
 		}
-		x.decVar(varname+"."+t2.Name, t2.Type, false)
+		x.decVar(varname+"."+t2.Name, "", t2.Type, false)
 	}
 	x.line("default:")
 	// pass the slice here, so that the string will not escape, and maybe save allocation
@@ -1501,18 +1518,19 @@ func (x *genRunner) decStructArray(varname, lenvarname, breakString string, rtid
 	x.linef("var %shl%s bool = %s >= 0", tpfx, i, lenvarname) // has length
 	for _, si := range tisfi {
 		var t2 reflect.StructField
-		if si.i != -1 {
-			t2 = t.Field(int(si.i))
-		} else {
+		{
 			//we must accommodate anonymous fields, where the embedded field is a nil pointer in the value.
 			// t2 = t.FieldByIndex(si.is)
 			t2typ := t
 			varname3 := varname
-			for _, ix := range si.is {
+			for ij, ix := range si.is {
+				if uint8(ij) == si.nis {
+					break
+				}
 				for t2typ.Kind() == reflect.Ptr {
 					t2typ = t2typ.Elem()
 				}
-				t2 = t2typ.Field(ix)
+				t2 = t2typ.Field(int(ix))
 				t2typ = t2.Type
 				varname3 = varname3 + "." + t2.Name
 				if t2typ.Kind() == reflect.Ptr {
@@ -1527,7 +1545,7 @@ func (x *genRunner) decStructArray(varname, lenvarname, breakString string, rtid
 		x.linef("if %sb%s { z.DecSendContainerState(codecSelfer_containerArrayEnd%s); %s }",
 			tpfx, i, x.xs, breakString)
 		x.linef("z.DecSendContainerState(codecSelfer_containerArrayElem%s)", x.xs)
-		x.decVar(varname+"."+t2.Name, t2.Type, true)
+		x.decVar(varname+"."+t2.Name, "", t2.Type, true)
 	}
 	// read remaining values and throw away.
 	x.line("for {")

codec/helper.go

@@ -103,8 +103,10 @@ import (
 	"errors"
 	"fmt"
 	"math"
+	"os"
 	"reflect"
 	"sort"
+	"strconv"
 	"strings"
 	"sync"
 	"time"
@@ -112,32 +114,56 @@ import (
 
 const (
 	scratchByteArrayLen = 32
-	initCollectionCap   = 32 // 32 is defensive. 16 is preferred.
+	// initCollectionCap   = 16 // 32 is defensive. 16 is preferred.
 
 	// Support encoding.(Binary|Text)(Unm|M)arshaler.
 	// This constant flag will enable or disable it.
 	supportMarshalInterfaces = true
 
-	// Each Encoder or Decoder uses a cache of functions based on conditionals,
-	// so that the conditionals are not run every time.
-	//
-	// Either a map or a slice is used to keep track of the functions.
-	// The map is more natural, but has a higher cost than a slice/array.
-	// This flag (useMapForCodecCache) controls which is used.
-	//
-	// From benchmarks, slices with linear search perform better with < 32 entries.
-	// We have typically seen a high threshold of about 24 entries.
-	useMapForCodecCache = false
-
 	// for debugging, set this to false, to catch panic traces.
 	// Note that this will always cause rpc tests to fail, since they need io.EOF sent via panic.
 	recoverPanicToErr = true
 
-	// resetSliceElemToZeroValue: on decoding a slice, reset the element to a zero value first.
-	// concern: if the slice already contained some garbage, we will decode into that garbage.
-	// The chances of this are slim, so leave this "optimization".
-	// TODO: should this be true, to ensure that we always decode into a "zero" "empty" value?
-	resetSliceElemToZeroValue = false
+	// arrayCacheLen is the length of the cache used in encoder or decoder for
+	// allowing zero-alloc initialization.
+	arrayCacheLen = 8
+
+	// We tried an optimization, where we detect if a type is one of the known types
+	// we optimized for (e.g. int, []uint64, etc).
+	//
+	// However, we notice some worse performance when using this optimization.
+	// So we hide it behind a flag, to turn on if needed.
+	useLookupRecognizedTypes = false
+
+	// using recognized allows us to do d.decode(interface{}) instead of d.decodeValue(reflect.Value)
+	// when we can infer that the kind of the interface{} is one of the ones hard-coded in the
+	// type switch for known types or the ones defined by fast-path.
+	//
+	// However, it seems we get better performance when we don't recognize, and just let
+	// reflection handle it.
+	//
+	// Reasoning is as below:
+	// typeswitch is a binary search with a branch to a code-point.
+	// getdecfn is a binary search with a call to a function pointer.
+	//
+	// both are about the same.
+	//
+	// so: why prefer typeswitch?
+	//
+	// is recognized does the following:
+	// - lookup rtid
+	// - check if in sorted list
+	// - calls decode(type switch)
+	//   - 1 or 2 binary search to a point in code
+	//   - branch there
+	//
+	// vs getdecfn
+	// - lookup rtid
+	// - check in sorted list for a function pointer
+	// - calls it to decode using reflection (optimized)
+
+	// always set xDebug = false before releasing software
+	xDebug = true
 )
 
 var (
@@ -145,6 +171,20 @@ var (
 	zeroByteSlice = oneByteArr[:0:0]
 )
 
+var pool pooler
+
+func init() {
+	pool.init()
+}
+
+// type findCodecFnMode uint8
+
+// const (
+// 	findCodecFnModeMap findCodecFnMode = iota
+// 	findCodecFnModeBinarySearch
+// 	findCodecFnModeLinearSearch
+// )
+
 type charEncoding uint8
 
 const (
@@ -177,6 +217,36 @@ const (
 	// valueTypeInvalid = 0xff
 )
 
+func (x valueType) String() string {
+	switch x {
+	case valueTypeNil:
+		return "Nil"
+	case valueTypeInt:
+		return "Int"
+	case valueTypeUint:
+		return "Uint"
+	case valueTypeFloat:
+		return "Float"
+	case valueTypeBool:
+		return "Bool"
+	case valueTypeString:
+		return "String"
+	case valueTypeSymbol:
+		return "Symbol"
+	case valueTypeBytes:
+		return "Bytes"
+	case valueTypeMap:
+		return "Map"
+	case valueTypeArray:
+		return "Array"
+	case valueTypeTimestamp:
+		return "Timestamp"
+	case valueTypeExt:
+		return "Ext"
+	}
+	return strconv.FormatInt(int64(x), 10)
+}
+
 type seqType uint8
 
 const (
@@ -216,25 +286,21 @@ const rgetMaxRecursion = 2
 
 // Anecdotally, we believe most types have <= 12 fields.
 // Java's PMD rules set TooManyFields threshold to 15.
-const rgetPoolTArrayLen = 12
+const typeInfoLoadArrayLen = 12
 
-type rgetT struct {
+type typeInfoLoad struct {
 	fNames   []string
 	encNames []string
 	etypes   []uintptr
 	sfis     []*structFieldInfo
 }
 
-type rgetPoolT struct {
-	fNames   [rgetPoolTArrayLen]string
-	encNames [rgetPoolTArrayLen]string
-	etypes   [rgetPoolTArrayLen]uintptr
-	sfis     [rgetPoolTArrayLen]*structFieldInfo
-	sfiidx   [rgetPoolTArrayLen]sfiIdx
-}
-
-var rgetPool = sync.Pool{
-	New: func() interface{} { return new(rgetPoolT) },
+type typeInfoLoadArray struct {
+	fNames   [typeInfoLoadArrayLen]string
+	encNames [typeInfoLoadArrayLen]string
+	etypes   [typeInfoLoadArrayLen]uintptr
+	sfis     [typeInfoLoadArrayLen]*structFieldInfo
+	sfiidx   [typeInfoLoadArrayLen]sfiIdx
 }
 
 type containerStateRecv interface {
@@ -306,21 +372,96 @@ var (
 var defTypeInfos = NewTypeInfos([]string{"codec", "json"})
 
 var immutableKindsSet = [32]bool{
-	reflect.Int:     true,
-	reflect.Int8:    true,
-	reflect.Int16:   true,
-	reflect.Int32:   true,
-	reflect.Int64:   true,
-	reflect.Uint:    true,
-	reflect.Uint8:   true,
-	reflect.Uint16:  true,
-	reflect.Uint32:  true,
-	reflect.Uint64:  true,
-	reflect.Uintptr: true,
-	reflect.Float32: true,
-	reflect.Float64: true,
-	reflect.Bool:    true,
-	reflect.String:  true,
+	// reflect.Invalid:  ,
+	reflect.Bool:       true,
+	reflect.Int:        true,
+	reflect.Int8:       true,
+	reflect.Int16:      true,
+	reflect.Int32:      true,
+	reflect.Int64:      true,
+	reflect.Uint:       true,
+	reflect.Uint8:      true,
+	reflect.Uint16:     true,
+	reflect.Uint32:     true,
+	reflect.Uint64:     true,
+	reflect.Uintptr:    true,
+	reflect.Float32:    true,
+	reflect.Float64:    true,
+	reflect.Complex64:  true,
+	reflect.Complex128: true,
+	// reflect.Array
+	// reflect.Chan
+	// reflect.Func: true,
+	// reflect.Interface
+	// reflect.Map
+	// reflect.Ptr
+	// reflect.Slice
+	reflect.String: true,
+	// reflect.Struct
+	// reflect.UnsafePointer
+}
+
+var recognizedRtids []uintptr
+var recognizedRtidPtrs []uintptr
+var recognizedRtidOrPtrs []uintptr
+
+func init() {
+	if !useLookupRecognizedTypes {
+		return
+	}
+	for _, v := range [...]interface{}{
+		float32(0),
+		float64(0),
+		uintptr(0),
+		uint(0),
+		uint8(0),
+		uint16(0),
+		uint32(0),
+		uint64(0),
+		uintptr(0),
+		int(0),
+		int8(0),
+		int16(0),
+		int32(0),
+		int64(0),
+		bool(false),
+		string(""),
+		Raw{},
+		[]byte(nil),
+	} {
+		rt := reflect.TypeOf(v)
+		recognizedRtids = append(recognizedRtids, rt2id(rt))
+		recognizedRtidPtrs = append(recognizedRtidPtrs, rt2id(reflect.PtrTo(rt)))
+	}
+}
+
+func containsU(s []uintptr, v uintptr) bool {
+	// return false // TODO: REMOVE
+	h, i, j := 0, 0, len(s)
+	for i < j {
+		h = i + (j-i)/2
+		if s[h] < v {
+			i = h + 1
+		} else {
+			j = h
+		}
+	}
+	if i < len(s) && s[i] == v {
+		return true
+	}
+	return false
+}
+
+func isRecognizedRtid(rtid uintptr) bool {
+	return containsU(recognizedRtids, rtid)
+}
+
+func isRecognizedRtidPtr(rtid uintptr) bool {
+	return containsU(recognizedRtidPtrs, rtid)
+}
+
+func isRecognizedRtidOrPtr(rtid uintptr) bool {
+	return containsU(recognizedRtidOrPtrs, rtid)
 }
 
 // Selfer defines methods by which a value can encode or decode itself.
@@ -356,6 +497,7 @@ type BasicHandle struct {
 	extHandle
 	EncodeOptions
 	DecodeOptions
+	noBuiltInTypeChecker
 }
 
 func (x *BasicHandle) getBasicHandle() *BasicHandle {
@@ -379,6 +521,7 @@ type Handle interface {
 	newEncDriver(w *Encoder) encDriver
 	newDecDriver(r *Decoder) decDriver
 	isBinary() bool
+	IsBuiltinType(rtid uintptr) bool
 }
 
 // Raw represents raw formatted bytes.
@@ -506,9 +649,13 @@ func (_ textEncodingType) isBinary() bool { return false }
 
 // noBuiltInTypes is embedded into many types which do not support builtins
 // e.g. msgpack, simple, cbor.
-type noBuiltInTypes struct{}
 
-func (_ noBuiltInTypes) IsBuiltinType(rt uintptr) bool           { return false }
+type noBuiltInTypeChecker struct{}
+
+func (_ noBuiltInTypeChecker) IsBuiltinType(rt uintptr) bool { return false }
+
+type noBuiltInTypes struct{ noBuiltInTypeChecker }
+
 func (_ noBuiltInTypes) EncodeBuiltin(rt uintptr, v interface{}) {}
 func (_ noBuiltInTypes) DecodeBuiltin(rt uintptr, v interface{}) {}
 
@@ -613,14 +760,14 @@ func (o extHandle) getExtForTag(tag uint64) *extTypeTagFn {
 	return nil
 }
 
+const maxLevelsEmbedding = 16
+
 type structFieldInfo struct {
 	encName   string // encode name
 	fieldName string // field name
 
-	// only one of 'i' or 'is' can be set. If 'i' is -1, then 'is' has been set.
-
-	is        []int // (recursive/embedded) field index in struct
-	i         int16 // field index in struct
+	is        [maxLevelsEmbedding]uint16 // (recursive/embedded) field index in struct
+	nis       uint8                      // num levels of embedding. if 1, then it's not embedded.
 	omitEmpty bool
 	toArray   bool // if field is _struct, is the toArray set?
 }
@@ -629,48 +776,31 @@ type structFieldInfo struct {
 // 	return si.encName == "" && len(si.is) == 0 && si.i == 0 && !si.omitEmpty && !si.toArray
 // }
 
+func (si *structFieldInfo) setToZeroValue(v reflect.Value) {
+	if v, valid := si.field(v, false); valid {
+		v.Set(reflect.Zero(v.Type()))
+	}
+}
+
 // rv returns the field of the struct.
 // If anonymous, it returns an Invalid
-func (si *structFieldInfo) field(v reflect.Value, update bool) (rv2 reflect.Value) {
-	if si.i != -1 {
-		v = v.Field(int(si.i))
-		return v
-	}
+func (si *structFieldInfo) field(v reflect.Value, update bool) (rv2 reflect.Value, valid bool) {
 	// replicate FieldByIndex
-	for _, x := range si.is {
-		for v.Kind() == reflect.Ptr {
-			if v.IsNil() {
-				if !update {
-					return
-				}
-				v.Set(reflect.New(v.Type().Elem()))
-			}
-			v = v.Elem()
+	for i, x := range si.is {
+		if uint8(i) == si.nis {
+			break
+		}
+		if v, valid = baseStructRv(v, update); !valid {
+			return
 		}
-		v = v.Field(x)
+		v = v.Field(int(x))
 	}
-	return v
+	return v, true
 }
 
-func (si *structFieldInfo) setToZeroValue(v reflect.Value) {
-	if si.i != -1 {
-		v = v.Field(int(si.i))
-		v.Set(reflect.Zero(v.Type()))
-		// v.Set(reflect.New(v.Type()).Elem())
-		// v.Set(reflect.New(v.Type()))
-	} else {
-		// replicate FieldByIndex
-		for _, x := range si.is {
-			for v.Kind() == reflect.Ptr {
-				if v.IsNil() {
-					return
-				}
-				v = v.Elem()
-			}
-			v = v.Field(x)
-		}
-		v.Set(reflect.Zero(v.Type()))
-	}
+func (si *structFieldInfo) fieldval(v reflect.Value, update bool) reflect.Value {
+	v, _ = si.field(v, update)
+	return v
 }
 
 func parseStructFieldInfo(fname string, stag string) *structFieldInfo {
@@ -714,6 +844,97 @@ func (p sfiSortedByEncName) Swap(i, j int) {
 	p[i], p[j] = p[j], p[i]
 }
 
+const structFieldNodeNumToCache = 4
+
+type structFieldNodeCache struct {
+	rv  [structFieldNodeNumToCache]reflect.Value
+	idx [structFieldNodeNumToCache]uint32
+	num uint8
+}
+
+func (x *structFieldNodeCache) get(key uint32) (fv reflect.Value, valid bool) {
+	// defer func() { fmt.Printf(">>>> found in cache2? %v\n", valid) }()
+	for i, k := range &x.idx {
+		if uint8(i) == x.num {
+			return // break
+		}
+		if key == k {
+			return x.rv[i], true
+		}
+	}
+	return
+}
+
+func (x *structFieldNodeCache) tryAdd(fv reflect.Value, key uint32) {
+	if x.num < structFieldNodeNumToCache {
+		x.rv[x.num] = fv
+		x.idx[x.num] = key
+		x.num++
+		return
+	}
+}
+
+type structFieldNode struct {
+	v      reflect.Value
+	cache2 structFieldNodeCache
+	cache3 structFieldNodeCache
+	update bool
+}
+
+func (x *structFieldNode) field(si *structFieldInfo) (fv reflect.Value) {
+	// return si.fieldval(x.v, x.update)
+	// Note: we only cache if nis=2 or nis=3 i.e. up to 2 levels of embedding
+	// This mostly saves us time on the repeated calls to v.Elem, v.Field, etc.
+	var valid bool
+	switch si.nis {
+	case 1:
+		fv = x.v.Field(int(si.is[0]))
+	case 2:
+		if fv, valid = x.cache2.get(uint32(si.is[0])); valid {
+			fv = fv.Field(int(si.is[1]))
+			return
+		}
+		fv = x.v.Field(int(si.is[0]))
+		if fv, valid = baseStructRv(fv, x.update); !valid {
+			return
+		}
+		x.cache2.tryAdd(fv, uint32(si.is[0]))
+		fv = fv.Field(int(si.is[1]))
+	case 3:
+		var key uint32 = uint32(si.is[0])<<16 | uint32(si.is[1])
+		if fv, valid = x.cache3.get(key); valid {
+			fv = fv.Field(int(si.is[2]))
+			return
+		}
+		fv = x.v.Field(int(si.is[0]))
+		if fv, valid = baseStructRv(fv, x.update); !valid {
+			return
+		}
+		fv = fv.Field(int(si.is[1]))
+		if fv, valid = baseStructRv(fv, x.update); !valid {
+			return
+		}
+		x.cache3.tryAdd(fv, key)
+		fv = fv.Field(int(si.is[2]))
+	default:
+		fv, _ = si.field(x.v, x.update)
+	}
+	return
+}
+
+func baseStructRv(v reflect.Value, update bool) (v2 reflect.Value, valid bool) {
+	for v.Kind() == reflect.Ptr {
+		if v.IsNil() {
+			if !update {
+				return
+			}
+			v.Set(reflect.New(v.Type().Elem()))
+		}
+		v = v.Elem()
+	}
+	return v, true
+}
+
 // typeInfo keeps information about each type referenced in the encode/decode sequence.
 //
 // During an encode/decode sequence, we work as below:
@@ -738,6 +959,8 @@ type typeInfo struct {
 	baseId    uintptr
 	baseIndir int8 // number of indirections to get to base
 
+	anyOmitEmpty bool
+
 	mbs bool // base type (T or *T) is a MapBySlice
 
 	bm        bool // base type (T or *T) is a binaryMarshaler
@@ -761,14 +984,16 @@ type typeInfo struct {
 	toArray bool // whether this (struct) type should be encoded as an array
 }
 
-// linear search. faster than binary search in my testing up to 16-field structs.
-const binarySearchThreshold = 8 // similar to what python does for hashtables
+// define length beyond which we do a binary search instead of a linear search.
+// From our testing, linear search seems faster than binary search up to 16-field structs.
+// However, we set to 8 similar to what python does for hashtables.
+const indexForEncNameBinarySearchThreshold = 8
 
 func (ti *typeInfo) indexForEncName(name string) int {
 	// NOTE: name may be a stringView, so don't pass it to another function.
 	//tisfi := ti.sfi
 	sfilen := len(ti.sfi)
-	if sfilen < binarySearchThreshold {
+	if sfilen < indexForEncNameBinarySearchThreshold {
 		for i, si := range ti.sfi {
 			if si.encName == name {
 				return i
@@ -829,7 +1054,6 @@ func (x *TypeInfos) structTag(t reflect.StructTag) (s string) {
 
 func (x *TypeInfos) find(sp *[]rtid2ti, rtid uintptr) (idx int, ti *typeInfo) {
 	// binary search. adapted from sort/search.go.
-	// fmt.Printf(">>>> calling typeinfos.find ... \n")
 	// if sp == nil {
 	// 	return -1, nil
 	// }
@@ -850,7 +1074,6 @@ func (x *TypeInfos) find(sp *[]rtid2ti, rtid uintptr) (idx int, ti *typeInfo) {
 }
 
 func (x *TypeInfos) get(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
-	// fmt.Printf(">>>> calling typeinfos.get ... \n")
 	sp := x.infos.load()
 	var idx int
 	if sp != nil {
@@ -916,13 +1139,13 @@ func (x *TypeInfos) get(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
 			ti.toArray = siInfo.toArray
 			omitEmpty = siInfo.omitEmpty
 		}
-		pi := rgetPool.Get()
-		pv := pi.(*rgetPoolT)
+		pp, pi := pool.tiLoad()
+		pv := pi.(*typeInfoLoadArray)
 		pv.etypes[0] = ti.baseId
-		vv := rgetT{pv.fNames[:0], pv.encNames[:0], pv.etypes[:1], pv.sfis[:0]}
+		vv := typeInfoLoad{pv.fNames[:0], pv.encNames[:0], pv.etypes[:1], pv.sfis[:0]}
 		x.rget(rt, rtid, omitEmpty, nil, &vv)
-		ti.sfip, ti.sfi = rgetResolveSFI(vv.sfis, pv.sfiidx[:0])
-		rgetPool.Put(pi)
+		ti.sfip, ti.sfi, ti.anyOmitEmpty = rgetResolveSFI(vv.sfis, pv.sfiidx[:0])
+		pp.Put(pi)
 	}
 	// sfi = sfip
 
@@ -930,7 +1153,6 @@ func (x *TypeInfos) get(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
 	x.mu.Lock()
 	sp = x.infos.load()
 	if sp == nil {
-		// fmt.Printf(">>>> in typeinfos.get: sp == nil\n")
 		pti = &ti
 		vs = []rtid2ti{{rtid, pti}}
 		x.infos.store(&vs)
@@ -947,12 +1169,11 @@ func (x *TypeInfos) get(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
 		}
 	}
 	x.mu.Unlock()
-	// fmt.Printf(">>>>>>> TypeInfos: Num Elements: %v\n", len(*(x.infos.load())))
 	return
 }
 
 func (x *TypeInfos) rget(rt reflect.Type, rtid uintptr, omitEmpty bool,
-	indexstack []int, pv *rgetT,
+	indexstack []uint16, pv *typeInfoLoad,
 ) {
 	// Read up fields and store how to access the value.
 	//
@@ -962,10 +1183,13 @@ func (x *TypeInfos) rget(rt reflect.Type, rtid uintptr, omitEmpty bool,
 	// Note: we consciously use slices, not a map, to simulate a set.
 	//       Typically, types have < 16 fields,
 	//       and iteration using equals is faster than maps there
-
+	flen := rt.NumField()
+	if flen > (1<<16 - 1) {
+		panic(fmt.Errorf("codec: types with more than %v fields are not supported - has %v fields", (1<<16 - 1), flen))
+	}
 LOOP:
-	for j, jlen := 0, rt.NumField(); j < jlen; j++ {
-		f := rt.Field(j)
+	for j, jlen := uint16(0), uint16(flen); j < jlen; j++ {
+		f := rt.Field(int(j))
 		fkind := f.Type.Kind()
 		// skip if a func type, or is unexported, or structTag value == "-"
 		switch fkind {
@@ -1017,7 +1241,7 @@ LOOP:
 					}
 					if processIt {
 						pv.etypes = append(pv.etypes, ftid)
-						indexstack2 := make([]int, len(indexstack)+1)
+						indexstack2 := make([]uint16, len(indexstack)+1)
 						copy(indexstack2, indexstack)
 						indexstack2[len(indexstack)] = j
 						// indexstack2 := append(append(make([]int, 0, len(indexstack)+4), indexstack...), j)
@@ -1049,15 +1273,12 @@ LOOP:
 		pv.encNames = append(pv.encNames, si.encName)
 
 		// si.ikind = int(f.Type.Kind())
-		if len(indexstack) == 0 {
-			si.i = int16(j)
-		} else {
-			si.i = -1
-			si.is = make([]int, len(indexstack)+1)
-			copy(si.is, indexstack)
-			si.is[len(indexstack)] = j
-			// si.is = append(append(make([]int, 0, len(indexstack)+4), indexstack...), j)
+		if len(indexstack) > maxLevelsEmbedding-1 {
+			panic(fmt.Errorf("codec: only supports up to %v depth of embedding - type has %v depth", maxLevelsEmbedding-1, len(indexstack)))
 		}
+		si.nis = uint8(len(indexstack)) + 1
+		copy(si.is[:], indexstack)
+		si.is[len(indexstack)] = j
 
 		if omitEmpty {
 			si.omitEmpty = true
@@ -1068,7 +1289,7 @@ LOOP:
 
 // resolves the struct field info got from a call to rget.
 // Returns a trimmed, unsorted and sorted []*structFieldInfo.
-func rgetResolveSFI(x []*structFieldInfo, pv []sfiIdx) (y, z []*structFieldInfo) {
+func rgetResolveSFI(x []*structFieldInfo, pv []sfiIdx) (y, z []*structFieldInfo, anyOmitEmpty bool) {
 	var n int
 	for i, v := range x {
 		xn := v.encName //TODO: fieldName or encName? use encName for now.
@@ -1105,6 +1326,9 @@ func rgetResolveSFI(x []*structFieldInfo, pv []sfiIdx) (y, z []*structFieldInfo)
 		if v == nil {
 			continue
 		}
+		if !anyOmitEmpty && v.omitEmpty {
+			anyOmitEmpty = true
+		}
 		y[n] = v
 		n++
 	}
@@ -1115,15 +1339,33 @@ func rgetResolveSFI(x []*structFieldInfo, pv []sfiIdx) (y, z []*structFieldInfo)
 	return
 }
 
+func xprintf(format string, a ...interface{}) {
+	if xDebug {
+		fmt.Fprintf(os.Stderr, format, a...)
+	}
+}
+
 func panicToErr(err *error) {
 	if recoverPanicToErr {
 		if x := recover(); x != nil {
-			//debug.PrintStack()
+			// if false && xDebug {
+			// 	fmt.Printf("panic'ing with: %v\n", x)
+			// 	debug.PrintStack()
+			// }
 			panicValToErr(x, err)
 		}
 	}
 }
 
+func panicToErrs2(err1, err2 *error) {
+	if recoverPanicToErr {
+		if x := recover(); x != nil {
+			panicValToErr(x, err1)
+			panicValToErr(x, err2)
+		}
+	}
+}
+
 // func doPanic(tag string, format string, params ...interface{}) {
 // 	params2 := make([]interface{}, len(params)+1)
 // 	params2[0] = tag
@@ -1151,6 +1393,247 @@ func isImmutableKind(k reflect.Kind) (v bool) {
 	// 	k == reflect.String
 }
 
+// ----
+
+type codecFnInfo struct {
+	ti    *typeInfo
+	xfFn  Ext
+	xfTag uint64
+	seq   seqType
+	addr  bool
+}
+
+// codecFn encapsulates the captured variables and the encode function.
+// This way, we only do some calculations one times, and pass to the
+// code block that should be called (encapsulated in a function)
+// instead of executing the checks every time.
+type codecFn struct {
+	i  codecFnInfo
+	fe func(*Encoder, *codecFnInfo, reflect.Value)
+	fd func(*Decoder, *codecFnInfo, reflect.Value)
+}
+
+type codecRtidFn struct {
+	rtid uintptr
+	fn   codecFn
+}
+
+type codecFner struct {
+	hh Handle
+	h  *BasicHandle
+	cs [arrayCacheLen]*[arrayCacheLen]codecRtidFn
+	s  []*[arrayCacheLen]codecRtidFn
+	sn uint32
+	be bool
+	js bool
+	cf [arrayCacheLen]codecRtidFn
+}
+
+func (c *codecFner) reset(hh Handle) {
+	c.hh = hh
+	c.h = hh.getBasicHandle()
+	_, c.js = hh.(*JsonHandle)
+	c.be = hh.isBinary()
+}
+
+func (c *codecFner) get(rt reflect.Type, checkFastpath, checkCodecSelfer bool) (fn *codecFn) {
+	rtid := rt2id(rt)
+	var j uint32
+	var sn uint32 = c.sn
+	if sn == 0 {
+		c.s = c.cs[:1]
+		c.s[0] = &c.cf
+		c.cf[0].rtid = rtid
+		fn = &(c.cf[0].fn)
+		c.sn = 1
+	} else {
+	LOOP1:
+		for _, x := range c.s {
+			for i := range x {
+				if j == sn {
+					break LOOP1
+				}
+				if x[i].rtid == rtid {
+					fn = &(x[i].fn)
+					return
+				}
+				j++
+			}
+		}
+		sx, sy := sn/arrayCacheLen, sn%arrayCacheLen
+		if sy == 0 {
+			c.s = append(c.s, &[arrayCacheLen]codecRtidFn{})
+		}
+		c.s[sx][sy].rtid = rtid
+		fn = &(c.s[sx][sy].fn)
+		c.sn++
+	}
+
+	ti := c.h.getTypeInfo(rtid, rt)
+	fi := &(fn.i)
+	fi.ti = ti
+
+	if checkCodecSelfer && ti.cs {
+		fn.fe = (*Encoder).selferMarshal
+		fn.fd = (*Decoder).selferUnmarshal
+	} else if rtid == rawTypId {
+		fn.fe = (*Encoder).raw
+		fn.fd = (*Decoder).raw
+	} else if rtid == rawExtTypId {
+		fn.fe = (*Encoder).rawExt
+		fn.fd = (*Decoder).rawExt
+		fn.i.addr = true
+	} else if c.hh.IsBuiltinType(rtid) {
+		fn.fe = (*Encoder).builtin
+		fn.fd = (*Decoder).builtin
+		fn.i.addr = true
+	} else if xfFn := c.h.getExt(rtid); xfFn != nil {
+		fi.xfTag, fi.xfFn = xfFn.tag, xfFn.ext
+		fn.fe = (*Encoder).ext
+		fn.fd = (*Decoder).ext
+		fn.i.addr = true
+	} else if supportMarshalInterfaces && c.be && ti.bm {
+		fn.fe = (*Encoder).binaryMarshal
+		fn.fd = (*Decoder).binaryUnmarshal
+	} else if supportMarshalInterfaces && !c.be && c.js && ti.jm {
+		//If JSON, we should check JSONMarshal before textMarshal
+		fn.fe = (*Encoder).jsonMarshal
+		fn.fd = (*Decoder).jsonUnmarshal
+	} else if supportMarshalInterfaces && !c.be && ti.tm {
+		fn.fe = (*Encoder).textMarshal
+		fn.fd = (*Decoder).textUnmarshal
+	} else {
+		rk := rt.Kind()
+		if fastpathEnabled && checkFastpath && (rk == reflect.Map || rk == reflect.Slice) {
+			if rt.PkgPath() == "" { // un-named slice or map
+				if idx := fastpathAV.index(rtid); idx != -1 {
+					fn.fe = fastpathAV[idx].encfn
+					fn.fd = fastpathAV[idx].decfn
+					fn.i.addr = true
+				}
+			} else {
+				// use mapping for underlying type if there
+				var rtu reflect.Type
+				if rk == reflect.Map {
+					rtu = reflect.MapOf(rt.Key(), rt.Elem())
+				} else {
+					rtu = reflect.SliceOf(rt.Elem())
+				}
+				rtuid := rt2id(rtu)
+				if idx := fastpathAV.index(rtuid); idx != -1 {
+					xfnf := fastpathAV[idx].encfn
+					xrt := fastpathAV[idx].rt
+					fn.fe = func(e *Encoder, xf *codecFnInfo, xrv reflect.Value) {
+						xfnf(e, xf, xrv.Convert(xrt))
+					}
+					fn.i.addr = true
+					xfnf2 := fastpathAV[idx].decfn
+					fn.fd = func(d *Decoder, xf *codecFnInfo, xrv reflect.Value) {
+						xfnf2(d, xf, xrv.Convert(reflect.PtrTo(xrt)))
+					}
+				}
+			}
+		}
+		if fn.fe == nil && fn.fd == nil {
+			switch rk {
+			case reflect.Bool:
+				fn.fe = (*Encoder).kBool
+				fn.fd = (*Decoder).kBool
+			case reflect.String:
+				fn.fe = (*Encoder).kString
+				fn.fd = (*Decoder).kString
+			case reflect.Int:
+				fn.fd = (*Decoder).kInt
+				fn.fe = (*Encoder).kInt
+			case reflect.Int8:
+				fn.fe = (*Encoder).kInt
+				fn.fd = (*Decoder).kInt8
+			case reflect.Int16:
+				fn.fe = (*Encoder).kInt
+				fn.fd = (*Decoder).kInt16
+			case reflect.Int32:
+				fn.fe = (*Encoder).kInt
+				fn.fd = (*Decoder).kInt32
+			case reflect.Int64:
+				fn.fe = (*Encoder).kInt
+				fn.fd = (*Decoder).kInt64
+			case reflect.Uint:
+				fn.fd = (*Decoder).kUint
+				fn.fe = (*Encoder).kUint
+			case reflect.Uint8:
+				fn.fe = (*Encoder).kUint
+				fn.fd = (*Decoder).kUint8
+			case reflect.Uint16:
+				fn.fe = (*Encoder).kUint
+				fn.fd = (*Decoder).kUint16
+			case reflect.Uint32:
+				fn.fe = (*Encoder).kUint
+				fn.fd = (*Decoder).kUint32
+			case reflect.Uint64:
+				fn.fe = (*Encoder).kUint
+				fn.fd = (*Decoder).kUint64
+				// case reflect.Ptr:
+				// 	fn.fd = (*Decoder).kPtr
+			case reflect.Uintptr:
+				fn.fe = (*Encoder).kUint
+				fn.fd = (*Decoder).kUintptr
+			case reflect.Float32:
+				fn.fe = (*Encoder).kFloat32
+				fn.fd = (*Decoder).kFloat32
+			case reflect.Float64:
+				fn.fe = (*Encoder).kFloat64
+				fn.fd = (*Decoder).kFloat64
+			case reflect.Invalid:
+				fn.fe = (*Encoder).kInvalid
+			case reflect.Chan:
+				fi.seq = seqTypeChan
+				fn.fe = (*Encoder).kSlice
+				fn.fd = (*Decoder).kSlice
+			case reflect.Slice:
+				fi.seq = seqTypeSlice
+				fn.fe = (*Encoder).kSlice
+				fn.fd = (*Decoder).kSlice
+			case reflect.Array:
+				fi.seq = seqTypeArray
+				fn.fe = (*Encoder).kSlice
+				fi.addr = false
+				rt2 := reflect.SliceOf(rt.Elem())
+				fn.fd = func(d *Decoder, xf *codecFnInfo, xrv reflect.Value) {
+					// println(">>>>>> decoding an array ... ")
+					d.cf.get(rt2, true, false).fd(d, xf, xrv.Slice(0, xrv.Len()))
+					// println(">>>>>> decoding an array ... DONE")
+				}
+				// fn.fd = (*Decoder).kArray
+			case reflect.Struct:
+				if ti.anyOmitEmpty {
+					fn.fe = (*Encoder).kStruct
+				} else {
+					fn.fe = (*Encoder).kStructNoOmitempty
+				}
+				fn.fd = (*Decoder).kStruct
+				// reflect.Ptr and reflect.Interface are handled already by preEncodeValue
+				// case reflect.Ptr:
+				// 	fn.fe = (*Encoder).kPtr
+				// case reflect.Interface:
+				// 	fn.fe = (*Encoder).kInterface
+			case reflect.Map:
+				fn.fe = (*Encoder).kMap
+				fn.fd = (*Decoder).kMap
+			case reflect.Interface:
+				// encode: reflect.Interface are handled already by preEncodeValue
+				fn.fd = (*Decoder).kInterface
+			default:
+				fn.fe = (*Encoder).kErr
+				fn.fd = (*Decoder).kErr
+			}
+		}
+	}
+
+	return
+}
+
+// ----
+
 // these functions must be inlinable, and not call anybody
 type checkOverflow struct{}
 
@@ -1208,6 +1691,7 @@ func isNaN(f float64) bool { return f != f }
 
 type intSlice []int64
 type uintSlice []uint64
+type uintptrSlice []uintptr
 type floatSlice []float64
 type boolSlice []bool
 type stringSlice []string
@@ -1221,6 +1705,10 @@ func (p uintSlice) Len() int           { return len(p) }
 func (p uintSlice) Less(i, j int) bool { return p[i] < p[j] }
 func (p uintSlice) Swap(i, j int)      { p[i], p[j] = p[j], p[i] }
 
+func (p uintptrSlice) Len() int           { return len(p) }
+func (p uintptrSlice) Less(i, j int) bool { return p[i] < p[j] }
+func (p uintptrSlice) Swap(i, j int)      { p[i], p[j] = p[j], p[i] }
+
 func (p floatSlice) Len() int { return len(p) }
 func (p floatSlice) Less(i, j int) bool {
 	return p[i] < p[j] || isNaN(p[i]) && !isNaN(p[j])
@@ -1317,7 +1805,6 @@ type set []uintptr
 
 func (s *set) add(v uintptr) (exists bool) {
 	// e.ci is always nil, or len >= 1
-	// defer func() { fmt.Printf("$$$$$$$$$$$ cirRef Add: %v, exists: %v\n", v, exists) }()
 	x := *s
 	if x == nil {
 		x = make([]uintptr, 1, 8)
@@ -1358,7 +1845,6 @@ func (s *set) add(v uintptr) (exists bool) {
 }
 
 func (s *set) remove(v uintptr) (exists bool) {
-	// defer func() { fmt.Printf("$$$$$$$$$$$ cirRef Rm: %v, exists: %v\n", v, exists) }()
 	x := *s
 	if len(x) == 0 {
 		return
@@ -1380,3 +1866,77 @@ func (s *set) remove(v uintptr) (exists bool) {
 	}
 	return
 }
+
+// ------
+
+// bitset types are better than [256]bool, because they permit the whole
+// bitset array being on a single cache line and use less memory.
+
+// given x > 0 and n > 0 and x is exactly 2^n, then pos/x === pos>>n AND pos%x === pos&(x-1).
+// consequently, pos/32 === pos>>5, pos/16 === pos>>4, pos/8 === pos>>3, pos%8 == pos&7
+
+type bitset256 [32]byte
+
+func (x *bitset256) set(pos byte) {
+	x[pos>>3] |= (1 << (pos & 7))
+}
+func (x *bitset256) unset(pos byte) {
+	x[pos>>3] &^= (1 << (pos & 7))
+}
+func (x *bitset256) isset(pos byte) bool {
+	return x[pos>>3]&(1<<(pos&7)) != 0
+}
+
+type bitset128 [16]byte
+
+func (x *bitset128) set(pos byte) {
+	x[pos>>3] |= (1 << (pos & 7))
+}
+func (x *bitset128) unset(pos byte) {
+	x[pos>>3] &^= (1 << (pos & 7))
+}
+func (x *bitset128) isset(pos byte) bool {
+	return x[pos>>3]&(1<<(pos&7)) != 0
+}
+
+// ------------
+
+type pooler struct {
+	// for stringRV
+	strRv8, strRv16, strRv32, strRv64, strRv128 sync.Pool
+	// for the decNaked
+	dn     sync.Pool
+	tiload sync.Pool
+}
+
+func (p *pooler) init() {
+	p.strRv8.New = func() interface{} { return new([8]stringRv) }
+	p.strRv16.New = func() interface{} { return new([16]stringRv) }
+	p.strRv32.New = func() interface{} { return new([32]stringRv) }
+	p.strRv64.New = func() interface{} { return new([64]stringRv) }
+	p.strRv128.New = func() interface{} { return new([128]stringRv) }
+	p.dn.New = func() interface{} { x := new(decNaked); x.init(); return x }
+	p.tiload.New = func() interface{} { return new(typeInfoLoadArray) }
+}
+
+func (p *pooler) stringRv8() (sp *sync.Pool, v interface{}) {
+	return &p.strRv8, p.strRv8.Get()
+}
+func (p *pooler) stringRv16() (sp *sync.Pool, v interface{}) {
+	return &p.strRv16, p.strRv16.Get()
+}
+func (p *pooler) stringRv32() (sp *sync.Pool, v interface{}) {
+	return &p.strRv32, p.strRv32.Get()
+}
+func (p *pooler) stringRv64() (sp *sync.Pool, v interface{}) {
+	return &p.strRv64, p.strRv64.Get()
+}
+func (p *pooler) stringRv128() (sp *sync.Pool, v interface{}) {
+	return &p.strRv128, p.strRv128.Get()
+}
+func (p *pooler) decNaked() (sp *sync.Pool, v interface{}) {
+	return &p.dn, p.dn.Get()
+}
+func (p *pooler) tiLoad() (sp *sync.Pool, v interface{}) {
+	return &p.tiload, p.tiload.Get()
+}

codec/helper_not_unsafe.go

@@ -30,6 +30,19 @@ func bytesView(v string) []byte {
 	return []byte(v)
 }
 
+func definitelyNil(v interface{}) bool {
+	return false
+	// rv := reflect.ValueOf(v)
+	// switch rv.Kind() {
+	// case reflect.Invalid:
+	// 	return true
+	// case reflect.Ptr, reflect.Interface, reflect.Chan, reflect.Slice, reflect.Map, reflect.Func:
+	// 	return rv.IsNil()
+	// default:
+	// 	return false
+	// }
+}
+
 // // keepAlive4BytesView maintains a reference to the input parameter for bytesView.
 // //
 // // Usage: call this at point where done with the bytes view.
@@ -48,13 +61,17 @@ func rt2id(rt reflect.Type) uintptr {
 	return reflect.ValueOf(rt).Pointer()
 }
 
+func rv2rtid(rv reflect.Value) uintptr {
+	return reflect.ValueOf(rv.Type()).Pointer()
+}
+
 // --------------------------
-type ptrToRvMap struct{}
+// type ptrToRvMap struct{}
 
-func (_ *ptrToRvMap) init() {}
-func (_ *ptrToRvMap) get(i interface{}) reflect.Value {
-	return reflect.ValueOf(i).Elem()
-}
+// func (_ *ptrToRvMap) init() {}
+// func (_ *ptrToRvMap) get(i interface{}) reflect.Value {
+// 	return reflect.ValueOf(i).Elem()
+// }
 
 // --------------------------
 type atomicTypeInfoSlice struct {
@@ -74,70 +91,66 @@ func (x *atomicTypeInfoSlice) store(p *[]rtid2ti) {
 }
 
 // --------------------------
-func (f *decFnInfo) raw(rv reflect.Value) {
-	rv.SetBytes(f.d.raw())
+func (d *Decoder) raw(f *codecFnInfo, rv reflect.Value) {
+	rv.SetBytes(d.rawBytes())
 }
 
-func (f *decFnInfo) kString(rv reflect.Value) {
-	rv.SetString(f.d.d.DecodeString())
+func (d *Decoder) kString(f *codecFnInfo, rv reflect.Value) {
+	rv.SetString(d.d.DecodeString())
 }
 
-func (f *decFnInfo) kBool(rv reflect.Value) {
-	rv.SetBool(f.d.d.DecodeBool())
+func (d *Decoder) kBool(f *codecFnInfo, rv reflect.Value) {
+	rv.SetBool(d.d.DecodeBool())
 }
 
-func (f *decFnInfo) kFloat32(rv reflect.Value) {
-	rv.SetFloat(f.d.d.DecodeFloat(true))
+func (d *Decoder) kFloat32(f *codecFnInfo, rv reflect.Value) {
+	rv.SetFloat(d.d.DecodeFloat(true))
 }
 
-func (f *decFnInfo) kFloat64(rv reflect.Value) {
-	rv.SetFloat(f.d.d.DecodeFloat(false))
+func (d *Decoder) kFloat64(f *codecFnInfo, rv reflect.Value) {
+	rv.SetFloat(d.d.DecodeFloat(false))
 }
 
-func (f *decFnInfo) kInt(rv reflect.Value) {
-	rv.SetInt(f.d.d.DecodeInt(intBitsize))
+func (d *Decoder) kInt(f *codecFnInfo, rv reflect.Value) {
+	rv.SetInt(d.d.DecodeInt(intBitsize))
 }
 
-func (f *decFnInfo) kInt8(rv reflect.Value) {
-	rv.SetInt(f.d.d.DecodeInt(8))
+func (d *Decoder) kInt8(f *codecFnInfo, rv reflect.Value) {
+	rv.SetInt(d.d.DecodeInt(8))
 }
 
-func (f *decFnInfo) kInt16(rv reflect.Value) {
-	rv.SetInt(f.d.d.DecodeInt(16))
+func (d *Decoder) kInt16(f *codecFnInfo, rv reflect.Value) {
+	rv.SetInt(d.d.DecodeInt(16))
 }
 
-func (f *decFnInfo) kInt32(rv reflect.Value) {
-	rv.SetInt(f.d.d.DecodeInt(32))
+func (d *Decoder) kInt32(f *codecFnInfo, rv reflect.Value) {
+	rv.SetInt(d.d.DecodeInt(32))
 }
 
-func (f *decFnInfo) kInt64(rv reflect.Value) {
-	rv.SetInt(f.d.d.DecodeInt(64))
+func (d *Decoder) kInt64(f *codecFnInfo, rv reflect.Value) {
+	rv.SetInt(d.d.DecodeInt(64))
 }
 
-func (f *decFnInfo) kUint(rv reflect.Value) {
-	rv.SetUint(f.d.d.DecodeUint(uintBitsize))
+func (d *Decoder) kUint(f *codecFnInfo, rv reflect.Value) {
+	rv.SetUint(d.d.DecodeUint(uintBitsize))
 }
 
-func (f *decFnInfo) kUintptr(rv reflect.Value) {
-	rv.SetUint(f.d.d.DecodeUint(uintBitsize))
+func (d *Decoder) kUintptr(f *codecFnInfo, rv reflect.Value) {
+	rv.SetUint(d.d.DecodeUint(uintBitsize))
 }
 
-func (f *decFnInfo) kUint8(rv reflect.Value) {
-	rv.SetUint(f.d.d.DecodeUint(8))
+func (d *Decoder) kUint8(f *codecFnInfo, rv reflect.Value) {
+	rv.SetUint(d.d.DecodeUint(8))
 }
 
-func (f *decFnInfo) kUint16(rv reflect.Value) {
-	rv.SetUint(f.d.d.DecodeUint(16))
+func (d *Decoder) kUint16(f *codecFnInfo, rv reflect.Value) {
+	rv.SetUint(d.d.DecodeUint(16))
 }
 
-func (f *decFnInfo) kUint32(rv reflect.Value) {
-	rv.SetUint(f.d.d.DecodeUint(32))
+func (d *Decoder) kUint32(f *codecFnInfo, rv reflect.Value) {
+	rv.SetUint(d.d.DecodeUint(32))
 }
 
-func (f *decFnInfo) kUint64(rv reflect.Value) {
-	rv.SetUint(f.d.d.DecodeUint(64))
+func (d *Decoder) kUint64(f *codecFnInfo, rv reflect.Value) {
+	rv.SetUint(d.d.DecodeUint(64))
 }
-
-// func i2rv(i interface{}) reflect.Value {
-// 	return reflect.ValueOf(i)
-// }

codec/helper_unsafe.go

@@ -18,6 +18,8 @@ import (
 
 // var zeroRTv [4]uintptr
 
+const unsafeFlagIndir = 1 << 7 // keep in sync with GO_ROOT/src/reflect/value.go
+
 type unsafeString struct {
 	Data uintptr
 	Len  int
@@ -60,6 +62,10 @@ func bytesView(v string) []byte {
 	return *(*[]byte)(unsafe.Pointer(&bx))
 }
 
+func definitelyNil(v interface{}) bool {
+	return (*unsafeIntf)(unsafe.Pointer(&v)).word == nil
+}
+
 // func keepAlive4BytesView(v string) {
 // 	runtime.KeepAlive(v)
 // }
@@ -68,21 +74,19 @@ func bytesView(v string) []byte {
 // 	runtime.KeepAlive(v)
 // }
 
-const _unsafe_rv2i_is_safe = false
-
 // TODO: consider a more generally-known optimization for reflect.Value ==> Interface
 //
 // Currently, we use this fragile method that taps into implememtation details from
 // the source go stdlib reflect/value.go,
 // and trims the implementation.
 func rv2i(rv reflect.Value) interface{} {
-	if _unsafe_rv2i_is_safe {
+	if false {
 		return rv.Interface()
 	}
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
 	// references that are single-words (map, ptr) may be double-referenced as flagIndir
 	kk := urv.flag & (1<<5 - 1)
-	if (kk == uintptr(reflect.Map) || kk == uintptr(reflect.Ptr)) && urv.flag&(1<<7) != 0 {
+	if (kk == uintptr(reflect.Map) || kk == uintptr(reflect.Ptr)) && urv.flag&unsafeFlagIndir != 0 {
 		return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: *(*unsafe.Pointer)(urv.ptr), typ: urv.typ}))
 	}
 	return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: urv.ptr, typ: urv.typ}))
@@ -92,6 +96,10 @@ func rt2id(rt reflect.Type) uintptr {
 	return uintptr(((*unsafeIntf)(unsafe.Pointer(&rt))).word)
 }
 
+func rv2rtid(rv reflect.Value) uintptr {
+	return uintptr((*unsafeReflectValue)(unsafe.Pointer(&rv)).typ)
+}
+
 // func rv0t(rt reflect.Type) reflect.Value {
 // 	ut := (*unsafeIntf)(unsafe.Pointer(&rt))
 // 	// we need to determine whether ifaceIndir, and then whether to just pass 0 as the ptr
@@ -99,48 +107,6 @@ func rt2id(rt reflect.Type) uintptr {
 // 	return *(*reflect.Value)(unsafe.Pointer(&uv})
 // }
 
-type ptrToRVKV struct {
-	k uintptr
-	v reflect.Value
-}
-
-type ptrToRvMap struct {
-	// m map[uintptr]reflect.Value
-	a [4]ptrToRVKV
-	v []ptrToRVKV
-}
-
-func (p *ptrToRvMap) init() {
-	// fmt.Printf(">>>> new ptr to rv map\n")
-	// p.m = make(map[uintptr]reflect.Value, 32)
-	p.v = p.a[:0]
-}
-
-func (p *ptrToRvMap) get(intf interface{}) (rv reflect.Value) {
-	word := uintptr(((*unsafeIntf)(unsafe.Pointer(&intf))).word)
-	// binary search. adapted from sort/search.go.
-	h, i, j := 0, 0, len(p.v)
-	for i < j {
-		h = i + (j-i)/2
-		if p.v[h].k < word {
-			i = h + 1
-		} else {
-			j = h
-		}
-	}
-	if i < len(p.v) && p.v[i].k == word {
-		return p.v[i].v
-	}
-
-	// insert into position i
-	// fmt.Printf(">>>> resetting rv for word: %x, interface: %v\n", word, intf)
-	rv = reflect.ValueOf(intf).Elem()
-	p.v = append(p.v, ptrToRVKV{})
-	copy(p.v[i+1:len(p.v)], p.v[i:len(p.v)-1])
-	p.v[i].k, p.v[i].v = word, rv
-	return
-}
-
 // --------------------------
 type atomicTypeInfoSlice struct {
 	v unsafe.Pointer
@@ -155,95 +121,90 @@ func (x *atomicTypeInfoSlice) store(p *[]rtid2ti) {
 }
 
 // --------------------------
-func (f *decFnInfo) raw(rv reflect.Value) {
+func (d *Decoder) raw(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*[]byte)(urv.ptr) = f.d.raw()
+	// if urv.flag&unsafeFlagIndir != 0 {
+	// 	urv.ptr = *(*unsafe.Pointer)(urv.ptr)
+	// }
+	*(*[]byte)(urv.ptr) = d.rawBytes()
 }
 
-func (f *decFnInfo) kString(rv reflect.Value) {
+func (d *Decoder) kString(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*string)(urv.ptr) = f.d.d.DecodeString()
+	*(*string)(urv.ptr) = d.d.DecodeString()
 }
 
-func (f *decFnInfo) kBool(rv reflect.Value) {
+func (d *Decoder) kBool(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*bool)(urv.ptr) = f.d.d.DecodeBool()
+	*(*bool)(urv.ptr) = d.d.DecodeBool()
 }
 
-func (f *decFnInfo) kFloat32(rv reflect.Value) {
+func (d *Decoder) kFloat32(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*float32)(urv.ptr) = float32(f.d.d.DecodeFloat(true))
+	*(*float32)(urv.ptr) = float32(d.d.DecodeFloat(true))
 }
 
-func (f *decFnInfo) kFloat64(rv reflect.Value) {
+func (d *Decoder) kFloat64(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*float64)(urv.ptr) = f.d.d.DecodeFloat(false)
+	*(*float64)(urv.ptr) = d.d.DecodeFloat(false)
 }
 
-func (f *decFnInfo) kInt(rv reflect.Value) {
+func (d *Decoder) kInt(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*int)(urv.ptr) = int(f.d.d.DecodeInt(intBitsize))
+	*(*int)(urv.ptr) = int(d.d.DecodeInt(intBitsize))
 }
 
-func (f *decFnInfo) kInt8(rv reflect.Value) {
+func (d *Decoder) kInt8(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*int8)(urv.ptr) = int8(f.d.d.DecodeInt(8))
+	*(*int8)(urv.ptr) = int8(d.d.DecodeInt(8))
 }
 
-func (f *decFnInfo) kInt16(rv reflect.Value) {
+func (d *Decoder) kInt16(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*int16)(urv.ptr) = int16(f.d.d.DecodeInt(16))
+	*(*int16)(urv.ptr) = int16(d.d.DecodeInt(16))
 }
 
-func (f *decFnInfo) kInt32(rv reflect.Value) {
+func (d *Decoder) kInt32(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*int32)(urv.ptr) = int32(f.d.d.DecodeInt(32))
+	*(*int32)(urv.ptr) = int32(d.d.DecodeInt(32))
 }
 
-func (f *decFnInfo) kInt64(rv reflect.Value) {
+func (d *Decoder) kInt64(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*int64)(urv.ptr) = f.d.d.DecodeInt(64)
+	*(*int64)(urv.ptr) = d.d.DecodeInt(64)
 }
 
-func (f *decFnInfo) kUint(rv reflect.Value) {
+func (d *Decoder) kUint(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*uint)(urv.ptr) = uint(f.d.d.DecodeUint(uintBitsize))
+	*(*uint)(urv.ptr) = uint(d.d.DecodeUint(uintBitsize))
 }
 
-func (f *decFnInfo) kUintptr(rv reflect.Value) {
+func (d *Decoder) kUintptr(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*uintptr)(urv.ptr) = uintptr(f.d.d.DecodeUint(uintBitsize))
+	*(*uintptr)(urv.ptr) = uintptr(d.d.DecodeUint(uintBitsize))
 }
 
-func (f *decFnInfo) kUint8(rv reflect.Value) {
+func (d *Decoder) kUint8(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*uint8)(urv.ptr) = uint8(f.d.d.DecodeUint(8))
+	*(*uint8)(urv.ptr) = uint8(d.d.DecodeUint(8))
 }
 
-func (f *decFnInfo) kUint16(rv reflect.Value) {
+func (d *Decoder) kUint16(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*uint16)(urv.ptr) = uint16(f.d.d.DecodeUint(16))
+	*(*uint16)(urv.ptr) = uint16(d.d.DecodeUint(16))
 }
 
-func (f *decFnInfo) kUint32(rv reflect.Value) {
+func (d *Decoder) kUint32(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*uint32)(urv.ptr) = uint32(f.d.d.DecodeUint(32))
+	*(*uint32)(urv.ptr) = uint32(d.d.DecodeUint(32))
 }
-func (f *decFnInfo) kUint64(rv reflect.Value) {
+
+func (d *Decoder) kUint64(f *codecFnInfo, rv reflect.Value) {
 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
-	*(*uint64)(urv.ptr) = f.d.d.DecodeUint(64)
+	*(*uint64)(urv.ptr) = d.d.DecodeUint(64)
 }
 
-// func (p *ptrToRvMap) get(i interface{}) (rv reflect.Value) {
-// 	word := uintptr(((*unsafeIntf)(unsafe.Pointer(&i))).word)
-// 	rv, exists := p.m[word]
-// 	if !exists {
-// 		fmt.Printf(">>>> resetting rv for word: %x, interface: %v\n", word, i)
-// 		rv = reflect.ValueOf(i).Elem()
-// 		p.m[word] = rv
-// 	}
-// 	return
-// }
+// ------------
 
 // func rt2id(rt reflect.Type) uintptr {
 // 	return uintptr(((*unsafeIntf)(unsafe.Pointer(&rt))).word)
@@ -455,8 +416,3 @@ func (f *decFnInfo) kUint64(rv reflect.Value) {
 // 	return *(*interface{})(unsafe.Pointer(&ui))
 // 	// return i
 // }
-
-// func i2rv(i interface{}) reflect.Value {
-// 	// u := *(*unsafeIntf)(unsafe.Pointer(&i))
-// 	return reflect.ValueOf(i)
-// }

codec/json.go

@@ -43,7 +43,7 @@ import (
 //--------------------------------
 
 var (
-	jsonLiterals = [...]byte{'t', 'r', 'u', 'e', 'f', 'a', 'l', 's', 'e', 'n', 'u', 'l', 'l'}
+	// jsonLiterals = [...]byte{'t', 'r', 'u', 'e', 'f', 'a', 'l', 's', 'e', 'n', 'u', 'l', 'l'}
 
 	jsonFloat64Pow10 = [...]float64{
 		1e0, 1e1, 1e2, 1e3, 1e4, 1e5, 1e6, 1e7, 1e8, 1e9,
@@ -59,10 +59,10 @@ var (
 	// jsonTabs and jsonSpaces are used as caches for indents
 	jsonTabs, jsonSpaces string
 
-	jsonCharHtmlSafeSet   [utf8.RuneSelf]bool
-	jsonCharSafeSet       [utf8.RuneSelf]bool
-	jsonCharWhitespaceSet [256]bool
-	jsonNumSet            [256]bool
+	jsonCharHtmlSafeSet   bitset128
+	jsonCharSafeSet       bitset128
+	jsonCharWhitespaceSet bitset256
+	jsonNumSet            bitset256
 )
 
 const (
@@ -100,25 +100,23 @@ func init() {
 	// populate the safe values as true: note: ASCII control characters are (0-31)
 	// jsonCharSafeSet:     all true except (0-31) " \
 	// jsonCharHtmlSafeSet: all true except (0-31) " \ < > &
-	for i := 32; i < utf8.RuneSelf; i++ {
+	var i byte
+	for i = 32; i < utf8.RuneSelf; i++ {
 		switch i {
 		case '"', '\\':
-			jsonCharSafeSet[i] = false
-			jsonCharHtmlSafeSet[i] = false
 		case '<', '>', '&':
-			jsonCharHtmlSafeSet[i] = false
-			jsonCharSafeSet[i] = true
+			jsonCharSafeSet.set(i) // = true
 		default:
-			jsonCharSafeSet[i] = true
-			jsonCharHtmlSafeSet[i] = true
+			jsonCharSafeSet.set(i)
+			jsonCharHtmlSafeSet.set(i)
 		}
 	}
-	for i := 0; i < 256; i++ {
+	for i = 0; i <= utf8.RuneSelf; i++ {
 		switch i {
 		case ' ', '\t', '\r', '\n':
-			jsonCharWhitespaceSet[i] = true
+			jsonCharWhitespaceSet.set(i)
 		case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'e', 'E', '.', '+', '-':
-			jsonNumSet[i] = true
+			jsonNumSet.set(i)
 		}
 	}
 }
@@ -145,20 +143,21 @@ type jsonEncDriver struct {
 
 func (e *jsonEncDriver) sendContainerState(c containerState) {
 	// determine whether to output separators
-	if c == containerMapKey {
+	switch c {
+	case containerMapKey:
 		if e.c != containerMapStart {
 			e.w.writen1(',')
 		}
 		if e.d {
 			e.writeIndent()
 		}
-	} else if c == containerMapValue {
+	case containerMapValue:
 		if e.d {
 			e.w.writen2(':', ' ')
 		} else {
 			e.w.writen1(':')
 		}
-	} else if c == containerMapEnd {
+	case containerMapEnd:
 		if e.d {
 			e.dl--
 			if e.c != containerMapStart {
@@ -166,14 +165,14 @@ func (e *jsonEncDriver) sendContainerState(c containerState) {
 			}
 		}
 		e.w.writen1('}')
-	} else if c == containerArrayElem {
+	case containerArrayElem:
 		if e.c != containerArrayStart {
 			e.w.writen1(',')
 		}
 		if e.d {
 			e.writeIndent()
 		}
-	} else if c == containerArrayEnd {
+	case containerArrayEnd:
 		if e.d {
 			e.dl--
 			if e.c != containerArrayStart {
@@ -201,14 +200,14 @@ func (e *jsonEncDriver) writeIndent() {
 }
 
 func (e *jsonEncDriver) EncodeNil() {
-	e.w.writeb(jsonLiterals[9:13]) // null
+	e.w.writen4('n', 'u', 'l', 'l') // e.w.writeb(jsonLiterals[9:13]) // null
 }
 
 func (e *jsonEncDriver) EncodeBool(b bool) {
 	if b {
-		e.w.writeb(jsonLiterals[0:4]) // true
+		e.w.writen4('t', 'r', 'u', 'e') // e.w.writeb(jsonLiterals[0:4]) // true
 	} else {
-		e.w.writeb(jsonLiterals[4:9]) // false
+		e.w.writen5('f', 'a', 'l', 's', 'e') // e.w.writeb(jsonLiterals[4:9]) // false
 	}
 }
 
@@ -217,7 +216,6 @@ func (e *jsonEncDriver) EncodeFloat32(f float32) {
 }
 
 func (e *jsonEncDriver) EncodeFloat64(f float64) {
-	// e.w.writestr(strconv.FormatFloat(f, 'E', -1, 64))
 	e.encodeFloat(f, 64)
 }
 
@@ -230,7 +228,6 @@ func (e *jsonEncDriver) encodeFloat(f float64, numbits int) {
 }
 
 func (e *jsonEncDriver) EncodeInt(v int64) {
-	// if e.h.IntegerAsString == 'A' || e.h.IntegerAsString == 'L' && (v > 1<<53 || v < -(1<<53)) {
 	if x := e.h.IntegerAsString; x == 'A' || x == 'L' && (v > 1<<53 || v < -(1<<53)) {
 		e.w.writen1('"')
 		e.w.writeb(strconv.AppendInt(e.b[:0], v, 10))
@@ -241,7 +238,6 @@ func (e *jsonEncDriver) EncodeInt(v int64) {
 }
 
 func (e *jsonEncDriver) EncodeUint(v uint64) {
-	// if e.h.IntegerAsString == 'A' || e.h.IntegerAsString == 'L' && v > 1<<53 {
 	if x := e.h.IntegerAsString; x == 'A' || x == 'L' && v > 1<<53 {
 		e.w.writen1('"')
 		e.w.writeb(strconv.AppendUint(e.b[:0], v, 10))
@@ -253,7 +249,7 @@ func (e *jsonEncDriver) EncodeUint(v uint64) {
 
 func (e *jsonEncDriver) EncodeExt(rv interface{}, xtag uint64, ext Ext, en *Encoder) {
 	if v := ext.ConvertExt(rv); v == nil {
-		e.w.writeb(jsonLiterals[9:13]) // null // e.EncodeNil()
+		e.w.writen4('n', 'u', 'l', 'l') // e.w.writeb(jsonLiterals[9:13]) // null // e.EncodeNil()
 	} else {
 		en.encode(v)
 	}
@@ -262,7 +258,7 @@ func (e *jsonEncDriver) EncodeExt(rv interface{}, xtag uint64, ext Ext, en *Enco
 func (e *jsonEncDriver) EncodeRawExt(re *RawExt, en *Encoder) {
 	// only encodes re.Value (never re.Data)
 	if re.Value == nil {
-		e.w.writeb(jsonLiterals[9:13]) // null // e.EncodeNil()
+		e.w.writen4('n', 'u', 'l', 'l') // e.w.writeb(jsonLiterals[9:13]) // null // e.EncodeNil()
 	} else {
 		en.encode(re.Value)
 	}
@@ -285,12 +281,10 @@ func (e *jsonEncDriver) EncodeMapStart(length int) {
 }
 
 func (e *jsonEncDriver) EncodeString(c charEncoding, v string) {
-	// e.w.writestr(strconv.Quote(v))
 	e.quoteStr(v)
 }
 
 func (e *jsonEncDriver) EncodeSymbol(v string) {
-	// e.EncodeString(c_UTF8, v)
 	e.quoteStr(v)
 }
 
@@ -312,7 +306,6 @@ func (e *jsonEncDriver) EncodeStringBytes(c charEncoding, v []byte) {
 		e.w.writeb(e.bs)
 		e.w.writen1('"')
 	} else {
-		// e.EncodeString(c, string(v))
 		e.quoteStr(stringView(v))
 	}
 }
@@ -332,7 +325,7 @@ func (e *jsonEncDriver) quoteStr(s string) {
 		// also encode < > & to prevent security holes when served to some browsers.
 		if b := s[i]; b < utf8.RuneSelf {
 			// if 0x20 <= b && b != '\\' && b != '"' && b != '<' && b != '>' && b != '&' {
-			if jsonCharHtmlSafeSet[b] || (e.h.HTMLCharsAsIs && jsonCharSafeSet[b]) {
+			if jsonCharHtmlSafeSet.isset(b) || (e.h.HTMLCharsAsIs && jsonCharSafeSet.isset(b)) {
 				i++
 				continue
 			}
@@ -400,6 +393,8 @@ type jsonDecDriver struct {
 	// tok is used to store the token read right after skipWhiteSpace.
 	tok uint8
 
+	fnull bool // found null from appendStringAsBytes
+
 	bstr [8]byte  // scratch used for string \UXXX parsing
 	b    [64]byte // scratch, used for parsing strings or numbers
 	b2   [64]byte // scratch, used only for decodeBytes (after base64)
@@ -412,7 +407,7 @@ type jsonDecDriver struct {
 
 func jsonIsWS(b byte) bool {
 	// return b == ' ' || b == '\t' || b == '\r' || b == '\n'
-	return jsonCharWhitespaceSet[b]
+	return jsonCharWhitespaceSet.isset(b)
 }
 
 func (d *jsonDecDriver) uncacheRead() {
@@ -427,19 +422,20 @@ func (d *jsonDecDriver) sendContainerState(c containerState) {
 		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 	var xc uint8 // char expected
-	if c == containerMapKey {
+	switch c {
+	case containerMapKey:
 		if d.c != containerMapStart {
 			xc = ','
 		}
-	} else if c == containerMapValue {
+	case containerMapValue:
 		xc = ':'
-	} else if c == containerMapEnd {
+	case containerMapEnd:
 		xc = '}'
-	} else if c == containerArrayElem {
+	case containerArrayElem:
 		if d.c != containerArrayStart {
 			xc = ','
 		}
-	} else if c == containerArrayEnd {
+	case containerArrayEnd:
 		xc = ']'
 	}
 	if xc != 0 {
@@ -458,11 +454,29 @@ func (d *jsonDecDriver) CheckBreak() bool {
 	return d.tok == '}' || d.tok == ']'
 }
 
-func (d *jsonDecDriver) readStrIdx(fromIdx, toIdx uint8) {
-	bs := d.r.readx(int(toIdx - fromIdx))
+// func (d *jsonDecDriver) readLiteralIdx(fromIdx, toIdx uint8) {
+// 	bs := d.r.readx(int(toIdx - fromIdx))
+// 	d.tok = 0
+// 	if jsonValidateSymbols && !bytes.Equal(bs, jsonLiterals[fromIdx:toIdx]) {
+// 		d.d.errorf("json: expecting %s: got %s", jsonLiterals[fromIdx:toIdx], bs)
+// 		return
+// 	}
+// }
+
+func (d *jsonDecDriver) readSymbol3(v1, v2, v3 uint8) {
+	b1, b2, b3 := d.r.readn3()
+	d.tok = 0
+	if jsonValidateSymbols && (b1 != v1 || b2 != v2 || b3 != v3) {
+		d.d.errorf("json: expecting %c, %c, %c: got %c, %c, %c", b1, b2, b3, v1, v2, v3)
+		return
+	}
+}
+
+func (d *jsonDecDriver) readSymbol4(v1, v2, v3, v4 uint8) {
+	b1, b2, b3, b4 := d.r.readn4()
 	d.tok = 0
-	if jsonValidateSymbols && !bytes.Equal(bs, jsonLiterals[fromIdx:toIdx]) {
-		d.d.errorf("json: expecting %s: got %s", jsonLiterals[fromIdx:toIdx], bs)
+	if jsonValidateSymbols && (b1 != v1 || b2 != v2 || b3 != v3 || b4 != v4) {
+		d.d.errorf("json: expecting %c, %c, %c, %c: got %c, %c, %c, %c", b1, b2, b3, b4, v1, v2, v3, v4)
 		return
 	}
 }
@@ -472,7 +486,7 @@ func (d *jsonDecDriver) TryDecodeAsNil() bool {
 		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 	if d.tok == 'n' {
-		d.readStrIdx(10, 13) // ull
+		d.readSymbol3('u', 'l', 'l') // d.readLiteralIdx(10, 13) // ull
 		return true
 	}
 	return false
@@ -483,11 +497,11 @@ func (d *jsonDecDriver) DecodeBool() bool {
 		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 	if d.tok == 'f' {
-		d.readStrIdx(5, 9) // alse
+		d.readSymbol4('a', 'l', 's', 'e') // d.readLiteralIdx(5, 9) // alse
 		return false
 	}
 	if d.tok == 't' {
-		d.readStrIdx(1, 4) // rue
+		d.readSymbol3('r', 'u', 'e') // d.readLiteralIdx(1, 4) // rue
 		return true
 	}
 	d.d.errorf("json: decode bool: got first char %c", d.tok)
@@ -548,10 +562,8 @@ func (d *jsonDecDriver) decNumBytes() (bs []byte) {
 	} else {
 		d.r.unreadn1()
 		bs = d.r.readTo(d.bs[:0], &jsonNumSet)
-		// bs = d.r.readbUntilAny(d.bs[:0], " \t\n:,{}[]")
 	}
 	d.tok = 0
-	// fmt.Printf(">>>> decNumBytes: returning: '%s'\n", bs)
 	return bs
 }
 
@@ -567,10 +579,6 @@ func (d *jsonDecDriver) DecodeUint(bitsize uint8) (u uint64) {
 
 func (d *jsonDecDriver) DecodeInt(bitsize uint8) (i int64) {
 	bs := d.decNumBytes()
-	// if bytes.ContainsAny(bs, ".eE") {
-	// 	d.d.errorf("json: decoding int, but found one or more of the chars: .eE: %s", bs)
-	// 	return
-	// }
 	i, err := strconv.ParseInt(stringView(bs), 10, int(bitsize))
 	if err != nil {
 		d.d.errorf("json: decode int from %s: %v", bs, err)
@@ -614,10 +622,15 @@ func (d *jsonDecDriver) DecodeBytes(bs []byte, zerocopy bool) (bsOut []byte) {
 		return
 	}
 	d.appendStringAsBytes()
-	// if appendStringAsBytes returned a zero-len slice, then treat as nil.
-	// This should only happen for null, and "".
+	// base64 encodes []byte{} as "", and we encode nil []byte as null.
+	// Consequently, base64 should decode null as a nil []byte, and "" as an empty []byte{}.
+	// appendStringAsBytes returns a zero-len slice for both, so as not to reset d.bs.
+	// However, it sets a fnull field to true, so we can check if a null was found.
 	if len(d.bs) == 0 {
-		return nil
+		if d.fnull {
+			return nil
+		}
+		return []byte{}
 	}
 	bs0 := d.bs
 	slen := base64.StdEncoding.DecodedLen(len(bs0))
@@ -660,19 +673,21 @@ func (d *jsonDecDriver) appendStringAsBytes() {
 		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 
+	d.fnull = false
 	if d.tok != '"' {
 		// d.d.errorf("json: expect char '%c' but got char '%c'", '"', d.tok)
 		// handle non-string scalar: null, true, false or a number
 		switch d.tok {
 		case 'n':
-			d.readStrIdx(10, 13) // ull
+			d.readSymbol3('u', 'l', 'l') // d.readLiteralIdx(10, 13) // ull
 			d.bs = d.bs[:0]
+			d.fnull = true
 		case 'f':
-			d.readStrIdx(5, 9) // alse
+			d.readSymbol4('a', 'l', 's', 'e') // d.readLiteralIdx(5, 9) // alse
 			d.bs = d.bs[:5]
 			copy(d.bs, "false")
 		case 't':
-			d.readStrIdx(1, 4) // rue
+			d.readSymbol3('r', 'u', 'e') // d.readLiteralIdx(1, 4) // rue
 			d.bs = d.bs[:4]
 			copy(d.bs, "true")
 		default:
@@ -688,24 +703,26 @@ func (d *jsonDecDriver) appendStringAsBytes() {
 	r := d.r
 	var cs []byte
 	v := d.bs[:0]
-	// var c uint8
+	var c uint8
 	for i := 0; ; i++ {
 		if i == len(cs) {
 			cs = r.readUntil(d.b2[:0], '"')
 			i = 0
 		}
-		if cs[i] == '"' {
+		c = cs[i]
+		if c == '"' {
 			break
 		}
-		if cs[i] != '\\' {
-			v = append(v, cs[i])
+		if c != '\\' {
+			v = append(v, c)
 			continue
 		}
 		// cs[i] == '\\'
 		i++
-		switch cs[i] {
+		c = cs[i]
+		switch c {
 		case '"', '\\', '/', '\'':
-			v = append(v, cs[i])
+			v = append(v, c)
 		case 'b':
 			v = append(v, '\b')
 		case 'f':
@@ -719,9 +736,7 @@ func (d *jsonDecDriver) appendStringAsBytes() {
 		case 'u':
 			rr := d.jsonU4Arr([4]byte{cs[i+1], cs[i+2], cs[i+3], cs[i+4]})
 			i += 4
-			// fmt.Printf("$$$$$$$$$: is surrogate: %v\n", utf16.IsSurrogate(rr))
 			if utf16.IsSurrogate(rr) {
-				// fmt.Printf(">>>> checking utf16 surrogate\n")
 				if !(cs[i+1] == '\\' && cs[i+2] == 'u') {
 					d.d.errorf(`json: unquoteStr: invalid unicode sequence. Expecting \u`)
 					return
@@ -733,7 +748,7 @@ func (d *jsonDecDriver) appendStringAsBytes() {
 			w2 := utf8.EncodeRune(d.bstr[:], rr)
 			v = append(v, d.bstr[:w2]...)
 		default:
-			d.d.errorf("json: unsupported escaped value: %c", cs[i])
+			d.d.errorf("json: unsupported escaped value: %c", c)
 		}
 	}
 	d.bs = v
@@ -755,12 +770,11 @@ func (d *jsonDecDriver) jsonU4Arr(bs [4]byte) (r rune) {
 		}
 		u = u*16 + uint32(v)
 	}
-	// fmt.Printf(">>>>>>>> jsonU4Arr: %v, %s\n", rune(u), string(rune(u)))
 	return rune(u)
 }
 
 func (d *jsonDecDriver) DecodeNaked() {
-	z := &d.d.n
+	z := d.d.n
 	// var decodeFurther bool
 
 	if d.tok == 0 {
@@ -768,24 +782,20 @@ func (d *jsonDecDriver) DecodeNaked() {
 	}
 	switch d.tok {
 	case 'n':
-		d.readStrIdx(10, 13) // ull
+		d.readSymbol3('u', 'l', 'l') // d.readLiteralIdx(10, 13) // ull
 		z.v = valueTypeNil
 	case 'f':
-		d.readStrIdx(5, 9) // alse
+		d.readSymbol4('a', 'l', 's', 'e') // d.readLiteralIdx(5, 9) // alse
 		z.v = valueTypeBool
 		z.b = false
 	case 't':
-		d.readStrIdx(1, 4) // rue
+		d.readSymbol3('r', 'u', 'e') // d.readLiteralIdx(1, 4) // rue
 		z.v = valueTypeBool
 		z.b = true
 	case '{':
-		z.v = valueTypeMap
-		// d.tok = 0 // don't consume. kInterfaceNaked will call ReadMapStart
-		// decodeFurther = true
+		z.v = valueTypeMap // don't consume. kInterfaceNaked will call ReadMapStart
 	case '[':
-		z.v = valueTypeArray
-		// d.tok = 0 // don't consume. kInterfaceNaked will call ReadArrayStart
-		// decodeFurther = true
+		z.v = valueTypeArray // don't consume. kInterfaceNaked will call ReadArrayStart
 	case '"':
 		z.v = valueTypeString
 		z.s = d.DecodeString()
@@ -821,7 +831,6 @@ func (d *jsonDecDriver) DecodeNaked() {
 				return
 			}
 		}
-		// fmt.Printf("DecodeNaked: Number: %T, %v\n", v, v)
 	}
 	// if decodeFurther {
 	// 	d.s.sc.retryRead()
@@ -829,14 +838,6 @@ func (d *jsonDecDriver) DecodeNaked() {
 	return
 }
 
-// func jsonAcceptNonWS(b byte) bool {
-// 	return !jsonCharWhitespaceSet[b]
-// }
-
-// func jsonAcceptDQuote(b byte) bool {
-// 	return b == '"'
-// }
-
 //----------------------
 
 // JsonHandle is a handle for JSON encoding format.

codec/msgpack.go

@@ -286,7 +286,7 @@ func (d *msgpackDecDriver) DecodeNaked() {
 		d.readNextBd()
 	}
 	bd := d.bd
-	n := &d.d.n
+	n := d.d.n
 	var decodeFurther bool
 
 	switch bd {
@@ -529,12 +529,42 @@ func (d *msgpackDecDriver) DecodeBytes(bs []byte, zerocopy bool) (bsOut []byte)
 	if !d.bdRead {
 		d.readNextBd()
 	}
+
+	// DecodeBytes could be from: bin str fixstr fixarray array ...
 	var clen int
-	if bd := d.bd; bd == mpBin8 || bd == mpBin16 || bd == mpBin32 {
-		clen = d.readContainerLen(msgpackContainerBin)
-	} else {
+	vt := d.ContainerType()
+	switch vt {
+	case valueTypeBytes:
+		// valueTypeBytes may be a mpBin or an mpStr container
+		if bd := d.bd; bd == mpBin8 || bd == mpBin16 || bd == mpBin32 {
+			clen = d.readContainerLen(msgpackContainerBin)
+		} else {
+			clen = d.readContainerLen(msgpackContainerStr)
+		}
+	case valueTypeString:
 		clen = d.readContainerLen(msgpackContainerStr)
+	case valueTypeArray:
+		clen = d.readContainerLen(msgpackContainerList)
+		// ensure everything after is one byte each
+		for i := 0; i < clen; i++ {
+			d.readNextBd()
+			if d.bd == mpNil {
+				bs = append(bs, 0)
+			} else if d.bd == mpUint8 {
+				bs = append(bs, d.r.readn1())
+			} else {
+				d.d.errorf("cannot read non-byte into a byte array")
+				return
+			}
+		}
+		d.bdRead = false
+		return bs
+	default:
+		d.d.errorf("invalid container type: expecting bin|str|array")
+		return
 	}
+
+	// these are (bin|str)(8|16|32)
 	// println("DecodeBytes: clen: ", clen)
 	d.bdRead = false
 	// bytes may be nil, so handle it. if nil, clen=-1.

codec/noop.go

@@ -1,6 +1,8 @@
 // Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
 // Use of this source code is governed by a MIT license found in the LICENSE file.
 
+// +build ignore
+
 package codec
 
 import (

codec/shared_test.go

@@ -43,6 +43,7 @@ package codec
 import (
 	"bytes"
 	"flag"
+	"io"
 	"sync"
 )
 
@@ -54,8 +55,24 @@ type testHED struct {
 	D *Decoder
 }
 
+type ioReaderWrapper struct {
+	r io.Reader
+}
+
+func (x ioReaderWrapper) Read(p []byte) (n int, err error) {
+	return x.r.Read(p)
+}
+
+type ioWriterWrapper struct {
+	w io.Writer
+}
+
+func (x ioWriterWrapper) Write(p []byte) (n int, err error) {
+	return x.w.Write(p)
+}
+
 var (
-	testNoopH    = NoopHandle(8)
+	// testNoopH    = NoopHandle(8)
 	testMsgpackH = &MsgpackHandle{}
 	testBincH    = &BincHandle{}
 	testSimpleH  = &SimpleHandle{}
@@ -73,9 +90,10 @@ var (
 
 // flag variables used by tests (and bench)
 var (
+	testDepth int
+
 	testVerbose        bool
 	testInitDebug      bool
-	testUseIoEncDec    bool
 	testStructToArray  bool
 	testCanonical      bool
 	testUseReset       bool
@@ -84,13 +102,23 @@ var (
 	testInternStr      bool
 	testUseMust        bool
 	testCheckCircRef   bool
-	testJsonIndent     int
-	testMaxInitLen     int
+
+	testUseIoEncDec  bool
+	testUseIoWrapper bool
+
+	testJsonIndent int
+	testMaxInitLen int
 
 	testJsonHTMLCharsAsIs bool
 	testJsonPreferFloat   bool
 )
 
+// variables that are not flags, but which can configure the handles
+var (
+	testEncodeOptions EncodeOptions
+	testDecodeOptions DecodeOptions
+)
+
 // flag variables used by bench
 var (
 	benchDoInitBench      bool
@@ -106,7 +134,8 @@ var (
 func init() {
 	testHEDs = make([]testHED, 0, 32)
 	testHandles = append(testHandles,
-		testNoopH, testMsgpackH, testBincH, testSimpleH,
+		// testNoopH,
+		testMsgpackH, testBincH, testSimpleH,
 		testCborH, testJsonH)
 	testInitFlags()
 	benchInitFlags()
@@ -114,9 +143,11 @@ func init() {
 
 func testInitFlags() {
 	// delete(testDecOpts.ExtFuncs, timeTyp)
+	flag.IntVar(&testDepth, "tsd", 0, "Test Struc Depth")
 	flag.BoolVar(&testVerbose, "tv", false, "Test Verbose")
 	flag.BoolVar(&testInitDebug, "tg", false, "Test Init Debug")
 	flag.BoolVar(&testUseIoEncDec, "ti", false, "Use IO Reader/Writer for Marshal/Unmarshal")
+	flag.BoolVar(&testUseIoWrapper, "tiw", false, "Wrap the IO Reader/Writer with a base pass-through reader/writer")
 	flag.BoolVar(&testStructToArray, "ts", false, "Set StructToArray option")
 	flag.BoolVar(&testWriteNoSymbols, "tn", false, "Set NoSymbols option")
 	flag.BoolVar(&testCanonical, "tc", false, "Set Canonical option")
@@ -181,7 +212,11 @@ func testCodecEncode(ts interface{}, bsIn []byte,
 	}
 	if testUseIoEncDec {
 		buf = fn(bsIn)
-		e.Reset(buf)
+		if testUseIoWrapper {
+			e.Reset(ioWriterWrapper{buf})
+		} else {
+			e.Reset(buf)
+		}
 	} else {
 		bs = bsIn
 		e.ResetBytes(&bs)
@@ -199,15 +234,19 @@ func testCodecEncode(ts interface{}, bsIn []byte,
 
 func testCodecDecode(bs []byte, ts interface{}, h Handle) (err error) {
 	var d *Decoder
-	var buf *bytes.Reader
+	// var buf *bytes.Reader
 	if testUseReset {
 		d = testHEDGet(h).D
 	} else {
 		d = NewDecoder(nil, h)
 	}
 	if testUseIoEncDec {
-		buf = bytes.NewReader(bs)
-		d.Reset(buf)
+		buf := bytes.NewReader(bs)
+		if testUseIoWrapper {
+			d.Reset(ioReaderWrapper{buf})
+		} else {
+			d.Reset(buf)
+		}
 	} else {
 		d.ResetBytes(bs)
 	}

codec/simple.go

@@ -433,7 +433,7 @@ func (d *simpleDecDriver) DecodeNaked() {
 		d.readNextBd()
 	}
 
-	n := &d.d.n
+	n := d.d.n
 	var decodeFurther bool
 
 	switch d.bd {

codec/time.go

@@ -197,24 +197,24 @@ func decodeTime(bs []byte) (tt time.Time, err error) {
 	return
 }
 
-func timeLocUTCName(tzint int16) string {
-	if tzint == 0 {
-		return "UTC"
-	}
-	var tzname = []byte("UTC+00:00")
-	//tzname := fmt.Sprintf("UTC%s%02d:%02d", tzsign, tz/60, tz%60) //perf issue using Sprintf. inline below.
-	//tzhr, tzmin := tz/60, tz%60 //faster if u convert to int first
-	var tzhr, tzmin int16
-	if tzint < 0 {
-		tzname[3] = '-' // (TODO: verify. this works here)
-		tzhr, tzmin = -tzint/60, (-tzint)%60
-	} else {
-		tzhr, tzmin = tzint/60, tzint%60
-	}
-	tzname[4] = timeDigits[tzhr/10]
-	tzname[5] = timeDigits[tzhr%10]
-	tzname[7] = timeDigits[tzmin/10]
-	tzname[8] = timeDigits[tzmin%10]
-	return string(tzname)
-	//return time.FixedZone(string(tzname), int(tzint)*60)
-}
+// func timeLocUTCName(tzint int16) string {
+// 	if tzint == 0 {
+// 		return "UTC"
+// 	}
+// 	var tzname = []byte("UTC+00:00")
+// 	//tzname := fmt.Sprintf("UTC%s%02d:%02d", tzsign, tz/60, tz%60) //perf issue using Sprintf. inline below.
+// 	//tzhr, tzmin := tz/60, tz%60 //faster if u convert to int first
+// 	var tzhr, tzmin int16
+// 	if tzint < 0 {
+// 		tzname[3] = '-' // (TODO: verify. this works here)
+// 		tzhr, tzmin = -tzint/60, (-tzint)%60
+// 	} else {
+// 		tzhr, tzmin = tzint/60, tzint%60
+// 	}
+// 	tzname[4] = timeDigits[tzhr/10]
+// 	tzname[5] = timeDigits[tzhr%10]
+// 	tzname[7] = timeDigits[tzmin/10]
+// 	tzname[8] = timeDigits[tzmin%10]
+// 	return string(tzname)
+// 	//return time.FixedZone(string(tzname), int(tzint)*60)
+// }

codec/values_flex_test.go

@@ -0,0 +1,69 @@
+/* // +build testing */
+
+// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
+// Use of this source code is governed by a MIT license found in the LICENSE file.
+
+package codec
+
+// This file contains values used by tests and benchmarks.
+
+type TestStrucFlex struct {
+	_struct struct{} `codec:",omitempty"` //set omitempty for every field
+	testStrucCommon
+
+	Mis     map[int]string
+	Miwu64s map[int]wrapUint64Slice
+	Mfwss   map[float64]wrapStringSlice
+	Mf32wss map[float32]wrapStringSlice
+	Mui2wss map[uint64]wrapStringSlice
+	Msu2wss map[stringUint64T]wrapStringSlice
+
+	//M map[interface{}]interface{}  `json:"-",bson:"-"`
+	Mtsptr     map[string]*TestStrucFlex
+	Mts        map[string]TestStrucFlex
+	Its        []*TestStrucFlex
+	Nteststruc *TestStrucFlex
+}
+
+func newTestStrucFlex(depth int, bench, useInterface, useStringKeyOnly bool) (ts *TestStrucFlex) {
+	ts = &TestStrucFlex{
+		Miwu64s: map[int]wrapUint64Slice{
+			5: []wrapUint64{1, 2, 3, 4, 5},
+			3: []wrapUint64{1, 2, 3},
+		},
+
+		Mf32wss: map[float32]wrapStringSlice{
+			5.0: []wrapString{"1.0", "2.0", "3.0", "4.0", "5.0"},
+			3.0: []wrapString{"1.0", "2.0", "3.0"},
+		},
+
+		Mui2wss: map[uint64]wrapStringSlice{
+			5: []wrapString{"1.0", "2.0", "3.0", "4.0", "5.0"},
+			3: []wrapString{"1.0", "2.0", "3.0"},
+		},
+
+		Mfwss: map[float64]wrapStringSlice{
+			5.0: []wrapString{"1.0", "2.0", "3.0", "4.0", "5.0"},
+			3.0: []wrapString{"1.0", "2.0", "3.0"},
+		},
+		Mis: map[int]string{
+			1:   "one",
+			22:  "twenty two",
+			-44: "minus forty four",
+		},
+	}
+	populateTestStrucCommon(&ts.testStrucCommon, bench, useInterface, useStringKeyOnly)
+	if depth > 0 {
+		depth--
+		if ts.Mtsptr == nil {
+			ts.Mtsptr = make(map[string]*TestStrucFlex)
+		}
+		if ts.Mts == nil {
+			ts.Mts = make(map[string]TestStrucFlex)
+		}
+		ts.Mtsptr["0"] = newTestStrucFlex(depth, bench, useInterface, useStringKeyOnly)
+		ts.Mts["0"] = *(ts.Mtsptr["0"])
+		ts.Its = append(ts.Its, ts.Mtsptr["0"])
+	}
+	return
+}

codec/values_test.go

@@ -14,7 +14,17 @@ import (
 	"time"
 )
 
-var testStrucTime = time.Date(2012, 2, 2, 2, 2, 2, 2000, time.UTC).UTC()
+type wrapSliceUint64 []uint64
+type wrapSliceString []string
+type wrapUint64 uint64
+type wrapString string
+type wrapUint64Slice []wrapUint64
+type wrapStringSlice []wrapString
+
+type stringUint64T struct {
+	S string
+	U uint64
+}
 
 type AnonInTestStruc struct {
 	AS        string
@@ -36,16 +46,29 @@ type AnonInTestStrucIntf struct {
 	T      time.Time
 }
 
-type TestStruc struct {
-	_struct struct{} `codec:",omitempty"` //set omitempty for every field
+type testSimpleFields struct {
+	S string
+
+	I64 int64
+	I32 int32
+	I16 int16
+	I8  int8
+
+	I64n int64
+	I32n int32
+	I16n int16
+	I8n  int8
 
-	S    string
-	I64  int64
-	I16  int16
 	Ui64 uint64
+	Ui32 uint32
+	Ui16 uint16
 	Ui8  uint8
-	B    bool
-	By   uint8 // byte: msgp doesn't like byte
+
+	F64 float64
+	F32 float32
+
+	B  bool
+	By uint8 // byte: msgp doesn't like byte
 
 	Sslice    []string
 	I64slice  []int64
@@ -59,22 +82,80 @@ type TestStruc struct {
 
 	// TODO: test these separately, specifically for reflection and codecgen.
 	// Unfortunately, ffjson doesn't support these. Its compilation even fails.
-	// Ui64array      [4]uint64
-	// Ui64slicearray [][4]uint64
 
-	AnonInTestStruc
+	Ui64array       [4]uint64
+	Ui64slicearray  []*[4]uint64
+	WrapSliceInt64  wrapSliceUint64
+	WrapSliceString wrapSliceString
+
+	Msi64 map[string]int64
+}
+
+type testStrucCommon struct {
+	S string
+
+	I64 int64
+	I32 int32
+	I16 int16
+	I8  int8
+
+	I64n int64
+	I32n int32
+	I16n int16
+	I8n  int8
+
+	Ui64 uint64
+	Ui32 uint32
+	Ui16 uint16
+	Ui8  uint8
+
+	F64 float64
+	F32 float32
+
+	B  bool
+	By uint8 // byte: msgp doesn't like byte
+
+	Sslice    []string
+	I64slice  []int64
+	I16slice  []int16
+	Ui64slice []uint64
+	Ui8slice  []uint8
+	Bslice    []bool
+	Byslice   []byte
+
+	Iptrslice []*int64
+
+	// TODO: test these separately, specifically for reflection and codecgen.
+	// Unfortunately, ffjson doesn't support these. Its compilation even fails.
+
+	Ui64array       [4]uint64
+	Ui64slicearray  []*[4]uint64
+	WrapSliceInt64  wrapSliceUint64
+	WrapSliceString wrapSliceString
 
-	//M map[interface{}]interface{}  `json:"-",bson:"-"`
 	Msi64 map[string]int64
 
+	Simplef testSimpleFields
+
+	AnonInTestStruc
+
+	NotAnon AnonInTestStruc
+
 	// make this a ptr, so that it could be set or not.
 	// for comparison (e.g. with msgp), give it a struct tag (so it is not inlined),
-	// make this one omitempty (so it is included if nil).
+	// make this one omitempty (so it is excluded if nil).
 	*AnonInTestStrucIntf `codec:",omitempty"`
 
-	Nmap       map[string]bool //don't set this, so we can test for nil
-	Nslice     []byte          //don't set this, so we can test for nil
-	Nint64     *int64          //don't set this, so we can test for nil
+	Nmap   map[string]bool //don't set this, so we can test for nil
+	Nslice []byte          //don't set this, so we can test for nil
+	Nint64 *int64          //don't set this, so we can test for nil
+}
+
+type TestStruc struct {
+	_struct struct{} `codec:",omitempty"` //set omitempty for every field
+
+	testStrucCommon
+
 	Mtsptr     map[string]*TestStruc
 	Mts        map[string]TestStruc
 	Its        []*TestStruc
@@ -89,17 +170,94 @@ type tLowerFirstLetter struct {
 	b []byte
 }
 
-func newTestStruc(depth int, bench bool, useInterface, useStringKeyOnly bool) (ts *TestStruc) {
+// Some other types
+
+type Sstring string
+type Bbool bool
+type Sstructsmall struct {
+	A int
+}
+
+type Sstructbig struct {
+	A int
+	B bool
+	c string
+	// Sval Sstruct
+	Ssmallptr *Sstructsmall
+	Ssmall    *Sstructsmall
+	Sptr      *Sstructbig
+}
+
+type SstructbigMapBySlice struct {
+	_struct struct{} `codec:",toarray"`
+	A       int
+	B       bool
+	c       string
+	// Sval Sstruct
+	Ssmallptr *Sstructsmall
+	Ssmall    *Sstructsmall
+	Sptr      *Sstructbig
+}
+
+type Sinterface interface {
+	Noop()
+}
+
+var testStrucTime = time.Date(2012, 2, 2, 2, 2, 2, 2000, time.UTC).UTC()
+
+func populateTestStrucCommon(ts *testStrucCommon, bench, useInterface, useStringKeyOnly bool) {
 	var i64a, i64b, i64c, i64d int64 = 64, 6464, 646464, 64646464
 
-	ts = &TestStruc{
-		S:    "some string",
+	var a = AnonInTestStruc{
+		// There's more leeway in altering this.
+		AS:    "A-String",
+		AI64:  -64646464,
+		AI16:  1616,
+		AUi64: 64646464,
+		// (U+1D11E)G-clef character may be represented in json as "\uD834\uDD1E".
+		// single reverse solidus character may be represented in json as "\u005C".
+		// include these in ASslice below.
+		ASslice: []string{"Aone", "Atwo", "Athree",
+			"Afour.reverse_solidus.\u005c", "Afive.Gclef.\U0001d11E"},
+		AI64slice: []int64{1, -22, 333, -4444, 55555, -666666},
+		AMSU16:    map[string]uint16{"1": 1, "22": 2, "333": 3, "4444": 4},
+		AF64slice: []float64{
+			11.11e-11, -11.11e+11,
+			2.222E+12, -2.222E-12,
+			-555.55E-5, 555.55E+5,
+			666.66E-6, -666.66E+6,
+			7777.7777E-7, -7777.7777E-7,
+			-8888.8888E+8, 8888.8888E+8,
+			-99999.9999E+9, 99999.9999E+9,
+			// these below are hairy enough to need strconv.ParseFloat
+			33.33E-33, -33.33E+33,
+			44.44e+44, -44.44e-44,
+		},
+	}
+
+	*ts = testStrucCommon{
+		S: "some string",
+
+		// set the numbers close to the limits
+		I8:   math.MaxInt8 * 2 / 3,  // 8,
+		I8n:  math.MinInt8 * 2 / 3,  // 8,
+		I16:  math.MaxInt16 * 2 / 3, // 16,
+		I16n: math.MinInt16 * 2 / 3, // 16,
+		I32:  math.MaxInt32 * 2 / 3, // 32,
+		I32n: math.MinInt32 * 2 / 3, // 32,
 		I64:  math.MaxInt64 * 2 / 3, // 64,
-		I16:  1616,
-		Ui64: uint64(int64(math.MaxInt64 * 2 / 3)), // 64, //don't use MaxUint64, as bson can't write it
-		Ui8:  160,
-		B:    true,
-		By:   5,
+		I64n: math.MinInt64 * 2 / 3, // 64,
+
+		Ui64: math.MaxUint64 * 2 / 3, // 64
+		Ui32: math.MaxUint32 * 2 / 3, // 32
+		Ui16: math.MaxUint16 * 2 / 3, // 16
+		Ui8:  math.MaxUint8 * 2 / 3,  // 8
+
+		F32: 3.402823e+38, // max representable float32 without losing precision
+		F64: 3.40281991833838838338e+53,
+
+		B:  true,
+		By: 5,
 
 		Sslice:    []string{"one", "two", "three"},
 		I64slice:  []int64{1111, 2222, 3333},
@@ -113,33 +271,70 @@ func newTestStruc(depth int, bench bool, useInterface, useStringKeyOnly bool) (t
 			"one": 1,
 			"two": 2,
 		},
-		AnonInTestStruc: AnonInTestStruc{
-			// There's more leeway in altering this.
-			AS:    "A-String",
-			AI64:  -64646464,
-			AI16:  1616,
-			AUi64: 64646464,
-			// (U+1D11E)G-clef character may be represented in json as "\uD834\uDD1E".
-			// single reverse solidus character may be represented in json as "\u005C".
-			// include these in ASslice below.
-			ASslice: []string{"Aone", "Atwo", "Athree",
-				"Afour.reverse_solidus.\u005c", "Afive.Gclef.\U0001d11E"},
-			AI64slice: []int64{1, -22, 333, -4444, 55555, -666666},
-			AMSU16:    map[string]uint16{"1": 1, "22": 2, "333": 3, "4444": 4},
-			AF64slice: []float64{
-				11.11e-11, -11.11e+11,
-				2.222E+12, -2.222E-12,
-				-555.55E-5, 555.55E+5,
-				666.66E-6, -666.66E+6,
-				7777.7777E-7, -7777.7777E-7,
-				-8888.8888E+8, 8888.8888E+8,
-				-99999.9999E+9, 99999.9999E+9,
-				// these below are hairy enough to need strconv.ParseFloat
-				33.33E-33, -33.33E+33,
-				44.44e+44, -44.44e-44,
+
+		Ui64array: [4]uint64{4, 16, 64, 256},
+
+		WrapSliceInt64:  []uint64{4, 16, 64, 256},
+		WrapSliceString: []string{"4", "16", "64", "256"},
+
+		// DecodeNaked bombs here, because the stringUint64T is decoded as a map,
+		// and a map cannot be the key type of a map.
+		// Thus, don't initialize this here.
+		// Msu2wss: map[stringUint64T]wrapStringSlice{
+		// 	{"5", 5}: []wrapString{"1", "2", "3", "4", "5"},
+		// 	{"3", 3}: []wrapString{"1", "2", "3"},
+		// },
+
+		// make Simplef same as top-level
+		Simplef: testSimpleFields{
+			S: "some string",
+
+			// set the numbers close to the limits
+			I8:   math.MaxInt8 * 2 / 3,  // 8,
+			I8n:  math.MinInt8 * 2 / 3,  // 8,
+			I16:  math.MaxInt16 * 2 / 3, // 16,
+			I16n: math.MinInt16 * 2 / 3, // 16,
+			I32:  math.MaxInt32 * 2 / 3, // 32,
+			I32n: math.MinInt32 * 2 / 3, // 32,
+			I64:  math.MaxInt64 * 2 / 3, // 64,
+			I64n: math.MinInt64 * 2 / 3, // 64,
+
+			Ui64: math.MaxUint64 * 2 / 3, // 64
+			Ui32: math.MaxUint32 * 2 / 3, // 32
+			Ui16: math.MaxUint16 * 2 / 3, // 16
+			Ui8:  math.MaxUint8 * 2 / 3,  // 8
+
+			F32: 3.402823e+38, // max representable float32 without losing precision
+			F64: 3.40281991833838838338e+53,
+
+			B:  true,
+			By: 5,
+
+			Sslice:    []string{"one", "two", "three"},
+			I64slice:  []int64{1111, 2222, 3333},
+			I16slice:  []int16{44, 55, 66},
+			Ui64slice: []uint64{12121212, 34343434, 56565656},
+			Ui8slice:  []uint8{210, 211, 212},
+			Bslice:    []bool{true, false, true, false},
+			Byslice:   []byte{13, 14, 15},
+
+			Msi64: map[string]int64{
+				"one": 1,
+				"two": 2,
 			},
+
+			Ui64array: [4]uint64{4, 16, 64, 256},
+
+			WrapSliceInt64:  []uint64{4, 16, 64, 256},
+			WrapSliceString: []string{"4", "16", "64", "256"},
 		},
+
+		AnonInTestStruc: a,
+		NotAnon:         a,
 	}
+
+	ts.Ui64slicearray = []*[4]uint64{&ts.Ui64array, &ts.Ui64array}
+
 	if useInterface {
 		ts.AnonInTestStrucIntf = &AnonInTestStrucIntf{
 			Islice: []interface{}{"true", true, "no", false, uint64(288), float64(0.4)},
@@ -165,6 +360,11 @@ func newTestStruc(depth int, bench bool, useInterface, useStringKeyOnly bool) (t
 	if !useStringKeyOnly {
 		// ts.AnonInTestStruc.AMU32F64 = map[uint32]float64{1: 1, 2: 2, 3: 3} // Json/Bson barf
 	}
+}
+
+func newTestStruc(depth int, bench, useInterface, useStringKeyOnly bool) (ts *TestStruc) {
+	ts = &TestStruc{}
+	populateTestStrucCommon(&ts.testStrucCommon, bench, useInterface, useStringKeyOnly)
 	if depth > 0 {
 		depth--
 		if ts.Mtsptr == nil {
@@ -179,36 +379,3 @@ func newTestStruc(depth int, bench bool, useInterface, useStringKeyOnly bool) (t
 	}
 	return
 }
-
-// Some other types
-
-type Sstring string
-type Bbool bool
-type Sstructsmall struct {
-	A int
-}
-
-type Sstructbig struct {
-	A int
-	B bool
-	c string
-	// Sval Sstruct
-	Ssmallptr *Sstructsmall
-	Ssmall    *Sstructsmall
-	Sptr      *Sstructbig
-}
-
-type SstructbigMapBySlice struct {
-	_struct struct{} `codec:",toarray"`
-	A       int
-	B       bool
-	c       string
-	// Sval Sstruct
-	Ssmallptr *Sstructsmall
-	Ssmall    *Sstructsmall
-	Sptr      *Sstructbig
-}
-
-type Sinterface interface {
-	Noop()
-}

codec/z.go

@@ -0,0 +1,23 @@
+// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
+// Use of this source code is governed by a MIT license found in the LICENSE file.
+
+package codec
+
+import "sort"
+
+// TODO: this is brittle, as it depends on z.go's init() being called last.
+// The current build tools all honor that files are passed in lexical order.
+// However, we should consider using an init_channel,
+// that each person doing init will write to.
+
+func init() {
+	if !useLookupRecognizedTypes {
+		return
+	}
+	sort.Sort(uintptrSlice(recognizedRtids))
+	sort.Sort(uintptrSlice(recognizedRtidPtrs))
+	recognizedRtidOrPtrs = make([]uintptr, len(recognizedRtids)+len(recognizedRtidPtrs))
+	copy(recognizedRtidOrPtrs, recognizedRtids)
+	copy(recognizedRtidOrPtrs[len(recognizedRtids):], recognizedRtidPtrs)
+	sort.Sort(uintptrSlice(recognizedRtidOrPtrs))
+}

codec/z_all_test.go

@@ -35,6 +35,9 @@ func testSuite(t *testing.T, f func(t *testing.T)) {
 
 	testReinit() // so flag.Parse() is called first, and never called again
 
+	testDecodeOptions = DecodeOptions{}
+	testEncodeOptions = EncodeOptions{}
+
 	testUseMust = false
 	testCanonical = false
 	testUseMust = false
@@ -47,6 +50,7 @@ func testSuite(t *testing.T, f func(t *testing.T)) {
 	testUseReset = false
 	testMaxInitLen = 0
 	testJsonIndent = 0
+	testUseIoWrapper = false
 	testReinit()
 	t.Run("optionsFalse", f)
 
@@ -70,8 +74,28 @@ func testSuite(t *testing.T, f func(t *testing.T)) {
 	testCheckCircRef = true
 	testJsonHTMLCharsAsIs = true
 	testUseReset = true
+	testDecodeOptions.MapValueReset = true
 	testReinit()
 	t.Run("optionsTrue", f)
+
+	testUseIoWrapper = true
+	testReinit()
+	t.Run("optionsTrue-ioWrapper", f)
+
+	testDepth = 6
+	testReinit()
+	t.Run("optionsTrue-deepstruct", f)
+
+	// The following here MUST be tested individually, as they create
+	// side effects i.e. the decoded value is different.
+	// testDecodeOptions.MapValueReset = true // ok - no side effects
+	// testDecodeOptions.InterfaceReset = true // error??? because we do deepEquals to verify
+	// testDecodeOptions.ErrorIfNoField = true // error, as expected, as fields not there
+	// testDecodeOptions.ErrorIfNoArrayExpand = true // no error, but no error case either
+	// testDecodeOptions.PreferArrayOverSlice = true // error??? because slice != array.
+	// .... however, update deepEqual to take this option
+	// testReinit()
+	// t.Run("optionsTrue-resetOptions", f)
 }
 
 /*
@@ -84,6 +108,7 @@ find . -name "$z" | xargs grep -e '^func Test' | \
 func testCodecGroup(t *testing.T) {
 	// println("running testcodecsuite")
 	// <setup code>
+
 	t.Run("TestBincCodecsTable", TestBincCodecsTable)
 	t.Run("TestBincCodecsMisc", TestBincCodecsMisc)
 	t.Run("TestBincCodecsEmbeddedPointer", TestBincCodecsEmbeddedPointer)