codec: major update with MAJOR PERFORMANCE improvements

Along with these changes, we support go 1.4+ across the board, and test across
the last 3 go releases (go 1.7+) and tip before each github push.

The changes articulated below are contained in this push:

expand support for careful contained use of unsafe for performance

    This mostly involved creating unsafe and safe versions of a few functions, and
    using them across the codebase.

    - []byte-->string or string-->[]byte when we know there is no write done.
      e.g. map lookup, looking up a struct field given a field name, etc
    - rt2id(...): getting an id for a type usable as an id
    - rv2i(...):  getting an interface from a reflect.Value in our narrow context for encoding/decoding
    - ptrToRvMap: caching reflect.Value got for the same pointers
    - using atomic pointers to reduce sync.RWMutex contention (lock-free model)
    - fast conversion from reflect.Value to a pointer when we know exactly what the type is.

    Note that we only support unsafe for the 3 most recent go releases.
    This is because use of unsafe requires intimate knowledge of implementation details.

Simplify codebase during decode, making things consistent and improving performance

    - Use single conditional to remove duplicate codes for decoding into maps, arrays, slices and chans
    - Make changes across: reflection, fast-path and codecgen modes

    Litany of changes
    - prepare safe and unsafe variants for *decFnInfo methods
    - clean up struct for decoding into structs and collections (maps, slices, arrays, chans)
      so they are consistent in the reflection path.
      Previously, i had separate paths for length-prefix vs separator-based collections.
    - Update fast-path and codecgen template generation scripts to make all code mirror themselves
    - Apply some optimizations along the way (specifically for decode)
    - removed some unnecessary reflection calls in kMap
    - add some CanSet() checks and reuse reflect.Value (if immutablekind) during decode

use safe and appengine tags to judiciously use "unsafe" by default for performance

    The tags are set up this way, so that a user must explicitly pass
    one of the following tags, to jse the standard "slower" apis.

json: use lookup table for safe characters to improve performance

    The previous check for characters inside of a JSON string that needed
    to be escaped performed seven different boolean comparisons before
    determining that a ASCII character did not need to be escaped. Most
    characters do not need to be escaped, so this check can be done in a
    more performant way *typically* using a single/simple memory table lookup.

    We use [128]bool to represent safe and unsafe characters, and look up
    there instead of doing a computation.

use sync/atomic to lookup typeinfo without synchronization

    Also, move from using a map to just do a binary search when looking
    up typeInfo. This should be faster (arguably).

optimize for map with string keys to avoid allocation if possible

re-introduce decDriver.DecodeStringAsBytes

    This reduces allocation significantly and is clearer.

    DecodeString and DecodeStringAsBytes are 2 versions of the same code.

add native helper functions for performant json parsing
improved json interaction with codec package for better performance.

    One way to improve json performance is to bypass the multiple readn1
    calls in a tight loop, and just have a single interface call to do the
    combo functionality.

    json does some things a lot during decoding:
    - skip whitespace characters
    - constructs numbers
    - read strings

    doing these one character at a time incur interface indirection and indirect function
    call overhead, as it all goes through decDriver interface to an implementation like *bytesDecDriver.

    We can make some major gains by exposing "combo-like" methods that expose this functionality
    as a single call (as opposed to calls that do one byte at a time).
        // skip will skip any byte that matches, and return the first non-matching byte
        skip(accept *[256]bool) (token byte)
        // readTo will read any byte that matches, stopping once no-longer matching.
        readTo(in []byte, accept *[256]bool) (out []byte)
        // readUntil will read, only stopping once it matches the 'stop' byte.
        readUntil(in []byte, stop byte) (out []byte)

    Currently, only json handle has use of these methods, giving it much better perf.

    In addition, since we can just request a block of text, and work on it directly.
    we might as well remove the logic that does strconv.ParseFloat at iterator time.
    Instead, we just grab the block of text representing a number, and pass that to the
    strconv.ParseXXX functions.

    This resulted in the removal of all the string->number logic. Which is A-ok.

    passing through interface calls to an implementation (function and indirection).

    We saw performance gains when using [256]bool arrays in place of multiple comparisons.
    We continue to leverage that pattern to get some performance improvements without the
    need to pass callback functions for matching bytes. This is good because the call-backs
    were costing us over 10% in performance loss.

    Performance gains are very good now over the last few code iterations.

re-packaged all language-specific code in appropriately named files

json: add PreferFloat option to default to decoding numbers at float

test suite and organization

    We now use test suites to define and run our tests, making
    most of what tests.sh and bench.sh did unnecessary.

    We now drive scripts from run.sh, and those just call
    go test ... for running tests and benchmarks.

    This required some re-factoring in the tests to make them all work.

    organize tests so they depend on helper functions, allowing for many suites.

    This affords XSuite, CodecSuite and AllSuite .

    It also allows us separate the tests from benchmarks, and further separate benchmarks into
    codec+stdlib only, and external ones.

codecgen: remove unsafe - delegating to codec's safe/unsafe instead.

    codecgen will now use whatever tag (safe or unsafe) the codec package is built with.

    To do this, expose stringView to codecgen (as genHelperDecoder.StringView(...)
    and use that within the generated code.

conditional build: to support go1.4+

    support go1.4+ and use goversion_* to conditionally compile things based on go version

codecgen: gen.go is only used by codecgen execution, behind build tag "codecgen.exec"

    Do this by
    - adding the build tag "codecgen.exec" to gen.go
    - moving the definition of GenVersion to gen-helper.go.tmpl (which is used at runtime)
      which requires adding a "Version" field to genInternal struct
    - passing the "codecgen.exec" tag as one of the values to "go run ..."

    Also, codecgen depends on fastpath support.
    Document that, and update run.sh appropriately.

    Also, values_codecgen_generated_test.go MUST not have the 'x' build tag set.
    That 'x' build tag is only for codebases which are not codec based.

    Code coverage is currently at 71%, which is just above our goals.

----------

Testing
    - expand list of float values to test
    - add a true large string for testing
    - add testUseIOEnc support to json-iterator benchmark
    - add a test suite for running all codec tests and grabbing codecoverage metrics
    - add test for json indent
    - exercise all paths in fast-path using full-featured generated type

Misc:
    - add json-iterator and easyjson to the benchmarks

codec/0doc.go

@@ -2,8 +2,8 @@
 // Use of this source code is governed by a MIT license found in the LICENSE file.
 
 /*
-High Performance, Feature-Rich Idiomatic Go codec/encoding library for 
-binc, msgpack, cbor, json.
+High Performance, Feature-Rich Idiomatic Go 1.4+ codec/encoding library for
+binc, msgpack, cbor, json
 
 Supported Serialization formats are:
 
@@ -11,21 +11,17 @@ Supported Serialization formats are:
   - binc:    http://github.com/ugorji/binc
   - cbor:    http://cbor.io http://tools.ietf.org/html/rfc7049
   - json:    http://json.org http://tools.ietf.org/html/rfc7159
-  - simple: 
+  - simple:
 
 To install:
 
     go get github.com/ugorji/go/codec
 
-This package understands the 'unsafe' tag, to allow using unsafe semantics:
-
-  - When decoding into a struct, you need to read the field name as a string 
-    so you can find the struct field it is mapped to.
-    Using `unsafe` will bypass the allocation and copying overhead of []byte->string conversion.
-
-To install using unsafe, pass the 'unsafe' tag:
-
-    go get -tags=unsafe github.com/ugorji/go/codec
+This package will carefully use 'unsafe' for performance reasons in specific places.
+You can build without unsafe use by passing the safe or appengine tag
+i.e. 'go install -tags=safe ...'. Note that unsafe is only supported for the last 3
+go sdk versions e.g. current go release is go 1.9, so we support unsafe use only from
+go 1.7+ . This is because supporting unsafe requires knowledge of implementation details.
 
 For detailed usage information, read the primer at http://ugorji.net/blog/go-codec-primer .
 
@@ -38,9 +34,9 @@ Rich Feature Set includes:
   - Very High Performance.
     Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
   - Multiple conversions:
-    Package coerces types where appropriate 
+    Package coerces types where appropriate
     e.g. decode an int in the stream into a float, etc.
-  - Corner Cases: 
+  - Corner Cases:
     Overflows, nil maps/slices, nil values in streams are handled correctly
   - Standard field renaming via tags
   - Support for omitting empty fields during an encoding
@@ -56,7 +52,7 @@ Rich Feature Set includes:
   - Fast (no-reflection) encoding/decoding of common maps and slices
   - Code-generation for faster performance.
   - Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
-  - Support indefinite-length formats to enable true streaming 
+  - Support indefinite-length formats to enable true streaming
     (for formats which support it e.g. json, cbor)
   - Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes.
     This mostly applies to maps, where iteration order is non-deterministic.
@@ -68,12 +64,12 @@ Rich Feature Set includes:
   - Encode/Decode from/to chan types (for iterative streaming support)
   - Drop-in replacement for encoding/json. `json:` key in struct tag supported.
   - Provides a RPC Server and Client Codec for net/rpc communication protocol.
-  - Handle unique idiosyncrasies of codecs e.g. 
-    - For messagepack, configure how ambiguities in handling raw bytes are resolved 
-    - For messagepack, provide rpc server/client codec to support 
+  - Handle unique idiosyncrasies of codecs e.g.
+    - For messagepack, configure how ambiguities in handling raw bytes are resolved
+    - For messagepack, provide rpc server/client codec to support
       msgpack-rpc protocol defined at:
       https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
-  
+
 Extension Support
 
 Users can register a function to handle the encoding or decoding of

codec/README.md

@@ -15,17 +15,11 @@ To install:
 
     go get github.com/ugorji/go/codec
 
-This package understands the `unsafe` tag, to allow using unsafe semantics:
-
-  - When decoding into a struct, you need to read the field name as a string 
-    so you can find the struct field it is mapped to.
-    Using `unsafe` will bypass the allocation and copying overhead of `[]byte->string` conversion.
-
-To use it, you must pass the `unsafe` tag during install:
-
-```
-go install -tags=unsafe github.com/ugorji/go/codec 
-```
+This package will carefully use 'unsafe' for performance reasons in specific places.
+You can build without unsafe use by passing the safe or appengine tag
+i.e. 'go install -tags=safe ...'. Note that unsafe is only supported for the last 3
+go sdk versions e.g. current go release is go 1.9, so we support unsafe use only from
+go 1.7+ . This is because supporting unsafe requires knowledge of implementation details.
 
 Online documentation: http://godoc.org/github.com/ugorji/go/codec  
 Detailed Usage/How-to Primer: http://ugorji.net/blog/go-codec-primer
@@ -36,8 +30,13 @@ the standard library (ie json, xml, gob, etc).
 Rich Feature Set includes:
 
   - Simple but extremely powerful and feature-rich API
+  - Support for go1.4 and above, while selectively using newer APIs for later releases
+  - Good code coverage ( > 70% )
   - Very High Performance.
     Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
+  - Careful selected use of 'unsafe' for targeted performance gains.
+    100% mode exists where 'unsafe' is not used at all.
+  - Lock-free (sans mutex) concurrency for scaling to 100's of cores
   - Multiple conversions:
     Package coerces types where appropriate 
     e.g. decode an int in the stream into a float, etc.
@@ -146,3 +145,22 @@ Typical usage model:
     //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
     client := rpc.NewClientWithCodec(rpcCodec)
 
+## Running Tests
+
+To run tests, use the following:
+
+    go test
+
+To run the full suite of tests, use the following:
+
+    go test -tags alltests -run Suite
+
+You can run the tag 'safe' to run tests or build in safe mode. e.g.
+
+    go test -tags safe -run Json
+    go test -tags "alltests safe" -run Suite
+
+## Running Benchmarks
+
+Please see http://github.com/ugorji/go-codec-bench .
+

codec/binc.go

@@ -728,11 +728,12 @@ func (d *bincDecDriver) DecodeString() (s string) {
 	return
 }
 
-func (d *bincDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut []byte) {
-	if isstring {
-		bsOut, _ = d.decStringAndBytes(bs, false, zerocopy)
-		return
-	}
+func (d *bincDecDriver) DecodeStringAsBytes() (s []byte) {
+	s, _ = d.decStringAndBytes(d.b[:], false, true)
+	return
+}
+
+func (d *bincDecDriver) DecodeBytes(bs []byte, zerocopy bool) (bsOut []byte) {
 	if !d.bdRead {
 		d.readNextBd()
 	}
@@ -789,7 +790,7 @@ func (d *bincDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs []b
 		}
 		xbs = d.r.readx(l)
 	} else if d.vd == bincVdByteArray {
-		xbs = d.DecodeBytes(nil, false, true)
+		xbs = d.DecodeBytes(nil, true)
 	} else {
 		d.d.errorf("Invalid d.vd for extensions (Expecting extensions or byte array). Got: 0x%x", d.vd)
 		return
@@ -858,7 +859,7 @@ func (d *bincDecDriver) DecodeNaked() {
 		n.s = d.DecodeString()
 	case bincVdByteArray:
 		n.v = valueTypeBytes
-		n.l = d.DecodeBytes(nil, false, false)
+		n.l = d.DecodeBytes(nil, false)
 	case bincVdTimestamp:
 		n.v = valueTypeTimestamp
 		tt, err := decodeTime(d.r.readx(int(d.vs)))

codec/cbor.go

@@ -407,7 +407,7 @@ func (d *cborDecDriver) decAppendIndefiniteBytes(bs []byte) []byte {
 	return bs
 }
 
-func (d *cborDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut []byte) {
+func (d *cborDecDriver) DecodeBytes(bs []byte, zerocopy bool) (bsOut []byte) {
 	if !d.bdRead {
 		d.readNextBd()
 	}
@@ -434,7 +434,11 @@ func (d *cborDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut [
 }
 
 func (d *cborDecDriver) DecodeString() (s string) {
-	return string(d.DecodeBytes(d.b[:], true, true))
+	return string(d.DecodeBytes(d.b[:], true))
+}
+
+func (d *cborDecDriver) DecodeStringAsBytes() (s []byte) {
+	return d.DecodeBytes(d.b[:], true)
 }
 
 func (d *cborDecDriver) DecodeExt(rv interface{}, xtag uint64, ext Ext) (realxtag uint64) {
@@ -485,7 +489,7 @@ func (d *cborDecDriver) DecodeNaked() {
 		n.f = d.DecodeFloat(false)
 	case cborBdIndefiniteBytes:
 		n.v = valueTypeBytes
-		n.l = d.DecodeBytes(nil, false, false)
+		n.l = d.DecodeBytes(nil, false)
 	case cborBdIndefiniteString:
 		n.v = valueTypeString
 		n.s = d.DecodeString()
@@ -510,7 +514,7 @@ func (d *cborDecDriver) DecodeNaked() {
 			n.i = d.DecodeInt(64)
 		case d.bd >= cborBaseBytes && d.bd < cborBaseString:
 			n.v = valueTypeBytes
-			n.l = d.DecodeBytes(nil, false, false)
+			n.l = d.DecodeBytes(nil, false)
 		case d.bd >= cborBaseString && d.bd < cborBaseArray:
 			n.v = valueTypeString
 			n.s = d.DecodeString()

codec/codec_test.go

@@ -134,6 +134,14 @@ func (x *TestABC2) UnmarshalText(data []byte) (err error) {
 	// _, err = fmt.Sscanf(string(data), "%s %s %s", &x.A, &x.B, &x.C)
 }
 
+type TestSimplish struct {
+	Ii int
+	Ss string
+	Ar [2]*TestSimplish
+	Sl []*TestSimplish
+	Mm map[string]*TestSimplish
+}
+
 type TestRpcABC struct {
 	A, B, C string
 }
@@ -339,6 +347,8 @@ func testInit() {
 
 	testJsonH.Indent = int8(testJsonIndent)
 	testJsonH.HTMLCharsAsIs = testJsonHTMLCharsAsIs
+	testJsonH.PreferFloat = testJsonPreferFloat
+
 	testMsgpackH.RawToString = true
 
 	// testMsgpackH.AddExt(byteSliceTyp, 0, testMsgpackH.BinaryEncodeExt, testMsgpackH.BinaryDecodeExt)
@@ -348,7 +358,20 @@ func testInit() {
 	// use different flavors of XXXExt calls, including deprecated ones.
 	// NOTE:
 	// DO NOT set extensions for JsonH, so we can test json(M|Unm)arshal support.
+	var (
+		timeExtEncFn = func(rv reflect.Value) (bs []byte, err error) {
+			defer panicToErr(&err)
+			bs = timeExt{}.WriteExt(rv.Interface())
+			return
+		}
+		timeExtDecFn = func(rv reflect.Value, bs []byte) (err error) {
+			defer panicToErr(&err)
+			timeExt{}.ReadExt(rv.Interface(), bs)
+			return
+		}
+	)
 	testSimpleH.AddExt(timeTyp, 1, timeExtEncFn, timeExtDecFn)
+
 	testMsgpackH.SetBytesExt(timeTyp, 1, timeExt{})
 	testCborH.SetInterfaceExt(timeTyp, 1, &testUnixNanoTimeExt{})
 	// testJsonH.SetInterfaceExt(timeTyp, 1, &testUnixNanoTimeExt{})
@@ -407,9 +430,22 @@ func testInit() {
 				},
 				[]interface{}{true, false},
 			},
-			"int32":        int32(32323232),
-			"bool":         true,
-			"LONG STRING":  "123456789012345678901234567890123456789012345678901234567890",
+			"int32": int32(32323232),
+			"bool":  true,
+			"LONG STRING": `
+1234567890 1234567890 
+1234567890 1234567890 
+1234567890 1234567890 
+ABCDEDFGHIJKLMNOPQRSTUVWXYZ 
+abcdedfghijklmnopqrstuvwxyz 
+ABCDEDFGHIJKLMNOPQRSTUVWXYZ 
+abcdedfghijklmnopqrstuvwxyz 
+"ABCDEDFGHIJKLMNOPQRSTUVWXYZ" 
+'	a tab	'
+\a\b\c\d\e
+\b\f\n\r\t all literally
+ugorji
+`,
 			"SHORT STRING": "1234567890",
 		},
 		map[interface{}]interface{}{
@@ -489,7 +525,7 @@ func testMarshal(v interface{}, h Handle) (bs []byte, err error) {
 func testMarshalErr(v interface{}, h Handle, t *testing.T, name string) (bs []byte, err error) {
 	if bs, err = testMarshal(v, h); err != nil {
 		logT(t, "Error encoding %s: %v, Err: %v", name, v, err)
-		t.FailNow()
+		failT(t)
 	}
 	return
 }
@@ -497,7 +533,17 @@ func testMarshalErr(v interface{}, h Handle, t *testing.T, name string) (bs []by
 func testUnmarshalErr(v interface{}, data []byte, h Handle, t *testing.T, name string) (err error) {
 	if err = testUnmarshal(v, data, h); err != nil {
 		logT(t, "Error Decoding into %s: %v, Err: %v", name, v, err)
-		t.FailNow()
+		failT(t)
+	}
+	return
+}
+
+func testDeepEqualErr(v1, v2 interface{}, t *testing.T, name string) (err error) {
+	if err = deepEqual(v1, v2); err == nil {
+		logT(t, "%s: values equal", name)
+	} else {
+		logT(t, "%s: values not equal: %v. 1: %v, 2: %v", name, err, v1, v2)
+		failT(t)
 	}
 	return
 }
@@ -621,13 +667,13 @@ func testCodecMiscOne(t *testing.T, h Handle) {
 	// var i *int32
 	// if err = testUnmarshal(b, i, nil); err == nil {
 	// 	logT(t, "------- Expecting error because we cannot unmarshal to int32 nil ptr")
-	// 	t.FailNow()
+	// 	failT(t)
 	// }
 	var i2 int32 = 0
 	err = testUnmarshalErr(&i2, b, h, t, "int32-ptr")
 	if i2 != int32(32) {
 		logT(t, "------- didn't unmarshal to 32: Received: %d", i2)
-		t.FailNow()
+		failT(t)
 	}
 
 	// func TestMsgpackDecodePtr(t *testing.T) {
@@ -635,7 +681,7 @@ func testCodecMiscOne(t *testing.T, h Handle) {
 	b, err = testMarshalErr(ts, h, t, "pointer-to-struct")
 	if len(b) < 40 {
 		logT(t, "------- Size must be > 40. Size: %d", len(b))
-		t.FailNow()
+		failT(t)
 	}
 	if h.isBinary() {
 		logT(t, "------- b: %v", b)
@@ -646,7 +692,7 @@ func testCodecMiscOne(t *testing.T, h Handle) {
 	err = testUnmarshalErr(ts2, b, h, t, "pointer-to-struct")
 	if ts2.I64 != math.MaxInt64*2/3 {
 		logT(t, "------- Unmarshal wrong. Expect I64 = 64. Got: %v", ts2.I64)
-		t.FailNow()
+		failT(t)
 	}
 
 	// func TestMsgpackIntfDecode(t *testing.T) {
@@ -660,7 +706,7 @@ func testCodecMiscOne(t *testing.T, h Handle) {
 
 	if m2["A"] != 2 || m2["B"] != 3 {
 		logT(t, "m2 not as expected: expecting: %v, got: %v", m, m2)
-		t.FailNow()
+		failT(t)
 	}
 	// log("m: %v, m2: %v, p: %v, p2: %v", m, m2, p, p2)
 	checkEqualT(t, p, p2, "p=p2")
@@ -669,13 +715,13 @@ func testCodecMiscOne(t *testing.T, h Handle) {
 		logT(t, "p and p2 match")
 	} else {
 		logT(t, "Not Equal: %v. p: %v, p2: %v", err, p, p2)
-		t.FailNow()
+		failT(t)
 	}
 	if err = deepEqual(m, m2); err == nil {
 		logT(t, "m and m2 match")
 	} else {
 		logT(t, "Not Equal: %v. m: %v, m2: %v", err, m, m2)
-		t.FailNow()
+		failT(t)
 	}
 
 	// func TestMsgpackDecodeStructSubset(t *testing.T) {
@@ -705,7 +751,7 @@ func testCodecMiscOne(t *testing.T, h Handle) {
 		bs, err = testMarshalErr(tarr1, h, t, "tarr1")
 		if err != nil {
 			logT(t, "Error marshalling: %v", err)
-			t.FailNow()
+			failT(t)
 		}
 		if _, ok := h.(*JsonHandle); ok {
 			logT(t, "Marshal as: %s", bs)
@@ -815,7 +861,7 @@ func testCodecRpcOne(t *testing.T, rr Rpc, h Handle, doRequest bool, exitSleepMs
 	// rpc needs EOF, which is sent via a panic, and so must be recovered.
 	if !recoverPanicToErr {
 		logT(t, "EXPECTED. set recoverPanicToErr=true, since rpc needs EOF")
-		t.FailNow()
+		failT(t)
 	}
 	srv := rpc.NewServer()
 	srv.Register(testRpcInt)
@@ -958,7 +1004,7 @@ func doTestMapEncodeForCanonical(t *testing.T, name string, h Handle) {
 	e2.MustEncode(v2)
 	if !bytes.Equal(b1, b2) {
 		logT(t, "Unequal bytes: %v VS %v", b1, b2)
-		t.FailNow()
+		failT(t)
 	}
 }
 
@@ -1012,13 +1058,13 @@ func doTestEncCircularRef(t *testing.T, name string, h Handle) {
 	err = NewEncoderBytes(&bs, h).Encode(&t3)
 	if err == nil {
 		logT(t, "expecting error due to circular reference. found none")
-		t.FailNow()
+		failT(t)
 	}
 	if x := err.Error(); strings.Contains(x, "circular") || strings.Contains(x, "cyclic") {
 		logT(t, "error detected as expected: %v", x)
 	} else {
 		logT(t, "error detected was not as expected: %v", x)
-		t.FailNow()
+		failT(t)
 	}
 }
 
@@ -1046,7 +1092,7 @@ func doTestAnonCycle(t *testing.T, name string, h Handle) {
 
 	// just check that you can get typeInfo for T1
 	rt := reflect.TypeOf((*TestAnonCycleT1)(nil)).Elem()
-	rtid := reflect.ValueOf(rt).Pointer()
+	rtid := rt2id(rt)
 	pti := h.getBasicHandle().getTypeInfo(rtid, rt)
 	logT(t, "pti: %v", pti)
 }
@@ -1153,7 +1199,7 @@ func doTestRawValue(t *testing.T, name string, h Handle) {
 	// logT(t, "Encoded %v, decoded %v", i, i2)
 	if i != i2 {
 		logT(t, "Error: encoded %v, decoded %v", i, i2)
-		t.FailNow()
+		failT(t)
 	}
 }
 
@@ -1166,7 +1212,7 @@ func doTestPythonGenStreams(t *testing.T, name string, h Handle) {
 	tmpdir, err := ioutil.TempDir("", "golang-"+name+"-test")
 	if err != nil {
 		logT(t, "-------- Unable to create temp directory\n")
-		t.FailNow()
+		failT(t)
 	}
 	defer os.RemoveAll(tmpdir)
 	logT(t, "tmpdir: %v", tmpdir)
@@ -1177,7 +1223,7 @@ func doTestPythonGenStreams(t *testing.T, name string, h Handle) {
 	if cmdout, err = cmd.CombinedOutput(); err != nil {
 		logT(t, "-------- Error running test.py testdata. Err: %v", err)
 		logT(t, "         %v", string(cmdout))
-		t.FailNow()
+		failT(t)
 	}
 
 	bh := h.getBasicHandle()
@@ -1298,12 +1344,96 @@ func doTestMsgpackRpcSpecPythonClientToGoSvc(t *testing.T) {
 	if cmdout, err = cmd.CombinedOutput(); err != nil {
 		logT(t, "-------- Error running test.py rpc-client-go-service. Err: %v", err)
 		logT(t, "         %v", string(cmdout))
-		t.FailNow()
+		failT(t)
 	}
 	checkEqualT(t, string(cmdout),
 		fmt.Sprintf("%#v\n%#v\n", []string{"A1", "B2", "C3"}, TestRpcABC{"Aa", "Bb", "Cc"}), "cmdout=")
 }
 
+func testRandomFillRV(v reflect.Value) {
+	fneg := func() int64 {
+		i := rand.Intn(1)
+		if i == 1 {
+			return 1
+		}
+		return -1
+	}
+
+	switch v.Kind() {
+	case reflect.Invalid:
+	case reflect.Ptr:
+		if v.IsNil() {
+			v.Set(reflect.New(v.Type().Elem()))
+		}
+		testRandomFillRV(v.Elem())
+	case reflect.Interface:
+		if v.IsNil() {
+			v.Set(reflect.ValueOf("nothing"))
+		} else {
+			testRandomFillRV(v.Elem())
+		}
+	case reflect.Struct:
+		for i, n := 0, v.NumField(); i < n; i++ {
+			testRandomFillRV(v.Field(i))
+		}
+	case reflect.Slice:
+		if v.IsNil() {
+			v.Set(reflect.MakeSlice(v.Type(), 4, 4))
+		}
+		fallthrough
+	case reflect.Array:
+		for i, n := 0, v.Len(); i < n; i++ {
+			testRandomFillRV(v.Index(i))
+		}
+	case reflect.Map:
+		if v.IsNil() {
+			v.Set(reflect.MakeMap(v.Type()))
+		}
+		if v.Len() == 0 {
+			kt, vt := v.Type().Key(), v.Type().Elem()
+			for i := 0; i < 4; i++ {
+				k0 := reflect.New(kt).Elem()
+				v0 := reflect.New(vt).Elem()
+				testRandomFillRV(k0)
+				testRandomFillRV(v0)
+				v.SetMapIndex(k0, v0)
+			}
+		} else {
+			for _, k := range v.MapKeys() {
+				testRandomFillRV(v.MapIndex(k))
+			}
+		}
+	case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
+		v.SetInt(fneg() * rand.Int63n(127))
+	case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
+		v.SetUint(uint64(rand.Int63n(255)))
+	case reflect.Bool:
+		if fneg() == 1 {
+			v.SetBool(true)
+		} else {
+			v.SetBool(false)
+		}
+	case reflect.Float32, reflect.Float64:
+		v.SetFloat(float64(fneg()) * float64(rand.Float32()))
+	case reflect.String:
+		v.SetString(strings.Repeat(strconv.FormatInt(rand.Int63n(99), 10), rand.Intn(8)))
+	default:
+		panic(fmt.Errorf("testRandomFillRV: unsupported type: %v", v.Kind()))
+	}
+}
+
+func testMammoth(t *testing.T, name string, h Handle) {
+	testOnce.Do(testInitAll)
+	var m, m2 TestMammoth
+	testRandomFillRV(reflect.ValueOf(&m).Elem())
+	b, _ := testMarshalErr(&m, h, t, "mammoth-"+name)
+	testUnmarshalErr(&m2, b, h, t, "mammoth-"+name)
+	// fmt.Printf("m2: %v", &m2)
+	testDeepEqualErr(&m, &m2, t, "mammoth-"+name)
+}
+
+// -----------------
+
 func TestBincCodecsTable(t *testing.T) {
 	testCodecTableOne(t, testBincH)
 }
@@ -1320,6 +1450,10 @@ func TestBincStdEncIntf(t *testing.T) {
 	doTestStdEncIntf(t, "binc", testBincH)
 }
 
+func TestBincMammoth(t *testing.T) {
+	testMammoth(t, "binc", testBincH)
+}
+
 func TestSimpleCodecsTable(t *testing.T) {
 	testCodecTableOne(t, testSimpleH)
 }
@@ -1336,6 +1470,10 @@ func TestSimpleStdEncIntf(t *testing.T) {
 	doTestStdEncIntf(t, "simple", testSimpleH)
 }
 
+func TestSimpleMammoth(t *testing.T) {
+	testMammoth(t, "simple", testSimpleH)
+}
+
 func TestMsgpackCodecsTable(t *testing.T) {
 	testCodecTableOne(t, testMsgpackH)
 }
@@ -1352,6 +1490,10 @@ func TestMsgpackStdEncIntf(t *testing.T) {
 	doTestStdEncIntf(t, "msgpack", testMsgpackH)
 }
 
+func TestMsgpackMammoth(t *testing.T) {
+	testMammoth(t, "msgpack", testMsgpackH)
+}
+
 func TestCborCodecsTable(t *testing.T) {
 	testCodecTableOne(t, testCborH)
 }
@@ -1376,6 +1518,10 @@ func TestCborStdEncIntf(t *testing.T) {
 	doTestStdEncIntf(t, "cbor", testCborH)
 }
 
+func TestCborMammoth(t *testing.T) {
+	testMammoth(t, "cbor", testCborH)
+}
+
 func TestJsonCodecsTable(t *testing.T) {
 	testCodecTableOne(t, testJsonH)
 }
@@ -1396,6 +1542,10 @@ func TestJsonStdEncIntf(t *testing.T) {
 	doTestStdEncIntf(t, "json", testJsonH)
 }
 
+func TestJsonMammoth(t *testing.T) {
+	testMammoth(t, "json", testJsonH)
+}
+
 // ----- Raw ---------
 func TestJsonRaw(t *testing.T) {
 	doTestRawValue(t, "json", testJsonH)
@@ -1485,6 +1635,119 @@ func TestJsonDecodeNonStringScalarInStringContext(t *testing.T) {
 	}
 }
 
+func TestJsonEncodeIndent(t *testing.T) {
+	v := TestSimplish{
+		Ii: -794,
+		Ss: `A Man is
+after the new line
+	after new line and tab
+`,
+	}
+	v2 := v
+	v.Mm = make(map[string]*TestSimplish)
+	for i := 0; i < len(v.Ar); i++ {
+		v3 := v2
+		v3.Ii += (i * 4)
+		v3.Ss = fmt.Sprintf("%d - %s", v3.Ii, v3.Ss)
+		if i%2 == 0 {
+			v.Ar[i] = &v3
+		}
+		// v3 = v2
+		v.Sl = append(v.Sl, &v3)
+		v.Mm[strconv.FormatInt(int64(i), 10)] = &v3
+	}
+	oldcan := testJsonH.Canonical
+	oldIndent := testJsonH.Indent
+	oldS2A := testJsonH.StructToArray
+	defer func() {
+		testJsonH.Canonical = oldcan
+		testJsonH.Indent = oldIndent
+		testJsonH.StructToArray = oldS2A
+	}()
+	testJsonH.Canonical = true
+	testJsonH.Indent = -1
+	testJsonH.StructToArray = false
+	var bs []byte
+	NewEncoderBytes(&bs, testJsonH).MustEncode(&v)
+	txt1Tab := string(bs)
+	bs = nil
+	testJsonH.Indent = 120
+	NewEncoderBytes(&bs, testJsonH).MustEncode(&v)
+	txtSpaces := string(bs)
+	// fmt.Printf("\n-----------\n%s\n------------\n%s\n-------------\n", txt1Tab, txtSpaces)
+
+	goldenResultTab := `{
+	"Ar": [
+		{
+			"Ar": [
+				null,
+				null
+			],
+			"Ii": -794,
+			"Mm": null,
+			"Sl": null,
+			"Ss": "-794 - A Man is\nafter the new line\n\tafter new line and tab\n"
+		},
+		null
+	],
+	"Ii": -794,
+	"Mm": {
+		"0": {
+			"Ar": [
+				null,
+				null
+			],
+			"Ii": -794,
+			"Mm": null,
+			"Sl": null,
+			"Ss": "-794 - A Man is\nafter the new line\n\tafter new line and tab\n"
+		},
+		"1": {
+			"Ar": [
+				null,
+				null
+			],
+			"Ii": -790,
+			"Mm": null,
+			"Sl": null,
+			"Ss": "-790 - A Man is\nafter the new line\n\tafter new line and tab\n"
+		}
+	},
+	"Sl": [
+		{
+			"Ar": [
+				null,
+				null
+			],
+			"Ii": -794,
+			"Mm": null,
+			"Sl": null,
+			"Ss": "-794 - A Man is\nafter the new line\n\tafter new line and tab\n"
+		},
+		{
+			"Ar": [
+				null,
+				null
+			],
+			"Ii": -790,
+			"Mm": null,
+			"Sl": null,
+			"Ss": "-790 - A Man is\nafter the new line\n\tafter new line and tab\n"
+		}
+	],
+	"Ss": "A Man is\nafter the new line\n\tafter new line and tab\n"
+}`
+
+	if txt1Tab != goldenResultTab {
+		logT(t, "decoded indented with tabs != expected: \nexpected: %s\nencoded: %s", goldenResultTab, txt1Tab)
+		failT(t)
+	}
+	if txtSpaces != strings.Replace(goldenResultTab, "\t", strings.Repeat(" ", 120), -1) {
+		logT(t, "decoded indented with spaces != expected: \nexpected: %s\nencoded: %s", goldenResultTab, txtSpaces)
+		failT(t)
+	}
+}
+
 // TODO:
 //   Add Tests for:
 //   - decoding empty list/map in stream into a nil slice/map
@@ -1499,7 +1762,6 @@ func TestJsonDecodeNonStringScalarInStringContext(t *testing.T) {
 //   - struct tags:
 //     on anonymous fields, _struct (all fields), etc
 //   - codecgen of struct containing channels.
-//   - bad input with large array length prefix
 //
-//   Cleanup tests:
-//   - The are brittle in their handling of validation and skipping
+//  Add negative tests for failure conditions:
+//   - bad input with large array length prefix

codec/codecgen/gen.go

@@ -62,7 +62,7 @@ func CodecGenTempWrite{{ .RandString }}() {
 	var t{{ $index }} {{ . }}
 	typs = append(typs, reflect.TypeOf(t{{ $index }}))
 {{ end }}
-	{{ if not .CodecPkgFiles }}{{ .CodecPkgName }}.{{ end }}Gen(&out, "{{ .BuildTag }}", "{{ .PackageName }}", "{{ .RandString }}", {{ .UseUnsafe }}, {{ if not .CodecPkgFiles }}{{ .CodecPkgName }}.{{ end }}NewTypeInfos(strings.Split("{{ .StructTags }}", ",")), typs...)
+	{{ if not .CodecPkgFiles }}{{ .CodecPkgName }}.{{ end }}Gen(&out, "{{ .BuildTag }}", "{{ .PackageName }}", "{{ .RandString }}", {{ if not .CodecPkgFiles }}{{ .CodecPkgName }}.{{ end }}NewTypeInfos(strings.Split("{{ .StructTags }}", ",")), typs...)
 	bout, err := format.Source(out.Bytes())
 	if err != nil {
 		fout.Write(out.Bytes())
@@ -82,7 +82,7 @@ func CodecGenTempWrite{{ .RandString }}() {
 // Tool then executes: "go run __frun__" which creates fout.
 // fout contains Codec(En|De)codeSelf implementations for every type T.
 //
-func Generate(outfile, buildTag, codecPkgPath string, uid int64, useUnsafe bool, goRunTag string,
+func Generate(outfile, buildTag, codecPkgPath string, uid int64, goRunTag string,
 	st string, regexName *regexp.Regexp, notRegexName *regexp.Regexp, deleteTempFile bool, infiles ...string) (err error) {
 	// For each file, grab AST, find each type, and write a call to it.
 	if len(infiles) == 0 {
@@ -121,14 +121,12 @@ func Generate(outfile, buildTag, codecPkgPath string, uid int64, useUnsafe bool,
 		StructTags      string
 		Types           []string
 		CodecPkgFiles   bool
-		UseUnsafe       bool
 	}
 	tv := tmplT{
 		CodecPkgName:    genCodecPkg,
 		OutFile:         outfile,
 		CodecImportPath: codecPkgPath,
 		BuildTag:        buildTag,
-		UseUnsafe:       useUnsafe,
 		RandString:      strconv.FormatInt(uid, 10),
 		StructTags:      st,
 	}
@@ -256,7 +254,7 @@ func Generate(outfile, buildTag, codecPkgPath string, uid int64, useUnsafe bool,
 	os.Remove(outfile)
 
 	// execute go run frun
-	cmd := exec.Command("go", "run", "-tags="+goRunTag, frunMainName) //, frunPkg.Name())
+	cmd := exec.Command("go", "run", "-tags", "codecgen.exec safe "+goRunTag, frunMainName) //, frunPkg.Name())
 	var buf bytes.Buffer
 	cmd.Stdout = &buf
 	cmd.Stderr = &buf
@@ -313,10 +311,10 @@ func main() {
 	rt := flag.String("rt", "", "tags for go run")
 	st := flag.String("st", "codec,json", "struct tag keys to introspect")
 	x := flag.Bool("x", false, "keep temp file")
-	u := flag.Bool("u", false, "Use unsafe, e.g. to avoid unnecessary allocation on []byte->string")
+	_ = flag.Bool("u", false, "*IGNORED - kept for backwards compatibility*: Allow unsafe use")
 	d := flag.Int64("d", 0, "random identifier for use in generated code")
 	flag.Parse()
-	if err := Generate(*o, *t, *c, *d, *u, *rt, *st,
+	if err := Generate(*o, *t, *c, *d, *rt, *st,
 		regexp.MustCompile(*r), regexp.MustCompile(*nr), !*x, flag.Args()...); err != nil {
 		fmt.Fprintf(os.Stderr, "codecgen error: %v\n", err)
 		os.Exit(1)

codec/encode.go

@@ -313,21 +313,21 @@ type encFnInfo struct {
 }
 
 func (f *encFnInfo) builtin(rv reflect.Value) {
-	f.e.e.EncodeBuiltin(f.ti.rtid, rv.Interface())
+	f.e.e.EncodeBuiltin(f.ti.rtid, rv2i(rv))
 }
 
 func (f *encFnInfo) raw(rv reflect.Value) {
-	f.e.raw(rv.Interface().(Raw))
+	f.e.raw(rv2i(rv).(Raw))
 }
 
 func (f *encFnInfo) rawExt(rv reflect.Value) {
-	// rev := rv.Interface().(RawExt)
+	// rev := rv2i(rv).(RawExt)
 	// f.e.e.EncodeRawExt(&rev, f.e)
 	var re *RawExt
 	if rv.CanAddr() {
-		re = rv.Addr().Interface().(*RawExt)
+		re = rv2i(rv.Addr()).(*RawExt)
 	} else {
-		rev := rv.Interface().(RawExt)
+		rev := rv2i(rv).(RawExt)
 		re = &rev
 	}
 	f.e.e.EncodeRawExt(re, f.e)
@@ -338,21 +338,21 @@ func (f *encFnInfo) ext(rv reflect.Value) {
 	if k := rv.Kind(); (k == reflect.Struct || k == reflect.Array) && rv.CanAddr() {
 		rv = rv.Addr()
 	}
-	f.e.e.EncodeExt(rv.Interface(), f.xfTag, f.xfFn, f.e)
+	f.e.e.EncodeExt(rv2i(rv), f.xfTag, f.xfFn, f.e)
 }
 
 func (f *encFnInfo) getValueForMarshalInterface(rv reflect.Value, indir int8) (v interface{}, proceed bool) {
 	if indir == 0 {
-		v = rv.Interface()
+		v = rv2i(rv)
 	} else if indir == -1 {
 		// If a non-pointer was passed to Encode(), then that value is not addressable.
 		// Take addr if addressable, else copy value to an addressable value.
 		if rv.CanAddr() {
-			v = rv.Addr().Interface()
+			v = rv2i(rv.Addr())
 		} else {
 			rv2 := reflect.New(rv.Type())
 			rv2.Elem().Set(rv)
-			v = rv2.Interface()
+			v = rv2i(rv2)
 			// fmt.Printf("rv.Type: %v, rv2.Type: %v, v: %v\n", rv.Type(), rv2.Type(), v)
 		}
 	} else {
@@ -363,7 +363,7 @@ func (f *encFnInfo) getValueForMarshalInterface(rv reflect.Value, indir int8) (v
 			}
 			rv = rv.Elem()
 		}
-		v = rv.Interface()
+		v = rv2i(rv)
 	}
 	return v, true
 }
@@ -383,7 +383,7 @@ func (f *encFnInfo) binaryMarshal(rv reflect.Value) {
 
 func (f *encFnInfo) textMarshal(rv reflect.Value) {
 	if v, proceed := f.getValueForMarshalInterface(rv, f.ti.tmIndir); proceed {
-		// debugf(">>>> encoding.TextMarshaler: %T", rv.Interface())
+		// debugf(">>>> encoding.TextMarshaler: %T", rv2i(rv))
 		bs, fnerr := v.(encoding.TextMarshaler).MarshalText()
 		f.e.marshal(bs, fnerr, false, c_UTF8)
 	}
@@ -476,10 +476,10 @@ func (f *encFnInfo) kSlice(rv reflect.Value) {
 			bs := e.b[:0]
 			// do not use range, so that the number of elements encoded
 			// does not change, and encoding does not hang waiting on someone to close chan.
-			// for b := range rv.Interface().(<-chan byte) {
+			// for b := range rv2i(rv).(<-chan byte) {
 			// 	bs = append(bs, b)
 			// }
-			ch := rv.Interface().(<-chan byte)
+			ch := rv2i(rv).(<-chan byte)
 			for i := 0; i < l; i++ {
 				bs = append(bs, <-ch)
 			}
@@ -507,7 +507,7 @@ func (f *encFnInfo) kSlice(rv reflect.Value) {
 		// a concrete type and kInterface will bomb.
 		var fn *encFn
 		if rtelem.Kind() != reflect.Interface {
-			rtelemid := reflect.ValueOf(rtelem).Pointer()
+			rtelemid := rt2id(rtelem)
 			fn = e.getEncFn(rtelemid, rtelem, true, true)
 		}
 		// TODO: Consider perf implication of encoding odd index values as symbols if type is string
@@ -680,7 +680,7 @@ func (f *encFnInfo) kMap(rv reflect.Value) {
 	ti := f.ti
 	rtkey := ti.rt.Key()
 	rtval := ti.rt.Elem()
-	rtkeyid := reflect.ValueOf(rtkey).Pointer()
+	rtkeyid := rt2id(rtkey)
 	// keyTypeIsString := f.ti.rt.Key().Kind() == reflect.String
 	var keyTypeIsString = rtkeyid == stringTypId
 	if keyTypeIsString {
@@ -690,7 +690,7 @@ func (f *encFnInfo) kMap(rv reflect.Value) {
 			rtkey = rtkey.Elem()
 		}
 		if rtkey.Kind() != reflect.Interface {
-			rtkeyid = reflect.ValueOf(rtkey).Pointer()
+			rtkeyid = rt2id(rtkey)
 			keyFn = e.getEncFn(rtkeyid, rtkey, true, true)
 		}
 	}
@@ -698,7 +698,7 @@ func (f *encFnInfo) kMap(rv reflect.Value) {
 		rtval = rtval.Elem()
 	}
 	if rtval.Kind() != reflect.Interface {
-		rtvalid := reflect.ValueOf(rtval).Pointer()
+		rtvalid := rt2id(rtval)
 		valFn = e.getEncFn(rtvalid, rtval, true, true)
 	}
 	mks := rv.MapKeys()
@@ -1216,7 +1216,7 @@ func (e *Encoder) doEncodeValue(rv reflect.Value, fn *encFn, sptr uintptr,
 	}
 	if fn == nil {
 		rt := rv.Type()
-		rtid := reflect.ValueOf(rt).Pointer()
+		rtid := rt2id(rt)
 		// fn = e.getEncFn(rtid, rt, true, true)
 		fn = e.getEncFn(rtid, rt, checkFastpath, checkCodecSelfer)
 	}
@@ -1240,7 +1240,7 @@ func (e *Encoder) encodeValue(rv reflect.Value, fn *encFn) {
 }
 
 func (e *Encoder) getEncFn(rtid uintptr, rt reflect.Type, checkFastpath, checkCodecSelfer bool) (fn *encFn) {
-	// rtid := reflect.ValueOf(rt).Pointer()
+	// rtid := rt2id(rt)
 	var ok bool
 	if useMapForCodecCache {
 		fn, ok = e.f[rtid]
@@ -1310,7 +1310,7 @@ func (e *Encoder) getEncFn(rtid uintptr, rt reflect.Type, checkFastpath, checkCo
 				} else {
 					rtu = reflect.SliceOf(rt.Elem())
 				}
-				rtuid := reflect.ValueOf(rtu).Pointer()
+				rtuid := rt2id(rtu)
 				if idx := fastpathAV.index(rtuid); idx != -1 {
 					xfnf := fastpathAV[idx].encfn
 					xrt := fastpathAV[idx].rt

codec/fast-path.go.tmpl

@@ -86,7 +86,7 @@ func init() {
 	i := 0
 	fn := func(v interface{}, fe func(*encFnInfo, reflect.Value), fd func(*decFnInfo, reflect.Value)) (f fastpathE) {
 		xrt := reflect.TypeOf(v)
-		xptr := reflect.ValueOf(xrt).Pointer()
+		xptr := rt2id(xrt)
 		fastpathAV[i] = fastpathE{xptr, xrt, fe, fd}
 		i++
 		return
@@ -156,9 +156,9 @@ func fastpathEncodeTypeSwitchMap(iv interface{}, e *Encoder) bool {
 
 func (f *encFnInfo) {{ .MethodNamePfx "fastpathEnc" false }}R(rv reflect.Value) {
 	if f.ti.mbs {
-		fastpathTV.{{ .MethodNamePfx "EncAsMap" false }}V(rv.Interface().([]{{ .Elem }}), fastpathCheckNilFalse, f.e)
+		fastpathTV.{{ .MethodNamePfx "EncAsMap" false }}V(rv2i(rv).([]{{ .Elem }}), fastpathCheckNilFalse, f.e)
 	} else {
-		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(rv.Interface().([]{{ .Elem }}), fastpathCheckNilFalse, f.e)
+		fastpathTV.{{ .MethodNamePfx "Enc" false }}V(rv2i(rv).([]{{ .Elem }}), fastpathCheckNilFalse, f.e)
 	}
 }
 func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v []{{ .Elem }}, checkNil bool, e *Encoder) {
@@ -206,7 +206,7 @@ func (_ fastpathT) {{ .MethodNamePfx "EncAsMap" false }}V(v []{{ .Elem }}, check
 {{range .Values}}{{if not .Primitive}}{{if .MapKey }}
 
 func (f *encFnInfo) {{ .MethodNamePfx "fastpathEnc" false }}R(rv reflect.Value) {
-	fastpathTV.{{ .MethodNamePfx "Enc" false }}V(rv.Interface().(map[{{ .MapKey }}]{{ .Elem }}), fastpathCheckNilFalse, f.e)
+	fastpathTV.{{ .MethodNamePfx "Enc" false }}V(rv2i(rv).(map[{{ .MapKey }}]{{ .Elem }}), fastpathCheckNilFalse, f.e)
 }
 func (_ fastpathT) {{ .MethodNamePfx "Enc" false }}V(v map[{{ .MapKey }}]{{ .Elem }}, checkNil bool, e *Encoder) {
 	ee := e.e
@@ -306,13 +306,13 @@ Slices can change if they
 func (f *decFnInfo) {{ .MethodNamePfx "fastpathDec" false }}R(rv reflect.Value) { 
 	array := f.seq == seqTypeArray
 	if !array && rv.CanAddr() { {{/* // CanSet => CanAddr + Exported */}}
-		vp := rv.Addr().Interface().(*[]{{ .Elem }})
+		vp := rv2i(rv.Addr()).(*[]{{ .Elem }})
 		v, changed := fastpathTV.{{ .MethodNamePfx "Dec" false }}V(*vp, fastpathCheckNilFalse, !array, f.d)
 		if changed {
 			*vp = v
 		}
 	} else {
-		v := rv.Interface().([]{{ .Elem }})
+		v := rv2i(rv).([]{{ .Elem }})
 		fastpathTV.{{ .MethodNamePfx "Dec" false }}V(v, fastpathCheckNilFalse, false, f.d)
 	}
 }
@@ -327,16 +327,13 @@ func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v []{{ .Elem }}, checkNil b
 	dd := d.d
 	{{/* // if dd.isContainerType(valueTypeNil) { dd.TryDecodeAsNil() */}}
 	if checkNil && dd.TryDecodeAsNil() {
-		if v != nil {
-			changed = true 
-		}
+		if v != nil { changed = true }
 		return nil, changed 
 	}
-
 	slh, containerLenS := d.decSliceHelperStart()
 	if containerLenS == 0 {
 		if canChange {
-			if v == nil {
+			if v == nil { 
 				v = []{{ .Elem }}{}
 			} else if len(v) != 0 {
 				v = v[:0]
@@ -346,98 +343,62 @@ func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v []{{ .Elem }}, checkNil b
 		slh.End()
 		return v, changed
 	}
-	
-	if containerLenS > 0 {
-		x2read := containerLenS
-		var xtrunc bool 
+
+	hasLen := containerLenS > 0
+	var xlen int 
+	if hasLen && canChange {
 		if containerLenS > cap(v) {
-			if canChange { {{/*
-				// fast-path is for "basic" immutable types, so no need to copy them over
-				// s := make([]{{ .Elem }}, decInferLen(containerLenS, d.h.MaxInitLen))
-				// copy(s, v[:cap(v)])
-				// v = s */}}
-				var xlen int 
-                xlen, xtrunc = decInferLen(containerLenS, d.h.MaxInitLen, {{ .Size }})
-				if xtrunc {
-					if xlen <= cap(v) {
-						v = v[:xlen]
-					} else {
-						v = make([]{{ .Elem }}, xlen)
-					}
-				} else {
-					v = make([]{{ .Elem }}, xlen)
-				}
-				changed = true
+			xlen = decInferLen(containerLenS, d.h.MaxInitLen, {{ .Size }})
+			if xlen <= cap(v) {
+				v = v[:xlen]
 			} else {
-				d.arrayCannotExpand(len(v), containerLenS)
+				v = make([]{{ .Elem }}, xlen)
 			}
-			x2read = len(v)
+			changed = true 
 		} else if containerLenS != len(v) {
-			if canChange {
-				v = v[:containerLenS]
-				changed = true
-			}
-		} {{/* // all checks done. cannot go past len. */}}
-		j := 0
-		for ; j < x2read; j++ {
-			slh.ElemContainerState(j)
-			{{ if eq .Elem "interface{}" }}d.decode(&v[j]){{ else }}v[j] = {{ decmd .Elem }}{{ end }}
+			v = v[:containerLenS]
+			changed = true
 		}
-		if xtrunc { {{/* // means canChange=true, changed=true already. */}}
-			for ; j < containerLenS; j++ {
-				v = append(v, {{ zerocmd .Elem }})
-				slh.ElemContainerState(j)
-				{{ if eq .Elem "interface{}" }}d.decode(&v[j]){{ else }}v[j] = {{ decmd .Elem }}{{ end }}
-			}
-		} else if !canChange {
-			for ; j < containerLenS; j++ {
-				slh.ElemContainerState(j)
-				d.swallow()
+	}
+	j := 0
+	for ; (hasLen && j < containerLenS) || !(hasLen || dd.CheckBreak()); j++ {
+		if j == 0 && len(v) == 0 {
+			if hasLen {
+				xlen = decInferLen(containerLenS, d.h.MaxInitLen, {{ .Size }})
+			} else {
+				xlen = 8
 			}
+			v = make([]{{ .Elem }}, xlen)
+			changed = true 
 		}
-	} else {
-		breakFound := dd.CheckBreak() {{/* check break first, so we can initialize v with a capacity of 4 if necessary */}}
-		if breakFound {
+		// if indefinite, etc, then expand the slice if necessary
+		var decodeIntoBlank bool
+		if j >= len(v) {
 			if canChange {
-				if v == nil {
-					v = []{{ .Elem }}{}
-				} else if len(v) != 0 {
-					v = v[:0]
-				}
+				v = append(v, {{ zerocmd .Elem }})
 				changed = true
-			}
-			slh.End()
-			return v, changed
-		}
-		if cap(v) == 0 {
-			v = make([]{{ .Elem }}, 1, 4)
-			changed = true
-		}
-		j := 0	
-		for ; !breakFound; j++ {
-			if j >= len(v) { 
-				if canChange {
-					v = append(v, {{ zerocmd .Elem }})
-					changed = true
-				} else {
-					d.arrayCannotExpand(len(v), j+1)
-				}
-			}
-			slh.ElemContainerState(j)
-			if j < len(v) { {{/* // all checks done. cannot go past len. */}}
-				{{ if eq .Elem "interface{}" }}d.decode(&v[j])
-				{{ else }}v[j] = {{ decmd .Elem }}{{ end }}
 			} else {
-				d.swallow()
+				d.arrayCannotExpand(len(v), j+1)
+				decodeIntoBlank = true
 			}
-			breakFound = dd.CheckBreak()
 		}
-		if canChange && j < len(v) {
+		slh.ElemContainerState(j)
+		if decodeIntoBlank {
+			d.swallow()
+		} else {
+			{{ if eq .Elem "interface{}" }}d.decode(&v[j]){{ else }}v[j] = {{ decmd .Elem }}{{ end }}
+		}
+	}
+	if canChange {
+		if j < len(v) {
 			v = v[:j]
 			changed = true
+		} else if j == 0 && v == nil {
+			v = make([]{{ .Elem }}, 0)
+			changed = true
 		}
 	}
-	slh.End() 
+	slh.End()
 	return v, changed 
 }
 
@@ -452,13 +413,13 @@ Maps can change if they are
 */}}
 func (f *decFnInfo) {{ .MethodNamePfx "fastpathDec" false }}R(rv reflect.Value) { 
 	if rv.CanAddr() {
-		vp := rv.Addr().Interface().(*map[{{ .MapKey }}]{{ .Elem }})
+		vp := rv2i(rv.Addr()).(*map[{{ .MapKey }}]{{ .Elem }})
 		v, changed := fastpathTV.{{ .MethodNamePfx "Dec" false }}V(*vp, fastpathCheckNilFalse, true, f.d)
 		if changed {
 			*vp = v
 		}
 	} else {
-		v := rv.Interface().(map[{{ .MapKey }}]{{ .Elem }})
+		v := rv2i(rv).(map[{{ .MapKey }}]{{ .Elem }})
 		fastpathTV.{{ .MethodNamePfx "Dec" false }}V(v, fastpathCheckNilFalse, false, f.d)
 	}
 }
@@ -474,50 +435,35 @@ func (_ fastpathT) {{ .MethodNamePfx "Dec" false }}V(v map[{{ .MapKey }}]{{ .Ele
 	cr := d.cr
 	{{/* // if dd.isContainerType(valueTypeNil) {dd.TryDecodeAsNil() */}}
 	if checkNil && dd.TryDecodeAsNil() {
-		if v != nil {
-			changed = true
-		} 
+		if v != nil { changed = true } 
 		return nil, changed
 	}
-
 	containerLen := dd.ReadMapStart()
 	if canChange && v == nil {
-		xlen, _ := decInferLen(containerLen, d.h.MaxInitLen, {{ .Size }})
+		xlen := decInferLen(containerLen, d.h.MaxInitLen, {{ .Size }})
 		v = make(map[{{ .MapKey }}]{{ .Elem }}, xlen)
 		changed = true
 	}
+	if containerLen == 0 {
+		if cr != nil { cr.sendContainerState(containerMapEnd) }
+		return v, changed
+	}
 	{{ if eq .Elem "interface{}" }}mapGet := !d.h.MapValueReset && !d.h.InterfaceReset{{end}}
 	var mk {{ .MapKey }}
 	var mv {{ .Elem }}
-	if containerLen > 0 {
-		for j := 0; j < containerLen; j++ {
-			if cr != nil { cr.sendContainerState(containerMapKey) }
-			{{ if eq .MapKey "interface{}" }}mk = nil 
-			d.decode(&mk)
-			if bv, bok := mk.([]byte); bok {
-				mk = d.string(bv) {{/* // maps cannot have []byte as key. switch to string. */}}
-			}{{ else }}mk = {{ decmd .MapKey }}{{ end }}
-			if cr != nil { cr.sendContainerState(containerMapValue) }
-			{{ if eq .Elem "interface{}" }}if mapGet { mv = v[mk] } else { mv = nil }
-			d.decode(&mv){{ else }}mv = {{ decmd .Elem }}{{ end }}
-			if v != nil {
-				v[mk] = mv
-			}
-		}
-	} else if containerLen < 0 {
-		for j := 0; !dd.CheckBreak(); j++ {
-			if cr != nil { cr.sendContainerState(containerMapKey) }
-			{{ if eq .MapKey "interface{}" }}mk = nil 
-			d.decode(&mk)
-			if bv, bok := mk.([]byte); bok {
-				mk = d.string(bv) {{/* // maps cannot have []byte as key. switch to string. */}}
-			}{{ else }}mk = {{ decmd .MapKey }}{{ end }}
-			if cr != nil { cr.sendContainerState(containerMapValue) }
-			{{ if eq .Elem "interface{}" }}if mapGet { mv = v[mk] } else { mv = nil }
-			d.decode(&mv){{ else }}mv = {{ decmd .Elem }}{{ end }}
-			if v != nil {
-				v[mk] = mv
-			}
+	hasLen := containerLen > 0
+	for j := 0; (hasLen && j < containerLen) || !(hasLen || dd.CheckBreak()); j++ {
+		if cr != nil { cr.sendContainerState(containerMapKey) }
+		{{ if eq .MapKey "interface{}" }}mk = nil 
+		d.decode(&mk)
+		if bv, bok := mk.([]byte); bok {
+			mk = d.string(bv) {{/* // maps cannot have []byte as key. switch to string. */}}
+		}{{ else }}mk = {{ decmd .MapKey }}{{ end }}
+		if cr != nil { cr.sendContainerState(containerMapValue) }
+		{{ if eq .Elem "interface{}" }}if mapGet { mv = v[mk] } else { mv = nil }
+		d.decode(&mv){{ else }}mv = {{ decmd .Elem }}{{ end }}
+		if v != nil {
+			v[mk] = mv
 		}
 	}
 	if cr != nil { cr.sendContainerState(containerMapEnd) }

codec/gen-dec-array.go.tmpl

@@ -13,92 +13,58 @@ if {{var "l"}} == 0 {
 		{{var "v"}} = make({{ .CTyp }}, 0)
 		{{var "c"}} = true
 	} {{end}}
-} else if {{var "l"}} > 0 {
-	{{if isChan }}if {{var "v"}} == nil {
-		{{var "rl"}}, _ = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
-		{{var "v"}} = make({{ .CTyp }}, {{var "rl"}})
-		{{var "c"}} = true
-	}
-	for {{var "r"}} := 0; {{var "r"}} < {{var "l"}}; {{var "r"}}++ {
-		{{var "h"}}.ElemContainerState({{var "r"}})
-		var {{var "t"}} {{ .Typ }}
-		{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
-		{{var "v"}} <- {{var "t"}}
-	}
-	{{ else }}	var {{var "rr"}}, {{var "rl"}} int {{/* // num2read, length of slice/array/chan */}}
-	var {{var "rt"}} bool {{/* truncated */}}
-	_, _ = {{var "rl"}}, {{var "rt"}}
-	{{var "rr"}} = {{var "l"}} // len({{var "v"}})
+} else {
+	{{var "hl"}} := {{var "l"}} > 0
+	var {{var "rl"}} int 
+	{{if isSlice }} if {{var "hl"}} {
 	if {{var "l"}} > cap({{var "v"}}) {
-		{{if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "l"}})
-		{{ else }}{{if not .Immutable }}
-		{{var "rg"}} := len({{var "v"}}) > 0
-		{{var "v2"}} := {{var "v"}} {{end}}
-		{{var "rl"}}, {{var "rt"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
-		if {{var "rt"}} {
-			if {{var "rl"}} <= cap({{var "v"}}) {
-				{{var "v"}} = {{var "v"}}[:{{var "rl"}}]
-			} else {
-				{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
-			}
+		{{var "rl"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
+		if {{var "rl"}} <= cap({{var "v"}}) {
+			{{var "v"}} = {{var "v"}}[:{{var "rl"}}]
 		} else {
 			{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
 		}
 		{{var "c"}} = true
-		{{var "rr"}} = len({{var "v"}}) {{if not .Immutable }}
-			if {{var "rg"}} { copy({{var "v"}}, {{var "v2"}}) } {{end}} {{end}}{{/* end not Immutable, isArray */}}
-	} {{if isSlice }} else if {{var "l"}} != len({{var "v"}}) {
+	} else if {{var "l"}} != len({{var "v"}}) {
 		{{var "v"}} = {{var "v"}}[:{{var "l"}}]
 		{{var "c"}} = true
-	} {{end}}	{{/* end isSlice:47 */}} 
-	{{var "j"}} := 0
-	for ; {{var "j"}} < {{var "rr"}} ; {{var "j"}}++ {
-		{{var "h"}}.ElemContainerState({{var "j"}})
-		{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
-	}
-	{{if isArray }}for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
-		{{var "h"}}.ElemContainerState({{var "j"}})
-		z.DecSwallow()
 	}
-	{{ else }}if {{var "rt"}} {
-		for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
-			{{var "v"}} = append({{var "v"}}, {{ zero}})
-			{{var "h"}}.ElemContainerState({{var "j"}})
-			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
-		}
-	} {{end}} {{/* end isArray:56 */}}
-	{{end}} {{/* end isChan:16 */}}
-} else { {{/* len < 0 */}}
+	} {{end}}
 	{{var "j"}} := 0
-	for ; !r.CheckBreak(); {{var "j"}}++ {
-		{{if isChan }}
-		{{var "h"}}.ElemContainerState({{var "j"}})
-		var {{var "t"}} {{ .Typ }}
-		{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
-		{{var "v"}} <- {{var "t"}} 
-		{{ else }}
+	for ; ({{var "hl"}} && {{var "j"}} < {{var "l"}}) || !({{var "hl"}} || r.CheckBreak()); {{var "j"}}++ {
+		if {{var "j"}} == 0 && len({{var "v"}}) == 0 {
+			if {{var "hl"}} {
+				{{var "rl"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
+			} else {
+				{{var "rl"}} = 8
+			}
+			{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
+			{{var "c"}} = true 
+		}
+		// if indefinite, etc, then expand the slice if necessary
+		var {{var "db"}} bool
 		if {{var "j"}} >= len({{var "v"}}) {
-			{{if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "j"}}+1)
-			{{ else }}{{var "v"}} = append({{var "v"}}, {{zero}})// var {{var "z"}} {{ .Typ }}
-			{{var "c"}} = true {{end}}
+			{{if isSlice }} {{var "v"}} = append({{var "v"}}, {{ zero }}); {{var "c"}} = true
+			{{end}} {{if isArray}} z.DecArrayCannotExpand(len(v), {{var "j"}}+1); {{var "db"}} = true
+			{{end}}
 		}
 		{{var "h"}}.ElemContainerState({{var "j"}})
-		if {{var "j"}} < len({{var "v"}}) {
-			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
-		} else {
+		if {{var "db"}} {
 			z.DecSwallow()
+		} else {
+			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
 		}
-		{{end}}
 	}
-	{{if isSlice }}if {{var "j"}} < len({{var "v"}}) {
+	{{if isSlice}} if {{var "j"}} < len({{var "v"}}) {
 		{{var "v"}} = {{var "v"}}[:{{var "j"}}]
 		{{var "c"}} = true
 	} else if {{var "j"}} == 0 && {{var "v"}} == nil {
-		{{var "v"}} = []{{ .Typ }}{}
+		{{var "v"}} = make([]{{ .Typ }}, 0)
 		{{var "c"}} = true
-	}{{end}}
+	} {{end}}
 }
 {{var "h"}}.End()
 {{if not isArray }}if {{var "c"}} { 
 	*{{ .Varname }} = {{var "v"}}
 }{{end}}
+

codec/gen-dec-map.go.tmpl

@@ -2,7 +2,7 @@
 {{var "l"}} := r.ReadMapStart()
 {{var "bh"}} := z.DecBasicHandle()
 if {{var "v"}} == nil {
-	{{var "rl"}}, _ := z.DecInferLen({{var "l"}}, {{var "bh"}}.MaxInitLen, {{ .Size }})
+	{{var "rl"}} := z.DecInferLen({{var "l"}}, {{var "bh"}}.MaxInitLen, {{ .Size }})
 	{{var "v"}} = make(map[{{ .KTyp }}]{{ .Typ }}, {{var "rl"}})
 	*{{ .Varname }} = {{var "v"}}
 }
@@ -14,8 +14,9 @@ if {{var "bh"}}.MapValueReset {
 	{{else if decElemKindIntf}}if !{{var "bh"}}.InterfaceReset { {{var "mg"}} = true }
 	{{else if not decElemKindImmutable}}{{var "mg"}} = true
 	{{end}} }
-if {{var "l"}} > 0  {
-for {{var "j"}} := 0; {{var "j"}} < {{var "l"}}; {{var "j"}}++ {
+if {{var "l"}} != 0 {
+{{var "hl"}} := {{var "l"}} > 0 
+	for {{var "j"}} := 0; ({{var "hl"}} && {{var "j"}} < {{var "l"}}) || !({{var "hl"}} || r.CheckBreak()); {{var "j"}}++ {
 	z.DecSendContainerState(codecSelfer_containerMapKey{{ .Sfx }})
 	{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
 {{ if eq .KTyp "interface{}" }}{{/* // special case if a byte array. */}}if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
@@ -34,25 +35,5 @@ for {{var "j"}} := 0; {{var "j"}} < {{var "l"}}; {{var "j"}}++ {
 		{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
 	}
 }
-} else if {{var "l"}} < 0  {
-for {{var "j"}} := 0; !r.CheckBreak(); {{var "j"}}++ {
-	z.DecSendContainerState(codecSelfer_containerMapKey{{ .Sfx }})
-	{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
-{{ if eq .KTyp "interface{}" }}{{/* // special case if a byte array. */}}if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
-		{{var "mk"}} = string({{var "bv"}})
-	}{{ end }}{{if decElemKindPtr}}
-	{{var "ms"}} = true {{ end }}
-	if {{var "mg"}} {
-		{{if decElemKindPtr}}{{var "mv"}}, {{var "mok"}} = {{var "v"}}[{{var "mk"}}] 
-		if {{var "mok"}} {
-			{{var "ms"}} = false
-		} {{else}}{{var "mv"}} = {{var "v"}}[{{var "mk"}}] {{end}}
-	} {{if not decElemKindImmutable}}else { {{var "mv"}} = {{decElemZero}} }{{end}}
-	z.DecSendContainerState(codecSelfer_containerMapValue{{ .Sfx }})
-	{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
-	if {{if decElemKindPtr}} {{var "ms"}} && {{end}} {{var "v"}} != nil {
-		{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
-	}
-}
 } // else len==0: TODO: Should we clear map entries?
 z.DecSendContainerState(codecSelfer_containerMapEnd{{ .Sfx }})

codec/gen-helper.generated.go

@@ -15,6 +15,9 @@ import (
 	"reflect"
 )
 
+// GenVersion is the current version of codecgen.
+const GenVersion = 6
+
 // This file is used to generate helper code for codecgen.
 // The values here i.e. genHelper(En|De)coder are not to be used directly by
 // library users. They WILL change continuously and without notice.
@@ -26,12 +29,14 @@ import (
 // to perform encoding or decoding of primitives or known slice or map types.
 
 // GenHelperEncoder is exported so that it can be used externally by codecgen.
+//
 // Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.
 func GenHelperEncoder(e *Encoder) (genHelperEncoder, encDriver) {
 	return genHelperEncoder{e: e}, e.e
 }
 
 // GenHelperDecoder is exported so that it can be used externally by codecgen.
+//
 // Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.
 func GenHelperDecoder(d *Decoder) (genHelperDecoder, decDriver) {
 	return genHelperDecoder{d: d}, d.d
@@ -112,7 +117,7 @@ func (f genHelperEncoder) EncExt(v interface{}) (r bool) {
 	if rt.Kind() == reflect.Ptr {
 		rt = rt.Elem()
 	}
-	rtid := reflect.ValueOf(rt).Pointer()
+	rtid := rt2id(rt)
 	if xfFn := f.e.h.getExt(rtid); xfFn != nil {
 		f.e.e.EncodeExt(v, xfFn.tag, xfFn.ext, f.e)
 		return true
@@ -172,7 +177,7 @@ func (f genHelperDecoder) DecArrayCannotExpand(sliceLen, streamLen int) {
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecTextUnmarshal(tm encoding.TextUnmarshaler) {
-	fnerr := tm.UnmarshalText(f.d.d.DecodeBytes(f.d.b[:], true, true))
+	fnerr := tm.UnmarshalText(f.d.d.DecodeStringAsBytes())
 	if fnerr != nil {
 		panic(fnerr)
 	}
@@ -180,7 +185,7 @@ func (f genHelperDecoder) DecTextUnmarshal(tm encoding.TextUnmarshaler) {
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecJSONUnmarshal(tm jsonUnmarshaler) {
-	// bs := f.dd.DecodeBytes(f.d.b[:], true, true)
+	// bs := f.dd.DecodeStringAsBytes()
 	// grab the bytes to be read, as UnmarshalJSON needs the full JSON so as to unmarshal it itself.
 	fnerr := tm.UnmarshalJSON(f.d.nextValueBytes())
 	if fnerr != nil {
@@ -190,7 +195,7 @@ func (f genHelperDecoder) DecJSONUnmarshal(tm jsonUnmarshaler) {
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecBinaryUnmarshal(bm encoding.BinaryUnmarshaler) {
-	fnerr := bm.UnmarshalBinary(f.d.d.DecodeBytes(nil, false, true))
+	fnerr := bm.UnmarshalBinary(f.d.d.DecodeBytes(nil, true))
 	if fnerr != nil {
 		panic(fnerr)
 	}
@@ -222,7 +227,7 @@ func (f genHelperDecoder) HasExtensions() bool {
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecExt(v interface{}) (r bool) {
 	rt := reflect.TypeOf(v).Elem()
-	rtid := reflect.ValueOf(rt).Pointer()
+	rtid := rt2id(rt)
 	if xfFn := f.d.h.getExt(rtid); xfFn != nil {
 		f.d.d.DecodeExt(v, xfFn.tag, xfFn.ext)
 		return true
@@ -231,10 +236,15 @@ func (f genHelperDecoder) DecExt(v interface{}) (r bool) {
 }
 
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
-func (f genHelperDecoder) DecInferLen(clen, maxlen, unit int) (rvlen int, truncated bool) {
+func (f genHelperDecoder) DecInferLen(clen, maxlen, unit int) (rvlen int) {
 	return decInferLen(clen, maxlen, unit)
 }
 
+// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
+func (f genHelperDecoder) StringView(v []byte) string {
+	return stringView(v)
+}
+
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecSendContainerState(c containerState) {
 	if f.d.cr != nil {

codec/gen-helper.go.tmpl

@@ -15,6 +15,9 @@ import (
 	"reflect"
 )
 
+// GenVersion is the current version of codecgen.
+const GenVersion = {{ .Version }} 
+
 // This file is used to generate helper code for codecgen. 
 // The values here i.e. genHelper(En|De)coder are not to be used directly by 
 // library users. They WILL change continuously and without notice.
@@ -26,12 +29,14 @@ import (
 // to perform encoding or decoding of primitives or known slice or map types.
 
 // GenHelperEncoder is exported so that it can be used externally by codecgen.
+// 
 // Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.
 func GenHelperEncoder(e *Encoder) (genHelperEncoder, encDriver) {
 	return genHelperEncoder{e:e}, e.e 
 }
 
 // GenHelperDecoder is exported so that it can be used externally by codecgen.
+// 
 // Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.
 func GenHelperDecoder(d *Decoder) (genHelperDecoder, decDriver) {
 	return genHelperDecoder{d:d}, d.d 
@@ -103,7 +108,7 @@ func (f genHelperEncoder) EncExt(v interface{}) (r bool) {
 	if rt.Kind() == reflect.Ptr {
 		rt = rt.Elem()
 	}
-	rtid := reflect.ValueOf(rt).Pointer()
+	rtid := rt2id(rt)
 	if xfFn := f.e.h.getExt(rtid); xfFn != nil {
 		f.e.e.EncodeExt(v, xfFn.tag, xfFn.ext, f.e)
 		return true
@@ -154,14 +159,14 @@ func (f genHelperDecoder) DecArrayCannotExpand(sliceLen, streamLen int) {
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecTextUnmarshal(tm encoding.TextUnmarshaler) {
-	fnerr := tm.UnmarshalText(f.d.d.DecodeBytes(f.d.b[:], true, true))
+	fnerr := tm.UnmarshalText(f.d.d.DecodeStringAsBytes())
 	if fnerr != nil {
 		panic(fnerr)
 	}
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecJSONUnmarshal(tm jsonUnmarshaler) {
-	// bs := f.dd.DecodeBytes(f.d.b[:], true, true)
+	// bs := f.dd.DecodeStringAsBytes()
 	// grab the bytes to be read, as UnmarshalJSON needs the full JSON so as to unmarshal it itself.
 	fnerr := tm.UnmarshalJSON(f.d.nextValueBytes())
 	if fnerr != nil {
@@ -170,7 +175,7 @@ func (f genHelperDecoder) DecJSONUnmarshal(tm jsonUnmarshaler) {
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecBinaryUnmarshal(bm encoding.BinaryUnmarshaler) {
-	fnerr := bm.UnmarshalBinary(f.d.d.DecodeBytes(nil, false, true))
+	fnerr := bm.UnmarshalBinary(f.d.d.DecodeBytes(nil, true))
 	if fnerr != nil {
 		panic(fnerr)
 	}
@@ -197,7 +202,7 @@ func (f genHelperDecoder) HasExtensions() bool {
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecExt(v interface{}) (r bool) {
 	rt := reflect.TypeOf(v).Elem()
-	rtid := reflect.ValueOf(rt).Pointer()
+	rtid := rt2id(rt)
 	if xfFn := f.d.h.getExt(rtid); xfFn != nil {
 		f.d.d.DecodeExt(v, xfFn.tag, xfFn.ext)
 		return true
@@ -205,10 +210,14 @@ func (f genHelperDecoder) DecExt(v interface{}) (r bool) {
 	return false 
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
-func (f genHelperDecoder) DecInferLen(clen, maxlen, unit int) (rvlen int, truncated bool) {
+func (f genHelperDecoder) DecInferLen(clen, maxlen, unit int) (rvlen int) {
 	return decInferLen(clen, maxlen, unit)
 }
 // FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
+func (f genHelperDecoder) StringView(v []byte) string {
+	return stringView(v)
+}
+// FOR USE BY CODECGEN ONLY. IT *WILL* CHANGE WITHOUT NOTICE. *DO NOT USE*
 func (f genHelperDecoder) DecSendContainerState(c containerState) {
 	if f.d.cr != nil {
 		f.d.cr.sendContainerState(c)

codec/gen.generated.go

@@ -10,7 +10,7 @@ const genDecMapTmpl = `
 {{var "l"}} := r.ReadMapStart()
 {{var "bh"}} := z.DecBasicHandle()
 if {{var "v"}} == nil {
-	{{var "rl"}}, _ := z.DecInferLen({{var "l"}}, {{var "bh"}}.MaxInitLen, {{ .Size }})
+	{{var "rl"}} := z.DecInferLen({{var "l"}}, {{var "bh"}}.MaxInitLen, {{ .Size }})
 	{{var "v"}} = make(map[{{ .KTyp }}]{{ .Typ }}, {{var "rl"}})
 	*{{ .Varname }} = {{var "v"}}
 }
@@ -22,8 +22,9 @@ if {{var "bh"}}.MapValueReset {
 	{{else if decElemKindIntf}}if !{{var "bh"}}.InterfaceReset { {{var "mg"}} = true }
 	{{else if not decElemKindImmutable}}{{var "mg"}} = true
 	{{end}} }
-if {{var "l"}} > 0  {
-for {{var "j"}} := 0; {{var "j"}} < {{var "l"}}; {{var "j"}}++ {
+if {{var "l"}} != 0 {
+{{var "hl"}} := {{var "l"}} > 0 
+	for {{var "j"}} := 0; ({{var "hl"}} && {{var "j"}} < {{var "l"}}) || !({{var "hl"}} || r.CheckBreak()); {{var "j"}}++ {
 	z.DecSendContainerState(codecSelfer_containerMapKey{{ .Sfx }})
 	{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
 {{ if eq .KTyp "interface{}" }}{{/* // special case if a byte array. */}}if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
@@ -42,26 +43,6 @@ for {{var "j"}} := 0; {{var "j"}} < {{var "l"}}; {{var "j"}}++ {
 		{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
 	}
 }
-} else if {{var "l"}} < 0  {
-for {{var "j"}} := 0; !r.CheckBreak(); {{var "j"}}++ {
-	z.DecSendContainerState(codecSelfer_containerMapKey{{ .Sfx }})
-	{{ $x := printf "%vmk%v" .TempVar .Rand }}{{ decLineVarK $x }}
-{{ if eq .KTyp "interface{}" }}{{/* // special case if a byte array. */}}if {{var "bv"}}, {{var "bok"}} := {{var "mk"}}.([]byte); {{var "bok"}} {
-		{{var "mk"}} = string({{var "bv"}})
-	}{{ end }}{{if decElemKindPtr}}
-	{{var "ms"}} = true {{ end }}
-	if {{var "mg"}} {
-		{{if decElemKindPtr}}{{var "mv"}}, {{var "mok"}} = {{var "v"}}[{{var "mk"}}] 
-		if {{var "mok"}} {
-			{{var "ms"}} = false
-		} {{else}}{{var "mv"}} = {{var "v"}}[{{var "mk"}}] {{end}}
-	} {{if not decElemKindImmutable}}else { {{var "mv"}} = {{decElemZero}} }{{end}}
-	z.DecSendContainerState(codecSelfer_containerMapValue{{ .Sfx }})
-	{{ $x := printf "%vmv%v" .TempVar .Rand }}{{ decLineVar $x }}
-	if {{if decElemKindPtr}} {{var "ms"}} && {{end}} {{var "v"}} != nil {
-		{{var "v"}}[{{var "mk"}}] = {{var "mv"}}
-	}
-}
 } // else len==0: TODO: Should we clear map entries?
 z.DecSendContainerState(codecSelfer_containerMapEnd{{ .Sfx }})
 `
@@ -82,94 +63,60 @@ if {{var "l"}} == 0 {
 		{{var "v"}} = make({{ .CTyp }}, 0)
 		{{var "c"}} = true
 	} {{end}}
-} else if {{var "l"}} > 0 {
-	{{if isChan }}if {{var "v"}} == nil {
-		{{var "rl"}}, _ = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
-		{{var "v"}} = make({{ .CTyp }}, {{var "rl"}})
-		{{var "c"}} = true
-	}
-	for {{var "r"}} := 0; {{var "r"}} < {{var "l"}}; {{var "r"}}++ {
-		{{var "h"}}.ElemContainerState({{var "r"}})
-		var {{var "t"}} {{ .Typ }}
-		{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
-		{{var "v"}} <- {{var "t"}}
-	}
-	{{ else }}	var {{var "rr"}}, {{var "rl"}} int {{/* // num2read, length of slice/array/chan */}}
-	var {{var "rt"}} bool {{/* truncated */}}
-	_, _ = {{var "rl"}}, {{var "rt"}}
-	{{var "rr"}} = {{var "l"}} // len({{var "v"}})
+} else {
+	{{var "hl"}} := {{var "l"}} > 0
+	var {{var "rl"}} int 
+	{{if isSlice }} if {{var "hl"}} {
 	if {{var "l"}} > cap({{var "v"}}) {
-		{{if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "l"}})
-		{{ else }}{{if not .Immutable }}
-		{{var "rg"}} := len({{var "v"}}) > 0
-		{{var "v2"}} := {{var "v"}} {{end}}
-		{{var "rl"}}, {{var "rt"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
-		if {{var "rt"}} {
-			if {{var "rl"}} <= cap({{var "v"}}) {
-				{{var "v"}} = {{var "v"}}[:{{var "rl"}}]
-			} else {
-				{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
-			}
+		{{var "rl"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
+		if {{var "rl"}} <= cap({{var "v"}}) {
+			{{var "v"}} = {{var "v"}}[:{{var "rl"}}]
 		} else {
 			{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
 		}
 		{{var "c"}} = true
-		{{var "rr"}} = len({{var "v"}}) {{if not .Immutable }}
-			if {{var "rg"}} { copy({{var "v"}}, {{var "v2"}}) } {{end}} {{end}}{{/* end not Immutable, isArray */}}
-	} {{if isSlice }} else if {{var "l"}} != len({{var "v"}}) {
+	} else if {{var "l"}} != len({{var "v"}}) {
 		{{var "v"}} = {{var "v"}}[:{{var "l"}}]
 		{{var "c"}} = true
-	} {{end}}	{{/* end isSlice:47 */}} 
-	{{var "j"}} := 0
-	for ; {{var "j"}} < {{var "rr"}} ; {{var "j"}}++ {
-		{{var "h"}}.ElemContainerState({{var "j"}})
-		{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
-	}
-	{{if isArray }}for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
-		{{var "h"}}.ElemContainerState({{var "j"}})
-		z.DecSwallow()
 	}
-	{{ else }}if {{var "rt"}} {
-		for ; {{var "j"}} < {{var "l"}} ; {{var "j"}}++ {
-			{{var "v"}} = append({{var "v"}}, {{ zero}})
-			{{var "h"}}.ElemContainerState({{var "j"}})
-			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
-		}
-	} {{end}} {{/* end isArray:56 */}}
-	{{end}} {{/* end isChan:16 */}}
-} else { {{/* len < 0 */}}
+	} {{end}}
 	{{var "j"}} := 0
-	for ; !r.CheckBreak(); {{var "j"}}++ {
-		{{if isChan }}
-		{{var "h"}}.ElemContainerState({{var "j"}})
-		var {{var "t"}} {{ .Typ }}
-		{{ $x := printf "%st%s" .TempVar .Rand }}{{ decLineVar $x }}
-		{{var "v"}} <- {{var "t"}} 
-		{{ else }}
+	for ; ({{var "hl"}} && {{var "j"}} < {{var "l"}}) || !({{var "hl"}} || r.CheckBreak()); {{var "j"}}++ {
+		if {{var "j"}} == 0 && len({{var "v"}}) == 0 {
+			if {{var "hl"}} {
+				{{var "rl"}} = z.DecInferLen({{var "l"}}, z.DecBasicHandle().MaxInitLen, {{ .Size }})
+			} else {
+				{{var "rl"}} = 8
+			}
+			{{var "v"}} = make([]{{ .Typ }}, {{var "rl"}})
+			{{var "c"}} = true 
+		}
+		// if indefinite, etc, then expand the slice if necessary
+		var {{var "db"}} bool
 		if {{var "j"}} >= len({{var "v"}}) {
-			{{if isArray }}z.DecArrayCannotExpand(len({{var "v"}}), {{var "j"}}+1)
-			{{ else }}{{var "v"}} = append({{var "v"}}, {{zero}})// var {{var "z"}} {{ .Typ }}
-			{{var "c"}} = true {{end}}
+			{{if isSlice }} {{var "v"}} = append({{var "v"}}, {{ zero }}); {{var "c"}} = true
+			{{end}} {{if isArray}} z.DecArrayCannotExpand(len(v), {{var "j"}}+1); {{var "db"}} = true
+			{{end}}
 		}
 		{{var "h"}}.ElemContainerState({{var "j"}})
-		if {{var "j"}} < len({{var "v"}}) {
-			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
-		} else {
+		if {{var "db"}} {
 			z.DecSwallow()
+		} else {
+			{{ $x := printf "%[1]vv%[2]v[%[1]vj%[2]v]" .TempVar .Rand }}{{ decLineVar $x }}
 		}
-		{{end}}
 	}
-	{{if isSlice }}if {{var "j"}} < len({{var "v"}}) {
+	{{if isSlice}} if {{var "j"}} < len({{var "v"}}) {
 		{{var "v"}} = {{var "v"}}[:{{var "j"}}]
 		{{var "c"}} = true
 	} else if {{var "j"}} == 0 && {{var "v"}} == nil {
-		{{var "v"}} = []{{ .Typ }}{}
+		{{var "v"}} = make([]{{ .Typ }}, 0)
 		{{var "c"}} = true
-	}{{end}}
+	} {{end}}
 }
 {{var "h"}}.End()
 {{if not isArray }}if {{var "c"}} { 
 	*{{ .Varname }} = {{var "v"}}
 }{{end}}
+
 `
 

codec/gen.go

@@ -1,3 +1,5 @@
+// +build codecgen.exec
+
 // Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
 // Use of this source code is governed by a MIT license found in the LICENSE file.
 
@@ -80,6 +82,10 @@ import (
 // Note:
 //   It was a conscious decision to have gen.go always explicitly call EncodeNil or TryDecodeAsNil.
 //   This way, there isn't a function call overhead just to see that we should not enter a block of code.
+//
+// Note:
+//   codecgen-generated code depends on the variables defined by fast-path.generated.go.
+//   consequently, you cannot run with tags "codecgen notfastpath".
 
 // GenVersion is the current version of codecgen.
 //
@@ -94,7 +100,8 @@ import (
 //     changes in signature of some unpublished helper methods and codecgen cmdline arguments.
 // v4: Removed separator support from (en|de)cDriver, and refactored codec(gen)
 // v5: changes to support faster json decoding. Let encoder/decoder maintain state of collections.
-const GenVersion = 5
+// v6: removed unsafe from gen, and now uses codecgen.exec tag
+const genVersion = 6
 
 const (
 	genCodecPkg        = "codec1978"
@@ -126,7 +133,6 @@ var (
 	genExpectArrayOrMapErr = errors.New("unexpected type. Expecting array/map/slice")
 	genBase64enc           = base64.NewEncoding("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789__")
 	genQNameRegex          = regexp.MustCompile(`[A-Za-z_.]+`)
-	genCheckVendor         bool
 )
 
 // genRunner holds some state used during a Gen run.
@@ -147,8 +153,7 @@ type genRunner struct {
 	is map[reflect.Type]struct{} // types seen during import search
 	bp string                    // base PkgPath, for which we are generating for
 
-	cpfx   string // codec package prefix
-	unsafe bool   // is unsafe to be used in generated code?
+	cpfx string // codec package prefix
 
 	tm map[reflect.Type]struct{} // types for which enc/dec must be generated
 	ts []reflect.Type            // types for which enc/dec must be generated
@@ -163,8 +168,8 @@ type genRunner struct {
 // Gen will write a complete go file containing Selfer implementations for each
 // type passed. All the types must be in the same package.
 //
-// Library users: *DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.*
-func Gen(w io.Writer, buildTags, pkgName, uid string, useUnsafe bool, ti *TypeInfos, typ ...reflect.Type) {
+// Library users: DO NOT USE IT DIRECTLY. IT WILL CHANGE CONTINOUSLY WITHOUT NOTICE.
+func Gen(w io.Writer, buildTags, pkgName, uid string, ti *TypeInfos, typ ...reflect.Type) {
 	// All types passed to this method do not have a codec.Selfer method implemented directly.
 	// codecgen already checks the AST and skips any types that define the codec.Selfer methods.
 	// Consequently, there's no need to check and trim them if they implement codec.Selfer
@@ -173,19 +178,18 @@ func Gen(w io.Writer, buildTags, pkgName, uid string, useUnsafe bool, ti *TypeIn
 		return
 	}
 	x := genRunner{
-		unsafe: useUnsafe,
-		w:      w,
-		t:      typ,
-		te:     make(map[uintptr]bool),
-		td:     make(map[uintptr]bool),
-		im:     make(map[string]reflect.Type),
-		imn:    make(map[string]string),
-		is:     make(map[reflect.Type]struct{}),
-		tm:     make(map[reflect.Type]struct{}),
-		ts:     []reflect.Type{},
-		bp:     genImportPath(typ[0]),
-		xs:     uid,
-		ti:     ti,
+		w:   w,
+		t:   typ,
+		te:  make(map[uintptr]bool),
+		td:  make(map[uintptr]bool),
+		im:  make(map[string]reflect.Type),
+		imn: make(map[string]string),
+		is:  make(map[reflect.Type]struct{}),
+		tm:  make(map[reflect.Type]struct{}),
+		ts:  []reflect.Type{},
+		bp:  genImportPath(typ[0]),
+		xs:  uid,
+		ti:  ti,
 	}
 	if x.ti == nil {
 		x.ti = defTypeInfos
@@ -234,11 +238,8 @@ func Gen(w io.Writer, buildTags, pkgName, uid string, useUnsafe bool, ti *TypeIn
 		x.linef("%s \"%s\"", x.imn[k], k)
 	}
 	// add required packages
-	for _, k := range [...]string{"reflect", "unsafe", "runtime", "fmt", "errors"} {
+	for _, k := range [...]string{"reflect", "runtime", "fmt", "errors"} {
 		if _, ok := x.im[k]; !ok {
-			if k == "unsafe" && !x.unsafe {
-				continue
-			}
 			x.line("\"" + k + "\"")
 		}
 	}
@@ -265,20 +266,16 @@ func Gen(w io.Writer, buildTags, pkgName, uid string, useUnsafe bool, ti *TypeIn
 	x.line(")")
 	x.line("")
 
-	if x.unsafe {
-		x.line("type codecSelferUnsafeString" + x.xs + " struct { Data uintptr; Len int}")
-		x.line("")
-	}
 	x.hn = "codecSelfer" + x.xs
 	x.line("type " + x.hn + " struct{}")
 	x.line("")
 
 	x.varsfxreset()
 	x.line("func init() {")
-	x.linef("if %sGenVersion != %v {", x.cpfx, GenVersion)
+	x.linef("if %sGenVersion != %v {", x.cpfx, genVersion)
 	x.line("_, file, _, _ := runtime.Caller(0)")
 	x.line(`err := fmt.Errorf("codecgen version mismatch: current: %v, need %v. Re-generate file: %v", `)
-	x.linef(`%v, %sGenVersion, file)`, GenVersion, x.cpfx)
+	x.linef(`%v, %sGenVersion, file)`, genVersion, x.cpfx)
 	x.line("panic(err)")
 	x.linef("}")
 	x.line("if false { // reference the types, but skip this branch at build/run time")
@@ -289,10 +286,6 @@ func Gen(w io.Writer, buildTags, pkgName, uid string, useUnsafe bool, ti *TypeIn
 		x.linef("var v%v %s.%s", n, x.imn[k], t.Name())
 		n++
 	}
-	if x.unsafe {
-		x.linef("var v%v unsafe.Pointer", n)
-		n++
-	}
 	if n > 0 {
 		x.out("_")
 		for i := 1; i < n; i++ {
@@ -315,7 +308,7 @@ func Gen(w io.Writer, buildTags, pkgName, uid string, useUnsafe bool, ti *TypeIn
 	}
 
 	for _, t := range x.ts {
-		rtid := reflect.ValueOf(t).Pointer()
+		rtid := rt2id(t)
 		// generate enc functions for all these slice/map types.
 		x.varsfxreset()
 		x.linef("func (x %s) enc%s(v %s%s, e *%sEncoder) {", x.hn, x.genMethodNameT(t), x.arr2str(t, "*"), x.genTypeName(t), x.cpfx)
@@ -545,21 +538,21 @@ func (x *genRunner) selfer(encode bool) {
 		x.out(fnSigPfx)
 		x.line(") codecDecodeSelfFromMap(l int, d *" + x.cpfx + "Decoder) {")
 		x.genRequiredMethodVars(false)
-		x.decStructMap(genTopLevelVarName, "l", reflect.ValueOf(t0).Pointer(), t0, genStructMapStyleConsolidated)
+		x.decStructMap(genTopLevelVarName, "l", rt2id(t0), t0, genStructMapStyleConsolidated)
 		x.line("}")
 		x.line("")
 	} else {
 		x.out(fnSigPfx)
 		x.line(") codecDecodeSelfFromMapLenPrefix(l int, d *" + x.cpfx + "Decoder) {")
 		x.genRequiredMethodVars(false)
-		x.decStructMap(genTopLevelVarName, "l", reflect.ValueOf(t0).Pointer(), t0, genStructMapStyleLenPrefix)
+		x.decStructMap(genTopLevelVarName, "l", rt2id(t0), t0, genStructMapStyleLenPrefix)
 		x.line("}")
 		x.line("")
 
 		x.out(fnSigPfx)
 		x.line(") codecDecodeSelfFromMapCheckBreak(l int, d *" + x.cpfx + "Decoder) {")
 		x.genRequiredMethodVars(false)
-		x.decStructMap(genTopLevelVarName, "l", reflect.ValueOf(t0).Pointer(), t0, genStructMapStyleCheckBreak)
+		x.decStructMap(genTopLevelVarName, "l", rt2id(t0), t0, genStructMapStyleCheckBreak)
 		x.line("}")
 		x.line("")
 	}
@@ -568,7 +561,7 @@ func (x *genRunner) selfer(encode bool) {
 	x.out(fnSigPfx)
 	x.line(") codecDecodeSelfFromArray(l int, d *" + x.cpfx + "Decoder) {")
 	x.genRequiredMethodVars(false)
-	x.decStructArray(genTopLevelVarName, "l", "return", reflect.ValueOf(t0).Pointer(), t0)
+	x.decStructArray(genTopLevelVarName, "l", "return", rt2id(t0), t0)
 	x.line("}")
 	x.line("")
 
@@ -645,7 +638,7 @@ func (x *genRunner) encVar(varname string, t reflect.Type) {
 // enc will encode a variable (varname) of type t,
 // except t is of kind reflect.Struct or reflect.Array, wherein varname is of type ptrTo(T) (to prevent copying)
 func (x *genRunner) enc(varname string, t reflect.Type) {
-	rtid := reflect.ValueOf(t).Pointer()
+	rtid := rt2id(t)
 	// We call CodecEncodeSelf if one of the following are honored:
 	//   - the type already implements Selfer, call that
 	//   - the type has a Selfer implementation just created, use that
@@ -1098,7 +1091,7 @@ func (x *genRunner) dec(varname string, t reflect.Type) {
 	// assumptions:
 	//   - the varname is to a pointer already. No need to take address of it
 	//   - t is always a baseType T (not a *T, etc).
-	rtid := reflect.ValueOf(t).Pointer()
+	rtid := rt2id(t)
 	tptr := reflect.PtrTo(t)
 	if x.checkForSelfer(t, varname) {
 		if t.Implements(selferTyp) || tptr.Implements(selferTyp) {
@@ -1231,7 +1224,7 @@ func (x *genRunner) dec(varname string, t reflect.Type) {
 		// - if elements are primitives or Selfers, call dedicated function on each member.
 		// - else call Encoder.encode(XXX) on it.
 		if rtid == uint8SliceTypId {
-			x.line("*" + varname + " = r.DecodeBytes(*(*[]byte)(" + varname + "), false, false)")
+			x.line("*" + varname + " = r.DecodeBytes(*(*[]byte)(" + varname + "), false)")
 		} else if fastpathAV.index(rtid) != -1 {
 			g := x.newGenV(t)
 			x.line("z.F." + g.MethodNamePfx("Dec", false) + "X(" + varname + ", false, d)")
@@ -1318,11 +1311,11 @@ func (x *genRunner) decTryAssignPrimitive(varname string, t reflect.Type) (tryAs
 
 func (x *genRunner) decListFallback(varname string, rtid uintptr, t reflect.Type) {
 	if t.AssignableTo(uint8SliceTyp) {
-		x.line("*" + varname + " = r.DecodeBytes(*((*[]byte)(" + varname + ")), false, false)")
+		x.line("*" + varname + " = r.DecodeBytes(*((*[]byte)(" + varname + ")), false)")
 		return
 	}
 	if t.Kind() == reflect.Array && t.Elem().Kind() == reflect.Uint8 {
-		x.linef("r.DecodeBytes( ((*[%s]byte)(%s))[:], false, true)", t.Len(), varname)
+		x.linef("r.DecodeBytes( ((*[%s]byte)(%s))[:], true)", t.Len(), varname)
 		return
 	}
 	type tstruc struct {
@@ -1469,17 +1462,6 @@ func (x *genRunner) decStructMap(varname, lenvarname string, rtid uintptr, t ref
 	i := x.varsfx()
 	kName := tpfx + "s" + i
 
-	// We thought to use ReadStringAsBytes, as go compiler might optimize the copy out.
-	// However, using that was more expensive, as it seems that the switch expression
-	// is evaluated each time.
-	//
-	// We could depend on decodeString using a temporary/shared buffer internally.
-	// However, this model of creating a byte array, and using explicitly is faster,
-	// and allows optional use of unsafe []byte->string conversion without alloc.
-
-	// Also, ensure that the slice array doesn't escape.
-	// That will help escape analysis prevent allocation when it gets better.
-
 	// x.line("var " + kName + "Arr = [32]byte{} // default string to decode into")
 	// x.line("var " + kName + "Slc = " + kName + "Arr[:] // default slice to decode into")
 	// use the scratch buffer to avoid allocation (most field names are < 32).
@@ -1499,15 +1481,9 @@ func (x *genRunner) decStructMap(varname, lenvarname string, rtid uintptr, t ref
 		x.line("} else { if r.CheckBreak() { break }; }")
 	}
 	x.linef("z.DecSendContainerState(codecSelfer_containerMapKey%s)", x.xs)
-	x.line(kName + "Slc = r.DecodeBytes(" + kName + "Slc, true, true)")
+	x.line(kName + "Slc = r.DecodeStringAsBytes()")
 	// let string be scoped to this loop alone, so it doesn't escape.
-	if x.unsafe {
-		x.line(kName + "SlcHdr := codecSelferUnsafeString" + x.xs + "{uintptr(unsafe.Pointer(&" +
-			kName + "Slc[0])), len(" + kName + "Slc)}")
-		x.line(kName + " := *(*string)(unsafe.Pointer(&" + kName + "SlcHdr))")
-	} else {
-		x.line(kName + " := string(" + kName + "Slc)")
-	}
+	x.line(kName + " := string(" + kName + "Slc)")
 	x.linef("z.DecSendContainerState(codecSelfer_containerMapValue%s)", x.xs)
 	x.decStructMapSwitch(kName, varname, rtid, t)
 
@@ -1776,8 +1752,8 @@ func genIsImmutable(t reflect.Type) (v bool) {
 }
 
 type genInternal struct {
-	Values []genV
-	Unsafe bool
+	Version int
+	Values  []genV
 }
 
 func (x genInternal) FastpathLen() (l int) {
@@ -1891,7 +1867,7 @@ func stripVendor(s string) string {
 }
 
 // var genInternalMu sync.Mutex
-var genInternalV genInternal
+var genInternalV = genInternal{Version: genVersion}
 var genInternalTmplFuncs template.FuncMap
 var genInternalOnce sync.Once
 
@@ -1954,7 +1930,7 @@ func genInternalInit() {
 		"float64":     8,
 		"bool":        1,
 	}
-	var gt genInternal
+	var gt = genInternal{Version: genVersion}
 
 	// For each slice or map type, there must be a (symmetrical) Encode and Decode fast-path function
 	for _, s := range types {
@@ -1986,11 +1962,10 @@ func genInternalInit() {
 // It is run by the program author alone.
 // Unfortunately, it has to be exported so that it can be called from a command line tool.
 // *** DO NOT USE ***
-func genInternalGoFile(r io.Reader, w io.Writer, safe bool) (err error) {
+func genInternalGoFile(r io.Reader, w io.Writer) (err error) {
 	genInternalOnce.Do(genInternalInit)
 
 	gt := genInternalV
-	gt.Unsafe = !safe
 
 	t := template.New("").Funcs(genInternalTmplFuncs)
 

codec/decode_go.go → codec/goversion_arrayof_gte_go15.go

@@ -9,8 +9,6 @@ import "reflect"
 
 const reflectArrayOfSupported = true
 
-func reflectArrayOf(rvn reflect.Value) (rvn2 reflect.Value) {
-	rvn2 = reflect.New(reflect.ArrayOf(rvn.Len(), intfTyp)).Elem()
-	reflect.Copy(rvn2, rvn)
-	return
+func reflectArrayOf(count int, elem reflect.Type) reflect.Type {
+	return reflect.ArrayOf(count, elem)
 }

codec/decode_go14.go → codec/goversion_arrayof_lt_go15.go

@@ -9,6 +9,6 @@ import "reflect"
 
 const reflectArrayOfSupported = false
 
-func reflectArrayOf(rvn reflect.Value) (rvn2 reflect.Value) {
-	panic("reflect.ArrayOf unsupported")
+func reflectArrayOf(count int, elem reflect.Type) reflect.Type {
+	panic("codec: reflect.ArrayOf unsupported in this go version")
 }

codec/goversion_makemap_gte_go19.go

@@ -0,0 +1,15 @@
+// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
+// Use of this source code is governed by a MIT license found in the LICENSE file.
+
+// +build go1.9
+
+package codec
+
+import "reflect"
+
+func makeMapReflect(t reflect.Type, size int) reflect.Value {
+	if size < 0 {
+		return reflect.MakeMapWithSize(t, 4)
+	}
+	return reflect.MakeMapWithSize(t, size)
+}

codec/goversion_makemap_lt_go19.go

@@ -0,0 +1,12 @@
+// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
+// Use of this source code is governed by a MIT license found in the LICENSE file.
+
+// +build !go1.9
+
+package codec
+
+import "reflect"
+
+func makeMapReflect(t reflect.Type, size int) reflect.Value {
+	return reflect.MakeMap(t)
+}

codec/goversion_unsupported_lt_go14.go

@@ -0,0 +1,17 @@
+// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
+// Use of this source code is governed by a MIT license found in the LICENSE file.
+
+// +build !go1.4
+
+package codec
+
+// This codec package will only work for go1.4 and above.
+// This is for the following reasons:
+//   - go 1.4 was released in 2014
+//   - go runtime is written fully in go
+//   - interface only holds pointers
+//   - reflect.Value is stabilized as 3 words
+
+func init() {
+	panic("codec: go 1.3 and below are not supported")
+}

codec/gen_15.go → codec/goversion_vendor_eq_go15.go

@@ -7,6 +7,4 @@ package codec
 
 import "os"
 
-func init() {
-	genCheckVendor = os.Getenv("GO15VENDOREXPERIMENT") == "1"
-}
+var genCheckVendor = os.Getenv("GO15VENDOREXPERIMENT") == "1"

codec/gen_16.go → codec/goversion_vendor_eq_go16.go

@@ -1,12 +1,10 @@
 // Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
 // Use of this source code is governed by a MIT license found in the LICENSE file.
 
-// +build go1.6
+// +build go1.6,!go1.7
 
 package codec
 
 import "os"
 
-func init() {
-	genCheckVendor = os.Getenv("GO15VENDOREXPERIMENT") != "0"
-}
+var genCheckVendor = os.Getenv("GO15VENDOREXPERIMENT") != "0"

codec/gen_17.go → codec/goversion_vendor_gte_go17.go

@@ -5,6 +5,4 @@
 
 package codec
 
-func init() {
-	genCheckVendor = true
-}
+const genCheckVendor = true

codec/goversion_vendor_lt_go15.go

@@ -0,0 +1,8 @@
+// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
+// Use of this source code is governed by a MIT license found in the LICENSE file.
+
+// +build !go1.5
+
+package codec
+
+var genCheckVendor = false

codec/helper.go

@@ -133,11 +133,11 @@ const (
 	// Note that this will always cause rpc tests to fail, since they need io.EOF sent via panic.
 	recoverPanicToErr = true
 
-	// if resetSliceElemToZeroValue, then on decoding a slice, reset the element to a zero value first.
-	// Only concern is that, if the slice already contained some garbage, we will decode into that garbage.
+	// resetSliceElemToZeroValue: on decoding a slice, reset the element to a zero value first.
+	// concern: if the slice already contained some garbage, we will decode into that garbage.
 	// The chances of this are slim, so leave this "optimization".
 	// TODO: should this be true, to ensure that we always decode into a "zero" "empty" value?
-	resetSliceElemToZeroValue bool = false
+	resetSliceElemToZeroValue = false
 )
 
 var (
@@ -250,6 +250,8 @@ type jsonUnmarshaler interface {
 	UnmarshalJSON([]byte) error
 }
 
+// type byteAccepter func(byte) bool
+
 var (
 	bigen               = binary.BigEndian
 	structInfoFieldName = "_struct"
@@ -278,17 +280,17 @@ var (
 
 	selferTyp = reflect.TypeOf((*Selfer)(nil)).Elem()
 
-	uint8SliceTypId = reflect.ValueOf(uint8SliceTyp).Pointer()
-	rawExtTypId     = reflect.ValueOf(rawExtTyp).Pointer()
-	rawTypId        = reflect.ValueOf(rawTyp).Pointer()
-	intfTypId       = reflect.ValueOf(intfTyp).Pointer()
-	timeTypId       = reflect.ValueOf(timeTyp).Pointer()
-	stringTypId     = reflect.ValueOf(stringTyp).Pointer()
+	uint8SliceTypId = rt2id(uint8SliceTyp)
+	rawExtTypId     = rt2id(rawExtTyp)
+	rawTypId        = rt2id(rawTyp)
+	intfTypId       = rt2id(intfTyp)
+	timeTypId       = rt2id(timeTyp)
+	stringTypId     = rt2id(stringTyp)
 
-	mapStrIntfTypId  = reflect.ValueOf(mapStrIntfTyp).Pointer()
-	mapIntfIntfTypId = reflect.ValueOf(mapIntfIntfTyp).Pointer()
-	intfSliceTypId   = reflect.ValueOf(intfSliceTyp).Pointer()
-	// mapBySliceTypId  = reflect.ValueOf(mapBySliceTyp).Pointer()
+	mapStrIntfTypId  = rt2id(mapStrIntfTyp)
+	mapIntfIntfTypId = rt2id(mapIntfIntfTyp)
+	intfSliceTypId   = rt2id(intfSliceTyp)
+	// mapBySliceTypId  = rt2id(mapBySliceTyp)
 
 	intBitsize  uint8 = uint8(reflect.TypeOf(int(0)).Bits())
 	uintBitsize uint8 = uint8(reflect.TypeOf(uint(0)).Bits())
@@ -303,6 +305,24 @@ var (
 
 var defTypeInfos = NewTypeInfos([]string{"codec", "json"})
 
+var immutableKindsSet = [32]bool{
+	reflect.Int:     true,
+	reflect.Int8:    true,
+	reflect.Int16:   true,
+	reflect.Int32:   true,
+	reflect.Int64:   true,
+	reflect.Uint:    true,
+	reflect.Uint8:   true,
+	reflect.Uint16:  true,
+	reflect.Uint32:  true,
+	reflect.Uint64:  true,
+	reflect.Uintptr: true,
+	reflect.Float32: true,
+	reflect.Float64: true,
+	reflect.Bool:    true,
+	reflect.String:  true,
+}
+
 // Selfer defines methods by which a value can encode or decode itself.
 //
 // Any type which implements Selfer will be able to encode or decode itself.
@@ -343,10 +363,10 @@ func (x *BasicHandle) getBasicHandle() *BasicHandle {
 }
 
 func (x *BasicHandle) getTypeInfo(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
-	if x.TypeInfos != nil {
-		return x.TypeInfos.get(rtid, rt)
+	if x.TypeInfos == nil {
+		return defTypeInfos.get(rtid, rt)
 	}
-	return defTypeInfos.get(rtid, rt)
+	return x.TypeInfos.get(rtid, rt)
 }
 
 // Handle is the interface for a specific encoding format.
@@ -476,9 +496,6 @@ func (x *setExtWrapper) UpdateExt(dest interface{}, v interface{}) {
 	x.i.UpdateExt(dest, v)
 }
 
-// type errorString string
-// func (x errorString) Error() string { return string(x) }
-
 type binaryEncodingType struct{}
 
 func (_ binaryEncodingType) isBinary() bool { return true }
@@ -559,7 +576,7 @@ func (o *extHandle) SetExt(rt reflect.Type, tag uint64, ext Ext) (err error) {
 		return
 	}
 
-	rtid := reflect.ValueOf(rt).Pointer()
+	rtid := rt2id(rt)
 	for _, v := range *o {
 		if v.rtid == rtid {
 			v.tag, v.ext = tag, ext
@@ -711,6 +728,7 @@ type typeInfo struct {
 
 	rt   reflect.Type
 	rtid uintptr
+	// rv0  reflect.Value // saved zero value, used if immutableKind
 
 	numMeth uint16 // number of methods
 
@@ -743,42 +761,49 @@ type typeInfo struct {
 	toArray bool // whether this (struct) type should be encoded as an array
 }
 
+// linear search. faster than binary search in my testing up to 16-field structs.
+const binarySearchThreshold = 8 // similar to what python does for hashtables
+
 func (ti *typeInfo) indexForEncName(name string) int {
 	// NOTE: name may be a stringView, so don't pass it to another function.
 	//tisfi := ti.sfi
-	const binarySearchThreshold = 16
-	if sfilen := len(ti.sfi); sfilen < binarySearchThreshold {
-		// linear search. faster than binary search in my testing up to 16-field structs.
+	sfilen := len(ti.sfi)
+	if sfilen < binarySearchThreshold {
 		for i, si := range ti.sfi {
 			if si.encName == name {
 				return i
 			}
 		}
-	} else {
-		// binary search. adapted from sort/search.go.
-		h, i, j := 0, 0, sfilen
-		for i < j {
-			h = i + (j-i)/2
-			if ti.sfi[h].encName < name {
-				i = h + 1
-			} else {
-				j = h
-			}
-		}
-		if i < sfilen && ti.sfi[i].encName == name {
-			return i
+		return -1
+	}
+	// binary search. adapted from sort/search.go.
+	h, i, j := 0, 0, sfilen
+	for i < j {
+		h = i + (j-i)/2
+		if ti.sfi[h].encName < name {
+			i = h + 1
+		} else {
+			j = h
 		}
 	}
+	if i < sfilen && ti.sfi[i].encName == name {
+		return i
+	}
 	return -1
 }
 
+type rtid2ti struct {
+	rtid uintptr
+	ti   *typeInfo
+}
+
 // TypeInfos caches typeInfo for each type on first inspection.
 //
 // It is configured with a set of tag keys, which are used to get
 // configuration for the type.
 type TypeInfos struct {
-	infos map[uintptr]*typeInfo
-	mu    sync.RWMutex
+	infos atomicTypeInfoSlice // formerly map[uintptr]*typeInfo, now *[]rtid2ti
+	mu    sync.Mutex
 	tags  []string
 }
 
@@ -787,7 +812,7 @@ type TypeInfos struct {
 // This allows users customize the struct tag keys which contain configuration
 // of their types.
 func NewTypeInfos(tags []string) *TypeInfos {
-	return &TypeInfos{tags: tags, infos: make(map[uintptr]*typeInfo, 64)}
+	return &TypeInfos{tags: tags}
 }
 
 func (x *TypeInfos) structTag(t reflect.StructTag) (s string) {
@@ -802,20 +827,46 @@ func (x *TypeInfos) structTag(t reflect.StructTag) (s string) {
 	return
 }
 
+func (x *TypeInfos) find(sp *[]rtid2ti, rtid uintptr) (idx int, ti *typeInfo) {
+	// binary search. adapted from sort/search.go.
+	// fmt.Printf(">>>> calling typeinfos.find ... \n")
+	// if sp == nil {
+	// 	return -1, nil
+	// }
+	s := *sp
+	h, i, j := 0, 0, len(s)
+	for i < j {
+		h = i + (j-i)/2
+		if s[h].rtid < rtid {
+			i = h + 1
+		} else {
+			j = h
+		}
+	}
+	if i < len(s) && s[i].rtid == rtid {
+		return i, s[i].ti
+	}
+	return i, nil
+}
+
 func (x *TypeInfos) get(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
-	var ok bool
-	x.mu.RLock()
-	pti, ok = x.infos[rtid]
-	x.mu.RUnlock()
-	if ok {
-		return
+	// fmt.Printf(">>>> calling typeinfos.get ... \n")
+	sp := x.infos.load()
+	var idx int
+	if sp != nil {
+		idx, pti = x.find(sp, rtid)
+		if pti != nil {
+			return
+		}
 	}
 
 	// do not hold lock while computing this.
 	// it may lead to duplication, but that's ok.
 	ti := typeInfo{rt: rt, rtid: rtid}
-	ti.numMeth = uint16(rt.NumMethod())
+	// ti.rv0 = reflect.Zero(rt)
 
+	ti.numMeth = uint16(rt.NumMethod())
+	var ok bool
 	var indir int8
 	if ok, indir = implementsIntf(rt, binaryMarshalerTyp); ok {
 		ti.bm, ti.bmIndir = true, indir
@@ -854,7 +905,7 @@ func (x *TypeInfos) get(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
 		ti.baseId = rtid
 	} else {
 		ti.base = pt
-		ti.baseId = reflect.ValueOf(pt).Pointer()
+		ti.baseId = rt2id(pt)
 		ti.baseIndir = ptIndir
 	}
 
@@ -875,12 +926,28 @@ func (x *TypeInfos) get(rtid uintptr, rt reflect.Type) (pti *typeInfo) {
 	}
 	// sfi = sfip
 
+	var vs []rtid2ti
 	x.mu.Lock()
-	if pti, ok = x.infos[rtid]; !ok {
+	sp = x.infos.load()
+	if sp == nil {
+		// fmt.Printf(">>>> in typeinfos.get: sp == nil\n")
 		pti = &ti
-		x.infos[rtid] = pti
+		vs = []rtid2ti{{rtid, pti}}
+		x.infos.store(&vs)
+	} else {
+		idx, pti = x.find(sp, rtid)
+		if pti == nil {
+			s := *sp
+			pti = &ti
+			vs = make([]rtid2ti, len(s)+1)
+			copy(vs, s[:idx])
+			vs[idx] = rtid2ti{rtid, pti}
+			copy(vs[idx+1:], s[idx:])
+			x.infos.store(&vs)
+		}
 	}
 	x.mu.Unlock()
+	// fmt.Printf(">>>>>>> TypeInfos: Num Elements: %v\n", len(*(x.infos.load())))
 	return
 }
 
@@ -932,7 +999,7 @@ LOOP:
 				}
 				if ft.Kind() == reflect.Struct {
 					// if etypes contains this, don't call rget again (as fields are already seen here)
-					ftid := reflect.ValueOf(ft).Pointer()
+					ftid := rt2id(ft)
 					// We cannot recurse forever, but we need to track other field depths.
 					// So - we break if we see a type twice (not the first time).
 					// This should be sufficient to handle an embedded type that refers to its
@@ -1065,22 +1132,23 @@ func panicToErr(err *error) {
 // }
 
 func isImmutableKind(k reflect.Kind) (v bool) {
-	return false ||
-		k == reflect.Int ||
-		k == reflect.Int8 ||
-		k == reflect.Int16 ||
-		k == reflect.Int32 ||
-		k == reflect.Int64 ||
-		k == reflect.Uint ||
-		k == reflect.Uint8 ||
-		k == reflect.Uint16 ||
-		k == reflect.Uint32 ||
-		k == reflect.Uint64 ||
-		k == reflect.Uintptr ||
-		k == reflect.Float32 ||
-		k == reflect.Float64 ||
-		k == reflect.Bool ||
-		k == reflect.String
+	return immutableKindsSet[k]
+	// return false ||
+	// 	k == reflect.Int ||
+	// 	k == reflect.Int8 ||
+	// 	k == reflect.Int16 ||
+	// 	k == reflect.Int32 ||
+	// 	k == reflect.Int64 ||
+	// 	k == reflect.Uint ||
+	// 	k == reflect.Uint8 ||
+	// 	k == reflect.Uint16 ||
+	// 	k == reflect.Uint32 ||
+	// 	k == reflect.Uint64 ||
+	// 	k == reflect.Uintptr ||
+	// 	k == reflect.Float32 ||
+	// 	k == reflect.Float64 ||
+	// 	k == reflect.Bool ||
+	// 	k == reflect.String
 }
 
 // these functions must be inlinable, and not call anybody

codec/helper_internal.go

@@ -219,24 +219,3 @@ func growCap(oldCap, unit, num int) (newCap int) {
 	}
 	return
 }
-
-func expandSliceValue(s reflect.Value, num int) reflect.Value {
-	if num <= 0 {
-		return s
-	}
-	l0 := s.Len()
-	l1 := l0 + num // new slice length
-	if l1 < l0 {
-		panic("ExpandSlice: slice overflow")
-	}
-	c0 := s.Cap()
-	if l1 <= c0 {
-		return s.Slice(0, l1)
-	}
-	st := s.Type()
-	c1 := growCap(c0, int(st.Elem().Size()), num)
-	s2 := reflect.MakeSlice(st, l1, c1)
-	// println("expandslicevalue: cap-old: ", c0, ", cap-new: ", c1, ", len-new: ", l1)
-	reflect.Copy(s2, s)
-	return s2
-}

codec/helper_not_unsafe.go

@@ -1,10 +1,15 @@
-// +build !unsafe
+// +build !go1.7 safe appengine
 
 // Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
 // Use of this source code is governed by a MIT license found in the LICENSE file.
 
 package codec
 
+import (
+	"reflect"
+	"sync/atomic"
+)
+
 // stringView returns a view of the []byte as a string.
 // In unsafe mode, it doesn't incur allocation and copying caused by conversion.
 // In regular safe mode, it is an allocation and copy.
@@ -25,12 +30,114 @@ func bytesView(v string) []byte {
 	return []byte(v)
 }
 
-// keepAlive4BytesView maintains a reference to the input parameter for bytesView.
-//
-// Usage: call this at point where done with the bytes view.
-func keepAlive4BytesView(v string) {}
+// // keepAlive4BytesView maintains a reference to the input parameter for bytesView.
+// //
+// // Usage: call this at point where done with the bytes view.
+// func keepAlive4BytesView(v string) {}
 
-// keepAlive4BytesView maintains a reference to the input parameter for stringView.
-//
-// Usage: call this at point where done with the string view.
-func keepAlive4StringView(v []byte) {}
+// // keepAlive4BytesView maintains a reference to the input parameter for stringView.
+// //
+// // Usage: call this at point where done with the string view.
+// func keepAlive4StringView(v []byte) {}
+
+func rv2i(rv reflect.Value) interface{} {
+	return rv.Interface()
+}
+
+func rt2id(rt reflect.Type) uintptr {
+	return reflect.ValueOf(rt).Pointer()
+}
+
+// --------------------------
+type ptrToRvMap struct{}
+
+func (_ *ptrToRvMap) init() {}
+func (_ *ptrToRvMap) get(i interface{}) reflect.Value {
+	return reflect.ValueOf(i).Elem()
+}
+
+// --------------------------
+type atomicTypeInfoSlice struct {
+	v atomic.Value
+}
+
+func (x *atomicTypeInfoSlice) load() *[]rtid2ti {
+	i := x.v.Load()
+	if i == nil {
+		return nil
+	}
+	return i.(*[]rtid2ti)
+}
+
+func (x *atomicTypeInfoSlice) store(p *[]rtid2ti) {
+	x.v.Store(p)
+}
+
+// --------------------------
+func (f *decFnInfo) raw(rv reflect.Value) {
+	rv.SetBytes(f.d.raw())
+}
+
+func (f *decFnInfo) kString(rv reflect.Value) {
+	rv.SetString(f.d.d.DecodeString())
+}
+
+func (f *decFnInfo) kBool(rv reflect.Value) {
+	rv.SetBool(f.d.d.DecodeBool())
+}
+
+func (f *decFnInfo) kFloat32(rv reflect.Value) {
+	rv.SetFloat(f.d.d.DecodeFloat(true))
+}
+
+func (f *decFnInfo) kFloat64(rv reflect.Value) {
+	rv.SetFloat(f.d.d.DecodeFloat(false))
+}
+
+func (f *decFnInfo) kInt(rv reflect.Value) {
+	rv.SetInt(f.d.d.DecodeInt(intBitsize))
+}
+
+func (f *decFnInfo) kInt8(rv reflect.Value) {
+	rv.SetInt(f.d.d.DecodeInt(8))
+}
+
+func (f *decFnInfo) kInt16(rv reflect.Value) {
+	rv.SetInt(f.d.d.DecodeInt(16))
+}
+
+func (f *decFnInfo) kInt32(rv reflect.Value) {
+	rv.SetInt(f.d.d.DecodeInt(32))
+}
+
+func (f *decFnInfo) kInt64(rv reflect.Value) {
+	rv.SetInt(f.d.d.DecodeInt(64))
+}
+
+func (f *decFnInfo) kUint(rv reflect.Value) {
+	rv.SetUint(f.d.d.DecodeUint(uintBitsize))
+}
+
+func (f *decFnInfo) kUintptr(rv reflect.Value) {
+	rv.SetUint(f.d.d.DecodeUint(uintBitsize))
+}
+
+func (f *decFnInfo) kUint8(rv reflect.Value) {
+	rv.SetUint(f.d.d.DecodeUint(8))
+}
+
+func (f *decFnInfo) kUint16(rv reflect.Value) {
+	rv.SetUint(f.d.d.DecodeUint(16))
+}
+
+func (f *decFnInfo) kUint32(rv reflect.Value) {
+	rv.SetUint(f.d.d.DecodeUint(32))
+}
+
+func (f *decFnInfo) kUint64(rv reflect.Value) {
+	rv.SetUint(f.d.d.DecodeUint(64))
+}
+
+// func i2rv(i interface{}) reflect.Value {
+// 	return reflect.ValueOf(i)
+// }

codec/helper_unsafe.go

@@ -1,4 +1,6 @@
-// +build unsafe
+// +build !safe
+// +build !appengine
+// +build go1.7
 
 // Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
 // Use of this source code is governed by a MIT license found in the LICENSE file.
@@ -6,13 +8,16 @@
 package codec
 
 import (
-	"runtime"
+	"reflect"
+	"sync/atomic"
 	"unsafe"
 )
 
 // This file has unsafe variants of some helper methods.
 // NOTE: See helper_not_unsafe.go for the usage information.
 
+// var zeroRTv [4]uintptr
+
 type unsafeString struct {
 	Data uintptr
 	Len  int
@@ -24,6 +29,17 @@ type unsafeSlice struct {
 	Cap  int
 }
 
+type unsafeIntf struct {
+	typ  unsafe.Pointer
+	word unsafe.Pointer
+}
+
+type unsafeReflectValue struct {
+	typ  unsafe.Pointer
+	ptr  unsafe.Pointer
+	flag uintptr
+}
+
 func stringView(v []byte) string {
 	if len(v) == 0 {
 		return ""
@@ -44,10 +60,403 @@ func bytesView(v string) []byte {
 	return *(*[]byte)(unsafe.Pointer(&bx))
 }
 
-func keepAlive4BytesView(v string) {
-	runtime.KeepAlive(v)
+// func keepAlive4BytesView(v string) {
+// 	runtime.KeepAlive(v)
+// }
+
+// func keepAlive4StringView(v []byte) {
+// 	runtime.KeepAlive(v)
+// }
+
+const _unsafe_rv2i_is_safe = false
+
+// TODO: consider a more generally-known optimization for reflect.Value ==> Interface
+//
+// Currently, we use this fragile method that taps into implememtation details from
+// the source go stdlib reflect/value.go,
+// and trims the implementation.
+func rv2i(rv reflect.Value) interface{} {
+	if _unsafe_rv2i_is_safe {
+		return rv.Interface()
+	}
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	// references that are single-words (map, ptr) may be double-referenced as flagIndir
+	kk := urv.flag & (1<<5 - 1)
+	if (kk == uintptr(reflect.Map) || kk == uintptr(reflect.Ptr)) && urv.flag&(1<<7) != 0 {
+		return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: *(*unsafe.Pointer)(urv.ptr), typ: urv.typ}))
+	}
+	return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: urv.ptr, typ: urv.typ}))
+}
+
+func rt2id(rt reflect.Type) uintptr {
+	return uintptr(((*unsafeIntf)(unsafe.Pointer(&rt))).word)
+}
+
+// func rv0t(rt reflect.Type) reflect.Value {
+// 	ut := (*unsafeIntf)(unsafe.Pointer(&rt))
+// 	// we need to determine whether ifaceIndir, and then whether to just pass 0 as the ptr
+// 	uv := unsafeReflectValue{ut.word, &zeroRTv, flag(rt.Kind())}
+// 	return *(*reflect.Value)(unsafe.Pointer(&uv})
+// }
+
+type ptrToRVKV struct {
+	k uintptr
+	v reflect.Value
+}
+
+type ptrToRvMap struct {
+	// m map[uintptr]reflect.Value
+	a [4]ptrToRVKV
+	v []ptrToRVKV
+}
+
+func (p *ptrToRvMap) init() {
+	// fmt.Printf(">>>> new ptr to rv map\n")
+	// p.m = make(map[uintptr]reflect.Value, 32)
+	p.v = p.a[:0]
+}
+
+func (p *ptrToRvMap) get(intf interface{}) (rv reflect.Value) {
+	word := uintptr(((*unsafeIntf)(unsafe.Pointer(&intf))).word)
+	// binary search. adapted from sort/search.go.
+	h, i, j := 0, 0, len(p.v)
+	for i < j {
+		h = i + (j-i)/2
+		if p.v[h].k < word {
+			i = h + 1
+		} else {
+			j = h
+		}
+	}
+	if i < len(p.v) && p.v[i].k == word {
+		return p.v[i].v
+	}
+
+	// insert into position i
+	// fmt.Printf(">>>> resetting rv for word: %x, interface: %v\n", word, intf)
+	rv = reflect.ValueOf(intf).Elem()
+	p.v = append(p.v, ptrToRVKV{})
+	copy(p.v[i+1:len(p.v)], p.v[i:len(p.v)-1])
+	p.v[i].k, p.v[i].v = word, rv
+	return
+}
+
+// --------------------------
+type atomicTypeInfoSlice struct {
+	v unsafe.Pointer
+}
+
+func (x *atomicTypeInfoSlice) load() *[]rtid2ti {
+	return (*[]rtid2ti)(atomic.LoadPointer(&x.v))
+}
+
+func (x *atomicTypeInfoSlice) store(p *[]rtid2ti) {
+	atomic.StorePointer(&x.v, unsafe.Pointer(p))
+}
+
+// --------------------------
+func (f *decFnInfo) raw(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*[]byte)(urv.ptr) = f.d.raw()
+}
+
+func (f *decFnInfo) kString(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*string)(urv.ptr) = f.d.d.DecodeString()
+}
+
+func (f *decFnInfo) kBool(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*bool)(urv.ptr) = f.d.d.DecodeBool()
+}
+
+func (f *decFnInfo) kFloat32(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*float32)(urv.ptr) = float32(f.d.d.DecodeFloat(true))
 }
 
-func keepAlive4StringView(v []byte) {
-	runtime.KeepAlive(v)
+func (f *decFnInfo) kFloat64(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*float64)(urv.ptr) = f.d.d.DecodeFloat(false)
 }
+
+func (f *decFnInfo) kInt(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*int)(urv.ptr) = int(f.d.d.DecodeInt(intBitsize))
+}
+
+func (f *decFnInfo) kInt8(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*int8)(urv.ptr) = int8(f.d.d.DecodeInt(8))
+}
+
+func (f *decFnInfo) kInt16(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*int16)(urv.ptr) = int16(f.d.d.DecodeInt(16))
+}
+
+func (f *decFnInfo) kInt32(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*int32)(urv.ptr) = int32(f.d.d.DecodeInt(32))
+}
+
+func (f *decFnInfo) kInt64(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*int64)(urv.ptr) = f.d.d.DecodeInt(64)
+}
+
+func (f *decFnInfo) kUint(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*uint)(urv.ptr) = uint(f.d.d.DecodeUint(uintBitsize))
+}
+
+func (f *decFnInfo) kUintptr(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*uintptr)(urv.ptr) = uintptr(f.d.d.DecodeUint(uintBitsize))
+}
+
+func (f *decFnInfo) kUint8(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*uint8)(urv.ptr) = uint8(f.d.d.DecodeUint(8))
+}
+
+func (f *decFnInfo) kUint16(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*uint16)(urv.ptr) = uint16(f.d.d.DecodeUint(16))
+}
+
+func (f *decFnInfo) kUint32(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*uint32)(urv.ptr) = uint32(f.d.d.DecodeUint(32))
+}
+func (f *decFnInfo) kUint64(rv reflect.Value) {
+	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+	*(*uint64)(urv.ptr) = f.d.d.DecodeUint(64)
+}
+
+// func (p *ptrToRvMap) get(i interface{}) (rv reflect.Value) {
+// 	word := uintptr(((*unsafeIntf)(unsafe.Pointer(&i))).word)
+// 	rv, exists := p.m[word]
+// 	if !exists {
+// 		fmt.Printf(">>>> resetting rv for word: %x, interface: %v\n", word, i)
+// 		rv = reflect.ValueOf(i).Elem()
+// 		p.m[word] = rv
+// 	}
+// 	return
+// }
+
+// func rt2id(rt reflect.Type) uintptr {
+// 	return uintptr(((*unsafeIntf)(unsafe.Pointer(&rt))).word)
+// 	// var i interface{} = rt
+// 	// // ui := (*unsafeIntf)(unsafe.Pointer(&i))
+// 	// return ((*unsafeIntf)(unsafe.Pointer(&i))).word
+// }
+
+// func rv2i(rv reflect.Value) interface{} {
+// 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+// 	// non-reference type: already indir
+// 	// reference type: depend on flagIndir property ('cos maybe was double-referenced)
+// 	// const (unsafeRvFlagKindMask    = 1<<5 - 1 , unsafeRvFlagIndir       = 1 << 7 )
+// 	// rvk := reflect.Kind(urv.flag & (1<<5 - 1))
+// 	// if (rvk == reflect.Chan ||
+// 	// 	rvk == reflect.Func ||
+// 	// 	rvk == reflect.Interface ||
+// 	// 	rvk == reflect.Map ||
+// 	// 	rvk == reflect.Ptr ||
+// 	// 	rvk == reflect.UnsafePointer) && urv.flag&(1<<8) != 0 {
+// 	// 	fmt.Printf(">>>>> ---- double indirect reference: %v, %v\n", rvk, rv.Type())
+// 	// 	return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: *(*unsafe.Pointer)(urv.ptr), typ: urv.typ}))
+// 	// }
+// 	if urv.flag&(1<<5-1) == uintptr(reflect.Map) && urv.flag&(1<<7) != 0 {
+// 		// fmt.Printf(">>>>> ---- double indirect reference: %v, %v\n", rvk, rv.Type())
+// 		return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: *(*unsafe.Pointer)(urv.ptr), typ: urv.typ}))
+// 	}
+// 	// fmt.Printf(">>>>> ++++ direct reference: %v, %v\n", rvk, rv.Type())
+// 	return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: urv.ptr, typ: urv.typ}))
+// }
+
+// const (
+// 	unsafeRvFlagKindMask    = 1<<5 - 1
+// 	unsafeRvKindDirectIface = 1 << 5
+// 	unsafeRvFlagIndir       = 1 << 7
+// 	unsafeRvFlagAddr        = 1 << 8
+// 	unsafeRvFlagMethod      = 1 << 9
+
+// 	_USE_RV_INTERFACE bool = false
+// 	_UNSAFE_RV_DEBUG       = true
+// )
+
+// type unsafeRtype struct {
+// 	_    [2]uintptr
+// 	_    uint32
+// 	_    uint8
+// 	_    uint8
+// 	_    uint8
+// 	kind uint8
+// 	_    [2]uintptr
+// 	_    int32
+// }
+
+// func _rv2i(rv reflect.Value) interface{} {
+// 	// Note: From use,
+// 	//   - it's never an interface
+// 	//   - the only calls here are for ifaceIndir types.
+// 	//     (though that conditional is wrong)
+// 	//     To know for sure, we need the value of t.kind (which is not exposed).
+// 	//
+// 	// Need to validate the path: type is indirect ==> only value is indirect ==> default (value is direct)
+// 	//    - Type indirect, Value indirect: ==> numbers, boolean, slice, struct, array, string
+// 	//    - Type Direct,   Value indirect: ==> map???
+// 	//    - Type Direct,   Value direct:   ==> pointers, unsafe.Pointer, func, chan, map
+// 	//
+// 	// TRANSLATES TO:
+// 	//    if typeIndirect { } else if valueIndirect { } else { }
+// 	//
+// 	// Since we don't deal with funcs, then "flagNethod" is unset, and can be ignored.
+
+// 	if _USE_RV_INTERFACE {
+// 		return rv.Interface()
+// 	}
+// 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+
+// 	// if urv.flag&unsafeRvFlagMethod != 0 || urv.flag&unsafeRvFlagKindMask == uintptr(reflect.Interface) {
+// 	// 	println("***** IS flag method or interface: delegating to rv.Interface()")
+// 	// 	return rv.Interface()
+// 	// }
+
+// 	// if urv.flag&unsafeRvFlagKindMask == uintptr(reflect.Interface) {
+// 	// 	println("***** IS Interface: delegate to rv.Interface")
+// 	// 	return rv.Interface()
+// 	// }
+// 	// if urv.flag&unsafeRvFlagKindMask&unsafeRvKindDirectIface == 0 {
+// 	// 	if urv.flag&unsafeRvFlagAddr == 0 {
+// 	// 		println("***** IS ifaceIndir typ")
+// 	// 		// ui := unsafeIntf{word: urv.ptr, typ: urv.typ}
+// 	// 		// return *(*interface{})(unsafe.Pointer(&ui))
+// 	// 		// return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: urv.ptr, typ: urv.typ}))
+// 	// 	}
+// 	// } else if urv.flag&unsafeRvFlagIndir != 0 {
+// 	// 	println("***** IS flagindir")
+// 	// 	// return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: *(*unsafe.Pointer)(urv.ptr), typ: urv.typ}))
+// 	// } else {
+// 	// 	println("***** NOT flagindir")
+// 	// 	return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: urv.ptr, typ: urv.typ}))
+// 	// }
+// 	// println("***** default: delegate to rv.Interface")
+
+// 	urt := (*unsafeRtype)(unsafe.Pointer(urv.typ))
+// 	if _UNSAFE_RV_DEBUG {
+// 		fmt.Printf(">>>> start: %v: ", rv.Type())
+// 		fmt.Printf("%v - %v\n", *urv, *urt)
+// 	}
+// 	if urt.kind&unsafeRvKindDirectIface == 0 {
+// 		if _UNSAFE_RV_DEBUG {
+// 			fmt.Printf("**** +ifaceIndir type: %v\n", rv.Type())
+// 		}
+// 		// println("***** IS ifaceIndir typ")
+// 		// if true || urv.flag&unsafeRvFlagAddr == 0 {
+// 		// 	// println("    ***** IS NOT addr")
+// 		return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: urv.ptr, typ: urv.typ}))
+// 		// }
+// 	} else if urv.flag&unsafeRvFlagIndir != 0 {
+// 		if _UNSAFE_RV_DEBUG {
+// 			fmt.Printf("**** +flagIndir type: %v\n", rv.Type())
+// 		}
+// 		// println("***** IS flagindir")
+// 		return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: *(*unsafe.Pointer)(urv.ptr), typ: urv.typ}))
+// 	} else {
+// 		if _UNSAFE_RV_DEBUG {
+// 			fmt.Printf("**** -flagIndir type: %v\n", rv.Type())
+// 		}
+// 		// println("***** NOT flagindir")
+// 		return *(*interface{})(unsafe.Pointer(&unsafeIntf{word: urv.ptr, typ: urv.typ}))
+// 	}
+// 	// println("***** default: delegating to rv.Interface()")
+// 	// return rv.Interface()
+// }
+
+// var staticM0 = make(map[string]uint64)
+// var staticI0 = (int32)(-5)
+
+// func staticRv2iTest() {
+// 	i0 := (int32)(-5)
+// 	m0 := make(map[string]uint16)
+// 	m0["1"] = 1
+// 	for _, i := range []interface{}{
+// 		(int)(7),
+// 		(uint)(8),
+// 		(int16)(-9),
+// 		(uint16)(19),
+// 		(uintptr)(77),
+// 		(bool)(true),
+// 		float32(-32.7),
+// 		float64(64.9),
+// 		complex(float32(19), 5),
+// 		complex(float64(-32), 7),
+// 		[4]uint64{1, 2, 3, 4},
+// 		(chan<- int)(nil), // chan,
+// 		rv2i,              // func
+// 		io.Writer(ioutil.Discard),
+// 		make(map[string]uint),
+// 		(map[string]uint)(nil),
+// 		staticM0,
+// 		m0,
+// 		&m0,
+// 		i0,
+// 		&i0,
+// 		&staticI0,
+// 		&staticM0,
+// 		[]uint32{6, 7, 8},
+// 		"abc",
+// 		Raw{},
+// 		RawExt{},
+// 		&Raw{},
+// 		&RawExt{},
+// 		unsafe.Pointer(&i0),
+// 	} {
+// 		i2 := rv2i(reflect.ValueOf(i))
+// 		eq := reflect.DeepEqual(i, i2)
+// 		fmt.Printf(">>>> %v == %v? %v\n", i, i2, eq)
+// 	}
+// 	// os.Exit(0)
+// }
+
+// func init() {
+// 	staticRv2iTest()
+// }
+
+// func rv2i(rv reflect.Value) interface{} {
+// 	if _USE_RV_INTERFACE || rv.Kind() == reflect.Interface || rv.CanAddr() {
+// 		return rv.Interface()
+// 	}
+// 	// var i interface{}
+// 	// ui := (*unsafeIntf)(unsafe.Pointer(&i))
+// 	var ui unsafeIntf
+// 	urv := (*unsafeReflectValue)(unsafe.Pointer(&rv))
+// 	// fmt.Printf("urv: flag: %b, typ: %b, ptr: %b\n", urv.flag, uintptr(urv.typ), uintptr(urv.ptr))
+// 	if (urv.flag&unsafeRvFlagKindMask)&unsafeRvKindDirectIface == 0 {
+// 		if urv.flag&unsafeRvFlagAddr != 0 {
+// 			println("***** indirect and addressable! Needs typed move - delegate to rv.Interface()")
+// 			return rv.Interface()
+// 		}
+// 		println("****** indirect type/kind")
+// 		ui.word = urv.ptr
+// 	} else if urv.flag&unsafeRvFlagIndir != 0 {
+// 		println("****** unsafe rv flag indir")
+// 		ui.word = *(*unsafe.Pointer)(urv.ptr)
+// 	} else {
+// 		println("****** default: assign prt to word directly")
+// 		ui.word = urv.ptr
+// 	}
+// 	// ui.word = urv.ptr
+// 	ui.typ = urv.typ
+// 	// fmt.Printf("(pointers) ui.typ: %p, word: %p\n", ui.typ, ui.word)
+// 	// fmt.Printf("(binary)   ui.typ: %b, word: %b\n", uintptr(ui.typ), uintptr(ui.word))
+// 	return *(*interface{})(unsafe.Pointer(&ui))
+// 	// return i
+// }
+
+// func i2rv(i interface{}) reflect.Value {
+// 	// u := *(*unsafeIntf)(unsafe.Pointer(&i))
+// 	return reflect.ValueOf(i)
+// }

codec/json.go

@@ -34,7 +34,6 @@ package codec
 import (
 	"bytes"
 	"encoding/base64"
-	"fmt"
 	"reflect"
 	"strconv"
 	"unicode/utf16"
@@ -59,6 +58,11 @@ var (
 
 	// jsonTabs and jsonSpaces are used as caches for indents
 	jsonTabs, jsonSpaces string
+
+	jsonCharHtmlSafeSet   [utf8.RuneSelf]bool
+	jsonCharSafeSet       [utf8.RuneSelf]bool
+	jsonCharWhitespaceSet [256]bool
+	jsonNumSet            [256]bool
 )
 
 const (
@@ -78,19 +82,6 @@ const (
 	// P.S. Do not expect a significant decoding boost from this.
 	jsonValidateSymbols = true
 
-	// if jsonTruncateMantissa, truncate mantissa if trailing 0's.
-	// This is important because it could allow some floats to be decoded without
-	// deferring to strconv.ParseFloat.
-	jsonTruncateMantissa = true
-
-	// if mantissa >= jsonNumUintCutoff before multiplying by 10, this is an overflow
-	jsonNumUintCutoff = (1<<64-1)/uint64(10) + 1 // cutoff64(base)
-
-	// if mantissa >= jsonNumUintMaxVal, this is an overflow
-	jsonNumUintMaxVal = 1<<uint64(64) - 1
-
-	// jsonNumDigitsUint64Largest = 19
-
 	jsonSpacesOrTabsLen = 128
 )
 
@@ -105,6 +96,31 @@ func init() {
 		bs[i] = '\t'
 	}
 	jsonTabs = string(bs[:])
+
+	// populate the safe values as true: note: ASCII control characters are (0-31)
+	// jsonCharSafeSet:     all true except (0-31) " \
+	// jsonCharHtmlSafeSet: all true except (0-31) " \ < > &
+	for i := 32; i < utf8.RuneSelf; i++ {
+		switch i {
+		case '"', '\\':
+			jsonCharSafeSet[i] = false
+			jsonCharHtmlSafeSet[i] = false
+		case '<', '>', '&':
+			jsonCharHtmlSafeSet[i] = false
+			jsonCharSafeSet[i] = true
+		default:
+			jsonCharSafeSet[i] = true
+			jsonCharHtmlSafeSet[i] = true
+		}
+	}
+	for i := 0; i < 256; i++ {
+		switch i {
+		case ' ', '\t', '\r', '\n':
+			jsonCharWhitespaceSet[i] = true
+		case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'e', 'E', '.', '+', '-':
+			jsonNumSet[i] = true
+		}
+	}
 }
 
 type jsonEncDriver struct {
@@ -214,6 +230,7 @@ func (e *jsonEncDriver) encodeFloat(f float64, numbits int) {
 }
 
 func (e *jsonEncDriver) EncodeInt(v int64) {
+	// if e.h.IntegerAsString == 'A' || e.h.IntegerAsString == 'L' && (v > 1<<53 || v < -(1<<53)) {
 	if x := e.h.IntegerAsString; x == 'A' || x == 'L' && (v > 1<<53 || v < -(1<<53)) {
 		e.w.writen1('"')
 		e.w.writeb(strconv.AppendInt(e.b[:0], v, 10))
@@ -224,6 +241,7 @@ func (e *jsonEncDriver) EncodeInt(v int64) {
 }
 
 func (e *jsonEncDriver) EncodeUint(v uint64) {
+	// if e.h.IntegerAsString == 'A' || e.h.IntegerAsString == 'L' && v > 1<<53 {
 	if x := e.h.IntegerAsString; x == 'A' || x == 'L' && v > 1<<53 {
 		e.w.writen1('"')
 		e.w.writeb(strconv.AppendUint(e.b[:0], v, 10))
@@ -313,7 +331,8 @@ func (e *jsonEncDriver) quoteStr(s string) {
 		// encode all bytes < 0x20 (except \r, \n).
 		// also encode < > & to prevent security holes when served to some browsers.
 		if b := s[i]; b < utf8.RuneSelf {
-			if 0x20 <= b && b != '\\' && b != '"' && b != '<' && b != '>' && b != '&' {
+			// if 0x20 <= b && b != '\\' && b != '"' && b != '<' && b != '>' && b != '&' {
+			if jsonCharHtmlSafeSet[b] || (e.h.HTMLCharsAsIs && jsonCharSafeSet[b]) {
 				i++
 				continue
 			}
@@ -333,13 +352,6 @@ func (e *jsonEncDriver) quoteStr(s string) {
 				w.writen2('\\', 'f')
 			case '\t':
 				w.writen2('\\', 't')
-			case '<', '>', '&':
-				if e.h.HTMLCharsAsIs {
-					w.writen1(b)
-				} else {
-					w.writestr(`\u00`)
-					w.writen2(hex[b>>4], hex[b&0xF])
-				}
 			default:
 				w.writestr(`\u00`)
 				w.writen2(hex[b>>4], hex[b&0xF])
@@ -378,88 +390,6 @@ func (e *jsonEncDriver) quoteStr(s string) {
 	w.writen1('"')
 }
 
-//--------------------------------
-
-type jsonNum struct {
-	// bytes            []byte // may have [+-.eE0-9]
-	mantissa         uint64 // where mantissa ends, and maybe dot begins.
-	exponent         int16  // exponent value.
-	manOverflow      bool
-	neg              bool // started with -. No initial sign in the bytes above.
-	dot              bool // has dot
-	explicitExponent bool // explicit exponent
-}
-
-func (x *jsonNum) reset() {
-	x.manOverflow = false
-	x.neg = false
-	x.dot = false
-	x.explicitExponent = false
-	x.mantissa = 0
-	x.exponent = 0
-}
-
-// uintExp is called only if exponent > 0.
-func (x *jsonNum) uintExp() (n uint64, overflow bool) {
-	n = x.mantissa
-	e := x.exponent
-	if e >= int16(len(jsonUint64Pow10)) {
-		overflow = true
-		return
-	}
-	n *= jsonUint64Pow10[e]
-	if n < x.mantissa || n > jsonNumUintMaxVal {
-		overflow = true
-		return
-	}
-	return
-	// for i := int16(0); i < e; i++ {
-	// 	if n >= jsonNumUintCutoff {
-	// 		overflow = true
-	// 		return
-	// 	}
-	// 	n *= 10
-	// }
-	// return
-}
-
-// these constants are only used withn floatVal.
-// They are brought out, so that floatVal can be inlined.
-const (
-	jsonUint64MantissaBits = 52
-	jsonMaxExponent        = int16(len(jsonFloat64Pow10)) - 1
-)
-
-func (x *jsonNum) floatVal() (f float64, parseUsingStrConv bool) {
-	// We do not want to lose precision.
-	// Consequently, we will delegate to strconv.ParseFloat if any of the following happen:
-	//    - There are more digits than in math.MaxUint64: 18446744073709551615 (20 digits)
-	//      We expect up to 99.... (19 digits)
-	//    - The mantissa cannot fit into a 52 bits of uint64
-	//    - The exponent is beyond our scope ie beyong 22.
-	parseUsingStrConv = x.manOverflow ||
-		x.exponent > jsonMaxExponent ||
-		(x.exponent < 0 && -(x.exponent) > jsonMaxExponent) ||
-		x.mantissa>>jsonUint64MantissaBits != 0
-
-	if parseUsingStrConv {
-		return
-	}
-
-	// all good. so handle parse here.
-	f = float64(x.mantissa)
-	// fmt.Printf(".Float: uint64 value: %v, float: %v\n", m, f)
-	if x.neg {
-		f = -f
-	}
-	if x.exponent > 0 {
-		f *= jsonFloat64Pow10[x.exponent]
-	} else if x.exponent < 0 {
-		f /= jsonFloat64Pow10[-x.exponent]
-	}
-	return
-}
-
 type jsonDecDriver struct {
 	noBuiltInTypes
 	d *Decoder
@@ -477,26 +407,14 @@ type jsonDecDriver struct {
 
 	se setExtWrapper
 
-	n jsonNum
+	// n jsonNum
 }
 
 func jsonIsWS(b byte) bool {
-	return b == ' ' || b == '\t' || b == '\r' || b == '\n'
+	// return b == ' ' || b == '\t' || b == '\r' || b == '\n'
+	return jsonCharWhitespaceSet[b]
 }
 
-// // This will skip whitespace characters and return the next byte to read.
-// // The next byte determines what the value will be one of.
-// func (d *jsonDecDriver) skipWhitespace() {
-// 	// fast-path: do not enter loop. Just check first (in case no whitespace).
-// 	b := d.r.readn1()
-// 	if jsonIsWS(b) {
-// 		r := d.r
-// 		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
-// 		}
-// 	}
-// 	d.tok = b
-// }
-
 func (d *jsonDecDriver) uncacheRead() {
 	if d.tok != 0 {
 		d.r.unreadn1()
@@ -506,11 +424,7 @@ func (d *jsonDecDriver) uncacheRead() {
 
 func (d *jsonDecDriver) sendContainerState(c containerState) {
 	if d.tok == 0 {
-		var b byte
-		r := d.r
-		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
-		}
-		d.tok = b
+		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 	var xc uint8 // char expected
 	if c == containerMapKey {
@@ -539,37 +453,23 @@ func (d *jsonDecDriver) sendContainerState(c containerState) {
 
 func (d *jsonDecDriver) CheckBreak() bool {
 	if d.tok == 0 {
-		var b byte
-		r := d.r
-		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
-		}
-		d.tok = b
+		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
-	if d.tok == '}' || d.tok == ']' {
-		// d.tok = 0 // only checking, not consuming
-		return true
-	}
-	return false
+	return d.tok == '}' || d.tok == ']'
 }
 
 func (d *jsonDecDriver) readStrIdx(fromIdx, toIdx uint8) {
 	bs := d.r.readx(int(toIdx - fromIdx))
 	d.tok = 0
-	if jsonValidateSymbols {
-		if !bytes.Equal(bs, jsonLiterals[fromIdx:toIdx]) {
-			d.d.errorf("json: expecting %s: got %s", jsonLiterals[fromIdx:toIdx], bs)
-			return
-		}
+	if jsonValidateSymbols && !bytes.Equal(bs, jsonLiterals[fromIdx:toIdx]) {
+		d.d.errorf("json: expecting %s: got %s", jsonLiterals[fromIdx:toIdx], bs)
+		return
 	}
 }
 
 func (d *jsonDecDriver) TryDecodeAsNil() bool {
 	if d.tok == 0 {
-		var b byte
-		r := d.r
-		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
-		}
-		d.tok = b
+		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 	if d.tok == 'n' {
 		d.readStrIdx(10, 13) // ull
@@ -580,11 +480,7 @@ func (d *jsonDecDriver) TryDecodeAsNil() bool {
 
 func (d *jsonDecDriver) DecodeBool() bool {
 	if d.tok == 0 {
-		var b byte
-		r := d.r
-		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
-		}
-		d.tok = b
+		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 	if d.tok == 'f' {
 		d.readStrIdx(5, 9) // alse
@@ -600,11 +496,7 @@ func (d *jsonDecDriver) DecodeBool() bool {
 
 func (d *jsonDecDriver) ReadMapStart() int {
 	if d.tok == 0 {
-		var b byte
-		r := d.r
-		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
-		}
-		d.tok = b
+		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 	if d.tok != '{' {
 		d.d.errorf("json: expect char '%c' but got char '%c'", '{', d.tok)
@@ -616,11 +508,7 @@ func (d *jsonDecDriver) ReadMapStart() int {
 
 func (d *jsonDecDriver) ReadArrayStart() int {
 	if d.tok == 0 {
-		var b byte
-		r := d.r
-		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
-		}
-		d.tok = b
+		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 	if d.tok != '[' {
 		d.d.errorf("json: expect char '%c' but got char '%c'", '[', d.tok)
@@ -633,11 +521,7 @@ func (d *jsonDecDriver) ReadArrayStart() int {
 func (d *jsonDecDriver) ContainerType() (vt valueType) {
 	// check container type by checking the first char
 	if d.tok == 0 {
-		var b byte
-		r := d.r
-		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
-		}
-		d.tok = b
+		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 	if b := d.tok; b == '{' {
 		return valueTypeMap
@@ -653,264 +537,57 @@ func (d *jsonDecDriver) ContainerType() (vt valueType) {
 	// return false // "unreachable"
 }
 
-func (d *jsonDecDriver) decNum(storeBytes bool) {
-	// If it is has a . or an e|E, decode as a float; else decode as an int.
+func (d *jsonDecDriver) decNumBytes() (bs []byte) {
+	// stores num bytes in d.bs
 	if d.tok == 0 {
-		var b byte
-		r := d.r
-		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
-		}
-		d.tok = b
+		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
-	b := d.tok
-	var str bool
-	if b == '"' {
-		str = true
-		b = d.r.readn1()
-	}
-	if !(b == '+' || b == '-' || b == '.' || (b >= '0' && b <= '9')) {
-		d.d.errorf("json: decNum: got first char '%c'", b)
-		return
+	if d.tok == '"' {
+		bs = d.r.readUntil(d.b2[:0], '"')
+		bs = bs[:len(bs)-1]
+	} else {
+		d.r.unreadn1()
+		bs = d.r.readTo(d.bs[:0], &jsonNumSet)
+		// bs = d.r.readbUntilAny(d.bs[:0], " \t\n:,{}[]")
 	}
 	d.tok = 0
-
-	const cutoff = (1<<64-1)/uint64(10) + 1 // cutoff64(base)
-	const jsonNumUintMaxVal = 1<<uint64(64) - 1
-
-	n := &d.n
-	r := d.r
-	n.reset()
-	d.bs = d.bs[:0]
-
-	if str && storeBytes {
-		d.bs = append(d.bs, '"')
-	}
-
-	// The format of a number is as below:
-	// parsing:     sign? digit* dot? digit* e?  sign? digit*
-	// states:  0   1*    2      3*   4      5*  6     7
-	// We honor this state so we can break correctly.
-	var state uint8 = 0
-	var eNeg bool
-	var e int16
-	var eof bool
-LOOP:
-	for !eof {
-		// fmt.Printf("LOOP: b: %q\n", b)
-		switch b {
-		case '+':
-			switch state {
-			case 0:
-				state = 2
-				if storeBytes {
-					d.bs = append(d.bs, b)
-				}
-				b, eof = r.readn1eof()
-				continue
-			case 6: // typ = jsonNumFloat
-				state = 7
-			default:
-				break LOOP
-			}
-		case '-':
-			switch state {
-			case 0:
-				state = 2
-				n.neg = true
-				if storeBytes {
-					d.bs = append(d.bs, b)
-				}
-				b, eof = r.readn1eof()
-				continue
-			case 6: // typ = jsonNumFloat
-				eNeg = true
-				state = 7
-			default:
-				break LOOP
-			}
-		case '.':
-			switch state {
-			case 0, 2: // typ = jsonNumFloat
-				state = 4
-				n.dot = true
-			default:
-				break LOOP
-			}
-		case 'e', 'E':
-			switch state {
-			case 0, 2, 4: // typ = jsonNumFloat
-				state = 6
-				// n.mantissaEndIndex = int16(len(n.bytes))
-				n.explicitExponent = true
-			default:
-				break LOOP
-			}
-		case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9':
-			switch state {
-			case 0:
-				state = 2
-				fallthrough
-			case 2:
-				fallthrough
-			case 4:
-				if n.dot {
-					n.exponent--
-				}
-				if n.mantissa >= jsonNumUintCutoff {
-					n.manOverflow = true
-					break
-				}
-				v := uint64(b - '0')
-				n.mantissa *= 10
-				if v != 0 {
-					n1 := n.mantissa + v
-					if n1 < n.mantissa || n1 > jsonNumUintMaxVal {
-						n.manOverflow = true // n+v overflows
-						break
-					}
-					n.mantissa = n1
-				}
-			case 6:
-				state = 7
-				fallthrough
-			case 7:
-				if !(b == '0' && e == 0) {
-					e = e*10 + int16(b-'0')
-				}
-			default:
-				break LOOP
-			}
-		case '"':
-			if str {
-				if storeBytes {
-					d.bs = append(d.bs, '"')
-				}
-				b, eof = r.readn1eof()
-			}
-			break LOOP
-		default:
-			break LOOP
-		}
-		if storeBytes {
-			d.bs = append(d.bs, b)
-		}
-		b, eof = r.readn1eof()
-	}
-
-	if jsonTruncateMantissa && n.mantissa != 0 {
-		for n.mantissa%10 == 0 {
-			n.mantissa /= 10
-			n.exponent++
-		}
-	}
-
-	if e != 0 {
-		if eNeg {
-			n.exponent -= e
-		} else {
-			n.exponent += e
-		}
-	}
-
-	// d.n = n
-
-	if !eof {
-		if jsonUnreadAfterDecNum {
-			r.unreadn1()
-		} else {
-			if !jsonIsWS(b) {
-				d.tok = b
-			}
-		}
-	}
-	// fmt.Printf("1: n: bytes: %s, neg: %v, dot: %v, exponent: %v, mantissaEndIndex: %v\n",
-	// 	n.bytes, n.neg, n.dot, n.exponent, n.mantissaEndIndex)
-	return
+	// fmt.Printf(">>>> decNumBytes: returning: '%s'\n", bs)
+	return bs
 }
 
-func (d *jsonDecDriver) DecodeInt(bitsize uint8) (i int64) {
-	d.decNum(false)
-	n := &d.n
-	if n.manOverflow {
-		d.d.errorf("json: overflow integer after: %v", n.mantissa)
-		return
-	}
-	var u uint64
-	if n.exponent == 0 {
-		u = n.mantissa
-	} else if n.exponent < 0 {
-		d.d.errorf("json: fractional integer")
-		return
-	} else if n.exponent > 0 {
-		var overflow bool
-		if u, overflow = n.uintExp(); overflow {
-			d.d.errorf("json: overflow integer")
-			return
-		}
-	}
-	i = int64(u)
-	if n.neg {
-		i = -i
-	}
-	if chkOvf.Int(i, bitsize) {
-		d.d.errorf("json: overflow %v bits: %s", bitsize, d.bs)
+func (d *jsonDecDriver) DecodeUint(bitsize uint8) (u uint64) {
+	bs := d.decNumBytes()
+	u, err := strconv.ParseUint(stringView(bs), 10, int(bitsize))
+	if err != nil {
+		d.d.errorf("json: decode uint from %s: %v", bs, err)
 		return
 	}
-	// fmt.Printf("DecodeInt: %v\n", i)
 	return
 }
 
-// floatVal MUST only be called after a decNum, as d.bs now contains the bytes of the number
-func (d *jsonDecDriver) floatVal() (f float64) {
-	f, useStrConv := d.n.floatVal()
-	if useStrConv {
-		var err error
-		if f, err = strconv.ParseFloat(stringView(d.bs), 64); err != nil {
-			panic(fmt.Errorf("parse float: %s, %v", d.bs, err))
-		}
-		if d.n.neg {
-			f = -f
-		}
-	}
-	return
-}
-
-func (d *jsonDecDriver) DecodeUint(bitsize uint8) (u uint64) {
-	d.decNum(false)
-	n := &d.n
-	if n.neg {
-		d.d.errorf("json: unsigned integer cannot be negative")
-		return
-	}
-	if n.manOverflow {
-		d.d.errorf("json: overflow integer after: %v", n.mantissa)
-		return
-	}
-	if n.exponent == 0 {
-		u = n.mantissa
-	} else if n.exponent < 0 {
-		d.d.errorf("json: fractional integer")
-		return
-	} else if n.exponent > 0 {
-		var overflow bool
-		if u, overflow = n.uintExp(); overflow {
-			d.d.errorf("json: overflow integer")
-			return
-		}
-	}
-	if chkOvf.Uint(u, bitsize) {
-		d.d.errorf("json: overflow %v bits: %s", bitsize, d.bs)
+func (d *jsonDecDriver) DecodeInt(bitsize uint8) (i int64) {
+	bs := d.decNumBytes()
+	// if bytes.ContainsAny(bs, ".eE") {
+	// 	d.d.errorf("json: decoding int, but found one or more of the chars: .eE: %s", bs)
+	// 	return
+	// }
+	i, err := strconv.ParseInt(stringView(bs), 10, int(bitsize))
+	if err != nil {
+		d.d.errorf("json: decode int from %s: %v", bs, err)
 		return
 	}
-	// fmt.Printf("DecodeUint: %v\n", u)
 	return
 }
 
 func (d *jsonDecDriver) DecodeFloat(chkOverflow32 bool) (f float64) {
-	d.decNum(true)
-	f = d.floatVal()
-	if chkOverflow32 && chkOvf.Float32(f) {
-		d.d.errorf("json: overflow float32: %v, %s", f, d.bs)
+	bs := d.decNumBytes()
+	bitsize := 64
+	if chkOverflow32 {
+		bitsize = 32
+	}
+	f, err := strconv.ParseFloat(stringView(bs), bitsize)
+	if err != nil {
+		d.d.errorf("json: decode float from %s: %v", bs, err)
 		return
 	}
 	return
@@ -929,19 +606,14 @@ func (d *jsonDecDriver) DecodeExt(rv interface{}, xtag uint64, ext Ext) (realxta
 	return
 }
 
-func (d *jsonDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut []byte) {
+func (d *jsonDecDriver) DecodeBytes(bs []byte, zerocopy bool) (bsOut []byte) {
 	// if decoding into raw bytes, and the RawBytesExt is configured, use it to decode.
-	if !isstring && d.se.i != nil {
+	if d.se.i != nil {
 		bsOut = bs
 		d.DecodeExt(&bsOut, 0, &d.se)
 		return
 	}
 	d.appendStringAsBytes()
-	// if isstring, then just return the bytes, even if it is using the scratch buffer.
-	// the bytes will be converted to a string as needed.
-	if isstring {
-		return d.bs
-	}
 	// if appendStringAsBytes returned a zero-len slice, then treat as nil.
 	// This should only happen for null, and "".
 	if len(d.bs) == 0 {
@@ -967,22 +639,25 @@ func (d *jsonDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut [
 	return
 }
 
+const jsonAlwaysReturnInternString = false
+
 func (d *jsonDecDriver) DecodeString() (s string) {
 	d.appendStringAsBytes()
 	// if x := d.s.sc; x != nil && x.so && x.st == '}' { // map key
-	if d.c == containerMapKey {
+	if jsonAlwaysReturnInternString || d.c == containerMapKey {
 		return d.d.string(d.bs)
 	}
 	return string(d.bs)
 }
 
+func (d *jsonDecDriver) DecodeStringAsBytes() (s []byte) {
+	d.appendStringAsBytes()
+	return d.bs
+}
+
 func (d *jsonDecDriver) appendStringAsBytes() {
 	if d.tok == 0 {
-		var b byte
-		r := d.r
-		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
-		}
-		d.tok = b
+		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 
 	if d.tok != '"' {
@@ -1002,63 +677,72 @@ func (d *jsonDecDriver) appendStringAsBytes() {
 			copy(d.bs, "true")
 		default:
 			// try to parse a valid number
-			d.decNum(true)
+			bs := d.decNumBytes()
+			d.bs = d.bs[:len(bs)]
+			copy(d.bs, bs)
 		}
 		return
 	}
 
 	d.tok = 0
-
-	v := d.bs[:0]
-	var c uint8
 	r := d.r
-	for {
-		c = r.readn1()
-		if c == '"' {
+	var cs []byte
+	v := d.bs[:0]
+	// var c uint8
+	for i := 0; ; i++ {
+		if i == len(cs) {
+			cs = r.readUntil(d.b2[:0], '"')
+			i = 0
+		}
+		if cs[i] == '"' {
 			break
-		} else if c == '\\' {
-			c = r.readn1()
-			switch c {
-			case '"', '\\', '/', '\'':
-				v = append(v, c)
-			case 'b':
-				v = append(v, '\b')
-			case 'f':
-				v = append(v, '\f')
-			case 'n':
-				v = append(v, '\n')
-			case 'r':
-				v = append(v, '\r')
-			case 't':
-				v = append(v, '\t')
-			case 'u':
-				rr := d.jsonU4(false)
-				// fmt.Printf("$$$$$$$$$: is surrogate: %v\n", utf16.IsSurrogate(rr))
-				if utf16.IsSurrogate(rr) {
-					rr = utf16.DecodeRune(rr, d.jsonU4(true))
+		}
+		if cs[i] != '\\' {
+			v = append(v, cs[i])
+			continue
+		}
+		// cs[i] == '\\'
+		i++
+		switch cs[i] {
+		case '"', '\\', '/', '\'':
+			v = append(v, cs[i])
+		case 'b':
+			v = append(v, '\b')
+		case 'f':
+			v = append(v, '\f')
+		case 'n':
+			v = append(v, '\n')
+		case 'r':
+			v = append(v, '\r')
+		case 't':
+			v = append(v, '\t')
+		case 'u':
+			rr := d.jsonU4Arr([4]byte{cs[i+1], cs[i+2], cs[i+3], cs[i+4]})
+			i += 4
+			// fmt.Printf("$$$$$$$$$: is surrogate: %v\n", utf16.IsSurrogate(rr))
+			if utf16.IsSurrogate(rr) {
+				// fmt.Printf(">>>> checking utf16 surrogate\n")
+				if !(cs[i+1] == '\\' && cs[i+2] == 'u') {
+					d.d.errorf(`json: unquoteStr: invalid unicode sequence. Expecting \u`)
+					return
 				}
-				w2 := utf8.EncodeRune(d.bstr[:], rr)
-				v = append(v, d.bstr[:w2]...)
-			default:
-				d.d.errorf("json: unsupported escaped value: %c", c)
+				i += 2
+				rr = utf16.DecodeRune(rr, d.jsonU4Arr([4]byte{cs[i+1], cs[i+2], cs[i+3], cs[i+4]}))
+				i += 4
 			}
-		} else {
-			v = append(v, c)
+			w2 := utf8.EncodeRune(d.bstr[:], rr)
+			v = append(v, d.bstr[:w2]...)
+		default:
+			d.d.errorf("json: unsupported escaped value: %c", cs[i])
 		}
 	}
 	d.bs = v
 }
 
-func (d *jsonDecDriver) jsonU4(checkSlashU bool) rune {
-	r := d.r
-	if checkSlashU && !(r.readn1() == '\\' && r.readn1() == 'u') {
-		d.d.errorf(`json: unquoteStr: invalid unicode sequence. Expecting \u`)
-		return 0
-	}
+func (d *jsonDecDriver) jsonU4Arr(bs [4]byte) (r rune) {
 	// u, _ := strconv.ParseUint(string(d.bstr[:4]), 16, 64)
 	var u uint32
-	for i := 0; i < 4; i++ {
-		v := r.readn1()
+	for _, v := range bs {
 		if '0' <= v && v <= '9' {
 			v = v - '0'
 		} else if 'a' <= v && v <= 'z' {
@@ -1071,6 +755,7 @@ func (d *jsonDecDriver) jsonU4(checkSlashU bool) rune {
 		}
 		u = u*16 + uint32(v)
 	}
+	// fmt.Printf(">>>>>>>> jsonU4Arr: %v, %s\n", rune(u), string(rune(u)))
 	return rune(u)
 }
 
@@ -1079,11 +764,7 @@ func (d *jsonDecDriver) DecodeNaked() {
 	// var decodeFurther bool
 
 	if d.tok == 0 {
-		var b byte
-		r := d.r
-		for b = r.readn1(); jsonIsWS(b); b = r.readn1() {
-		}
-		d.tok = b
+		d.tok = d.r.skip(&jsonCharWhitespaceSet)
 	}
 	switch d.tok {
 	case 'n':
@@ -1109,41 +790,35 @@ func (d *jsonDecDriver) DecodeNaked() {
 		z.v = valueTypeString
 		z.s = d.DecodeString()
 	default: // number
-		d.decNum(true)
-		n := &d.n
-		// if the string had a any of [.eE], then decode as float.
-		switch {
-		case n.explicitExponent, n.dot, n.exponent < 0, n.manOverflow:
+		bs := d.decNumBytes()
+		var err error
+		if len(bs) == 0 {
+			d.d.errorf("json: decode number from empty string")
+			return
+		} else if d.h.PreferFloat ||
+			bytes.IndexByte(bs, '.') != -1 ||
+			bytes.IndexByte(bs, 'e') != -1 ||
+			bytes.IndexByte(bs, 'E') != -1 {
+			// } else if d.h.PreferFloat || bytes.ContainsAny(bs, ".eE") {
 			z.v = valueTypeFloat
-			z.f = d.floatVal()
-		case n.exponent == 0:
-			u := n.mantissa
-			switch {
-			case n.neg:
-				z.v = valueTypeInt
-				z.i = -int64(u)
-			case d.h.SignedInteger:
-				z.v = valueTypeInt
-				z.i = int64(u)
-			default:
-				z.v = valueTypeUint
-				z.u = u
+			z.f, err = strconv.ParseFloat(stringView(bs), 64)
+		} else if d.h.SignedInteger || bs[0] == '-' {
+			z.v = valueTypeInt
+			z.i, err = strconv.ParseInt(stringView(bs), 10, 64)
+		} else {
+			z.v = valueTypeUint
+			z.u, err = strconv.ParseUint(stringView(bs), 10, 64)
+		}
+		if err != nil {
+			if z.v == valueTypeInt || z.v == valueTypeUint {
+				if v, ok := err.(*strconv.NumError); ok && (v.Err == strconv.ErrRange || v.Err == strconv.ErrSyntax) {
+					z.v = valueTypeFloat
+					z.f, err = strconv.ParseFloat(stringView(bs), 64)
+				}
 			}
-		default:
-			u, overflow := n.uintExp()
-			switch {
-			case overflow:
-				z.v = valueTypeFloat
-				z.f = d.floatVal()
-			case n.neg:
-				z.v = valueTypeInt
-				z.i = -int64(u)
-			case d.h.SignedInteger:
-				z.v = valueTypeInt
-				z.i = int64(u)
-			default:
-				z.v = valueTypeUint
-				z.u = u
+			if err != nil {
+				d.d.errorf("json: decode number from %s: %v", bs, err)
+				return
 			}
 		}
 		// fmt.Printf("DecodeNaked: Number: %T, %v\n", v, v)
@@ -1154,6 +829,14 @@ func (d *jsonDecDriver) DecodeNaked() {
 	return
 }
 
+// func jsonAcceptNonWS(b byte) bool {
+// 	return !jsonCharWhitespaceSet[b]
+// }
+
+// func jsonAcceptDQuote(b byte) bool {
+// 	return b == '"'
+// }
+
 //----------------------
 
 // JsonHandle is a handle for JSON encoding format.
@@ -1203,6 +886,11 @@ type JsonHandle struct {
 	// By default, we encode them as \uXXX
 	// to prevent security holes when served from some browsers.
 	HTMLCharsAsIs bool
+
+	// PreferFloat says that we will default to decoding a number as a float.
+	// If not set, we will examine the characters of the number and decode as an
+	// integer type if it doesn't have any of the characters [.eE].
+	PreferFloat bool
 }
 
 func (h *JsonHandle) SetInterfaceExt(rt reflect.Type, tag uint64, ext InterfaceExt) (err error) {
@@ -1251,7 +939,7 @@ func (d *jsonDecDriver) reset() {
 		d.bs = d.bs[:0]
 	}
 	d.c, d.tok = 0, 0
-	d.n.reset()
+	// d.n.reset()
 }
 
 var jsonEncodeTerminate = []byte{' '}

codec/mammoth-test.go.tmpl

@@ -0,0 +1,33 @@
+// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
+// Use of this source code is governed by a MIT license found in the LICENSE file.
+
+// ************************************************************
+// DO NOT EDIT. 
+// THIS FILE IS AUTO-GENERATED from mammoth-test.go.tmpl
+// ************************************************************
+
+package codec
+
+// TestMammoth has all the different paths optimized in fast-path
+// It has all the primitives, slices and maps.
+// 
+// For each of those types, it has a pointer and a non-pointer field.
+
+type TestMammoth struct {
+
+{{range .Values }}{{if .Primitive }}{{/*
+*/}}{{ .MethodNamePfx "F" true }} {{ .Primitive }}
+{{ .MethodNamePfx "Fptr" true }} *{{ .Primitive }}
+{{end}}{{end}}
+
+{{range .Values }}{{if not .Primitive }}{{if not .MapKey }}{{/*
+*/}}{{ .MethodNamePfx "F" false }} []{{ .Elem }}
+{{ .MethodNamePfx "Fptr" false }} *[]{{ .Elem }}
+{{end}}{{end}}{{end}}
+
+{{range .Values }}{{if not .Primitive }}{{if .MapKey }}{{/*
+*/}}{{ .MethodNamePfx "F" false }} map[{{ .MapKey }}]{{ .Elem }}
+{{ .MethodNamePfx "Fptr" false }} *map[{{ .MapKey }}]{{ .Elem }}
+{{end}}{{end}}{{end}}
+
+}

codec/mammoth_generated_test.go

@@ -0,0 +1,593 @@
+// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
+// Use of this source code is governed by a MIT license found in the LICENSE file.
+
+// ************************************************************
+// DO NOT EDIT.
+// THIS FILE IS AUTO-GENERATED from mammoth-test.go.tmpl
+// ************************************************************
+
+package codec
+
+// TestMammoth has all the different paths optimized in fast-path
+// It has all the primitives, slices and maps.
+//
+// For each of those types, it has a pointer and a non-pointer field.
+
+type TestMammoth struct {
+	FIntf       interface{}
+	FptrIntf    *interface{}
+	FString     string
+	FptrString  *string
+	FFloat32    float32
+	FptrFloat32 *float32
+	FFloat64    float64
+	FptrFloat64 *float64
+	FUint       uint
+	FptrUint    *uint
+	FUint8      uint8
+	FptrUint8   *uint8
+	FUint16     uint16
+	FptrUint16  *uint16
+	FUint32     uint32
+	FptrUint32  *uint32
+	FUint64     uint64
+	FptrUint64  *uint64
+	FUintptr    uintptr
+	FptrUintptr *uintptr
+	FInt        int
+	FptrInt     *int
+	FInt8       int8
+	FptrInt8    *int8
+	FInt16      int16
+	FptrInt16   *int16
+	FInt32      int32
+	FptrInt32   *int32
+	FInt64      int64
+	FptrInt64   *int64
+	FBool       bool
+	FptrBool    *bool
+
+	FSliceIntf       []interface{}
+	FptrSliceIntf    *[]interface{}
+	FSliceString     []string
+	FptrSliceString  *[]string
+	FSliceFloat32    []float32
+	FptrSliceFloat32 *[]float32
+	FSliceFloat64    []float64
+	FptrSliceFloat64 *[]float64
+	FSliceUint       []uint
+	FptrSliceUint    *[]uint
+	FSliceUint16     []uint16
+	FptrSliceUint16  *[]uint16
+	FSliceUint32     []uint32
+	FptrSliceUint32  *[]uint32
+	FSliceUint64     []uint64
+	FptrSliceUint64  *[]uint64
+	FSliceUintptr    []uintptr
+	FptrSliceUintptr *[]uintptr
+	FSliceInt        []int
+	FptrSliceInt     *[]int
+	FSliceInt8       []int8
+	FptrSliceInt8    *[]int8
+	FSliceInt16      []int16
+	FptrSliceInt16   *[]int16
+	FSliceInt32      []int32
+	FptrSliceInt32   *[]int32
+	FSliceInt64      []int64
+	FptrSliceInt64   *[]int64
+	FSliceBool       []bool
+	FptrSliceBool    *[]bool
+
+	FMapIntfIntf          map[interface{}]interface{}
+	FptrMapIntfIntf       *map[interface{}]interface{}
+	FMapIntfString        map[interface{}]string
+	FptrMapIntfString     *map[interface{}]string
+	FMapIntfUint          map[interface{}]uint
+	FptrMapIntfUint       *map[interface{}]uint
+	FMapIntfUint8         map[interface{}]uint8
+	FptrMapIntfUint8      *map[interface{}]uint8
+	FMapIntfUint16        map[interface{}]uint16
+	FptrMapIntfUint16     *map[interface{}]uint16
+	FMapIntfUint32        map[interface{}]uint32
+	FptrMapIntfUint32     *map[interface{}]uint32
+	FMapIntfUint64        map[interface{}]uint64
+	FptrMapIntfUint64     *map[interface{}]uint64
+	FMapIntfUintptr       map[interface{}]uintptr
+	FptrMapIntfUintptr    *map[interface{}]uintptr
+	FMapIntfInt           map[interface{}]int
+	FptrMapIntfInt        *map[interface{}]int
+	FMapIntfInt8          map[interface{}]int8
+	FptrMapIntfInt8       *map[interface{}]int8
+	FMapIntfInt16         map[interface{}]int16
+	FptrMapIntfInt16      *map[interface{}]int16
+	FMapIntfInt32         map[interface{}]int32
+	FptrMapIntfInt32      *map[interface{}]int32
+	FMapIntfInt64         map[interface{}]int64
+	FptrMapIntfInt64      *map[interface{}]int64
+	FMapIntfFloat32       map[interface{}]float32
+	FptrMapIntfFloat32    *map[interface{}]float32
+	FMapIntfFloat64       map[interface{}]float64
+	FptrMapIntfFloat64    *map[interface{}]float64
+	FMapIntfBool          map[interface{}]bool
+	FptrMapIntfBool       *map[interface{}]bool
+	FMapStringIntf        map[string]interface{}
+	FptrMapStringIntf     *map[string]interface{}
+	FMapStringString      map[string]string
+	FptrMapStringString   *map[string]string
+	FMapStringUint        map[string]uint
+	FptrMapStringUint     *map[string]uint
+	FMapStringUint8       map[string]uint8
+	FptrMapStringUint8    *map[string]uint8
+	FMapStringUint16      map[string]uint16
+	FptrMapStringUint16   *map[string]uint16
+	FMapStringUint32      map[string]uint32
+	FptrMapStringUint32   *map[string]uint32
+	FMapStringUint64      map[string]uint64
+	FptrMapStringUint64   *map[string]uint64
+	FMapStringUintptr     map[string]uintptr
+	FptrMapStringUintptr  *map[string]uintptr
+	FMapStringInt         map[string]int
+	FptrMapStringInt      *map[string]int
+	FMapStringInt8        map[string]int8
+	FptrMapStringInt8     *map[string]int8
+	FMapStringInt16       map[string]int16
+	FptrMapStringInt16    *map[string]int16
+	FMapStringInt32       map[string]int32
+	FptrMapStringInt32    *map[string]int32
+	FMapStringInt64       map[string]int64
+	FptrMapStringInt64    *map[string]int64
+	FMapStringFloat32     map[string]float32
+	FptrMapStringFloat32  *map[string]float32
+	FMapStringFloat64     map[string]float64
+	FptrMapStringFloat64  *map[string]float64
+	FMapStringBool        map[string]bool
+	FptrMapStringBool     *map[string]bool
+	FMapFloat32Intf       map[float32]interface{}
+	FptrMapFloat32Intf    *map[float32]interface{}
+	FMapFloat32String     map[float32]string
+	FptrMapFloat32String  *map[float32]string
+	FMapFloat32Uint       map[float32]uint
+	FptrMapFloat32Uint    *map[float32]uint
+	FMapFloat32Uint8      map[float32]uint8
+	FptrMapFloat32Uint8   *map[float32]uint8
+	FMapFloat32Uint16     map[float32]uint16
+	FptrMapFloat32Uint16  *map[float32]uint16
+	FMapFloat32Uint32     map[float32]uint32
+	FptrMapFloat32Uint32  *map[float32]uint32
+	FMapFloat32Uint64     map[float32]uint64
+	FptrMapFloat32Uint64  *map[float32]uint64
+	FMapFloat32Uintptr    map[float32]uintptr
+	FptrMapFloat32Uintptr *map[float32]uintptr
+	FMapFloat32Int        map[float32]int
+	FptrMapFloat32Int     *map[float32]int
+	FMapFloat32Int8       map[float32]int8
+	FptrMapFloat32Int8    *map[float32]int8
+	FMapFloat32Int16      map[float32]int16
+	FptrMapFloat32Int16   *map[float32]int16
+	FMapFloat32Int32      map[float32]int32
+	FptrMapFloat32Int32   *map[float32]int32
+	FMapFloat32Int64      map[float32]int64
+	FptrMapFloat32Int64   *map[float32]int64
+	FMapFloat32Float32    map[float32]float32
+	FptrMapFloat32Float32 *map[float32]float32
+	FMapFloat32Float64    map[float32]float64
+	FptrMapFloat32Float64 *map[float32]float64
+	FMapFloat32Bool       map[float32]bool
+	FptrMapFloat32Bool    *map[float32]bool
+	FMapFloat64Intf       map[float64]interface{}
+	FptrMapFloat64Intf    *map[float64]interface{}
+	FMapFloat64String     map[float64]string
+	FptrMapFloat64String  *map[float64]string
+	FMapFloat64Uint       map[float64]uint
+	FptrMapFloat64Uint    *map[float64]uint
+	FMapFloat64Uint8      map[float64]uint8
+	FptrMapFloat64Uint8   *map[float64]uint8
+	FMapFloat64Uint16     map[float64]uint16
+	FptrMapFloat64Uint16  *map[float64]uint16
+	FMapFloat64Uint32     map[float64]uint32
+	FptrMapFloat64Uint32  *map[float64]uint32
+	FMapFloat64Uint64     map[float64]uint64
+	FptrMapFloat64Uint64  *map[float64]uint64
+	FMapFloat64Uintptr    map[float64]uintptr
+	FptrMapFloat64Uintptr *map[float64]uintptr
+	FMapFloat64Int        map[float64]int
+	FptrMapFloat64Int     *map[float64]int
+	FMapFloat64Int8       map[float64]int8
+	FptrMapFloat64Int8    *map[float64]int8
+	FMapFloat64Int16      map[float64]int16
+	FptrMapFloat64Int16   *map[float64]int16
+	FMapFloat64Int32      map[float64]int32
+	FptrMapFloat64Int32   *map[float64]int32
+	FMapFloat64Int64      map[float64]int64
+	FptrMapFloat64Int64   *map[float64]int64
+	FMapFloat64Float32    map[float64]float32
+	FptrMapFloat64Float32 *map[float64]float32
+	FMapFloat64Float64    map[float64]float64
+	FptrMapFloat64Float64 *map[float64]float64
+	FMapFloat64Bool       map[float64]bool
+	FptrMapFloat64Bool    *map[float64]bool
+	FMapUintIntf          map[uint]interface{}
+	FptrMapUintIntf       *map[uint]interface{}
+	FMapUintString        map[uint]string
+	FptrMapUintString     *map[uint]string
+	FMapUintUint          map[uint]uint
+	FptrMapUintUint       *map[uint]uint
+	FMapUintUint8         map[uint]uint8
+	FptrMapUintUint8      *map[uint]uint8
+	FMapUintUint16        map[uint]uint16
+	FptrMapUintUint16     *map[uint]uint16
+	FMapUintUint32        map[uint]uint32
+	FptrMapUintUint32     *map[uint]uint32
+	FMapUintUint64        map[uint]uint64
+	FptrMapUintUint64     *map[uint]uint64
+	FMapUintUintptr       map[uint]uintptr
+	FptrMapUintUintptr    *map[uint]uintptr
+	FMapUintInt           map[uint]int
+	FptrMapUintInt        *map[uint]int
+	FMapUintInt8          map[uint]int8
+	FptrMapUintInt8       *map[uint]int8
+	FMapUintInt16         map[uint]int16
+	FptrMapUintInt16      *map[uint]int16
+	FMapUintInt32         map[uint]int32
+	FptrMapUintInt32      *map[uint]int32
+	FMapUintInt64         map[uint]int64
+	FptrMapUintInt64      *map[uint]int64
+	FMapUintFloat32       map[uint]float32
+	FptrMapUintFloat32    *map[uint]float32
+	FMapUintFloat64       map[uint]float64
+	FptrMapUintFloat64    *map[uint]float64
+	FMapUintBool          map[uint]bool
+	FptrMapUintBool       *map[uint]bool
+	FMapUint8Intf         map[uint8]interface{}
+	FptrMapUint8Intf      *map[uint8]interface{}
+	FMapUint8String       map[uint8]string
+	FptrMapUint8String    *map[uint8]string
+	FMapUint8Uint         map[uint8]uint
+	FptrMapUint8Uint      *map[uint8]uint
+	FMapUint8Uint8        map[uint8]uint8
+	FptrMapUint8Uint8     *map[uint8]uint8
+	FMapUint8Uint16       map[uint8]uint16
+	FptrMapUint8Uint16    *map[uint8]uint16
+	FMapUint8Uint32       map[uint8]uint32
+	FptrMapUint8Uint32    *map[uint8]uint32
+	FMapUint8Uint64       map[uint8]uint64
+	FptrMapUint8Uint64    *map[uint8]uint64
+	FMapUint8Uintptr      map[uint8]uintptr
+	FptrMapUint8Uintptr   *map[uint8]uintptr
+	FMapUint8Int          map[uint8]int
+	FptrMapUint8Int       *map[uint8]int
+	FMapUint8Int8         map[uint8]int8
+	FptrMapUint8Int8      *map[uint8]int8
+	FMapUint8Int16        map[uint8]int16
+	FptrMapUint8Int16     *map[uint8]int16
+	FMapUint8Int32        map[uint8]int32
+	FptrMapUint8Int32     *map[uint8]int32
+	FMapUint8Int64        map[uint8]int64
+	FptrMapUint8Int64     *map[uint8]int64
+	FMapUint8Float32      map[uint8]float32
+	FptrMapUint8Float32   *map[uint8]float32
+	FMapUint8Float64      map[uint8]float64
+	FptrMapUint8Float64   *map[uint8]float64
+	FMapUint8Bool         map[uint8]bool
+	FptrMapUint8Bool      *map[uint8]bool
+	FMapUint16Intf        map[uint16]interface{}
+	FptrMapUint16Intf     *map[uint16]interface{}
+	FMapUint16String      map[uint16]string
+	FptrMapUint16String   *map[uint16]string
+	FMapUint16Uint        map[uint16]uint
+	FptrMapUint16Uint     *map[uint16]uint
+	FMapUint16Uint8       map[uint16]uint8
+	FptrMapUint16Uint8    *map[uint16]uint8
+	FMapUint16Uint16      map[uint16]uint16
+	FptrMapUint16Uint16   *map[uint16]uint16
+	FMapUint16Uint32      map[uint16]uint32
+	FptrMapUint16Uint32   *map[uint16]uint32
+	FMapUint16Uint64      map[uint16]uint64
+	FptrMapUint16Uint64   *map[uint16]uint64
+	FMapUint16Uintptr     map[uint16]uintptr
+	FptrMapUint16Uintptr  *map[uint16]uintptr
+	FMapUint16Int         map[uint16]int
+	FptrMapUint16Int      *map[uint16]int
+	FMapUint16Int8        map[uint16]int8
+	FptrMapUint16Int8     *map[uint16]int8
+	FMapUint16Int16       map[uint16]int16
+	FptrMapUint16Int16    *map[uint16]int16
+	FMapUint16Int32       map[uint16]int32
+	FptrMapUint16Int32    *map[uint16]int32
+	FMapUint16Int64       map[uint16]int64
+	FptrMapUint16Int64    *map[uint16]int64
+	FMapUint16Float32     map[uint16]float32
+	FptrMapUint16Float32  *map[uint16]float32
+	FMapUint16Float64     map[uint16]float64
+	FptrMapUint16Float64  *map[uint16]float64
+	FMapUint16Bool        map[uint16]bool
+	FptrMapUint16Bool     *map[uint16]bool
+	FMapUint32Intf        map[uint32]interface{}
+	FptrMapUint32Intf     *map[uint32]interface{}
+	FMapUint32String      map[uint32]string
+	FptrMapUint32String   *map[uint32]string
+	FMapUint32Uint        map[uint32]uint
+	FptrMapUint32Uint     *map[uint32]uint
+	FMapUint32Uint8       map[uint32]uint8
+	FptrMapUint32Uint8    *map[uint32]uint8
+	FMapUint32Uint16      map[uint32]uint16
+	FptrMapUint32Uint16   *map[uint32]uint16
+	FMapUint32Uint32      map[uint32]uint32
+	FptrMapUint32Uint32   *map[uint32]uint32
+	FMapUint32Uint64      map[uint32]uint64
+	FptrMapUint32Uint64   *map[uint32]uint64
+	FMapUint32Uintptr     map[uint32]uintptr
+	FptrMapUint32Uintptr  *map[uint32]uintptr
+	FMapUint32Int         map[uint32]int
+	FptrMapUint32Int      *map[uint32]int
+	FMapUint32Int8        map[uint32]int8
+	FptrMapUint32Int8     *map[uint32]int8
+	FMapUint32Int16       map[uint32]int16
+	FptrMapUint32Int16    *map[uint32]int16
+	FMapUint32Int32       map[uint32]int32
+	FptrMapUint32Int32    *map[uint32]int32
+	FMapUint32Int64       map[uint32]int64
+	FptrMapUint32Int64    *map[uint32]int64
+	FMapUint32Float32     map[uint32]float32
+	FptrMapUint32Float32  *map[uint32]float32
+	FMapUint32Float64     map[uint32]float64
+	FptrMapUint32Float64  *map[uint32]float64
+	FMapUint32Bool        map[uint32]bool
+	FptrMapUint32Bool     *map[uint32]bool
+	FMapUint64Intf        map[uint64]interface{}
+	FptrMapUint64Intf     *map[uint64]interface{}
+	FMapUint64String      map[uint64]string
+	FptrMapUint64String   *map[uint64]string
+	FMapUint64Uint        map[uint64]uint
+	FptrMapUint64Uint     *map[uint64]uint
+	FMapUint64Uint8       map[uint64]uint8
+	FptrMapUint64Uint8    *map[uint64]uint8
+	FMapUint64Uint16      map[uint64]uint16
+	FptrMapUint64Uint16   *map[uint64]uint16
+	FMapUint64Uint32      map[uint64]uint32
+	FptrMapUint64Uint32   *map[uint64]uint32
+	FMapUint64Uint64      map[uint64]uint64
+	FptrMapUint64Uint64   *map[uint64]uint64
+	FMapUint64Uintptr     map[uint64]uintptr
+	FptrMapUint64Uintptr  *map[uint64]uintptr
+	FMapUint64Int         map[uint64]int
+	FptrMapUint64Int      *map[uint64]int
+	FMapUint64Int8        map[uint64]int8
+	FptrMapUint64Int8     *map[uint64]int8
+	FMapUint64Int16       map[uint64]int16
+	FptrMapUint64Int16    *map[uint64]int16
+	FMapUint64Int32       map[uint64]int32
+	FptrMapUint64Int32    *map[uint64]int32
+	FMapUint64Int64       map[uint64]int64
+	FptrMapUint64Int64    *map[uint64]int64
+	FMapUint64Float32     map[uint64]float32
+	FptrMapUint64Float32  *map[uint64]float32
+	FMapUint64Float64     map[uint64]float64
+	FptrMapUint64Float64  *map[uint64]float64
+	FMapUint64Bool        map[uint64]bool
+	FptrMapUint64Bool     *map[uint64]bool
+	FMapUintptrIntf       map[uintptr]interface{}
+	FptrMapUintptrIntf    *map[uintptr]interface{}
+	FMapUintptrString     map[uintptr]string
+	FptrMapUintptrString  *map[uintptr]string
+	FMapUintptrUint       map[uintptr]uint
+	FptrMapUintptrUint    *map[uintptr]uint
+	FMapUintptrUint8      map[uintptr]uint8
+	FptrMapUintptrUint8   *map[uintptr]uint8
+	FMapUintptrUint16     map[uintptr]uint16
+	FptrMapUintptrUint16  *map[uintptr]uint16
+	FMapUintptrUint32     map[uintptr]uint32
+	FptrMapUintptrUint32  *map[uintptr]uint32
+	FMapUintptrUint64     map[uintptr]uint64
+	FptrMapUintptrUint64  *map[uintptr]uint64
+	FMapUintptrUintptr    map[uintptr]uintptr
+	FptrMapUintptrUintptr *map[uintptr]uintptr
+	FMapUintptrInt        map[uintptr]int
+	FptrMapUintptrInt     *map[uintptr]int
+	FMapUintptrInt8       map[uintptr]int8
+	FptrMapUintptrInt8    *map[uintptr]int8
+	FMapUintptrInt16      map[uintptr]int16
+	FptrMapUintptrInt16   *map[uintptr]int16
+	FMapUintptrInt32      map[uintptr]int32
+	FptrMapUintptrInt32   *map[uintptr]int32
+	FMapUintptrInt64      map[uintptr]int64
+	FptrMapUintptrInt64   *map[uintptr]int64
+	FMapUintptrFloat32    map[uintptr]float32
+	FptrMapUintptrFloat32 *map[uintptr]float32
+	FMapUintptrFloat64    map[uintptr]float64
+	FptrMapUintptrFloat64 *map[uintptr]float64
+	FMapUintptrBool       map[uintptr]bool
+	FptrMapUintptrBool    *map[uintptr]bool
+	FMapIntIntf           map[int]interface{}
+	FptrMapIntIntf        *map[int]interface{}
+	FMapIntString         map[int]string
+	FptrMapIntString      *map[int]string
+	FMapIntUint           map[int]uint
+	FptrMapIntUint        *map[int]uint
+	FMapIntUint8          map[int]uint8
+	FptrMapIntUint8       *map[int]uint8
+	FMapIntUint16         map[int]uint16
+	FptrMapIntUint16      *map[int]uint16
+	FMapIntUint32         map[int]uint32
+	FptrMapIntUint32      *map[int]uint32
+	FMapIntUint64         map[int]uint64
+	FptrMapIntUint64      *map[int]uint64
+	FMapIntUintptr        map[int]uintptr
+	FptrMapIntUintptr     *map[int]uintptr
+	FMapIntInt            map[int]int
+	FptrMapIntInt         *map[int]int
+	FMapIntInt8           map[int]int8
+	FptrMapIntInt8        *map[int]int8
+	FMapIntInt16          map[int]int16
+	FptrMapIntInt16       *map[int]int16
+	FMapIntInt32          map[int]int32
+	FptrMapIntInt32       *map[int]int32
+	FMapIntInt64          map[int]int64
+	FptrMapIntInt64       *map[int]int64
+	FMapIntFloat32        map[int]float32
+	FptrMapIntFloat32     *map[int]float32
+	FMapIntFloat64        map[int]float64
+	FptrMapIntFloat64     *map[int]float64
+	FMapIntBool           map[int]bool
+	FptrMapIntBool        *map[int]bool
+	FMapInt8Intf          map[int8]interface{}
+	FptrMapInt8Intf       *map[int8]interface{}
+	FMapInt8String        map[int8]string
+	FptrMapInt8String     *map[int8]string
+	FMapInt8Uint          map[int8]uint
+	FptrMapInt8Uint       *map[int8]uint
+	FMapInt8Uint8         map[int8]uint8
+	FptrMapInt8Uint8      *map[int8]uint8
+	FMapInt8Uint16        map[int8]uint16
+	FptrMapInt8Uint16     *map[int8]uint16
+	FMapInt8Uint32        map[int8]uint32
+	FptrMapInt8Uint32     *map[int8]uint32
+	FMapInt8Uint64        map[int8]uint64
+	FptrMapInt8Uint64     *map[int8]uint64
+	FMapInt8Uintptr       map[int8]uintptr
+	FptrMapInt8Uintptr    *map[int8]uintptr
+	FMapInt8Int           map[int8]int
+	FptrMapInt8Int        *map[int8]int
+	FMapInt8Int8          map[int8]int8
+	FptrMapInt8Int8       *map[int8]int8
+	FMapInt8Int16         map[int8]int16
+	FptrMapInt8Int16      *map[int8]int16
+	FMapInt8Int32         map[int8]int32
+	FptrMapInt8Int32      *map[int8]int32
+	FMapInt8Int64         map[int8]int64
+	FptrMapInt8Int64      *map[int8]int64
+	FMapInt8Float32       map[int8]float32
+	FptrMapInt8Float32    *map[int8]float32
+	FMapInt8Float64       map[int8]float64
+	FptrMapInt8Float64    *map[int8]float64
+	FMapInt8Bool          map[int8]bool
+	FptrMapInt8Bool       *map[int8]bool
+	FMapInt16Intf         map[int16]interface{}
+	FptrMapInt16Intf      *map[int16]interface{}
+	FMapInt16String       map[int16]string
+	FptrMapInt16String    *map[int16]string
+	FMapInt16Uint         map[int16]uint
+	FptrMapInt16Uint      *map[int16]uint
+	FMapInt16Uint8        map[int16]uint8
+	FptrMapInt16Uint8     *map[int16]uint8
+	FMapInt16Uint16       map[int16]uint16
+	FptrMapInt16Uint16    *map[int16]uint16
+	FMapInt16Uint32       map[int16]uint32
+	FptrMapInt16Uint32    *map[int16]uint32
+	FMapInt16Uint64       map[int16]uint64
+	FptrMapInt16Uint64    *map[int16]uint64
+	FMapInt16Uintptr      map[int16]uintptr
+	FptrMapInt16Uintptr   *map[int16]uintptr
+	FMapInt16Int          map[int16]int
+	FptrMapInt16Int       *map[int16]int
+	FMapInt16Int8         map[int16]int8
+	FptrMapInt16Int8      *map[int16]int8
+	FMapInt16Int16        map[int16]int16
+	FptrMapInt16Int16     *map[int16]int16
+	FMapInt16Int32        map[int16]int32
+	FptrMapInt16Int32     *map[int16]int32
+	FMapInt16Int64        map[int16]int64
+	FptrMapInt16Int64     *map[int16]int64
+	FMapInt16Float32      map[int16]float32
+	FptrMapInt16Float32   *map[int16]float32
+	FMapInt16Float64      map[int16]float64
+	FptrMapInt16Float64   *map[int16]float64
+	FMapInt16Bool         map[int16]bool
+	FptrMapInt16Bool      *map[int16]bool
+	FMapInt32Intf         map[int32]interface{}
+	FptrMapInt32Intf      *map[int32]interface{}
+	FMapInt32String       map[int32]string
+	FptrMapInt32String    *map[int32]string
+	FMapInt32Uint         map[int32]uint
+	FptrMapInt32Uint      *map[int32]uint
+	FMapInt32Uint8        map[int32]uint8
+	FptrMapInt32Uint8     *map[int32]uint8
+	FMapInt32Uint16       map[int32]uint16
+	FptrMapInt32Uint16    *map[int32]uint16
+	FMapInt32Uint32       map[int32]uint32
+	FptrMapInt32Uint32    *map[int32]uint32
+	FMapInt32Uint64       map[int32]uint64
+	FptrMapInt32Uint64    *map[int32]uint64
+	FMapInt32Uintptr      map[int32]uintptr
+	FptrMapInt32Uintptr   *map[int32]uintptr
+	FMapInt32Int          map[int32]int
+	FptrMapInt32Int       *map[int32]int
+	FMapInt32Int8         map[int32]int8
+	FptrMapInt32Int8      *map[int32]int8
+	FMapInt32Int16        map[int32]int16
+	FptrMapInt32Int16     *map[int32]int16
+	FMapInt32Int32        map[int32]int32
+	FptrMapInt32Int32     *map[int32]int32
+	FMapInt32Int64        map[int32]int64
+	FptrMapInt32Int64     *map[int32]int64
+	FMapInt32Float32      map[int32]float32
+	FptrMapInt32Float32   *map[int32]float32
+	FMapInt32Float64      map[int32]float64
+	FptrMapInt32Float64   *map[int32]float64
+	FMapInt32Bool         map[int32]bool
+	FptrMapInt32Bool      *map[int32]bool
+	FMapInt64Intf         map[int64]interface{}
+	FptrMapInt64Intf      *map[int64]interface{}
+	FMapInt64String       map[int64]string
+	FptrMapInt64String    *map[int64]string
+	FMapInt64Uint         map[int64]uint
+	FptrMapInt64Uint      *map[int64]uint
+	FMapInt64Uint8        map[int64]uint8
+	FptrMapInt64Uint8     *map[int64]uint8
+	FMapInt64Uint16       map[int64]uint16
+	FptrMapInt64Uint16    *map[int64]uint16
+	FMapInt64Uint32       map[int64]uint32
+	FptrMapInt64Uint32    *map[int64]uint32
+	FMapInt64Uint64       map[int64]uint64
+	FptrMapInt64Uint64    *map[int64]uint64
+	FMapInt64Uintptr      map[int64]uintptr
+	FptrMapInt64Uintptr   *map[int64]uintptr
+	FMapInt64Int          map[int64]int
+	FptrMapInt64Int       *map[int64]int
+	FMapInt64Int8         map[int64]int8
+	FptrMapInt64Int8      *map[int64]int8
+	FMapInt64Int16        map[int64]int16
+	FptrMapInt64Int16     *map[int64]int16
+	FMapInt64Int32        map[int64]int32
+	FptrMapInt64Int32     *map[int64]int32
+	FMapInt64Int64        map[int64]int64
+	FptrMapInt64Int64     *map[int64]int64
+	FMapInt64Float32      map[int64]float32
+	FptrMapInt64Float32   *map[int64]float32
+	FMapInt64Float64      map[int64]float64
+	FptrMapInt64Float64   *map[int64]float64
+	FMapInt64Bool         map[int64]bool
+	FptrMapInt64Bool      *map[int64]bool
+	FMapBoolIntf          map[bool]interface{}
+	FptrMapBoolIntf       *map[bool]interface{}
+	FMapBoolString        map[bool]string
+	FptrMapBoolString     *map[bool]string
+	FMapBoolUint          map[bool]uint
+	FptrMapBoolUint       *map[bool]uint
+	FMapBoolUint8         map[bool]uint8
+	FptrMapBoolUint8      *map[bool]uint8
+	FMapBoolUint16        map[bool]uint16
+	FptrMapBoolUint16     *map[bool]uint16
+	FMapBoolUint32        map[bool]uint32
+	FptrMapBoolUint32     *map[bool]uint32
+	FMapBoolUint64        map[bool]uint64
+	FptrMapBoolUint64     *map[bool]uint64
+	FMapBoolUintptr       map[bool]uintptr
+	FptrMapBoolUintptr    *map[bool]uintptr
+	FMapBoolInt           map[bool]int
+	FptrMapBoolInt        *map[bool]int
+	FMapBoolInt8          map[bool]int8
+	FptrMapBoolInt8       *map[bool]int8
+	FMapBoolInt16         map[bool]int16
+	FptrMapBoolInt16      *map[bool]int16
+	FMapBoolInt32         map[bool]int32
+	FptrMapBoolInt32      *map[bool]int32
+	FMapBoolInt64         map[bool]int64
+	FptrMapBoolInt64      *map[bool]int64
+	FMapBoolFloat32       map[bool]float32
+	FptrMapBoolFloat32    *map[bool]float32
+	FMapBoolFloat64       map[bool]float64
+	FptrMapBoolFloat64    *map[bool]float64
+	FMapBoolBool          map[bool]bool
+	FptrMapBoolBool       *map[bool]bool
+}

codec/msgpack.go

@@ -349,11 +349,11 @@ func (d *msgpackDecDriver) DecodeNaked() {
 				n.s = d.DecodeString()
 			} else {
 				n.v = valueTypeBytes
-				n.l = d.DecodeBytes(nil, false, false)
+				n.l = d.DecodeBytes(nil, false)
 			}
 		case bd == mpBin8, bd == mpBin16, bd == mpBin32:
 			n.v = valueTypeBytes
-			n.l = d.DecodeBytes(nil, false, false)
+			n.l = d.DecodeBytes(nil, false)
 		case bd == mpArray16, bd == mpArray32, bd >= mpFixArrayMin && bd <= mpFixArrayMax:
 			n.v = valueTypeArray
 			decodeFurther = true
@@ -525,12 +525,11 @@ func (d *msgpackDecDriver) DecodeBool() (b bool) {
 	return
 }
 
-func (d *msgpackDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut []byte) {
+func (d *msgpackDecDriver) DecodeBytes(bs []byte, zerocopy bool) (bsOut []byte) {
 	if !d.bdRead {
 		d.readNextBd()
 	}
 	var clen int
-	// ignore isstring. Expect that the bytes may be found from msgpackContainerStr or msgpackContainerBin
 	if bd := d.bd; bd == mpBin8 || bd == mpBin16 || bd == mpBin32 {
 		clen = d.readContainerLen(msgpackContainerBin)
 	} else {
@@ -553,7 +552,11 @@ func (d *msgpackDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOu
 }
 
 func (d *msgpackDecDriver) DecodeString() (s string) {
-	return string(d.DecodeBytes(d.b[:], true, true))
+	return string(d.DecodeBytes(d.b[:], true))
+}
+
+func (d *msgpackDecDriver) DecodeStringAsBytes() (s []byte) {
+	return d.DecodeBytes(d.b[:], true)
 }
 
 func (d *msgpackDecDriver) readNextBd() {
@@ -687,10 +690,10 @@ func (d *msgpackDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs
 	}
 	xbd := d.bd
 	if xbd == mpBin8 || xbd == mpBin16 || xbd == mpBin32 {
-		xbs = d.DecodeBytes(nil, false, true)
+		xbs = d.DecodeBytes(nil, true)
 	} else if xbd == mpStr8 || xbd == mpStr16 || xbd == mpStr32 ||
 		(xbd >= mpFixStrMin && xbd <= mpFixStrMax) {
-		xbs = d.DecodeBytes(nil, true, true)
+		xbs = d.DecodeStringAsBytes()
 	} else {
 		clen := d.readExtLen()
 		xtag = d.r.readn1()

codec/noop.go

@@ -105,10 +105,9 @@ func (h *noopDrv) DecodeUint(bitsize uint8) (ui uint64)       { return uint64(h.
 func (h *noopDrv) DecodeFloat(chkOverflow32 bool) (f float64) { return float64(h.m(95)) }
 func (h *noopDrv) DecodeBool() (b bool)                       { return h.m(2) == 0 }
 func (h *noopDrv) DecodeString() (s string)                   { return h.S[h.m(8)] }
+func (h *noopDrv) DecodeStringAsBytes() []byte                { return h.DecodeBytes(nil, true) }
 
-// func (h *noopDrv) DecodeStringAsBytes(bs []byte) []byte       { return h.DecodeBytes(bs) }
-
-func (h *noopDrv) DecodeBytes(bs []byte, isstring, zerocopy bool) []byte { return h.B[h.m(len(h.B))] }
+func (h *noopDrv) DecodeBytes(bs []byte, zerocopy bool) []byte { return h.B[h.m(len(h.B))] }
 
 func (h *noopDrv) ReadEnd() { h.end() }
 

codec/prebuild.go

@@ -1,3 +0,0 @@
-package codec
-
-//go:generate bash prebuild.sh

codec/prebuild.sh

@@ -1,205 +0,0 @@
-#!/bin/bash
-
-# _needgen is a helper function to tell if we need to generate files for msgp, codecgen.
-_needgen() {
-    local a="$1"
-    zneedgen=0
-    if [[ ! -e "$a" ]]
-    then
-        zneedgen=1
-        echo 1
-        return 0
-    fi 
-    for i in `ls -1 *.go.tmpl gen.go values_test.go`
-    do
-        if [[ "$a" -ot "$i" ]]
-        then
-            zneedgen=1
-            echo 1
-            return 0
-        fi 
-    done 
-    echo 0
-}
-
-# _build generates fast-path.go and gen-helper.go.
-# 
-# It is needed because there is some dependency between the generated code
-# and the other classes. Consequently, we have to totally remove the 
-# generated files and put stubs in place, before calling "go run" again
-# to recreate them.
-_build() {
-    if ! [[ "${zforce}" == "1" ||
-                "1" == $( _needgen "fast-path.generated.go" ) ||
-                "1" == $( _needgen "gen-helper.generated.go" ) ||
-                "1" == $( _needgen "gen.generated.go" ) ||
-                1 == 0 ]]
-    then
-        return 0
-    fi 
-
-    # echo "Running prebuild"
-    if [ "${zbak}" == "1" ] 
-    then
-        # echo "Backing up old generated files"
-        _zts=`date '+%m%d%Y_%H%M%S'`
-        _gg=".generated.go"
-        [ -e "gen-helper${_gg}" ] && mv gen-helper${_gg} gen-helper${_gg}__${_zts}.bak
-        [ -e "fast-path${_gg}" ] && mv fast-path${_gg} fast-path${_gg}__${_zts}.bak
-        [ -e "gen${_gg}" ] && mv gen${_gg} gen${_gg}__${_zts}.bak
-        # [ -e "safe${_gg}" ] && mv safe${_gg} safe${_gg}__${_zts}.bak
-        # [ -e "unsafe${_gg}" ] && mv unsafe${_gg} unsafe${_gg}__${_zts}.bak
-    fi 
-    rm -f gen-helper.generated.go fast-path.generated.go \
-       gen.generated.go \
-       *safe.generated.go *_generated_test.go *.generated_ffjson_expose.go
-
-    cat > gen.generated.go <<EOF
-// Copyright (c) 2012-2015 Ugorji Nwoke. All rights reserved.
-// Use of this source code is governed by a MIT license found in the LICENSE file.
-
-package codec
-
-// DO NOT EDIT. THIS FILE IS AUTO-GENERATED FROM gen-dec-(map|array).go.tmpl
-
-const genDecMapTmpl = \`
-EOF
-
-    cat >> gen.generated.go < gen-dec-map.go.tmpl
-
-    cat >> gen.generated.go <<EOF
-\`
-
-const genDecListTmpl = \`
-EOF
-
-    cat >> gen.generated.go < gen-dec-array.go.tmpl
-
-    cat >> gen.generated.go <<EOF
-\`
-
-EOF
-
-    cat > gen-from-tmpl.codec.generated.go <<EOF
-package codec 
-import "io"
-func GenInternalGoFile(r io.Reader, w io.Writer, safe bool) error {
-return genInternalGoFile(r, w, safe)
-}
-EOF
-    
-    cat > gen-from-tmpl.generated.go <<EOF
-//+build ignore
-
-package main
-
-//import "flag"
-import "${zpkg}"
-import "os"
-
-func run(fnameIn, fnameOut string, safe bool) {
-fin, err := os.Open(fnameIn)
-if err != nil { panic(err) }
-defer fin.Close()
-fout, err := os.Create(fnameOut)
-if err != nil { panic(err) }
-defer fout.Close()
-err = codec.GenInternalGoFile(fin, fout, safe)
-if err != nil { panic(err) }
-}
-
-func main() {
-// do not make safe/unsafe variants. 
-// Instead, depend on escape analysis, and place string creation and usage appropriately.
-// run("unsafe.go.tmpl", "safe.generated.go", true)
-// run("unsafe.go.tmpl", "unsafe.generated.go", false)
-run("fast-path.go.tmpl", "fast-path.generated.go", false)
-run("gen-helper.go.tmpl", "gen-helper.generated.go", false)
-}
-
-EOF
-    go run -tags=notfastpath gen-from-tmpl.generated.go && \
-        rm -f gen-from-tmpl.*generated.go 
-}
-
-_codegenerators() {
-    if [[ $zforce == "1" || 
-                "1" == $( _needgen "values_codecgen${zsfx}" ) ||
-                "1" == $( _needgen "values_msgp${zsfx}" ) ||
-                "1" == $( _needgen "values_ffjson${zsfx}" ) ||
-                1 == 0 ]] 
-    then
-        # codecgen creates some temporary files in the directory (main, pkg).
-        # Consequently, we should start msgp and ffjson first, and also put a small time latency before
-        # starting codecgen.
-        # Without this, ffjson chokes on one of the temporary files from codecgen.
-        if [[ $zexternal == "1" ]]
-        then 
-            echo "ffjson ... " && \
-                ffjson -w values_ffjson${zsfx} $zfin &
-            zzzIdFF=$!
-            echo "msgp ... " && \
-                msgp -tests=false -o=values_msgp${zsfx} -file=$zfin &
-            zzzIdMsgp=$!
-            
-            sleep 1 # give ffjson and msgp some buffer time. see note above.
-        fi
-        
-        echo "codecgen - !unsafe ... " && \
-            $zgobase/bin/codecgen -rt codecgen -t 'x,codecgen,!unsafe' -o values_codecgen${zsfx} -d 19780 $zfin &
-        zzzIdC=$!
-        echo "codecgen - unsafe ... " && \
-            $zgobase/bin/codecgen  -u -rt codecgen -t 'x,codecgen,unsafe' -o values_codecgen_unsafe${zsfx} -d 19781 $zfin &
-        zzzIdCU=$!
-        wait $zzzIdC $zzzIdCU $zzzIdMsgp $zzzIdFF && \
-            # remove (M|Unm)arshalJSON implementations, so they don't conflict with encoding/json bench \
-            if [[ $zexternal == "1" ]]
-            then
-                sed -i '' -e 's+ MarshalJSON(+ _MarshalJSON(+g' values_ffjson${zsfx} && \
-                    sed -i '' -e 's+ UnmarshalJSON(+ _UnmarshalJSON(+g' values_ffjson${zsfx}
-            fi && \
-            echo "generators done!" && \
-            true
-    fi 
-}
-
-# _init reads the arguments and sets up the flags
-_init() {
-OPTIND=1
-while getopts "fbxp:" flag
-do
-    case "x$flag" in 
-        'xf') zforce=1;;
-        'xb') zbak=1;;
-        'xx') zexternal=1;;
-        'xp') zpkg="${OPTARG}" ;;
-        *) echo "prebuild.sh accepts [-fbx] [-p basepkgname] only"; return 1;;
-    esac
-done
-shift $((OPTIND-1))
-OPTIND=1
-}
-
-# main script.
-# First ensure that this is being run from the basedir (i.e. dirname of script is .)
-# Sample usage:
-if [ "." = `dirname $0` ]
-then
-    zmydir=`pwd`
-    zfin="test_values.generated.go"
-    zsfx="_generated_test.go"
-    # zpkg="ugorji.net/codec"
-    zpkg=${zmydir##*/src/}
-    zgobase=${zmydir%%/src/*}
-    # rm -f *_generated_test.go 
-    rm -f codecgen-*.go && \
-        _init "$@" && \
-        _build && \
-        cp $zmydir/values_test.go $zmydir/$zfin && \
-        _codegenerators && \
-        echo prebuild done successfully
-    rm -f $zmydir/$zfin
-else
-    echo "Script must be run from the directory it resides in"
-fi 
-

codec/shared_test.go

@@ -21,7 +21,7 @@ package codec
 // Benchmarks should also take this parameter, to include the sereal, xdr, etc.
 // To run against codecgen, etc, make sure you pass extra parameters.
 // Example usage:
-//    go test "-tags=x codecgen unsafe" -bench=. <other parameters ...>
+//    go test "-tags=x codecgen" -bench=. <other parameters ...>
 //
 // To fully test everything:
 //    go test -tags=x -benchtime=100ms -tv -bg -bi  -brw -bu -v -run=. -bench=.
@@ -88,6 +88,7 @@ var (
 	testMaxInitLen     int
 
 	testJsonHTMLCharsAsIs bool
+	testJsonPreferFloat   bool
 )
 
 // flag variables used by bench
@@ -127,6 +128,7 @@ func testInitFlags() {
 	flag.BoolVar(&testUseMust, "tm", true, "Use Must(En|De)code")
 	flag.BoolVar(&testCheckCircRef, "tl", false, "Use Check Circular Ref")
 	flag.BoolVar(&testJsonHTMLCharsAsIs, "tas", false, "Set JSON HTMLCharsAsIs")
+	flag.BoolVar(&testJsonPreferFloat, "tjf", false, "Prefer Float in json")
 }
 
 func benchInitFlags() {
@@ -149,8 +151,16 @@ func testHEDGet(h Handle) *testHED {
 	return &testHEDs[len(testHEDs)-1]
 }
 
+func testReinit() {
+	testOnce = sync.Once{}
+	testHEDs = nil
+}
+
 func testInitAll() {
-	flag.Parse()
+	// only parse it once.
+	if !flag.Parsed() {
+		flag.Parse()
+	}
 	for _, f := range testPreInitFns {
 		f()
 	}

codec/simple.go

@@ -361,10 +361,14 @@ func (d *simpleDecDriver) decLen() int {
 }
 
 func (d *simpleDecDriver) DecodeString() (s string) {
-	return string(d.DecodeBytes(d.b[:], true, true))
+	return string(d.DecodeBytes(d.b[:], true))
 }
 
-func (d *simpleDecDriver) DecodeBytes(bs []byte, isstring, zerocopy bool) (bsOut []byte) {
+func (d *simpleDecDriver) DecodeStringAsBytes() (s []byte) {
+	return d.DecodeBytes(d.b[:], true)
+}
+
+func (d *simpleDecDriver) DecodeBytes(bs []byte, zerocopy bool) (bsOut []byte) {
 	if !d.bdRead {
 		d.readNextBd()
 	}
@@ -415,7 +419,7 @@ func (d *simpleDecDriver) decodeExtV(verifyTag bool, tag byte) (xtag byte, xbs [
 		}
 		xbs = d.r.readx(l)
 	case simpleVdByteArray, simpleVdByteArray + 1, simpleVdByteArray + 2, simpleVdByteArray + 3, simpleVdByteArray + 4:
-		xbs = d.DecodeBytes(nil, false, true)
+		xbs = d.DecodeBytes(nil, true)
 	default:
 		d.d.errorf("Invalid d.bd for extensions (Expecting extensions or byte array). Got: 0x%x", d.bd)
 		return
@@ -463,7 +467,7 @@ func (d *simpleDecDriver) DecodeNaked() {
 		n.s = d.DecodeString()
 	case simpleVdByteArray, simpleVdByteArray + 1, simpleVdByteArray + 2, simpleVdByteArray + 3, simpleVdByteArray + 4:
 		n.v = valueTypeBytes
-		n.l = d.DecodeBytes(nil, false, false)
+		n.l = d.DecodeBytes(nil, false)
 	case simpleVdExt, simpleVdExt + 1, simpleVdExt + 2, simpleVdExt + 3, simpleVdExt + 4:
 		n.v = valueTypeExt
 		l := d.decLen()

codec/tests.sh

@@ -1,107 +0,0 @@
-#!/bin/bash
-
-# Run all the different permutations of all the tests.
-# This helps ensure that nothing gets broken.
-
-_run() {
-    # 1. VARIATIONS: regular (t), canonical (c), IO R/W (i),
-    #                binc-nosymbols (n), struct2array (s), intern string (e),
-    #                json-indent (d), circular (l)
-    # 2. MODE: reflection (r), external (x), codecgen (g), unsafe (u), notfastpath (f)
-    # 3. OPTIONS: verbose (v), reset (z), must (m),
-    # 
-    # Use combinations of mode to get exactly what you want,
-    # and then pass the variations you need.
-
-    ztags=""
-    zargs=""
-    local OPTIND 
-    OPTIND=1
-    # "_xurtcinsvgzmefdl" ===  "_cdefgilmnrtsuvxz"
-    while getopts "_cdefgilmnrtsuvwxz" flag
-    do
-        case "x$flag" in 
-            'xr')  ;;
-            'xf') ztags="$ztags notfastpath" ;;
-            'xg') ztags="$ztags codecgen" ;;
-            'xx') ztags="$ztags x" ;;
-            'xu') ztags="$ztags unsafe" ;;
-            'xv') zargs="$zargs -tv" ;;
-            'xz') zargs="$zargs -tr" ;;
-            'xm') zargs="$zargs -tm" ;;
-            'xl') zargs="$zargs -tl" ;;
-            'xw') zargs="$zargs -tx=10" ;;
-            *) ;;
-        esac
-    done
-    # shift $((OPTIND-1))
-    printf '............. TAGS: %s .............\n' "$ztags"
-    # echo ">>>>>>> TAGS: $ztags"
-    
-    OPTIND=1
-    while getopts "_cdefgilmnrtsuvwxz" flag
-    do
-        case "x$flag" in 
-            'xt') printf ">>>>>>> REGULAR    : "; go test "-tags=$ztags" $zargs ; sleep 2 ;;
-            'xc') printf ">>>>>>> CANONICAL  : "; go test "-tags=$ztags" $zargs -tc; sleep 2 ;;
-            'xi') printf ">>>>>>> I/O        : "; go test "-tags=$ztags" $zargs -ti; sleep 2 ;;
-            'xn') printf ">>>>>>> NO_SYMBOLS : "; go test "-tags=$ztags" -run=Binc $zargs -tn; sleep 2 ;;
-            'xs') printf ">>>>>>> TO_ARRAY   : "; go test "-tags=$ztags" $zargs -ts; sleep 2 ;;
-            'xe') printf ">>>>>>> INTERN     : "; go test "-tags=$ztags" $zargs -te; sleep 2 ;;
-            'xd') printf ">>>>>>> INDENT     : ";
-                  go test "-tags=$ztags" -run=JsonCodecsTable -td=-1 $zargs;
-                  go test "-tags=$ztags" -run=JsonCodecsTable -td=8 $zargs;
-                  sleep 2 ;;
-            *) ;;
-        esac
-    done
-    shift $((OPTIND-1))
-
-    OPTIND=1
-}
-
-# echo ">>>>>>> RUNNING VARIATIONS OF TESTS"    
-if [[ "x$@" = "x"  || "x$@" = "x-A" ]]; then
-    # All: r, x, g, gu
-    _run "-_tcinsed_ml"  # regular
-    _run "-_tcinsed_ml_z" # regular with reset
-    _run "-w_tcinsed_ml"  # regular with max init len
-    _run "-_tcinsed_ml_f" # regular with no fastpath (notfastpath)
-    _run "-x_tcinsed_ml" # external
-    _run "-gx_tcinsed_ml" # codecgen: requires external
-    _run "-gxu_tcinsed_ml" # codecgen + unsafe
-elif [[ "x$@" = "x-Z" ]]; then
-    # Regular
-    _run "-_tcinsed_ml"  # regular
-    _run "-_tcinsed_ml_z" # regular with reset
-elif [[ "x$@" = "x-F" ]]; then
-    # regular with notfastpath
-    _run "-_tcinsed_ml_f"  # regular
-    _run "-_tcinsed_ml_zf" # regular with reset
-elif [[ "x$@" = "x-C" ]]; then
-    # codecgen
-    _run "-gx_tcinsed_ml" # codecgen: requires external
-    _run "-gxu_tcinsed_ml" # codecgen + unsafe
-    _run "-gxuw_tcinsed_ml" # codecgen + unsafe + maxinitlen
-elif [[ "x$@" = "x-X" ]]; then
-    # external
-    _run "-x_tcinsed_ml" # external
-elif [[ "x$@" = "x-h" || "x$@" = "x-?" ]]; then
-    cat <<EOF
-Usage: tests.sh [options...]
-  -A run through all tests (regular, external, codecgen)
-  -Z regular tests only 
-  -F regular tests only (without fastpath, so they run quickly)
-  -C codecgen only 
-  -X external only 
-  -h show help (usage)
-  -? same as -h
-  (no options) 
-      same as -A
-  (unrecognized options)
-      just pass on the options from the command line 
-EOF
-else
-    # e.g. ./tests.sh "-w_tcinsed_ml"
-    _run "$@"
-fi

codec/time.go

@@ -5,23 +5,10 @@ package codec
 
 import (
 	"fmt"
-	"reflect"
 	"time"
 )
 
-var (
-	timeDigits   = [...]byte{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'}
-	timeExtEncFn = func(rv reflect.Value) (bs []byte, err error) {
-		defer panicToErr(&err)
-		bs = timeExt{}.WriteExt(rv.Interface())
-		return
-	}
-	timeExtDecFn = func(rv reflect.Value, bs []byte) (err error) {
-		defer panicToErr(&err)
-		timeExt{}.ReadExt(rv.Interface(), bs)
-		return
-	}
-)
+var timeDigits = [...]byte{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'}
 
 type timeExt struct{}
 

codec/values_test.go

@@ -126,7 +126,18 @@ func newTestStruc(depth int, bench bool, useInterface, useStringKeyOnly bool) (t
 				"Afour.reverse_solidus.\u005c", "Afive.Gclef.\U0001d11E"},
 			AI64slice: []int64{1, -22, 333, -4444, 55555, -666666},
 			AMSU16:    map[string]uint16{"1": 1, "22": 2, "333": 3, "4444": 4},
-			AF64slice: []float64{11.11e-11, 22.22E+22, 33.33E-33, 44.44e+44, 555.55E-6, 666.66E6},
+			AF64slice: []float64{
+				11.11e-11, -11.11e+11,
+				2.222E+12, -2.222E-12,
+				-555.55E-5, 555.55E+5,
+				666.66E-6, -666.66E+6,
+				7777.7777E-7, -7777.7777E-7,
+				-8888.8888E+8, 8888.8888E+8,
+				-99999.9999E+9, 99999.9999E+9,
+				// these below are hairy enough to need strconv.ParseFloat
+				33.33E-33, -33.33E+33,
+				44.44e+44, -44.44e-44,
+			},
 		},
 	}
 	if useInterface {

codec/xml.go

@@ -0,0 +1,426 @@
+// +build ignore
+
+package codec
+
+import "reflect"
+
+/*
+
+A strict Non-validating namespace-aware XML 1.0 parser and (en|de)coder.
+
+We are attempting this due to perceived issues with encoding/xml:
+  - Complicated. It tried to do too much, and is not as simple to use as json.
+  - Due to over-engineering, reflection is over-used AND performance suffers:
+    java is 6X faster:http://fabsk.eu/blog/category/informatique/dev/golang/
+    even PYTHON performs better: http://outgoing.typepad.com/outgoing/2014/07/exploring-golang.html
+
+codec framework will offer the following benefits
+  - VASTLY improved performance (when using reflection-mode or codecgen)
+  - simplicity and consistency: with the rest of the supported formats
+  - all other benefits of codec framework (streaming, codegeneration, etc)
+
+codec is not a drop-in replacement for encoding/xml.
+It is a replacement, based on the simplicity and performance of codec.
+Look at it like JAXB for Go.
+
+Challenges:
+
+  - Need to output XML preamble, with all namespaces at the right location in the output.
+  - Each "end" block is dynamic, so we need to maintain a context-aware stack
+  - How to decide when to use an attribute VS an element
+  - How to handle chardata, attr, comment EXPLICITLY.
+  - Should it output fragments?
+    e.g. encoding a bool should just output true OR false, which is not well-formed XML.
+
+Extend the struct tag. See representative example:
+  type X struct {
+    ID uint8 codec:"xid|http://ugorji.net/x-namespace id,omitempty,toarray,attr,cdata"
+  }
+
+Based on this, we encode
+  - fields as elements, BUT encode as attributes if struct tag contains ",attr".
+  - text as entity-escaped text, BUT encode as CDATA if struct tag contains ",cdata".
+
+In this mode, we only encode as attribute if ",attr" is found, and only encode as CDATA
+if ",cdata" is found in the struct tag.
+
+To handle namespaces:
+  - XMLHandle is denoted as being namespace-aware.
+    Consequently, we WILL use the ns:name pair to encode and decode if defined, else use the plain name.
+  - *Encoder and *Decoder know whether the Handle "prefers" namespaces.
+  - add *Encoder.getEncName(*structFieldInfo).
+    No one calls *structFieldInfo.indexForEncName directly anymore
+  - add *Decoder.getStructFieldInfo(encName string) // encName here is either like abc, or h1:nsabc
+    No one accesses .encName anymore except in
+  - let encode.go and decode.go use these (for consistency)
+  - only problem exists for gen.go, where we create a big switch on encName.
+    Now, we also have to add a switch on strings.endsWith(kName, encNsName)
+    - gen.go will need to have many more methods, and then double-on the 2 switch loops like:
+      switch k {
+        case "abc" : x.abc()
+        case "def" : x.def()
+        default {
+          switch {
+            case !nsAware: panic(...)
+            case strings.endsWith("nsabc"): x.abc()
+            default: panic(...)
+          }
+        }
+     }
+
+The structure below accomodates this:
+
+  type typeInfo struct {
+    sfi []*structFieldInfo // sorted by encName
+    sfins // sorted by namespace
+    sfia  // sorted, to have those with attributes at the top. Needed to write XML appropriately.
+    sfip  // unsorted
+  }
+  type structFieldInfo struct {
+    encName
+    nsEncName
+    ns string
+    attr bool
+    cdata bool
+  }
+
+indexForEncName is now an internal helper function that takes a sorted array
+(one of ti.sfins or ti.sfi). It is only used by *Encoder.getStructFieldInfo(...)
+
+There will be a separate parser from the builder.
+The parser will have a method: next() xmlToken method.
+
+xmlToken has fields:
+  - type uint8: 0 | ElementStart | ElementEnd | AttrKey | AttrVal | Text
+  - value string
+  - ns string
+
+SEE: http://www.xml.com/pub/a/98/10/guide0.html?page=3#ENTDECL
+
+The following are skipped when parsing:
+  - External Entities (from external file)
+  - Notation Declaration e.g. <!NOTATION GIF87A SYSTEM "GIF">
+  - Entity Declarations & References
+  - XML Declaration (assume UTF-8)
+  - XML Directive i.e. <! ... >
+  - Other Declarations: Notation, etc.
+  - Comment
+  - Processing Instruction
+  - schema / DTD for validation:
+    We are not a VALIDATING parser. Validation is done elsewhere.
+    However, some parts of the DTD internal subset are used (SEE BELOW).
+    For Attribute List Declarations e.g.
+    <!ATTLIST foo:oldjoke name ID #REQUIRED label CDATA #IMPLIED status ( funny | notfunny ) 'funny' >
+    We considered using the ATTLIST to get "default" value, but not to validate the contents. (VETOED)
+
+The following XML features are supported
+  - Namespace
+  - Element
+  - Attribute
+  - cdata
+  - Unicode escape
+
+The following DTD (when as an internal sub-set) features are supported:
+  - Internal Entities e.g.
+    <!ELEMENT burns "ugorji is cool" > AND entities for the set: [<>&"']
+  - Parameter entities e.g.
+    <!ENTITY % personcontent "ugorji is cool"> <!ELEMENT burns (%personcontent;)*>
+
+At decode time, a structure containing the following is kept
+  - namespace mapping
+  - default attribute values
+  - all internal entities (<>&"' and others written in the document)
+
+When decode starts, it parses XML namespace declarations and creates a map in the
+xmlDecDriver. While parsing, that map continously gets updated.
+The only problem happens when a namespace declaration happens on the node that it defines.
+e.g. <hn:name xmlns:hn="http://www.ugorji.net" >
+To handle this, each Element must be fully parsed at a time,
+even if it amounts to multiple tokens which are returned one at a time on request.
+
+xmlns is a special attribute name.
+  - It is used to define namespaces, including the default
+  - It is never returned as an AttrKey or AttrVal.
+  *We may decide later to allow user to use it e.g. you want to parse the xmlns mappings into a field.*
+
+Number, bool, null, mapKey, etc can all be decoded from any xmlToken.
+This accomodates map[int]string for example.
+
+It should be possible to create a schema from the types,
+or vice versa (generate types from schema with appropriate tags).
+This is however out-of-scope from this parsing project.
+
+We should write all namespace information at the first point that it is referenced in the tree,
+and use the mapping for all child nodes and attributes. This means that state is maintained
+at a point in the tree. This also means that calls to Decode or MustDecode will reset some state.
+
+When decoding, it is important to keep track of entity references and default attribute values.
+It seems these can only be stored in the DTD components. We should honor them when decoding.
+
+Configuration for XMLHandle will look like this:
+
+  XMLHandle
+    DefaultNS string
+    // Encoding:
+    NS map[string]string // ns URI to key, used for encoding
+    // Decoding: in case ENTITY declared in external schema or dtd, store info needed here
+    Entities map[string]string // map of entity rep to character
+
+
+During encode, if a namespace mapping is not defined for a namespace found on a struct,
+then we create a mapping for it using nsN (where N is 1..1000000, and doesn't conflict
+with any other namespace mapping).
+
+Note that different fields in a struct can have different namespaces.
+However, all fields will default to the namespace on the _struct field (if defined).
+
+An XML document is a name, a map of attributes and a list of children.
+Consequently, we cannot "DecodeNaked" into a map[string]interface{} (for example).
+We have to "DecodeNaked" into something that resembles XML data.
+
+To support DecodeNaked (decode into nil interface{}) we have to define some "supporting" types:
+    type Name struct { // Prefered. Less allocations due to conversions.
+      Local string
+      Space string
+    }
+    type Element struct {
+      Name Name
+      Attrs map[Name]string
+      Children []interface{} // each child is either *Element or string
+    }
+Only two "supporting" types are exposed for XML: Name and Element.
+
+We considered 'type Name string' where Name is like "Space Local" (space-separated).
+We decided against it, because each creation of a name would lead to
+double allocation (first convert []byte to string, then concatenate them into a string).
+The benefit is that it is faster to read Attrs from a map. But given that Element is a value
+object, we want to eschew methods and have public exposed variables.
+
+We also considered the following, where xml types were not value objects, and we used
+intelligent accessor methods to extract information and for performance.
+*** WE DECIDED AGAINST THIS. ***
+    type Attr struct {
+      Name Name
+      Value string
+    }
+    // Element is a ValueObject: There are no accessor methods.
+    // Make element self-contained.
+    type Element struct {
+      Name Name
+      attrsMap map[string]string // where key is "Space Local"
+      attrs []Attr
+      childrenT []string
+      childrenE []Element
+      childrenI []int // each child is a index into T or E.
+    }
+    func (x *Element) child(i) interface{} // returns string or *Element
+
+Per XML spec and our default handling, white space is insignificant between elements,
+specifically between parent-child or siblings. White space occuring alone between start
+and end element IS significant. However, if xml:space='preserve', then we 'preserve'
+all whitespace. This is more critical when doing a DecodeNaked, but MAY not be as critical
+when decoding into a typed value.
+
+**Note: there is no xml: namespace. The xml: attributes were defined before namespaces.**
+**So treat them as just "directives" that should be interpreted to mean something**.
+
+On encoding, we don't add any prettifying markup (indenting, etc).
+
+A document or element can only be encoded/decoded from/to a struct. In this mode:
+  - struct name maps to element name (or tag-info from _struct field)
+  - fields are mapped to child elements or attributes
+
+A map is either encoded as attributes on current element, or as a set of child elements.
+Maps are encoded as attributes iff their keys and values are primitives (number, bool, string).
+
+A list is encoded as a set of child elements.
+
+Primitives (number, bool, string) are encoded as an element, attribute or text
+depending on the context.
+
+Extensions must encode themselves as a text string.
+
+Encoding is tough, specifically when encoding mappings, because we need to encode
+as either attribute or element. To do this, we need to default to encoding as attributes,
+and then let Encoder inform the Handle when to start encoding as nodes.
+i.e. Encoder does something like:
+
+    h.EncodeMapStart()
+    h.Encode(), h.Encode(), ...
+    h.EncodeMapNotAttrSignal() // this is not a bool, because it's a signal
+    h.Encode(), h.Encode(), ...
+    h.EncodeEnd()
+
+Only XMLHandle understands this, and will set itself to start encoding as elements.
+
+This support extends to maps. For example, if a struct field is a map, and it has
+the struct tag signifying it should be attr, then all its fields are encoded as attributes.
+e.g.
+
+    type X struct {
+       M map[string]int `codec:"m,attr"` // encode as attributes
+    }
+
+Question:
+  - if encoding a map, what if map keys have spaces in them???
+    Then they cannot be attributes or child elements. Error.
+
+Misc:
+
+  - For attribute values, normalize by trimming beginning and ending white space,
+    and converting every white space sequence to a single space.
+  - ATTLIST restrictions are enforced.
+    e.g. default value of xml:space, skipping xml:XYZ style attributes, etc.
+  - Consider supporting NON-STRICT mode (e.g. to handle HTML parsing).
+    Some elements e.g. br, hr, etc need not close and should be auto-closed
+    ... (see http://www.w3.org/TR/html4/loose.dtd)
+    An expansive set of entities are pre-defined.
+  - Have easy way to create a HTML parser:
+    add a HTML() method to XMLHandle, that will set Strict=false, specify AutoClose,
+    and add HTML Entities to the list.
+  - Support validating element/attribute XMLName before writing it.
+    Keep this behind a flag, which is set to false by default (for performance).
+    type XMLHandle struct {
+      CheckName bool
+    }
+
+ROADMAP (1 weeks):
+  - build encoder (1 day)
+  - build decoder (based off xmlParser) (1 day)
+  - implement xmlParser (2 days).
+    Look at encoding/xml for inspiration.
+  - integrate and TEST (1 days)
+  - write article and post it (1 day)
+
+
+*/
+
+// ----------- PARSER  -------------------
+
+type xmlTokenType uint8
+
+const (
+	_ xmlTokenType = iota << 1
+	xmlTokenElemStart
+	xmlTokenElemEnd
+	xmlTokenAttrKey
+	xmlTokenAttrVal
+	xmlTokenText
+)
+
+type xmlToken struct {
+	Type      xmlTokenType
+	Value     string
+	Namespace string // blank for AttrVal and Text
+}
+
+type xmlParser struct {
+	r    decReader
+	toks []xmlToken // list of tokens.
+	ptr  int        // ptr into the toks slice
+	done bool       // nothing else to parse. r now returns EOF.
+}
+
+func (x *xmlParser) next() (t *xmlToken) {
+	// once x.done, or x.ptr == len(x.toks) == 0, then return nil (to signify finish)
+	if !x.done && len(x.toks) == 0 {
+		x.nextTag()
+	}
+	// parses one element at a time (into possible many tokens)
+	if x.ptr < len(x.toks) {
+		t = &(x.toks[x.ptr])
+		x.ptr++
+		if x.ptr == len(x.toks) {
+			x.ptr = 0
+			x.toks = x.toks[:0]
+		}
+	}
+	return
+}
+
+// nextTag will parses the next element and fill up toks.
+// It set done flag if/once EOF is reached.
+func (x *xmlParser) nextTag() {
+	// TODO: implement.
+}
+
+// ----------- ENCODER -------------------
+
+type xmlEncDriver struct {
+	e  *Encoder
+	w  encWriter
+	h  *XMLHandle
+	b  [64]byte // scratch
+	bs []byte   // scratch
+	// s  jsonStack
+	noBuiltInTypes
+}
+
+// ----------- DECODER -------------------
+
+type xmlDecDriver struct {
+	d    *Decoder
+	h    *XMLHandle
+	r    decReader // *bytesDecReader decReader
+	ct   valueType // container type. one of unset, array or map.
+	bstr [8]byte   // scratch used for string \UXXX parsing
+	b    [64]byte  // scratch
+
+	// wsSkipped bool // whitespace skipped
+
+	// s jsonStack
+
+	noBuiltInTypes
+}
+
+// DecodeNaked will decode into an XMLNode
+
+// XMLName is a value object representing a namespace-aware NAME
+type XMLName struct {
+	Local string
+	Space string
+}
+
+// XMLNode represents a "union" of the different types of XML Nodes.
+// Only one of fields (Text or *Element) is set.
+type XMLNode struct {
+	Element *Element
+	Text    string
+}
+
+// XMLElement is a value object representing an fully-parsed XML element.
+type XMLElement struct {
+	Name  Name
+	Attrs map[XMLName]string
+	// Children is a list of child nodes, each being a *XMLElement or string
+	Children []XMLNode
+}
+
+// ----------- HANDLE  -------------------
+
+type XMLHandle struct {
+	BasicHandle
+	textEncodingType
+
+	DefaultNS string
+	NS        map[string]string // ns URI to key, for encoding
+	Entities  map[string]string // entity representation to string, for encoding.
+}
+
+func (h *XMLHandle) newEncDriver(e *Encoder) encDriver {
+	return &xmlEncDriver{e: e, w: e.w, h: h}
+}
+
+func (h *XMLHandle) newDecDriver(d *Decoder) decDriver {
+	// d := xmlDecDriver{r: r.(*bytesDecReader), h: h}
+	hd := xmlDecDriver{d: d, r: d.r, h: h}
+	hd.n.bytes = d.b[:]
+	return &hd
+}
+
+func (h *XMLHandle) SetInterfaceExt(rt reflect.Type, tag uint64, ext InterfaceExt) (err error) {
+	return h.SetExt(rt, tag, &setExtWrapper{i: ext})
+}
+
+var _ decDriver = (*xmlDecDriver)(nil)
+var _ encDriver = (*xmlEncDriver)(nil)

codec/z_all_test.go

@@ -0,0 +1,135 @@
+// +build alltests
+// +build go1.7
+
+package codec
+
+// Run this using:
+//   go test -tags=alltests -run=Suite -coverprofile=cov.out
+//   go tool cover -html=cov.out
+//
+// Because build tags are a build time parameter, we will have to test out the
+// different tags separately.
+// Tags: x codecgen safe appengine notfastpath
+//
+// These tags should be added to alltests, e.g.
+//   go test '-tags=alltests x codecgen' -run=Suite -coverprofile=cov.out
+//
+// To run all tests before submitting code, run:
+//    a=( "" "safe" "codecgen" "notfastpath" "codecgen notfastpath" "codecgen safe" "safe notfastpath" )
+//    for i in "${a[@]}"; do echo ">>>> TAGS: $i"; go test "-tags=alltests $i" -run=Suite; done
+//
+// This only works on go1.7 and above. This is when subtests and suites were supported.
+
+import "testing"
+
+// func TestMain(m *testing.M) {
+// 	println("calling TestMain")
+// 	// set some parameters
+// 	exitcode := m.Run()
+// 	os.Exit(exitcode)
+// }
+
+func testSuite(t *testing.T, f func(t *testing.T)) {
+	// find . -name "*_test.go" | xargs grep -e 'flag.' | cut -d '&' -f 2 | cut -d ',' -f 1 | grep -e '^test'
+	// Disregard the following: testVerbose, testInitDebug, testSkipIntf, testJsonIndent (Need a test for it)
+
+	testReinit() // so flag.Parse() is called first, and never called again
+
+	testUseMust = false
+	testCanonical = false
+	testUseMust = false
+	testInternStr = false
+	testUseIoEncDec = false
+	testStructToArray = false
+	testWriteNoSymbols = false
+	testCheckCircRef = false
+	testJsonHTMLCharsAsIs = false
+	testUseReset = false
+	testMaxInitLen = 0
+	testJsonIndent = 0
+	testReinit()
+	t.Run("optionsFalse", f)
+
+	testMaxInitLen = 10
+	testJsonIndent = 8
+	testReinit()
+	t.Run("initLen10-jsonSpaces", f)
+
+	testReinit()
+	testMaxInitLen = 10
+	testJsonIndent = -1
+	testReinit()
+	t.Run("initLen10-jsonTabs", f)
+
+	testCanonical = true
+	testUseMust = true
+	testInternStr = true
+	testUseIoEncDec = true
+	testStructToArray = true
+	testWriteNoSymbols = true
+	testCheckCircRef = true
+	testJsonHTMLCharsAsIs = true
+	testUseReset = true
+	testReinit()
+	t.Run("optionsTrue", f)
+}
+
+/*
+z='codec_test.go'
+find . -name "$z" | xargs grep -e '^func Test' | \
+    cut -d '(' -f 1 | cut -d ' ' -f 2 | \
+    while read f; do echo "t.Run(\"$f\", $f)"; done
+*/
+
+func testCodecGroup(t *testing.T) {
+	// println("running testcodecsuite")
+	// <setup code>
+	t.Run("TestBincCodecsTable", TestBincCodecsTable)
+	t.Run("TestBincCodecsMisc", TestBincCodecsMisc)
+	t.Run("TestBincCodecsEmbeddedPointer", TestBincCodecsEmbeddedPointer)
+	t.Run("TestBincStdEncIntf", TestBincStdEncIntf)
+	t.Run("TestBincMammoth", TestBincMammoth)
+	t.Run("TestSimpleCodecsTable", TestSimpleCodecsTable)
+	t.Run("TestSimpleCodecsMisc", TestSimpleCodecsMisc)
+	t.Run("TestSimpleCodecsEmbeddedPointer", TestSimpleCodecsEmbeddedPointer)
+	t.Run("TestSimpleStdEncIntf", TestSimpleStdEncIntf)
+	t.Run("TestSimpleMammoth", TestSimpleMammoth)
+	t.Run("TestMsgpackCodecsTable", TestMsgpackCodecsTable)
+	t.Run("TestMsgpackCodecsMisc", TestMsgpackCodecsMisc)
+	t.Run("TestMsgpackCodecsEmbeddedPointer", TestMsgpackCodecsEmbeddedPointer)
+	t.Run("TestMsgpackStdEncIntf", TestMsgpackStdEncIntf)
+	t.Run("TestMsgpackMammoth", TestMsgpackMammoth)
+	t.Run("TestCborCodecsTable", TestCborCodecsTable)
+	t.Run("TestCborCodecsMisc", TestCborCodecsMisc)
+	t.Run("TestCborCodecsEmbeddedPointer", TestCborCodecsEmbeddedPointer)
+	t.Run("TestCborMapEncodeForCanonical", TestCborMapEncodeForCanonical)
+	t.Run("TestCborCodecChan", TestCborCodecChan)
+	t.Run("TestCborStdEncIntf", TestCborStdEncIntf)
+	t.Run("TestCborMammoth", TestCborMammoth)
+	t.Run("TestJsonCodecsTable", TestJsonCodecsTable)
+	t.Run("TestJsonCodecsMisc", TestJsonCodecsMisc)
+	t.Run("TestJsonCodecsEmbeddedPointer", TestJsonCodecsEmbeddedPointer)
+	t.Run("TestJsonCodecChan", TestJsonCodecChan)
+	t.Run("TestJsonStdEncIntf", TestJsonStdEncIntf)
+	t.Run("TestJsonMammoth", TestJsonMammoth)
+	t.Run("TestJsonRaw", TestJsonRaw)
+	t.Run("TestBincRaw", TestBincRaw)
+	t.Run("TestMsgpackRaw", TestMsgpackRaw)
+	t.Run("TestSimpleRaw", TestSimpleRaw)
+	t.Run("TestCborRaw", TestCborRaw)
+	t.Run("TestAllEncCircularRef", TestAllEncCircularRef)
+	t.Run("TestAllAnonCycle", TestAllAnonCycle)
+	t.Run("TestBincRpcGo", TestBincRpcGo)
+	t.Run("TestSimpleRpcGo", TestSimpleRpcGo)
+	t.Run("TestMsgpackRpcGo", TestMsgpackRpcGo)
+	t.Run("TestCborRpcGo", TestCborRpcGo)
+	t.Run("TestJsonRpcGo", TestJsonRpcGo)
+	t.Run("TestMsgpackRpcSpec", TestMsgpackRpcSpec)
+	t.Run("TestBincUnderlyingType", TestBincUnderlyingType)
+	t.Run("TestJsonLargeInteger", TestJsonLargeInteger)
+	t.Run("TestJsonDecodeNonStringScalarInStringContext", TestJsonDecodeNonStringScalarInStringContext)
+	t.Run("TestJsonEncodeIndent", TestJsonEncodeIndent)
+	// <tear-down code>
+}
+
+func TestCodecSuite(t *testing.T) { testSuite(t, testCodecGroup) }