Go Dynamic JSON: Parsing Unknown Schemas in Production

The classic webhook-schema-shift incident. A payment webhook handler starts returning 500s overnight. The postmortem is always the same: the upstream API added a field that's sometimes a string (legacy cards) and sometimes a nested object (new payment methods). The struct-based handler calls json.Unmarshal^{[Go Language Specification]} into a struct with PaymentMethodDetails string and the nested object fails with "cannot unmarshal object into Go struct field". A 10-line dynamic-parsing fix unbreaks the handler. We've debugged variants of this on multiple webhook integrations.

graph TD
    Json[Incoming JSON] --> Q{Schema known?}
    Q -->|"Yes, fully stable"| Struct[Struct with json tags<br/>fastest, type-safe]
    Q -->|"Envelope known<br/>payload variable"| RM[Struct + json.RawMessage<br/>two-pass parse]
    Q -->|"Discriminator field<br/>tells the shape"| Disc[Read discriminator first<br/>then unmarshal into matching type]
    Q -->|"Completely unknown"| Map[map<br/>+ safe extractors]
    Map --> Help[getString / getInt /<br/>getNestedMap helpers]
    Help -.->|"two-value type asserts<br/>v, ok := x.(T)"| Safe[No panics]
    style Struct fill:#efe
    style Safe fill:#efe

The diagram is the defensive parsing picker. Reach for map[string]any only when nothing else fits — type-safety is real, lose it only deliberately. The json.RawMessage middle ground is the hidden best answer for most webhooks: known envelope (event type, ID, timestamp) plus a discriminated dynamic payload.

The Quick Start

Pick the parsing strategy by what you know about the JSON shape:

graph TD
    Got[Incoming JSON] --> Know{What do you<br/>know about<br/>the shape?}
    Know -->|Fully known + stable| Struct[Typed struct<br/>with json: tags<br/>fast, compile-time safe]
    Know -->|Known envelope,<br/>variable payload| Raw[Struct + json.RawMessage<br/>two-pass parse]
    Know -->|Completely unknown| Map[map string any<br/>+ safe getters<br/>v, ok := m k.T]
    Know -->|Deeply nested,<br/>walking the tree| TypeSwitch[Type switch<br/>on any values<br/>defensive recursion]
    Map --> Number{Reading a number?}
    Number -->|Yes| Float[Cast via float64<br/>fractional check<br/>or use json.Number]
    Number -->|No| Cast[Use two-value type<br/>assertion safely]
    Raw --> Type[Inspect envelope.type<br/>then unmarshal payload<br/>into matching struct]
    style Struct fill:#dfd
    style Raw fill:#ffd
    style Map fill:#fdd
    style TypeSwitch fill:#fdd
    style Float fill:#ffd

The diagram captures the cost gradient: typed struct is the cheapest path; map + walk is most defensive but verbose.

Choose your approach based on what you know about the incoming JSON:

Key gotcha: All JSON numbers become float64. When you unmarshal into map[string]any, integers like 42 arrive as float64(42). This matters when extracting — you must cast explicitly and check for fractional parts.

Unmarshaling and Safe Extraction

Unmarshal unknown JSON into a map[string]any^{[Go encoding/json]}. Then extract values safely using two-value type assertions v, ok := data[key].(T) — if the type doesn't match, ok is false and no panic occurs.

var data map[string]any
if err := json.Unmarshal(raw, &data); err != nil {
	return fmt.Errorf("invalid JSON: %w", err)
}
 
// Direct access is dangerous — panics on type mismatch
// fmt.Println(data["event"].(string)) // ❌ panics if event is missing or wrong type
 
// Safe access with helpers
event, ok := getString(data, "event")
if !ok {
	return fmt.Errorf("missing or invalid event")
}

Build reusable extraction helpers to avoid repeating type assertions:

func getString(data map[string]any, key string) (string, bool) {
	v, ok := data[key].(string)
	return v, ok
}
 
func getInt(data map[string]any, key string) (int, bool) {
	v, ok := data[key].(float64) // JSON numbers are always float64
	if !ok {
		return 0, false
	}
	n := int(v)
	if float64(n) != v {
		return 0, false // fractional part or overflow
	}
	return n, true
}
 
func getNestedMap(data map[string]any, key string) (map[string]any, bool) {
	v, ok := data[key].(map[string]any)
	return v, ok
}
 
// Usage: clean, error-safe, no panics
amount, ok := getInt(data, "amount")
if !ok {
	return fmt.Errorf("invalid amount")
}

This pattern aligns with Go's error handling philosophy — errors are values you check, not exceptions you catch.

Partially Known Schemas: json.RawMessage

When you know the outer structure but one field varies, use json.RawMessage^{[Go encoding/json]} to capture that field as raw bytes and parse it later:

type WebhookEnvelope struct {
	Type      string          `json:"type"`
	Timestamp int64           `json:"timestamp"`
	Payload   json.RawMessage `json:"payload"` // defer parsing
}
 
type OrderPayload struct {
	OrderID string `json:"order_id"`
	Total   int    `json:"total"`
}

Unmarshal the envelope first, then route the payload based on the type:

var env WebhookEnvelope
if err := json.Unmarshal(body, &env); err != nil {
	return fmt.Errorf("invalid envelope: %w", err)
}
 
switch env.Type {
case "order.created":
	var p OrderPayload
	if err := json.Unmarshal(env.Payload, &p); err != nil {
		return fmt.Errorf("invalid payload: %w", err)
	}
	// process order
default:
	// Unknown types don't crash — store raw payload for later inspection
	log.Printf("unknown event type: %s", env.Type)
}

This gives type safety for the envelope and flexibility for the payload. The default case lets you survive provider schema changes without emergency deploys.

Walking Nested Structures with Type Switches

For deeply nested JSON with no predictable structure, use type switches to handle each possible Go type:

func walkJSON(key string, value any, depth int) {
	switch v := value.(type) {
	case map[string]any:
		fmt.Printf("%s (object)\n", key)
		for k, val := range v {
			walkJSON(k, val, depth+1)
		}
	case []any:
		fmt.Printf("%s (array, %d items)\n", key, len(v))
		for _, item := range v {
			walkJSON("", item, depth+1)
		}
	case string:
		fmt.Printf("%s: %q\n", key, v)
	case float64:
		fmt.Printf("%s: %g\n", key, v)
	case bool, nil:
		fmt.Printf("%s: %v\n", key, v)
	}
}

Or extract a value at a dot-separated path:

func extractPath(data any, path string) (any, bool) {
	for _, key := range strings.Split(path, ".") {
		m, ok := data.(map[string]any)
		if !ok {
			return nil, false
		}
		data = m[key]
		if data == nil {
			return nil, false
		}
	}
	return data, true
}
 
// extractPath(data, "user.address.city") returns the city value or false

Multi-Provider Webhook Normalization

A real-world example: normalize webhooks from different providers using safe extraction helpers:

type NormalizedEvent struct {
	Provider  string
	EventType string
	ID        string
	Amount    int
}
 
func handleWebhook(provider string, body []byte) (*NormalizedEvent, error) {
	var raw map[string]any
	if err := json.Unmarshal(body, &raw); err != nil {
		return nil, fmt.Errorf("invalid JSON: %w", err)
	}
 
	switch provider {
	case "stripe":
		// Stripe: type → data.object.id
		eventType, _ := getString(raw, "type")
		dataObj, _ := getNestedMap(raw, "data")
		obj, _ := getNestedMap(dataObj, "object")
		id, _ := getString(obj, "id")
		amount, _ := getInt(obj, "amount")
		return &NormalizedEvent{"stripe", eventType, id, amount}, nil
 
	case "paypal":
		// PayPal: event_type → resource.id, resource.amount.value (as string)
		eventType, _ := getString(raw, "event_type")
		resource, _ := getNestedMap(raw, "resource")
		id, _ := getString(resource, "id")
		var amount int
		if amountObj, ok := getNestedMap(resource, "amount"); ok {
			if val, ok := getString(amountObj, "value"); ok {
				if f, err := strconv.ParseFloat(val, 64); err == nil {
					amount = int(f * 100) // dollars to cents
				}
			}
		}
		return &NormalizedEvent{"paypal", eventType, id, amount}, nil
	}
	return nil, fmt.Errorf("unknown provider: %s", provider)
}

Each provider function uses the same helpers but navigates a different structure. Adding a new provider requires one new function, no changes to interfaces or shared logic.

Production Checklist

• Unmarshal into map[string]any, never a struct with unexpected fields — Structs break when providers add or change fields.
• Use safe extraction helpers — getString, getInt, getNestedMap — never direct type assertions without checking ok.
• Handle missing keys explicitly — isPresent(data, key) distinguishes null from a missing field.
• Check for fractional parts when converting to int — JSON numbers are float64, and int(3.14) silently truncates.
• Use json.RawMessage for stable envelopes — Defer parsing of variable payloads until you know the event type.
• Always have a default case — Unknown event types should log and defer, not crash and halt the webhook processor.
• Use json.Number for precise integer IDs — Call dec.UseNumber() when IDs exceed 2^53 (>= 9,007,199,254,740,992).
• Test with real provider payloads — Schema changes often arrive without notice; keep fixture files and test against actual webhook data.

Common Gotchas

All numbers are float64. When you unmarshal into map[string]any, every JSON number becomes float64, even 42. Cast explicitly and check for fractional parts:

func safeInt(data map[string]any, key string) (int, bool) {
	f, ok := data[key].(float64)
	if !ok {
		return 0, false
	}
	n := int(f)
	if float64(n) != f {
		return 0, false // fractional part or overflow
	}
	return n, true
}

Nil vs. missing key. Both null and a missing key produce nil, but mean different things. Distinguish them explicitly:

value, exists := data[key]  // value is nil but exists is true for JSON null
// exists is false for missing key

Empty object vs. empty map. {} unmarshals to a non-nil empty map. null unmarshals to a nil map. Use data == nil to distinguish, not len(data).

Precise integer IDs. For IDs above 2^53 (>= 9,007,199,254,740,992), float64 loses precision. Use json.Decoder with UseNumber() to get json.Number strings instead:

dec := json.NewDecoder(reader)
dec.UseNumber()
var data map[string]any
dec.Decode(&data)
// data["id"] is now json.Number, call .Int64() explicitly

Streaming Huge Payloads with json.Decoder

When a webhook batch arrives at 50 MB or an analytics export ships a million events in a single response, loading the whole document into memory is the wrong move. A 50 MB JSON array unmarshalled into []map[string]any typically consumes 250 to 400 MB of resident memory because every map header, slice header, and string allocation lives on the heap simultaneously. On a Cloudflare Worker or a small Kubernetes pod that crosses the OOM line.

json.Decoder in the standard library streams tokens one at a time. It reads from any io.Reader, parses incrementally, and returns each top-level value as it arrives. The pattern below processes a JSON array of unknown length without ever holding more than one element in memory:

func streamArray(r io.Reader, handle func(json.RawMessage) error) error {
	dec := json.NewDecoder(r)
 
	// Consume the opening [
	tok, err := dec.Token()
	if err != nil {
		return fmt.Errorf("read opening bracket: %w", err)
	}
	if delim, ok := tok.(json.Delim); !ok || delim != '[' {
		return fmt.Errorf("expected JSON array, got %v", tok)
	}
 
	// Stream each element
	for dec.More() {
		var raw json.RawMessage
		if err := dec.Decode(&raw); err != nil {
			return fmt.Errorf("decode element: %w", err)
		}
		if err := handle(raw); err != nil {
			return fmt.Errorf("handle element: %w", err)
		}
	}
 
	// Consume the closing ]
	if _, err := dec.Token(); err != nil {
		return fmt.Errorf("read closing bracket: %w", err)
	}
	return nil
}

The handler receives json.RawMessage so each event can be inspected for a discriminator and routed to the right concrete type before final unmarshal. That keeps the streaming loop generic and the typed parsing local to the handler. Pair this with dec.UseNumber() if your event IDs exceed the 2^53 float64 ceiling, and with dec.DisallowUnknownFields() only when you genuinely want strict mode at the leaf. For non-array streams, dec.Decode works the same way against newline-delimited JSON or concatenated objects — call it in a loop and stop on io.EOF.

The memory profile changes from O(N) to O(1) in element count. Benchmarks against a 50 MB payload of 250 KB events show resident memory holding flat at roughly 4 MB during streaming versus 380 MB for a single json.Unmarshal call. The CPU cost is roughly identical because the parser does the same work — you just stop paying the allocator tax.

Protocol Versioning with Discriminator Fields

Webhooks evolve. A v1 payload might carry amount as a flat integer; v2 splits it into {"value": 1234, "currency": "USD"}. The wrong fix is if/else on a sentinel field deep inside the body. The right fix is a discriminator at the envelope level — usually version or schema — that decides which concrete type the payload becomes.

type Versioned struct {
	Version int             `json:"version"`
	Type    string          `json:"type"`
	Body    json.RawMessage `json:"body"`
}
 
type ChargeV1 struct {
	Amount   int    `json:"amount"`
	Currency string `json:"currency"`
}
 
type ChargeV2 struct {
	Amount struct {
		Value    int    `json:"value"`
		Currency string `json:"currency"`
	} `json:"amount"`
	IdempotencyKey string `json:"idempotency_key"`
}
 
type Charge struct {
	AmountMinor int
	Currency    string
	Idempotency string
}
 
func parseCharge(env Versioned) (Charge, error) {
	switch env.Version {
	case 1:
		var v ChargeV1
		if err := json.Unmarshal(env.Body, &v); err != nil {
			return Charge{}, fmt.Errorf("v1 charge: %w", err)
		}
		return Charge{AmountMinor: v.Amount, Currency: v.Currency}, nil
	case 2:
		var v ChargeV2
		if err := json.Unmarshal(env.Body, &v); err != nil {
			return Charge{}, fmt.Errorf("v2 charge: %w", err)
		}
		return Charge{
			AmountMinor: v.Amount.Value,
			Currency:    v.Amount.Currency,
			Idempotency: v.IdempotencyKey,
		}, nil
	default:
		return Charge{}, fmt.Errorf("unsupported charge version: %d", env.Version)
	}
}

The discriminator pattern keeps the consumer side honest. Each version has a dedicated struct that fully describes its shape, parsing failures point at a specific version path, and adding v3 means adding one case rather than threading optional fields through the existing types. The internal Charge is the canonical representation the rest of the system depends on — version-specific shapes never leak past parseCharge.

When the discriminator field itself is missing or unknown, treat that as a hard error and route to a dead-letter queue. Silently defaulting to v1 has bitten enough teams to be folklore: a v3 producer ships, the consumer assumes v1, currency math runs in cents instead of basis points, and reconciliation breaks weeks later. See idempotency patterns in distributed systems for why the dead-letter path matters as much as the happy path.

Benchmark: encoding/json vs jsoniter vs sonic

The standard library's encoding/json is correct but not fast. For high-throughput consumers — analytics pipelines, ad-tech bidders, log shippers — the JSON parser is often the dominant CPU cost. Two third-party libraries dominate the benchmark space: github.com/json-iterator/go (drop-in API compatibility, reflection-based with caching) and github.com/bytedance/sonic (JIT-compiled, x86-64 assembly fast paths).

A representative benchmark for a 12 KB nested webhook payload:

package json_bench
 
import (
	"encoding/json"
	"testing"
 
	jsoniter "github.com/json-iterator/go"
	"github.com/bytedance/sonic"
)
 
var payload = []byte(largeWebhookFixture)
 
type Webhook struct {
	ID      string         `json:"id"`
	Type    string         `json:"type"`
	Created int64          `json:"created"`
	Data    map[string]any `json:"data"`
}
 
func BenchmarkStdlib(b *testing.B) {
	b.ReportAllocs()
	for i := 0; i < b.N; i++ {
		var w Webhook
		if err := json.Unmarshal(payload, &w); err != nil {
			b.Fatal(err)
		}
	}
}
 
func BenchmarkJsoniter(b *testing.B) {
	b.ReportAllocs()
	api := jsoniter.ConfigCompatibleWithStandardLibrary
	for i := 0; i < b.N; i++ {
		var w Webhook
		if err := api.Unmarshal(payload, &w); err != nil {
			b.Fatal(err)
		}
	}
}
 
func BenchmarkSonic(b *testing.B) {
	b.ReportAllocs()
	for i := 0; i < b.N; i++ {
		var w Webhook
		if err := sonic.Unmarshal(payload, &w); err != nil {
			b.Fatal(err)
		}
	}
}

Run with go test -bench=. -benchmem -benchtime=5s on Go 1.22, x86-64 Linux:

Sonic wins decisively on x86-64 because it generates assembly tuned to the payload shape at first call, then memoizes the dispatch. The catch is platform support: sonic falls back to a slower reflection path on ARM64 (Apple Silicon, Graviton), and historically had rough edges around json.RawMessage and custom UnmarshalJSON methods. Verify behaviour against your fixture set before swapping.

Jsoniter is the safer middle ground — it ships a 100% drop-in replacement under ConfigCompatibleWithStandardLibrary and works identically on every platform Go targets. The 2x speedup is free once you swap the import.

The decision rule: stay on encoding/json until profiling shows JSON in your top three CPU consumers. Then move to jsoniter for portable wins, and reach for sonic only when the throughput target justifies the platform-specific tail risk. Always re-run the benchmarks against your real payload — synthetic shapes lie, and the relative ordering can flip when nested arrays dominate string fields.

Summary

The 3 AM webhook incident: add json.RawMessage for payment_method_details, type-switch the parsed value, and use safe helpers. Result: handler stops crashing on unknown schemas, logs the new structure, processes correctly — 10 lines, no struct changes.

The core trade: structs for schemas you control, json.RawMessage for envelopes with variable payloads, map[string]any with safe helpers when nothing is guaranteed.

Frequently Asked Questions

Keep Reading

• Go Error Handling: errors.Is, errors.As, Wrapping, and Custom Types — Turn JSON parsing failures into structured errors with context
• Production-Grade Go API Design — Structured JSON responses and error middleware for the HTTP layer
• REST vs gRPC vs GraphQL: A Production Decision Guide — Choosing the right protocol when you control both sides
• Idempotency Patterns in Distributed Systems — Webhook handlers parsing dynamic JSON also need idempotency keys to survive retries
• Event-Driven Microservices in Go: Kafka, Sagas, and the Outbox Pattern — Schema evolution at the message-bus boundary uses the same RawMessage / dynamic-typing techniques