gRPC and Protobuf: Building Service-to-Service APIs

The classic gRPC-behind-an-NLB incident. An order service starts throwing DEADLINE_EXCEEDED errors at significant rate. Most inventory pods sit near-idle, one runs hot. All traffic is pinned to a single replica.

Root cause: an AWS Network Load Balancer in front of gRPC. NLBs route TCP connections, not HTTP/2 streams^{[RFC 9113, 2022]}. Every gRPC client opens one persistent connection and multiplexes all RPCs over it^{[gRPC docs]}. The NLB picks a backend for that connection and locks in. Multiple calls from the same client all hit the same pod.

Switching to Envoy at L7 fixes it. The lesson: getting the protobuf schema right and the basic calls working is the easy part. Load balancing that understands HTTP/2, deadline propagation, health checks, and field evolution all require separate thought. We've debugged this on multiple gRPC-on-Kubernetes deployments.

This article covers the essentials: how Protobuf binary framing works, building unary and streaming RPCs in Go, and when gRPC wins over REST.

graph LR
    subgraph Client["gRPC Client"]
        C[App] --> Stub[Generated stub]
        Stub -->|RPC 1| Conn[(One HTTP/2 connection<br/>many multiplexed streams)]
        Stub -->|RPC 2| Conn
        Stub -->|RPC N| Conn
    end
    Conn -->|TCP connection| LB{Load balancer}
    LB -->|L4 NLB:<br/>connection-pinned| Bad[All RPCs to one pod 🔥]
    LB -->|L7 Envoy / Istio:<br/>per-stream routing| Good[Distributed across pods ✓]
    style Bad fill:#fee
    style Good fill:#efe
    style Conn fill:#eef

The diagram is the load-balancing lesson in one picture: gRPC's HTTP/2 multiplexing is a feature in the protocol layer and a bug at the L4 routing layer. Every gRPC-in-K8s production incident eventually traces back to which line of this diagram is wrong.

The Decision: gRPC vs REST

Choose gRPC when you control both client and server, need streaming, or require payload efficiency. Use REST for public APIs and unfamiliar teams.

Protobuf Wire Format

A unary gRPC call from byte to handler in one picture — the wire path that JSON-over-HTTP/1.1 simply does not have:

sequenceDiagram
    participant C as Go client<br/>generated stub
    participant TLS as HTTP/2 + TLS<br/>single connection
    participant S as Go server<br/>generated stub
    participant H as Handler
    Note over C,S: Connection setup once,<br/>then every call multiplexed
    C->>C: Marshal protobuf<br/>field-tag + varint + bytes
    C->>TLS: HEADERS frame<br/>:method=POST, :path=/orders.OrderService/CreateOrder
    C->>TLS: DATA frame<br/>compressed protobuf payload
    TLS->>S: HEADERS + DATA
    S->>S: Decode frame, route by :path
    S->>S: Unmarshal protobuf into request struct
    S->>H: handler(ctx, req)
    H-->>S: response struct
    S->>S: Marshal protobuf
    S->>TLS: HEADERS frame<br/>:status=200, content-type=application/grpc
    S->>TLS: DATA frame
    S->>TLS: TRAILERS<br/>grpc-status=0
    TLS->>C: response stream
    C->>C: Unmarshal into response struct

Protobuf doesn't send field names — only numeric tags compiled into stubs. JSON sends names on every request:

{"user_id": "abc-123", "email": "user@example.com", "created_at": 1710000000}

Protobuf binary:

[tag=1][type=2][len=7]["abc-123"]
[tag=3][type=2][len=16]["user@example.com"]
[tag=4][type=0][varint=1710000000]

Wire types: 0=varint, 1=64-bit, 2=length-delimited, 5=32-bit. Field tags are permanent — once tag 3 means email, it always does. Removing a field? Mark it reserved. Renaming? Safe (tags don't care). Adding? Use a new tag number.

Defining Services and Code Generation

The .proto file defines the contract. Everything else — client stubs, server interfaces, message types — is code-generated.

syntax = "proto3";
package commerce.inventory.v1;
 
option go_package = "github.com/yourco/inventory/gen/go/commerce/inventory/v1;inventorypb";
 
service InventoryService {
  rpc GetItem(GetItemRequest) returns (Item);           // unary
  rpc ListItems(ListItemsRequest) returns (stream Item); // server-streaming
  rpc RecordEvents(stream InventoryEvent) returns (EventSummary); // client-streaming
  rpc SyncStock(stream StockUpdate) returns (stream StockConfirmation); // bidirectional
}
 
message Item {
  string id = 1;
  string sku = 2;
  string name = 3;
  int64 quantity = 4;
  double price_usd = 5;
  reserved 6;      // removed field — never reuse this tag
}
 
message GetItemRequest {
  string id = 1;
}
 
message ListItemsRequest {
  int32 page_size = 1;
  string page_token = 2;
}
 
message InventoryEvent {
  string item_id = 1;
  int64 delta = 2;
  int64 timestamp_ms = 3;
}
 
message EventSummary {
  int32 events_processed = 1;
  int32 events_rejected = 2;
}
 
message StockUpdate {
  string item_id = 1;
  int64 new_quantity = 2;
}
 
message StockConfirmation {
  string item_id = 1;
  bool accepted = 2;
}

Generate Go code:

protoc \
  --go_out=. --go_opt=paths=source_relative \
  --go-grpc_out=. --go-grpc_opt=paths=source_relative \
  inventory/v1/inventory.proto

Output: inventory.pb.go (message types) and inventory_grpc.pb.go (client/server interfaces).

Building the gRPC Server

The generated interface enforces the contract at compile time. Embed UnimplementedInventoryServiceServer to safely handle future proto additions.

Unary RPC: Single Request, Single Response

type InventoryServer struct {
    pb.UnimplementedInventoryServiceServer
    db *pgxpool.Pool
}
 
func (s *InventoryServer) GetItem(ctx context.Context, req *pb.GetItemRequest) (*pb.Item, error) {
    if req.GetId() == "" {
        return nil, status.Error(codes.InvalidArgument, "id is required")
    }
 
    var item pb.Item
    err := s.db.QueryRow(ctx,
        `SELECT id, sku, name, quantity, price_usd FROM items WHERE id = $1`,
        req.GetId(),
    ).Scan(&item.Id, &item.Sku, &item.Name, &item.Quantity, &item.PriceUsd)
 
    if errors.Is(err, pgx.ErrNoRows) {
        return nil, status.Errorf(codes.NotFound, "item %q not found", req.GetId())
    }
    if err != nil {
        return nil, status.Error(codes.Internal, "failed to fetch item")
    }
 
    return &item, nil
}

Server-Streaming, Client-Streaming, and Bidirectional RPCs

// Server-streaming: send multiple responses
func (s *InventoryServer) ListItems(req *pb.ListItemsRequest, stream pb.InventoryService_ListItemsServer) error {
    rows, err := s.db.Query(stream.Context(), "SELECT ... FROM items")
    if err != nil {
        return status.Error(codes.Internal, "query failed")
    }
    defer rows.Close()
 
    for rows.Next() {
        if err := stream.Context().Err(); err != nil {
            return status.FromContextError(err).Err()
        }
        var item pb.Item
        if err := rows.Scan(&item.Id, &item.Sku, &item.Name, &item.Quantity, &item.PriceUsd); err != nil {
            return status.Error(codes.Internal, "scan failed")
        }
        if err := stream.Send(&item); err != nil {
            return err
        }
    }
    return rows.Err()
}
 
// Client-streaming: receive batch of requests, send one response
func (s *InventoryServer) RecordEvents(stream pb.InventoryService_RecordEventsServer) error {
    var processed, rejected int32
    for {
        event, err := stream.Recv()
        if err == io.EOF {
            break
        }
        if err != nil {
            return err
        }
        if event.GetDelta() == 0 {
            rejected++
            continue
        }
        processed++
    }
    return stream.SendAndClose(&pb.EventSummary{
        EventsProcessed: processed,
        EventsRejected:  rejected,
    })
}
 
// Bidirectional: echo confirmation for each update
func (s *InventoryServer) SyncStock(stream pb.InventoryService_SyncStockServer) error {
    for {
        update, err := stream.Recv()
        if err == io.EOF {
            return nil
        }
        if err != nil {
            return err
        }
        if err := stream.Send(&pb.StockConfirmation{
            ItemId:   update.GetItemId(),
            Accepted: true,
        }); err != nil {
            return err
        }
    }
}

Building a Server with Interceptors

Interceptors are gRPC middleware — the place for logging, auth, metrics, and tracing:

func loggingInterceptor(logger *slog.Logger) grpc.UnaryServerInterceptor {
    return func(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
        start := time.Now()
        resp, err := handler(ctx, req)
        logger.InfoContext(ctx, "rpc",
            slog.String("method", info.FullMethod),
            slog.Duration("duration", time.Since(start)),
            slog.String("status", status.Code(err).String()),
        )
        return resp, err
    }
}
 
srv := grpc.NewServer(
    grpc.ChainUnaryInterceptor(loggingInterceptor(logger)),
    grpc.ChainStreamInterceptor(streamLoggingInterceptor(logger)),
)
pb.RegisterInventoryServiceServer(srv, &InventoryServer{db: pool})

Production Checklist

Before shipping gRPC to production:

• Load balancing: Use Envoy/Istio (L7) not L4 load balancers. gRPC multiplexes all RPCs over one persistent HTTP/2 connection per client. An L4 balancer picks a backend for that connection and sticks with it — causing traffic skew. L7 load balancers route individual HTTP/2 streams, distributing load fairly.

• TLS: Always enable TLS. Plaintext gRPC exposes credentials and metadata in the clear.

• Health checking: Register grpc.health.v1.Health so Kubernetes readiness probes and Envoy can verify each service is alive and ready to receive traffic. Return NOT_SERVING during graceful shutdown to let the load balancer drain in-flight requests.

• Deadlines: Always derive deadlines from the incoming context, never from context.Background(). When you receive a 5-second deadline and spend 4 seconds, propagate only 1 second to downstream calls. Without this, timeouts don't cascade — slow services can starve fast ones.

• Status codes: Use semantic codes. InvalidArgument tells clients "don't retry" (bad input). Unavailable means "maybe later" (transient failure). Internal should be rare. The status code drives retry logic on the client side.

• Metadata: Propagate request IDs and trace context via gRPC metadata (equivalent to HTTP headers) in interceptors. This makes cross-service traces joinable without per-handler code.

• Retry policy: Define in service config at connection time, not in application code. Only retry UNAVAILABLE and RESOURCE_EXHAUSTED. Don't retry DeadlineExceeded — you'd burn the remaining deadline budget.

• Field evolution: Protobuf field tags are permanent. Mark removed fields reserved so they're never reused. Renaming fields is safe (tags don't care). Adding new fields uses new tag numbers. Incompatible changes require a new service version (v2, v3).

• mTLS: In Kubernetes, use Istio or Linkerd to inject mutual TLS automatically via sidecar proxies. No application code changes needed.

The retry policy that won't amplify cascades

Most teams hand-roll retries in application code, then debug why a downstream outage doubles their inbound RPS. The fix is to declare the policy in the service config so the gRPC client library enforces it once and the retry budget caps blast radius: ^{[gRPC docs]}

{
  "methodConfig": [
    {
      "name": [
        { "service": "orders.OrderService", "method": "CreateOrder" }
      ],
      "retryPolicy": {
        "maxAttempts": 4,
        "initialBackoff": "0.1s",
        "maxBackoff": "2s",
        "backoffMultiplier": 2.0,
        "retryableStatusCodes": [
          "UNAVAILABLE",
          "RESOURCE_EXHAUSTED"
        ]
      },
      "timeout": "5s"
    },
    {
      "name": [
        { "service": "orders.OrderService", "method": "GetOrder" }
      ],
      "retryPolicy": {
        "maxAttempts": 3,
        "initialBackoff": "0.05s",
        "maxBackoff": "1s",
        "backoffMultiplier": 2.0,
        "retryableStatusCodes": ["UNAVAILABLE"]
      },
      "timeout": "2s"
    }
  ],
  "retryThrottling": {
    "maxTokens": 100,
    "tokenRatio": 0.1
  }
}

Three details that matter in production:

1. Per-method policy. CreateOrder (writes) gets fewer attempts and a longer timeout than GetOrder (reads). Mutation retries cost more than read retries, so they should be cheaper to give up on.
2. retryThrottling block. This is the cascade circuit-breaker. Every successful call adds a token (capped at 100), every retry consumes 0.1 tokens. When the bucket empties, the client stops retrying entirely until the dependency recovers — preventing the runaway-amplification failure mode where a degraded backend gets hammered by retry traffic.
3. UNAVAILABLE and RESOURCE_EXHAUSTED only. Never retry DeadlineExceeded (you'd extend the timeout chain), never retry Internal (likely a real bug), never retry InvalidArgument (will fail the same way every time). The status-code semantics matter — that's why hand-rolled retries get this wrong.

Load this via grpc.WithDefaultServiceConfig(string(jsonBytes)) on the dial options; the library handles the rest.

Building a gRPC Client

func NewInventoryClient(addr string, tlsConfig *tls.Config) (pb.InventoryServiceClient, error) {
    conn, err := grpc.NewClient(addr,
        grpc.WithTransportCredentials(credentials.NewTLS(tlsConfig)),
        grpc.WithDefaultServiceConfig(`{
            "methodConfig": [{
                "name": [{"service": "commerce.inventory.v1.InventoryService"}],
                "retryPolicy": {
                    "maxAttempts": 3,
                    "initialBackoff": "0.1s",
                    "maxBackoff": "1s",
                    "backoffMultiplier": 2.0,
                    "retryableStatusCodes": ["UNAVAILABLE"]
                }
            }]
        }`),
    )
    if err != nil {
        return nil, fmt.Errorf("dial %s: %w", addr, err)
    }
    return pb.NewInventoryServiceClient(conn), nil
}
 
// Unary call
client := pb.NewInventoryServiceClient(conn)
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
item, err := client.GetItem(ctx, &pb.GetItemRequest{Id: "item-123"})
 
// Server-streaming call
stream, err := client.ListItems(ctx, &pb.ListItemsRequest{WarehouseId: "wh-1"})
for {
    item, err := stream.Recv()
    if err == io.EOF {
        break
    }
    if err != nil {
        return err
    }
    // process item
}

Bidirectional Streaming: A Real-World Chat Service

Bidirectional streams are the pattern most teams underuse. They suit any flow where both sides need to push data independently — chat, collaborative editing, market data feeds, and live dashboards. Unlike client- or server-streaming, neither side waits its turn; the HTTP/2 connection carries two independent message streams over the same logical RPC. ^{[gRPC docs]}

A chat-app example shows the shape clearly. The service definition declares one RPC where both request and response are streams:

syntax = "proto3";
package chat.v1;
 
option go_package = "github.com/yourco/chat/gen/go/chat/v1;chatpb";
 
service ChatService {
  rpc Connect(stream ClientMessage) returns (stream ServerMessage);
}
 
message ClientMessage {
  oneof payload {
    JoinRoom join = 1;
    LeaveRoom leave = 2;
    SendMessage send = 3;
    Heartbeat heartbeat = 4;
  }
}
 
message ServerMessage {
  oneof payload {
    RoomJoined joined = 1;
    MessagePosted posted = 2;
    UserPresence presence = 3;
    ServerError error = 4;
  }
  int64 server_time_ms = 10;
}
 
message JoinRoom { string room_id = 1; }
message LeaveRoom { string room_id = 1; }
message SendMessage { string room_id = 1; string body = 2; }
message Heartbeat { int64 client_time_ms = 1; }
message RoomJoined { string room_id = 1; repeated string member_ids = 2; }
message MessagePosted { string room_id = 1; string author_id = 2; string body = 3; }
message UserPresence { string user_id = 1; bool online = 2; }
message ServerError { string code = 1; string message = 2; }

The oneof block is the wire-level enum: at most one variant is set on any given message, the wire decoder enforces it, and unrecognised variants are dropped instead of crashing old clients. That property is what makes oneof evolution-safe — a v2 server can introduce a new variant tag without bricking v1 clients that have never heard of it.

The server implementation needs to handle two independent goroutines: one reading from the client, one writing fan-out from a room broker. Doing this correctly requires keeping a single writer goroutine per stream so concurrent sends don't interleave protobuf frames:

func (s *ChatServer) Connect(stream chatpb.ChatService_ConnectServer) error {
    ctx := stream.Context()
    userID, err := userIDFromMetadata(ctx)
    if err != nil {
        return status.Error(codes.Unauthenticated, "missing user identity")
    }
 
    out := make(chan *chatpb.ServerMessage, 32)
    sub := s.broker.Subscribe(userID, out)
    defer s.broker.Unsubscribe(sub)
 
    // Single writer — the only goroutine allowed to call stream.Send.
    var sendErr atomic.Value
    go func() {
        for msg := range out {
            msg.ServerTimeMs = time.Now().UnixMilli()
            if err := stream.Send(msg); err != nil {
                sendErr.Store(err)
                return
            }
        }
    }()
 
    // Reader loop — translates client messages into broker actions.
    for {
        if err, _ := sendErr.Load().(error); err != nil {
            return err
        }
        msg, err := stream.Recv()
        if err == io.EOF {
            close(out)
            return nil
        }
        if err != nil {
            close(out)
            return err
        }
        switch p := msg.Payload.(type) {
        case *chatpb.ClientMessage_Join:
            s.broker.Join(userID, p.Join.RoomId)
        case *chatpb.ClientMessage_Send:
            s.broker.Publish(p.Send.RoomId, userID, p.Send.Body)
        case *chatpb.ClientMessage_Heartbeat:
            // No-op: keepalive is handled by HTTP/2 PING frames.
        }
    }
}

Three production details that bite teams new to bidirectional streams. First, never call stream.Send from multiple goroutines — gRPC-Go's stream is not safe for concurrent sends, and frame interleaving will surface as cryptic deserialization errors on the client. Funnel writes through a single channel. Second, bound the per-connection buffer (the channel capacity above is 32). An unbounded buffer turns a slow consumer into a server-side memory leak. Third, enable HTTP/2 keepalives on the server with keepalive.ServerParameters{Time: 30 * time.Second, Timeout: 10 * time.Second} — without it, idle connections behind a stateful firewall get silently dropped and your chat server thinks clients are still subscribed.

Protobuf Evolution Rules

Protobuf is designed for forward and backward compatibility, but only if you follow a small number of rules religiously. The rules trace back to one principle: field tag numbers are the only durable identity on the wire. Names, types, and ordering are convenience for humans; the wire only carries tags.

Reserve removed fields. When you delete a field from a message, the tag number must never be reused — old clients still send it, and a new field at the same tag would silently corrupt their data. The compiler enforces this only if you tell it:

message Item {
  string id = 1;
  string sku = 2;
  string name = 3;
  int64 quantity = 4;
  double price_usd = 5;
 
  reserved 6, 7, 12 to 15;
  reserved "legacy_color", "discontinued_at";
}

Both numeric ranges and field names go into the reserved list. The name reservations protect against the subtle bug where someone resurrects an old field name with a fresh tag number — semantically wrong, syntactically valid without reserved. Generators warn when you violate either.

Migrate oneof carefully. oneof groups are a special case: the variants share semantic ground but each uses a distinct tag. Adding a new variant is safe; removing one means the tag must be reserved like any other field. Promoting a singular field into a oneof requires the existing tag to become a member of the new group — never assign it a new number. Demoting a oneof member to a top-level field works the same way in reverse; the tag travels with the field.

What counts as a breaking change versus an additive change is the daily judgment call. The following table captures the rules teams forget under deadline pressure:

The discipline is simple: every breaking change goes into a new package version (commerce.inventory.v2) with its own service. Run v1 and v2 side by side until clients migrate, then sunset v1 with a deprecation window measured in months — not days.

Buf: Linting and Breaking-Change Detection

protoc ships the codegen but nothing else. The missing piece — schema linting, dependency management, and breaking-change detection — is what buf (from Buf Build) provides. Adopting it is the difference between proto evolution rules existing as a wiki page and being enforced in CI.

A typical buf.yaml declares the module, lint rules, and breaking-change configuration:

version: v2
modules:
  - path: proto
lint:
  use:
    - STANDARD
  except:
    - PACKAGE_VERSION_SUFFIX
  enum_zero_value_suffix: _UNSPECIFIED
  service_suffix: Service
breaking:
  use:
    - WIRE_JSON
deps:
  - buf.build/googleapis/googleapis

STANDARD enables a curated lint set that flags the common mistakes: missing package version suffix, fields named in camelCase instead of snake_case, enums without an _UNSPECIFIED zero value, services without the Service suffix. The WIRE_JSON breaking-change category checks both binary wire compatibility and JSON serialization compatibility — the latter matters when the same proto is used over gRPC and REST gateways.

The CI pipeline that turns evolution rules into enforcement looks like this:

name: proto-checks
on:
  pull_request:
    paths: ['proto/**']
jobs:
  buf:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: bufbuild/buf-setup-action@v1
      - name: Lint
        run: buf lint
      - name: Format check
        run: buf format -d --exit-code
      - name: Breaking change detection
        run: buf breaking --against '.git#branch=main,subdir=proto'
      - name: Generate code
        run: buf generate
      - name: Verify generated code is committed
        run: git diff --exit-code

The buf breaking --against '.git#branch=main' invocation compares the proto files in the PR against the version on main and fails on any breaking change. This catches the accidents that proto evolution rules were written to prevent: a tag reuse, a field type swap, a removed oneof variant. Pair it with buf format so reviewers stop re-litigating indentation and buf generate so generated code stays in sync with the schema. Many teams also publish their schemas to the Buf Schema Registry so client repositories can pull typed stubs the same way they pull npm or Go modules — but the local CI checks above are the minimum bar before that becomes worthwhile.

Frequently Asked Questions

Keep Reading

• API Architecture: REST, gRPC, and GraphQL — Side-by-side implementations with DataLoader, decision frameworks, and migration strategies between API styles
• Production Go API Design — Middleware chains, request validation, and observability patterns for Go HTTP and gRPC services
• HTTP Protocol Evolution Guide — HTTP/1.1 through HTTP/3 and QUIC: the transport layer that gRPC depends on
• Building Resilient Distributed Systems with Go — Circuit breakers, timeouts, and retries that wrap every gRPC client in production
• Microservices Architecture Patterns — Service-to-service contracts: when gRPC is the right answer and when REST/event-driven beats it