Microservices Architecture: From Monolith to Production-Ready Services

Most teams that adopt microservices do it too early. They split a 20-endpoint application into 12 services before they have 12 engineers, then spend the next year debugging network failures and deployment pipelines slower than the monolith ever was.

The monolith only becomes a problem when specific signals appear: deploys take 45 minutes, two teams block each other in the codebase, or you need to scale search independently because it consumes 10x the CPU.

graph TD
    Start[Considering microservices?] --> Q1{Hit a concrete<br/>extraction signal?}
    Q1 -->|No| Mono[Stay modular monolith ✓]
    Q1 -->|Yes — which?| Sig{Signal type}
    Sig -->|Scaling imbalance| Ext[Extract that domain]
    Sig -->|Deploy bottleneck<br/>team blocking| Ext
    Sig -->|Tech mismatch<br/>Python ML, GPU, etc.| Ext
    Sig -->|Fault isolation<br/>blast radius| Ext
    Ext --> Test{Boundary clean<br/>inside monolith?}
    Test -->|No, hidden coupling| Fix[Fix coupling first<br/>still inside monolith]
    Fix --> Test
    Test -->|Yes| Net[Lift to network:<br/>gRPC behind feature flag]
    Net --> Resil[Add resilience trio:<br/>saga + outbox + circuit breaker]
    Resil --> Done[Production-ready service]
    style Mono fill:#efe
    style Done fill:#efe
    style Fix fill:#fee

The diagram is the extraction decision tree: don't move to the network until you've proved the boundary is clean inside a single deployment. Most failed microservices migrations are teams that skipped the "fix coupling inside the monolith" step — extracting to gRPC just made the coupling harder to see.

The modular monolith: boundaries without the network

Before extracting services, prove that your boundaries are correct inside a single deployment. A modular monolith organizes code into domain modules that communicate through explicit interfaces, not direct database access. This step eliminates the biggest class of failed microservices migrations: teams that extract services only to discover they're tightly coupled and still change together.

// Each module exposes an interface — not its internals
public interface OrderModule {
    OrderDTO createOrder(CreateOrderCommand cmd);
    OrderDTO getOrder(UUID orderId);
}
 
public interface InventoryModule {
    boolean reserveStock(UUID sku, int quantity);
    void releaseStock(UUID sku, int quantity);
}
 
// Modules interact through interfaces, never through shared tables
@Service
class OrderModuleImpl implements OrderModule {
    private final InventoryModule inventory; // injected interface
    private final OrderRepository orders;    // private to this module
 
    public OrderDTO createOrder(CreateOrderCommand cmd) {
        if (!inventory.reserveStock(cmd.sku(), cmd.quantity())) {
            throw new InsufficientStockException(cmd.sku());
        }
        return OrderDTO.from(orders.save(Order.from(cmd)));
    }
}

The critical rule: each module owns its database tables. The OrderModuleImpl accesses the orders table. The InventoryModule accesses the inventory table. No module reads another module's tables directly. This enforcement inside a monolith is the whole point — you learn if your boundaries actually work before paying the operational cost of running separate services.

If you cannot enforce this discipline inside a monolith, extracting to separate services will not fix it. The coupling will just be harder to see, slower to debug, and more expensive to operate.

Extract when signals justify it, not because it's trendy

These are the signals that justify extraction:

If none apply, stay with the monolith. Every service boundary adds network latency, partial failures, distributed tracing overhead, and on-call rotation. A team of 10 with one monolith moves faster than a team of 10 split across 5 services.

Service boundaries: business capabilities, not technical layers

Split by business capability, not technical layer. A "database service" and "API service" must change together for every feature, defeating microservices entirely.

Good boundary test: Can one team develop, test, deploy, and run this service without coordinating with other teams for 80%+ of changes? If no, you have a distributed monolith.

Data duplication across services is not a bug — it's the price of independent availability. When CustomerService changes an address, it publishes an event. OrderService ignores it (keeps historical snapshot); ShippingService consumes it for future deliveries. Result: CustomerService can be down without affecting orders and shipments.

graph TD
    GW["API Gateway"] -->|gRPC| CS["CustomerService<br/>owns: profiles, addresses"]
    GW -->|gRPC| OS["OrderService<br/>owns: carts, orders, payments"]
    GW -->|gRPC| SS["ShippingService<br/>owns: shipments, tracking"]

    CS -->|"address.changed event"| Kafka["Kafka"]
    OS -->|"order.placed event"| Kafka
    Kafka -->|consumes| SS
    Kafka -->|consumes| OS

    CS --- CDB[(Customer DB)]
    OS --- ODB[(Order DB)]
    SS --- SDB[(Shipping DB)]

Communication patterns: sync vs. async

Every inter-service call is synchronous (caller waits) or asynchronous (caller publishes and moves on).

For internal calls both under your control with < 100ms latency, gRPC is the default. Schema as contract, code generation, low enough latency that blocking doesn't cascade. HTTP/REST works but gRPC's binary framing and HTTP/2 multiplexing are safer.

// Inventory service — gRPC server
func (s *InventoryServer) ReserveStock(
    ctx context.Context, req *pb.ReserveRequest,
) (*pb.ReserveResponse, error) {
    err := s.store.Reserve(ctx, req.Sku, int(req.Quantity))
    if err != nil {
        if errors.Is(err, ErrInsufficientStock) {
            return &pb.ReserveResponse{Success: false}, nil
        }
        return nil, status.Errorf(codes.Internal, "reserve: %v", err)
    }
    return &pb.ReserveResponse{
        Success:       true,
        ReservationId: uuid.New().String(),
    }, nil
}

When the caller does not need an immediate response — for notifications, data replication, or workflows that span multiple services — publish an event. The producing service does not know or care who consumes it. New consumers can be added without modifying the producer.

// Order service publishes an event after creating an order
type OrderCreatedEvent struct {
    OrderID    string    `json:"order_id"`
    CustomerID string    `json:"customer_id"`
    Total      int64     `json:"total_cents"`
    CreatedAt  time.Time `json:"created_at"`
}
 
func (s *OrderService) CreateOrder(ctx context.Context, cmd CreateOrderCmd) error {
    order := Order{ID: uuid.New().String()}
 
    // Write order + outbox event in the same DB transaction
    return s.repo.WithTx(ctx, func(tx *sql.Tx) error {
        if err := s.repo.InsertOrder(ctx, tx, &order); err != nil {
            return err
        }
        return s.repo.InsertOutboxEvent(ctx, tx, OutboxEvent{
            AggregateID: order.ID,
            EventType:   "order.created",
            Payload:     OrderCreatedEvent{OrderID: order.ID},
        })
    })
}

The critical constraint: never do synchronous-only across services (you'll cascade failures when any service is slow) and never do asynchronous-only for queries (the client still needs the data).

Saga + outbox: distributed transactions you can actually debug

Two patterns carry the distributed-transaction story. Sagas break a multi-service operation ("create order, reserve inventory, charge payment") into steps, each with a compensating action that undoes it on downstream failure. Outbox solves the dual-write problem by writing the event into the same database transaction as the state change; a separate relay polls the outbox table (using FOR UPDATE SKIP LOCKED for concurrent relay instances) and publishes to Kafka.

The minimal outbox schema + relay-polling loop:

-- Outbox table — written in the same TX as business state
CREATE TABLE outbox_events (
    id          BIGSERIAL PRIMARY KEY,
    aggregate   TEXT NOT NULL,           -- e.g. 'order'
    aggregate_id TEXT NOT NULL,           -- e.g. order UUID
    event_type  TEXT NOT NULL,           -- e.g. 'OrderCreated'
    payload     JSONB NOT NULL,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at TIMESTAMPTZ           -- NULL until relay publishes
);
CREATE INDEX outbox_unpublished_idx
    ON outbox_events (id) WHERE published_at IS NULL;

// Relay loop — runs on multiple replicas concurrently;
// FOR UPDATE SKIP LOCKED makes parallel relays safe.
func (r *Relay) tick(ctx context.Context) error {
    return pgx.BeginTxFunc(ctx, r.db, pgx.TxOptions{}, func(tx pgx.Tx) error {
        rows, err := tx.Query(ctx, `
            SELECT id, event_type, payload FROM outbox_events
            WHERE published_at IS NULL
            ORDER BY id
            LIMIT 100
            FOR UPDATE SKIP LOCKED
        `)
        if err != nil { return err }
        defer rows.Close()
 
        for rows.Next() {
            var id int64; var typ string; var payload []byte
            if err := rows.Scan(&id, &typ, &payload); err != nil { return err }
            if err := r.kafka.Publish(ctx, typ, payload); err != nil { return err }
            if _, err := tx.Exec(ctx, `UPDATE outbox_events SET published_at = now() WHERE id = $1`, id); err != nil {
                return err
            }
        }
        return nil
    })
}

Two production rules visible in the code: (1) SKIP LOCKED lets multiple relay replicas run safely without a leader-election dance; (2) the publish-then-mark-published order means a crash between Kafka publish and DB UPDATE results in a duplicate publish — which is fine because consumers are idempotent.

The two compose: your service applies business state + writes to the outbox in one transaction, the relay publishes the event, and the saga orchestrator reacts. Never a state where the state changed but the event didn't fire (or vice versa). Two-phase commit (2PC) is a non-starter across services — it blocks resources and fails catastrophically when the coordinator crashes.

For the full Go orchestrator skeleton, compensation-on-failure code, outbox SQL schema, and relay-polling loop, see Event-Driven Microservices in Go: Kafka, Sagas, and the Outbox Pattern. The non-negotiables for sagas:

• Persist saga state to a database. An in-memory saga orchestrator that crashes mid-flight leaves orphaned work (reserved inventory, pending charges) with no recovery path.
• Retry compensations with exponential backoff, dead-letter after N attempts, and alert a human. Compensations fail too.
• Idempotent consumers are mandatory. At-least-once delivery + relay retries means every consumer sees duplicates.

Resilience: fail fast, isolate blast radius

Every cross-service call needs three patterns: circuit breaker (fail fast when a dependency is down rather than waiting on a timeout), bulkhead (separate connection pools per downstream so one slow service cannot exhaust all your workers), and timeouts (context deadline + per-call ceiling)^{[Beyer et al., 2016]}. Together they prevent a single slow dependency from cascading into every caller. Only retry idempotent operations — retrying CreateOrder without an idempotency key^{[Stripe idempotency]} duplicates the order.

Implementation details, the race condition most circuit-breaker code gets wrong (read-lock → write-lock transition), and the bulkhead semaphore pattern in Go are in Building Resilient Distributed Systems with Go: Circuit Breakers, Bulkheads, and Timeout Patterns.

For production implementations, field patterns, and configurations tuned to your latencies, see Building Resilient Distributed Systems with Go.

API Gateway and service mesh patterns

An API gateway sits between external clients and internal services, handling cross-cutting concerns: authentication, rate limiting, request routing. Keep it thin — route traffic, verify tokens, enforce limits. Do not aggregate responses in the gateway (a slow service will slow all requests). Use a Backend-for-Frontend (BFF) service for aggregation — it's a regular microservice that stitches responses together.

Production gotchas

Saga state in database. Persist saga state and run background retries. In-memory state is lost on crash.

Idempotency keys. Payment service maps (customer_id, key) → result; returns cached result on replay to prevent duplicate charges.

Probe window < breaker window. If probes run every 30s but breakers trip after 10s, service discovery reports healthy while clients hit open breakers — cascading failures.

Graceful degradation. If payment is down, reject if the client sees the error now; queue pending if you retry in background.

Pre-extraction checklist

• Boundaries proven in monolith (gRPC behind feature flag, nothing breaks)
• One concrete extraction signal (scaling, autonomy, fault isolation, deploy bottleneck)
• Sagas persist state to database; background job retries; dead-letter after N failures
• Outbox relay runs; events in same txn as state change
• Circuit breakers, bulkheads, timeouts all configured
• Distributed tracing wired (trace ID propagated, business context on spans)
• Health checks meaningful (not just process liveness)
• Each service owns its tables (no cross-service joins)

When to use each model

The safest path: start monolith → modular monolith → extract one service → repeat only when signals justify. Some modules never leave the monolith, and that's fine. A "residual monolith" is cheaper than 3 services with 3 CI/CD pipelines, 3 health dashboards, and 3 on-call rotations.

Teams that succeed can explain exactly why each boundary exists. If the answer is "best practice," it was premature. Extract only after proving boundaries work inside a monolith, setting up distributed tracing, and hiring engineers who understand resilience patterns.

The four cross-cutting code patterns every microservice fleet shares

Distributed tracing context propagation — the W3C traceparent header, set once at the edge and threaded through every downstream call. Without it, "the request was slow" becomes unfalsifiable in a 12-service mesh:

import (
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
)
 
// Wrap the inbound HTTP server: extracts traceparent from headers,
// starts a span named after the route, attaches it to ctx for handlers.
func TraceMiddleware(svc string) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return otelhttp.NewHandler(next, svc,
            otelhttp.WithTracerProvider(otel.GetTracerProvider()))
    }
}
 
// Wrap the outbound HTTP client: injects traceparent into outgoing requests.
func TracedClient(base *http.Transport) *http.Client {
    return &http.Client{Transport: otelhttp.NewTransport(base)}
}

Idempotent event consumer — the dedupe table the saga article mentions but rarely shows. Without it, a Kafka rebalance during deploy reprocesses up to one batch worth of events:

// ProcessOnce runs handler exactly once per (topic, key, offset) combination.
// The dedupe row is inserted in the SAME transaction as the side effect,
// so the handler's writes and the dedupe entry commit atomically.
func ProcessOnce(ctx context.Context, db *sql.DB, e Event, handler func(*sql.Tx) error) error {
    tx, err := db.BeginTx(ctx, nil)
    if err != nil { return err }
    defer tx.Rollback()
 
    res, err := tx.ExecContext(ctx, `
        INSERT INTO event_dedupe (topic, partition_key, offset_id, processed_at)
        VALUES ($1, $2, $3, now())
        ON CONFLICT DO NOTHING
    `, e.Topic, e.Key, e.Offset)
    if err != nil { return err }
 
    n, _ := res.RowsAffected()
    if n == 0 {
        return nil // already processed; skip handler, commit empty tx is fine
    }
    if err := handler(tx); err != nil { return err }
    return tx.Commit()
}

A health endpoint that knows the difference between liveness and readiness — the most common production bug is one endpoint that mixes them, causing a flaky downstream to restart your healthy pod:

// /healthz — process-only check. Does NOT verify dependencies. The Kubelet
// uses this to decide if the process is wedged and needs restart.
func Healthz(w http.ResponseWriter, _ *http.Request) {
    w.WriteHeader(http.StatusOK); _, _ = w.Write([]byte("ok"))
}
 
// /readyz — checks every critical dependency. Failure removes the pod from
// the Service endpoints (so traffic stops flowing in) but does NOT restart
// the process. Critical: each check has its own short timeout.
func Readyz(deps ...func(context.Context) error) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
        defer cancel()
        for _, dep := range deps {
            if err := dep(ctx); err != nil {
                http.Error(w, err.Error(), http.StatusServiceUnavailable)
                return
            }
        }
        w.WriteHeader(http.StatusOK)
    }
}

A service-mesh sidecar config that wires it all together — Istio's DestinationRule setting per-service circuit-breaker thresholds, paired with RetryPolicy that won't amplify cascades:

# istio-destinationrule.yaml — outbound resilience config for one service
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: orders-service
spec:
  host: orders.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        http2MaxRequests: 100
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 5      # eject host after 5 consecutive 5xx
      interval: 30s
      baseEjectionTime: 30s        # cool-off before re-adding
      maxEjectionPercent: 50       # never eject >50% of upstream pods
    retryPolicy:
      attempts: 3
      perTryTimeout: 1s
      retryOn: "5xx,reset,connect-failure"

The four pieces compose: tracing tells you where time went, dedupe makes consumers safe to replay, separated probes prevent restart cascades, and the destination rule ensures one slow upstream can't take everything down.

Frequently Asked Questions

Keep Reading

• Event-Driven Microservices in Go: Kafka, Sagas, and the Outbox Pattern — The full implementation of the event-driven patterns introduced here: saga orchestration, outbox relay, and idempotent consumers
• REST vs gRPC vs GraphQL: A Production Decision Guide — How to choose the right protocol for each communication pattern in your microservices architecture
• Building Resilient Distributed Systems with Go — Circuit breakers, bulkheads, and graceful degradation patterns that keep microservices running when dependencies fail