Finding and Fixing a 50,000 Goroutine Leak That Nearly Killed Production

The Symptoms That Everyone Ignored

It started innocently. A developer mentioned the API felt "sluggish" during sprint review. QA reported timeouts were "slightly higher." DevOps noted memory was "trending up but within limits."

Everyone had a piece of the puzzle. Nobody saw the picture.

Here's what we were looking at:

Week 1: 1,200 goroutines, 2.1GB RAM, 250ms p99 latency
Week 2: 3,400 goroutines, 3.8GB RAM, 380ms p99 latency  
Week 3: 8,900 goroutines, 7.2GB RAM, 610ms p99 latency
Week 4: 19,000 goroutines, 14GB RAM, 1.4s p99 latency
Week 5: 34,000 goroutines, 28GB RAM, 8.3s p99 latency
Week 6: 50,847 goroutines, 47GB RAM, 32s p99 latency ← You are here

Classic exponential growth. Classic "someone else's problem."

The Code That Looked Perfectly Fine

The leak was in our WebSocket notification system. Here's the simplified version:

func (s *NotificationService) Subscribe(userID string, ws *websocket.Conn) {
    ctx, cancel := context.WithCancel(context.Background())
    
    sub := &subscription{
        userID: userID,
        ws:     ws,
        cancel: cancel,
    }
    
    s.subscribers[userID] = sub
    
    // Start the message pump
    go s.pumpMessages(ctx, sub)
    
    // Start the heartbeat
    go s.heartbeat(ctx, sub)
}

func (s *NotificationService) pumpMessages(ctx context.Context, sub *subscription) {
    for {
        select {
        case <-ctx.Done():
            return
        case msg := <-sub.messages:
            sub.ws.WriteJSON(msg)  // What could go wrong? Two things, actually.
        }
    }
}

func (s *NotificationService) heartbeat(ctx context.Context, sub *subscription) {
    ticker := time.NewTicker(30 * time.Second)

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            sub.ws.WriteMessage(websocket.PingMessage, nil)  // Bug #4 - see below, not the goroutine leak
        }
    }
}

Looks reasonable, right? Context for cancellation. Cleanup on Done(). This passed code review from three senior engineers.

Enter goleak

After staring at code for 2 hours, I remembered Uber's goleak package. It's a test helper built specifically for catching goroutine leaks, rather than diagnosing them after the fact in production.

Installation took 30 seconds:

import (
    "go.uber.org/goleak"
)

func TestNotificationService_NoLeak(t *testing.T) {
    defer goleak.VerifyNone(t)
    
    // Your test here
    service := NewNotificationService()
    ws := mockWebSocket()
    service.Subscribe("user123", ws)
    
    // Simulate disconnect
    ws.Close()
    
    time.Sleep(100 * time.Millisecond) // Let cleanup happen
}

The test failed immediately:

found unexpected goroutines:
[Goroutine 18 in state select, with NotificationService.pumpMessages on top of the stack:
goroutine 18 [select]:
    service.(*NotificationService).pumpMessages(0xc0001d4000, {0x1038c20, 0xc0001d6000}, 0xc0001d8000)
        /app/notification.go:45 +0x85
    created by service.(*NotificationService).Subscribe
        /app/notification.go:32 +0x1a5

Goroutine 19 in state select, with NotificationService.heartbeat on top of the stack:
goroutine 19 [select]:
    service.(*NotificationService).heartbeat(0xc0001d4000, {0x1038c20, 0xc0001d6000}, 0xc0001d8000)
        /app/notification.go:58 +0x92
]

Two goroutines still running after the WebSocket closed. But why?

The Three Bugs That Created a Perfect Storm

Bug #1: Nobody Called Cancel

func (s *NotificationService) Subscribe(userID string, ws *websocket.Conn) {
    ctx, cancel := context.WithCancel(context.Background())
    
    // We save the subscription...
    sub := &subscription{
        userID: userID,
        ws:     ws,
        cancel: cancel,  // But who calls this?
    }
}

When the WebSocket disconnected, we never called cancel(). The goroutines lived forever, waiting for a context that would never close.

Bug #2: The Heartbeat Ticker Memory Leak

func (s *NotificationService) heartbeat(ctx context.Context, sub *subscription) {
    ticker := time.NewTicker(30 * time.Second)
    // WHERE IS ticker.Stop() ???
    
    for {
        select {
        case <-ctx.Done():
            return  // Ticker still running!
        case <-ticker.C:
            sub.ws.WriteMessage(websocket.PingMessage, nil)
        }
    }
}

In modern Go versions this example is less about GC correctness and more about making goroutine and time-based lifecycles explicit. In older Go versions, this pattern could contribute to real memory and resource leaks.

Bug #3: The Channel That Never Closed

type subscription struct {
    userID   string
    ws       *websocket.Conn
    messages chan Message  // Who closes this?
    cancel   context.CancelFunc
}

Writers kept sending to sub.messages. The channel grew. Memory grew. Pain grew.

Bug #4: A Data Race the Goroutine-Leak Framing Never Caught

This one wasn't in the original write-up, and it's not a goroutine leak at all - it's a separate, real bug sitting in the same two functions. pumpMessages calls sub.ws.WriteJSON(msg). heartbeat calls sub.ws.WriteMessage(websocket.PingMessage, nil). Those are two different goroutines calling write methods on the same *websocket.Conn at the same time - and per gorilla/websocket's own concurrency docs: "Applications are responsible for ensuring that no more than one goroutine calls the write methods (NextWriter, SetWriteDeadline, WriteMessage, WriteJSON, EnableWriteCompression, SetCompressionLevel) concurrently... The Close and WriteControl methods can be called concurrently with all other methods."

WriteMessage is on the "don't call me concurrently" list. WriteControl is explicitly exempted - and a ping is exactly what WriteControl is for. Reproduced against a mock Conn with the real data race detector (not just asserted from the docs) in the gist: hammering WriteJSON and WriteMessage concurrently reliably trips go test -race; the same test with WriteControl instead of WriteMessage stays clean. The fix below uses WriteControl for the ping - fixing the goroutine leak alone would have shipped this bug right back to production.

The Production Debugging Session From Hell

We couldn't just restart production. Too many active users. We needed to find and fix it live.

Step 1: Get a goroutine dump:

curl http://api-server:6060/debug/pprof/goroutine?debug=2 > goroutines.txt

Step 2: Analyze the patterns:

# Count goroutines by function
grep "^goroutine" goroutines.txt | sort | uniq -c | sort -rn

# Results:
# 25,423 NotificationService.pumpMessages
# 25,423 NotificationService.heartbeat
#     12 http.(*conn).serve
#      8 runtime.gcBgMarkWorker
#     ... normal stuff ...

50,846 goroutines in our notification service. We had about 1,000 active WebSocket connections.

If you're on Go 1.27+, you can skip most of the manual counting above. The runtime now ships a goroutineleak profile in runtime/pprof (experimental in 1.26, promoted to a regular profile in 1.27) that groups blocked goroutines by the pattern that's likely leaking them, instead of handing you a flat dump to grep through:

curl http://api-server:6060/debug/pprof/goroutineleak?debug=1 > leaks.txt

It won't replace goleak in tests - that's still the right tool for catching a leak before it ships - but for exactly the kind of live-production hunt in this section, it gets you from "50,846 goroutines, now what" to "here are the stacks that are actually stuck" without the grep-and-guess step.

Step 3: Find the pattern:

// Added emergency diagnostics endpoint
http.HandleFunc("/debug/subscriptions", func(w http.ResponseWriter, r *http.Request) {
    s.mu.Lock()
    defer s.mu.Unlock()
    
    active := 0
    for _, sub := range s.subscribers {
        // Try to ping the connection
        err := sub.ws.WriteControl(websocket.PingMessage, nil, time.Now().Add(time.Second))
        if err == nil {
            active++
        }
    }
    
    fmt.Fprintf(w, "Total subscriptions: %d\n", len(s.subscribers))
    fmt.Fprintf(w, "Active connections: %d\n", active)
    fmt.Fprintf(w, "Leaked goroutines: ~%d\n", (len(s.subscribers) - active) * 2)
})

Result:

Total subscriptions: 25,423
Active connections: 1,047
Leaked goroutines: ~48,752

Bingo. We were keeping subscriptions for dead connections.

The Fix That Saved the Weekend

Here's the fixed version:

func (s *NotificationService) Subscribe(userID string, ws *websocket.Conn) {
    ctx, cancel := context.WithCancel(context.Background())
    
    sub := &subscription{
        userID:   userID,
        ws:       ws,
        messages: make(chan Message, 10),
        cancel:   cancel,
    }
    
    s.mu.Lock()
    s.subscribers[userID] = sub
    s.mu.Unlock()
    
    // Critical: Setup cleanup handler
    ws.SetCloseHandler(func(code int, text string) error {
        s.Unsubscribe(userID)
        return nil
    })
    
    // Start goroutines
    go s.pumpMessages(ctx, sub)
    go s.heartbeat(ctx, sub)
    
    // Monitor the connection
    go s.monitorConnection(ctx, sub)
}

func (s *NotificationService) Unsubscribe(userID string) {
    s.mu.Lock()
    defer s.mu.Unlock()
    
    if sub, exists := s.subscribers[userID]; exists {
        sub.cancel()                    // Stop goroutines
        close(sub.messages)              // Close channel
        delete(s.subscribers, userID)   // Remove reference
    }
}

func (s *NotificationService) monitorConnection(ctx context.Context, sub *subscription) {
    defer s.Unsubscribe(sub.userID)  // Cleanup on exit
    
    for {
        select {
        case <-ctx.Done():
            return
        case <-time.After(1 * time.Minute):
            // Ping to detect broken connections
            if err := sub.ws.WriteControl(
                websocket.PingMessage, 
                nil, 
                time.Now().Add(10*time.Second),
            ); err != nil {
                return  // Connection dead, cleanup will run
            }
        }
    }
}

func (s *NotificationService) heartbeat(ctx context.Context, sub *subscription) {
    ticker := time.NewTicker(30 * time.Second)
    // Good practice to call Stop().
    // Starting from Go 1.23, unused tickers can be GC’d without Stop(),
    // but explicitly stopping makes intent clear and avoids surprises.
    defer ticker.Stop()
    
    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            select {
            case <-ctx.Done():
                return
            default:
                // WriteControl, not WriteMessage: it's the one write
                // method gorilla/websocket documents as safe to call
                // concurrently with pumpMessages' WriteJSON on the same
                // Conn - see Bug #4 above. WriteMessage here would still
                // compile and mostly work, right up until it doesn't.
                deadline := time.Now().Add(10 * time.Second)
                if err := sub.ws.WriteControl(websocket.PingMessage, nil, deadline); err != nil {
                    return  // Let monitor handle cleanup
                }
            }
        }
    }
}

The Gradual Recovery

We couldn't just deploy and pray. 50,000 goroutines don't just disappear.

Phase 1: Stop the bleeding (deployed immediately):

// Emergency goroutine limiter
if runtime.NumGoroutine() > 10000 {
    http.Error(w, "Server overloaded", 503)
    return
}

Phase 2: Clean up existing leaks (ran manually):

// One-time cleanup script
func emergencyCleanup() {
    for userID, sub := range s.subscribers {
        err := sub.ws.WriteControl(websocket.PingMessage, nil, time.Now().Add(time.Second))
        if err != nil {
            // Dead connection
            s.Unsubscribe(userID)
        }
    }
}

Phase 3: Monitor the recovery:

3:00 AM: 50,847 goroutines, 47GB RAM
3:15 AM: 45,231 goroutines, 43GB RAM (cleanup running)
3:30 AM: 32,109 goroutines, 31GB RAM
4:00 AM: 15,443 goroutines, 18GB RAM
5:00 AM: 3,221 goroutines, 5.4GB RAM
6:00 AM: 1,098 goroutines, 2.1GB RAM ← Normal

The Monitoring We Should Have Had

We added these alerts immediately:

// Prometheus metrics
var (
    goroutineGauge = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "go_goroutines_count",
            Help: "Current number of goroutines",
        },
    )
    
    subscriptionGauge = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "websocket_subscriptions_total",
            Help: "Total WebSocket subscriptions",
        },
    )
    
    activeConnectionsGauge = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "websocket_connections_active",
            Help: "Active WebSocket connections",
        },
    )
)

// Update every 10 seconds
go func() {
    for range time.Tick(10 * time.Second) {
        goroutineGauge.Set(float64(runtime.NumGoroutine()))
        
        s.mu.RLock()
        subscriptionGauge.Set(float64(len(s.subscribers)))
        s.mu.RUnlock()
        
        activeConnectionsGauge.Set(float64(s.countActiveConnections()))
    }
}()

Alert configuration:

- alert: GoroutineLeakSuspected
  expr: go_goroutines_count > 5000
  for: 10m
  annotations:
    summary: "Possible goroutine leak detected"
    
- alert: WebSocketLeakDetected  
  expr: websocket_subscriptions_total > websocket_connections_active * 1.5
  for: 5m
  annotations:
    summary: "WebSocket subscriptions exceeding active connections"

Security Considerations

Secure Goroutine Management

// Rate limit goroutine creation
type GoroutinePool struct {
    sem    chan struct{}
    wg     sync.WaitGroup
    ctx    context.Context
    cancel context.CancelFunc
}

func NewGoroutinePool(maxGoroutines int) *GoroutinePool {
    ctx, cancel := context.WithCancel(context.Background())
    return &GoroutinePool{
        sem:    make(chan struct{}, maxGoroutines),
        ctx:    ctx,
        cancel: cancel,
    }
}

func (p *GoroutinePool) Go(fn func()) error {
    select {
    case p.sem <- struct{}{}:
        p.wg.Add(1)
        go func() {
            defer func() {
                <-p.sem
                p.wg.Done()
            }()
            fn()
        }()
        return nil
    case <-p.ctx.Done():
        return fmt.Errorf("pool is shut down")
    default:
        return fmt.Errorf("goroutine limit reached")
    }
}

The Test That Would Have Caught This

The version below is what's actually in the gist and passes go test -race ./... - not a sketch of a test suite, the real one:

func TestFixedService_NoLeak(t *testing.T) {
    defer goleak.VerifyNone(t)

    svc := NewNotificationService()
    const users = 50
    conns := make([]*mockConn, users)

    for i := 0; i < users; i++ {
        conns[i] = &mockConn{}
        svc.Subscribe(fmt.Sprintf("user%d", i), conns[i])
    }

    time.Sleep(10 * time.Millisecond)

    for i := 0; i < users; i++ {
        conns[i].triggerClose()
    }

    time.Sleep(20 * time.Millisecond) // let cleanup actually run
}

The first version of this suite put the leaking service and this test in the same package. Running them together made TestFixedService_NoLeak fail - not because the fix was wrong, but because goleak.VerifyNone inspects every goroutine in the process, not just ones started by the current test. The previous test's deliberately-leaked goroutines (that's what it's testing) were still alive and tripped the leak check in a completely unrelated test right after it. The fix was moving the leaking demo into its own package, so it gets its own test binary and can't poison anything else's goleak check - worth knowing if you're using goleak seriously and not just in one isolated test file.

And the concurrent-write hazard from Bug #4, verified two ways - the fixed combination staying clean under -race:

func TestFixedService_ConcurrentWriteDoesNotRace(t *testing.T) {
    conn := &mockConn{}
    var wg sync.WaitGroup
    wg.Add(2)

    go func() {
        defer wg.Done()
        for i := 0; i < 200; i++ {
            conn.WriteJSON(Message{Text: "hi"})
        }
    }()
    go func() {
        defer wg.Done()
        for i := 0; i < 200; i++ {
            conn.WriteControl(PingMessage, nil, time.Now().Add(time.Second))
        }
    }()

    wg.Wait()
}

- and the buggy combination (WriteJSON + WriteMessage, what the original code actually did) reliably failing under -race, gated behind a build tag specifically because it's supposed to fail and shouldn't run in the default suite:

$ go test -race -tags racedemo -run TestBuggyWriteMessage_RacesWithWriteJSON ./...
==================
WARNING: DATA RACE
Read at 0x00c000114720 by goroutine 8:
  mockConn.WriteJSON()
Previous write at 0x00c000114720 by goroutine 9:
  mockConn.WriteMessage()
==================

Full setup, both packages, and the exact commands to reproduce all three results yourself are in the gist linked at the top of this article.

Lessons Burned Into My Brain

1. Every Goroutine Needs an Exit Strategy

// Bad: Fire and forget
go doSomething()

// Good: Controlled lifecycle
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go doSomething(ctx)

2. Tickers Are Not Garbage Collected

// This leaks
ticker := time.NewTicker(time.Second)

// This doesn't
ticker := time.NewTicker(time.Second)
defer ticker.Stop()

3. Monitor Goroutines Like Memory

If you monitor memory usage, monitor goroutine count. They're equally important.

4. Test for Leaks, Not Just Correctness

// Add to every concurrent test
defer goleak.VerifyNone(t)

5. WebSockets Are Goroutine Factories

Every WebSocket connection typically spawns 2-3 goroutines. 10,000 connections = 30,000 goroutines. Plan accordingly.

The Cost of This Bug

Illustrative, not a real invoice or ticket count: weeks of slowly degrading performance before anyone connected the dots, real engineering hours spent debugging it live, extra infrastructure cost from RAM scaling to keep the leak from taking the service down outright, and a genuinely bad night for whoever was on call. All from forgetting to call cancel() and ticker.Stop() - and, per Bug #4 above, a second bug the goroutine-leak framing alone would never have surfaced.

The Bottom Line

Goroutine leaks are memory leaks with extra steps. They're harder to spot, harder to debug, and cause weird cascading failures.

But they're easy to prevent:

1. Every go needs a way to stop
2. Every NewTicker needs a Stop()
3. Every make(chan) needs a close()
4. Every Subscribe needs an Unsubscribe

And for the love of all that is holy, use goleak in your tests.

P.S. We now have a pre-commit hook that looks for time.NewTicker without defer ticker.Stop(). It's caught a real handful of PRs since - each one a wake-up call it prevented, not a bug it merely documented after the fact.

What's measured vs illustrative in this article: the goroutine leak itself and its fix, the goleak-process-scope subtlety, and Bug #4's data race are all verified by actually running the tests in the gist - not narrated from memory. The production timeline, cost estimate, and specific goroutine/RAM/latency numbers throughout are scene-setting for a composite incident, not a real incident's exact logs.