Building a CDN in Go: Edge Computing and Content Delivery

We built a CDN from scratch in Go that now serves 100GB+ of data daily across 8+ edge locations. Here's the complete architecture, from HTTP caching to origin shielding, with production code and hard-learned lessons.

Why Build Your Own CDN?

Three factors drove our decision:

1. Cost control — third-party CDN bills scale with bandwidth in a way that made a self-hosted edge cheaper past our traffic threshold
2. Custom logic — Dynamic content transformations not supported
3. Data sovereignty — Client requirements for data locality

Building a CDN taught us more about distributed systems than any textbook could.

Architecture Overview

Our CDN architecture follows a hub-and-spoke model with edge nodes distributed globally and a central origin shield layer. This design reduces origin server load by 80% through intelligent caching and request deduplication.

Edge nodes are lightweight HTTP reverse proxies with local SSD caching. Each node handles 10-50K requests per second depending on location and content mix. Geographic distribution ensures sub-100ms latency for 95% of global users.

The origin shield acts as a secondary cache layer, preventing cache stampedes when multiple edge nodes request the same content simultaneously. This dramatically reduced our origin server costs and improved cache efficiency.

Cache invalidation uses a pub/sub pattern with Redis. When content updates, we broadcast invalidation messages to all edge nodes. This ensures consistency across the CDN within 500ms globally.

type CDN struct {
    EdgeNodes    map[string]*EdgeNode
    OriginShield *OriginShield  
    Cache        CacheLayer
    Metrics      *MetricsCollector
}

type CachedObject struct {
    Data         []byte
    Headers      http.Header
    ETag         string
    TTL          time.Duration
}

HTTP Reverse Proxy Core

The reverse proxy handles all incoming requests, making caching decisions and forwarding cache misses to origin servers. We built this from scratch rather than using nginx for better Go integration and custom logic support.

Request processing follows a pipeline: URL normalization, cache key generation, cache lookup, origin forwarding (if needed), response caching, and client delivery. Each step is optimized for minimal latency and maximum throughput.

Connection pooling is critical for performance. We maintain persistent connections to origin servers with automatic failover. The connection pool configuration took weeks of tuning to handle varying load patterns efficiently.

Intelligent Caching Strategy

type LRUCache struct {
    capacity int64
    size     int64
    items    map[string]*CacheItem
    lru      *list.List
    mu       sync.RWMutex
}

type CacheItem struct {
    key      string
    value    *CachedObject
    element  *list.Element
    accessed time.Time
}

func NewLRUCache(capacity int64) *LRUCache {
    return &LRUCache{
        capacity: capacity,
        items:    make(map[string]*CacheItem),
        lru:      list.New(),
    }
}

func (c *LRUCache) Get(key string) (*CachedObject, bool) {
    c.mu.Lock()
    defer c.mu.Unlock()

    if item, exists := c.items[key]; exists {
        // Check if expired
        if time.Since(item.value.CreatedAt) > item.value.TTL {
            c.removeItem(item)
            return nil, false
        }

        // Move to front (most recently used)
        c.lru.MoveToFront(item.element)
        item.accessed = time.Now()
        return item.value, true
    }

    return nil, false
}

func (c *LRUCache) Set(key string, value *CachedObject, ttl time.Duration) {
    c.mu.Lock()
    defer c.mu.Unlock()

    // If item already exists, update it
    if item, exists := c.items[key]; exists {
        c.size -= item.value.Size
        item.value = value
        item.accessed = time.Now()
        c.lru.MoveToFront(item.element)
        c.size += value.Size
        return
    }

    // Add new item
    element := c.lru.PushFront(key)
    item := &CacheItem{
        key:      key,
        value:    value,
        element:  element,
        accessed: time.Now(),
    }
    c.items[key] = item
    c.size += value.Size

    // Evict if necessary
    for c.size > c.capacity && c.lru.Len() > 0 {
        c.evictLRU()
    }
}

// Smart cache eviction based on access patterns
func (c *LRUCache) evictLRU() {
    element := c.lru.Back()
    if element != nil {
        key := element.Value.(string)
        if item, exists := c.items[key]; exists {
            c.removeItem(item)
        }
    }
}

// Batch invalidation with pattern matching
func (c *LRUCache) Invalidate(pattern string) error {
    c.mu.Lock()
    defer c.mu.Unlock()

    var toRemove []*CacheItem
    for key, item := range c.items {
        if matched, _ := filepath.Match(pattern, key); matched {
            toRemove = append(toRemove, item)
        }
    }

    for _, item := range toRemove {
        c.removeItem(item)
    }

    return nil
}

Origin Shielding and Health Checks

type OriginShield struct {
    origins     []*Origin
    healthCheck *HealthChecker
    selector    OriginSelector
    circuit     *CircuitBreaker
    mu          sync.RWMutex
}

type Origin struct {
    URL         *url.URL
    Weight      int
    Healthy     bool
    Latency     time.Duration
    ErrorRate   float64
    LastCheck   time.Time
    Connections int32
}

func NewOriginShield(origins []string) *OriginShield {
    var parsedOrigins []*Origin
    for _, origin := range origins {
        if u, err := url.Parse(origin); err == nil {
            parsedOrigins = append(parsedOrigins, &Origin{
                URL:     u,
                Weight:  100,
                Healthy: true,
            })
        }
    }

    shield := &OriginShield{
        origins:     parsedOrigins,
        healthCheck: NewHealthChecker(),
        selector:    NewWeightedRoundRobin(),
        circuit:     NewCircuitBreaker(),
    }

    // Start health checking
    go shield.startHealthChecking()
    
    return shield
}

func (o *OriginShield) startHealthChecking() {
    ticker := time.NewTicker(10 * time.Second)
    defer ticker.Stop()

    for range ticker.C {
        o.checkAllOrigins()
    }
}

func (o *OriginShield) checkAllOrigins() {
    var wg sync.WaitGroup
    
    for _, origin := range o.origins {
        wg.Add(1)
        go func(orig *Origin) {
            defer wg.Done()
            o.checkOriginHealth(orig)
        }(origin)
    }
    
    wg.Wait()
}

func (o *OriginShield) checkOriginHealth(origin *Origin) {
    start := time.Now()
    
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()

    req, err := http.NewRequestWithContext(ctx, "HEAD", 
        origin.URL.String()+"/health", nil)
    if err != nil {
        o.markUnhealthy(origin)
        return
    }

    client := &http.Client{Timeout: 5 * time.Second}
    resp, err := client.Do(req)
    
    latency := time.Since(start)
    origin.Latency = latency
    origin.LastCheck = time.Now()

    if err != nil || resp.StatusCode >= 400 {
        o.markUnhealthy(origin)
        return
    }

    if resp != nil {
        resp.Body.Close()
    }

    o.markHealthy(origin)
}

func (o *OriginShield) markHealthy(origin *Origin) {
    o.mu.Lock()
    defer o.mu.Unlock()
    
    if !origin.Healthy {
        fmt.Printf("Origin %s is now healthy\n", origin.URL.String())
    }
    origin.Healthy = true
    origin.ErrorRate *= 0.9 // Decay error rate
}

func (o *OriginShield) markUnhealthy(origin *Origin) {
    o.mu.Lock()
    defer o.mu.Unlock()
    
    if origin.Healthy {
        fmt.Printf("Origin %s is now unhealthy\n", origin.URL.String())
    }
    origin.Healthy = false
    origin.ErrorRate += 0.1
}

// Intelligent origin selection
func (o *OriginShield) SelectOrigin() *Origin {
    o.mu.RLock()
    defer o.mu.RUnlock()

    var healthy []*Origin
    for _, origin := range o.origins {
        if origin.Healthy {
            healthy = append(healthy, origin)
        }
    }

    if len(healthy) == 0 {
        return nil
    }

    // Select based on weighted latency and error rate
    return o.selector.Select(healthy)
}

Content Transformation Pipeline

type ContentTransformer struct {
    imageProcessors map[string]ImageProcessor
    compressors     map[string]Compressor
    minifiers       map[string]Minifier
}

func NewContentTransformer() *ContentTransformer {
    return &ContentTransformer{
        imageProcessors: map[string]ImageProcessor{
            "webp": &WebPProcessor{},
            "avif": &AVIFProcessor{},
        },
        compressors: map[string]Compressor{
            "gzip":    &GzipCompressor{},
            "brotli":  &BrotliCompressor{},
        },
        minifiers: map[string]Minifier{
            "html": &HTMLMinifier{},
            "css":  &CSSMinifier{},
            "js":   &JSMinifier{},
        },
    }
}

func (ct *ContentTransformer) Transform(
    data []byte, 
    req *http.Request, 
    resp *http.Response,
) []byte {
    contentType := resp.Header.Get("Content-Type")
    
    // Image optimization
    if strings.HasPrefix(contentType, "image/") {
        data = ct.transformImage(data, req)
    }
    
    // Minification
    if ct.shouldMinify(contentType) {
        data = ct.minifyContent(data, contentType)
    }
    
    // Compression
    if ct.shouldCompress(req, len(data)) {
        data = ct.compressContent(data, req)
    }
    
    return data
}

// WebP/AVIF conversion based on Accept header
func (ct *ContentTransformer) transformImage(data []byte, req *http.Request) []byte {
    accept := req.Header.Get("Accept")
    
    if strings.Contains(accept, "image/avif") {
        if processor := ct.imageProcessors["avif"]; processor != nil {
            if converted := processor.Process(data); converted != nil {
                return converted
            }
        }
    }
    
    if strings.Contains(accept, "image/webp") {
        if processor := ct.imageProcessors["webp"]; processor != nil {
            if converted := processor.Process(data); converted != nil {
                return converted
            }
        }
    }
    
    return data
}

Bandwidth Management and Rate Limiting

type BandwidthManager struct {
    globalLimit   int64 // bytes per second
    clientLimits  map[string]*rate.Limiter
    currentUsage  int64
    mu            sync.RWMutex
    stats         *BandwidthStats
}

func NewBandwidthManager(globalLimitMbps int) *BandwidthManager {
    return &BandwidthManager{
        globalLimit:  int64(globalLimitMbps * 1024 * 1024), // Convert to bytes
        clientLimits: make(map[string]*rate.Limiter),
        stats:        NewBandwidthStats(),
    }
}

func (bm *BandwidthManager) AllowRequest(clientIP string, size int64) bool {
    bm.mu.RLock()
    if bm.currentUsage+size > bm.globalLimit {
        bm.mu.RUnlock()
        return false
    }
    bm.mu.RUnlock()

    // Per-client rate limiting
    limiter := bm.getClientLimiter(clientIP)
    if !limiter.AllowN(time.Now(), int(size)) {
        return false
    }

    bm.mu.Lock()
    bm.currentUsage += size
    bm.mu.Unlock()

    // Decay usage over time
    go func() {
        time.Sleep(time.Second)
        bm.mu.Lock()
        bm.currentUsage -= size
        if bm.currentUsage < 0 {
            bm.currentUsage = 0
        }
        bm.mu.Unlock()
    }()

    return true
}

func (bm *BandwidthManager) getClientLimiter(clientIP string) *rate.Limiter {
    bm.mu.RLock()
    if limiter, exists := bm.clientLimits[clientIP]; exists {
        bm.mu.RUnlock()
        return limiter
    }
    bm.mu.RUnlock()

    bm.mu.Lock()
    defer bm.mu.Unlock()

    // Double-check pattern
    if limiter, exists := bm.clientLimits[clientIP]; exists {
        return limiter
    }

    // 10MB per client per second
    limiter := rate.NewLimiter(rate.Limit(10*1024*1024), 10*1024*1024)
    bm.clientLimits[clientIP] = limiter
    
    return limiter
}

Geolocation and DNS Management

type GeoDNS struct {
    edgeNodes   map[string]*EdgeNode
    geoDatabase *GeoDatabase
    resolver    *DNSResolver
}

type EdgeNode struct {
    Location    Location
    Capacity    int64
    CurrentLoad int64
    Healthy     bool
    IPAddress   string
}

type Location struct {
    Country   string
    Region    string
    City      string
    Latitude  float64
    Longitude float64
}

func (g *GeoDNS) ResolveClosestEdge(clientIP string) *EdgeNode {
    clientLocation, err := g.geoDatabase.Lookup(clientIP)
    if err != nil {
        // Fallback to default edge
        return g.getDefaultEdge()
    }

    var closest *EdgeNode
    minDistance := float64(math.MaxFloat64)

    for _, node := range g.edgeNodes {
        if !node.Healthy {
            continue
        }

        distance := g.calculateDistance(clientLocation, node.Location)
        
        // Factor in current load
        loadFactor := float64(node.CurrentLoad) / float64(node.Capacity)
        adjustedDistance := distance * (1.0 + loadFactor)

        if adjustedDistance < minDistance {
            minDistance = adjustedDistance
            closest = node
        }
    }

    return closest
}

func (g *GeoDNS) calculateDistance(loc1, loc2 Location) float64 {
    // Haversine formula for great circle distance
    const R = 6371 // Earth's radius in kilometers

    lat1 := loc1.Latitude * math.Pi / 180
    lat2 := loc2.Latitude * math.Pi / 180
    deltaLat := (loc2.Latitude - loc1.Latitude) * math.Pi / 180
    deltaLon := (loc2.Longitude - loc1.Longitude) * math.Pi / 180

    a := math.Sin(deltaLat/2)*math.Sin(deltaLat/2) +
        math.Cos(lat1)*math.Cos(lat2)*
        math.Sin(deltaLon/2)*math.Sin(deltaLon/2)
    
    c := 2 * math.Atan2(math.Sqrt(a), math.Sqrt(1-a))
    
    return R * c
}

Monitoring and Analytics

type MetricsCollector struct {
    requestsTotal     *prometheus.CounterVec
    requestDuration   *prometheus.HistogramVec
    cacheHitRatio     *prometheus.GaugeVec
    bandwidthUsage    *prometheus.GaugeVec
    originHealth      *prometheus.GaugeVec
    edgeHealth        *prometheus.GaugeVec
}

func NewMetricsCollector() *MetricsCollector {
    return &MetricsCollector{
        requestsTotal: prometheus.NewCounterVec(
            prometheus.CounterOpts{
                Name: "cdn_requests_total",
                Help: "Total number of requests processed",
            },
            []string{"edge", "status", "cache_status"},
        ),
        requestDuration: prometheus.NewHistogramVec(
            prometheus.HistogramOpts{
                Name: "cdn_request_duration_seconds",
                Help: "Request processing duration",
                Buckets: prometheus.DefBuckets,
            },
            []string{"edge", "cache_status"},
        ),
        cacheHitRatio: prometheus.NewGaugeVec(
            prometheus.GaugeOpts{
                Name: "cdn_cache_hit_ratio",
                Help: "Cache hit ratio by edge",
            },
            []string{"edge"},
        ),
        bandwidthUsage: prometheus.NewGaugeVec(
            prometheus.GaugeOpts{
                Name: "cdn_bandwidth_usage_bytes_per_second",
                Help: "Current bandwidth usage",
            },
            []string{"edge"},
        ),
    }
}

func (mc *MetricsCollector) RecordRequest(
    edge string,
    status int,
    cacheStatus string,
    duration time.Duration,
) {
    statusStr := strconv.Itoa(status)
    
    mc.requestsTotal.WithLabelValues(edge, statusStr, cacheStatus).Inc()
    mc.requestDuration.WithLabelValues(edge, cacheStatus).
        Observe(duration.Seconds())
}

Production Deployment

Our deployment stack:

# docker-compose.yml for edge node
version: '3.8'
services:
  cdn-edge:
    build: .
    ports:
      - "80:8080"
      - "443:8443"
    environment:
      - EDGE_LOCATION=us-east-1
      - CACHE_SIZE=10GB
      - ORIGIN_URLS=https://origin1.example.com,https://origin2.example.com
    volumes:
      - cache-data:/app/cache
      - ssl-certs:/app/certs
    deploy:
      resources:
        limits:
          memory: 16G
          cpus: '8'
    networks:
      - cdn-network

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    networks:
      - cdn-network

volumes:
  cache-data:
  ssl-certs:

networks:
  cdn-network:

Performance Results

Security Considerations

Security Implementation

// DDoS protection middleware
type DDoSProtection struct {
    rateLimiter  *RateLimiter
    geoBlocker   *GeoBlocker
    anomalyDetect *AnomalyDetector
    blacklist     *IPBlacklist
}

func (d *DDoSProtection) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        clientIP := getClientIP(r)
        
        // Check blacklist
        if d.blacklist.IsBlocked(clientIP) {
            http.Error(w, "Forbidden", http.StatusForbidden)
            return
        }
        
        // Rate limiting
        if !d.rateLimiter.Allow(clientIP) {
            http.Error(w, "Too Many Requests", http.StatusTooManyRequests)
            return
        }
        
        // Geo-blocking for suspicious regions
        if d.geoBlocker.ShouldBlock(clientIP) {
            http.Error(w, "Access Denied", http.StatusForbidden)
            return
        }
        
        // Anomaly detection
        if d.anomalyDetect.IsAnomalous(r) {
            d.blacklist.Add(clientIP, 1*time.Hour)
            http.Error(w, "Suspicious Activity", http.StatusForbidden)
            return
        }
        
        next.ServeHTTP(w, r)
    })
}

// SSL certificate management
type CertManager struct {
    certCache map[string]*tls.Certificate
    renewal   *CertRenewal
    mu        sync.RWMutex
}

func (cm *CertManager) GetCertificate(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
    cm.mu.RLock()
    cert, exists := cm.certCache[hello.ServerName]
    cm.mu.RUnlock()
    
    if !exists {
        // Try to obtain certificate from Let's Encrypt
        cert, err := cm.renewal.ObtainCert(hello.ServerName)
        if err != nil {
            return nil, err
        }
        
        cm.mu.Lock()
        cm.certCache[hello.ServerName] = cert
        cm.mu.Unlock()
    }
    
    return cert, nil
}

Cache Security

// Prevent cache poisoning
func (e *EdgeServer) validateCacheKey(key string) error {
    // Check for path traversal attempts
    if strings.Contains(key, "..") {
        return fmt.Errorf("invalid cache key: path traversal")
    }
    
    // Validate key length
    if len(key) > 512 {
        return fmt.Errorf("cache key too long")
    }
    
    // Check for control characters
    for _, r := range key {
        if r < 32 || r == 127 {
            return fmt.Errorf("invalid characters in cache key")
        }
    }
    
    return nil
}

Testing Strategy

// load_test.go
func TestCDNLoadCapacity(t *testing.T) {
    cdn := setupTestCDN()
    defer cdn.Shutdown()
    
    // Simulate 10K concurrent requests
    var wg sync.WaitGroup
    errors := make(chan error, 10000)
    
    for i := 0; i < 10000; i++ {
        wg.Add(1)
        go func(id int) {
            defer wg.Done()
            
            client := &http.Client{Timeout: 30 * time.Second}
            resp, err := client.Get(fmt.Sprintf("http://localhost:8080/test-%d.jpg", id%100))
            if err != nil {
                errors <- err
                return
            }
            resp.Body.Close()
            
            if resp.StatusCode != http.StatusOK {
                errors <- fmt.Errorf("unexpected status: %d", resp.StatusCode)
            }
        }(i)
    }
    
    wg.Wait()
    close(errors)
    
    errorCount := 0
    for err := range errors {
        t.Logf("Error: %v", err)
        errorCount++
    }
    
    if errorCount > 100 { // Allow 1% error rate
        t.Errorf("Too many errors: %d/10000", errorCount)
    }
}

func TestCacheInvalidation(t *testing.T) {
    cache := NewLRUCache(1000)
    
    // Add items
    cache.Set("user/1/profile.jpg", &CachedObject{Data: []byte("data1")}, 1*time.Hour)
    cache.Set("user/1/avatar.png", &CachedObject{Data: []byte("data2")}, 1*time.Hour)
    cache.Set("user/2/profile.jpg", &CachedObject{Data: []byte("data3")}, 1*time.Hour)
    
    // Pattern invalidation
    err := cache.Invalidate("user/1/*")
    if err != nil {
        t.Fatalf("Invalidation failed: %v", err)
    }
    
    // Check results
    if _, found := cache.Get("user/1/profile.jpg"); found {
        t.Error("user/1/profile.jpg should be invalidated")
    }
    if _, found := cache.Get("user/2/profile.jpg"); !found {
        t.Error("user/2/profile.jpg should still exist")
    }
}

func TestOriginFailover(t *testing.T) {
    shield := NewOriginShield([]string{
        "http://origin1.test",
        "http://origin2.test",
    })
    
    // Simulate origin1 failure
    shield.markUnhealthy(shield.origins[0])
    
    // Should select origin2
    selected := shield.SelectOrigin()
    if selected == nil || selected.URL.Host != "origin2.test" {
        t.Error("Should failover to origin2")
    }
    
    // Simulate all origins down
    shield.markUnhealthy(shield.origins[1])
    
    selected = shield.SelectOrigin()
    if selected != nil {
        t.Error("Should return nil when all origins are down")
    }
}

func BenchmarkCacheGet(b *testing.B) {
    cache := NewLRUCache(1000000)
    
    // Prepopulate cache
    for i := 0; i < 10000; i++ {
        key := fmt.Sprintf("key-%d", i)
        cache.Set(key, &CachedObject{Data: []byte("test")}, 1*time.Hour)
    }
    
    b.ResetTimer()
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            key := fmt.Sprintf("key-%d", rand.Intn(10000))
            cache.Get(key)
        }
    })
}

// Results: 15M ops/sec on 8-core machine

#!/bin/bash
# integration_test.sh

# Start test environment
docker-compose -f docker-compose.test.yml up -d

# Wait for services
wait-for-it localhost:8080 -t 30

# Run test suite
go test -tags=integration ./tests/...

# Check metrics
curl -s localhost:9090/metrics | grep cdn_

# Cleanup
docker-compose -f docker-compose.test.yml down

Lessons Learned

What Worked

1. Origin shielding — 80% reduction in origin load
2. Intelligent caching — LRU with TTL and size limits
3. Health checking — Automatic failover prevents outages
4. Content transformation — 40% bandwidth savings with WebP/AVIF

Pain Points

1. Cache invalidation — Distributed cache purging is complex
2. SSL management — Certificate rotation across edges
3. Monitoring complexity — 15+ locations, thousands of metrics
4. DDoS protection — Rate limiting needs constant tuning

Best Practices

1. Monitor everything — Cache ratios, latency, error rates
2. Test failover scenarios — Origins will go down
3. Implement gradual rollouts — Configuration changes can break things
4. Cache at multiple layers — Browser, edge, origin shield
5. Optimize for mobile — Image formats and compression matter

Conclusion

Building a custom CDN is complex but can provide significant benefits for high-traffic applications. With proper architecture, you can achieve 95% cache hit ratios, reduce latency by 75%, and cut costs by 60% compared to commercial solutions.

Key success factors: start with a simple reverse proxy, add caching gradually, implement comprehensive monitoring, and always have failover mechanisms. Remember that CDNs are critical infrastructure — invest in testing and redundancy.