Intelligent API Load Balancer

A high-performance, adaptive API Load Balancer built in Go. It supports multiple routing strategies, active/passive health checks, circuit breaking, rate limiting, and real-time observability.

Features

• Adaptive Routing:
  • Weighted Round-Robin
  • Least Connections
  • Adaptive Routing (based on latency, error rate, and active connections)
• Health Monitoring:
  • Active periodic health checks
  • Passive health detection (error tracking)
• Resilience:
  • Circuit Breaker (Closed -> Open -> Half-Open)
  • Token Bucket Rate Limiting
• Observability:
  • /metrics endpoint
  • Real-time Dashboard UI
• Admin Controls:
  • Dynamic backend management (Add/Remove) via API

Tech Stack

• Language: Go (Golang)
• Standard Lib: net/http, net/http/httputil, sync, context
• Configuration: YAML
• Deployment: Docker, Docker Compose

Getting Started

Prerequisites

• Go 1.21+
• Docker & Docker Compose

Running Locally

1. Clone the repository.
2. Install dependencies:

   go mod download

3. Run the load balancer:

   go run main.go

   The server will start on port 8080 (proxy) and 8081 (admin/metrics).

Running with Docker Compose

This will start the load balancer and 3 mock backend services.

docker-compose up --build

• Load Balancer: http://localhost:8080
• Dashboard: http://localhost:8081/dashboard/
• Metrics: http://localhost:8081/metrics

🌍 Real-World Scenarios & Test Cases

Here is how this project solves actual production problems, along with steps to verify them.

1. High Availability (The "Server Crash" Scenario)

Problem: In production, servers crash due to bugs, memory leaks, or hardware failures. You don't want your users to see error pages. Solution: The load balancer detects the failure and stops sending traffic to the dead server. Test It:

1. Start the system: docker-compose up --build
2. In a new terminal, run a loop: while true; do curl http://localhost:8080; echo; sleep 0.5; done
3. Kill one backend: docker stop api-load-balancer--backend1-1
4. Observation: You might see one error, but then all subsequent requests will instantly shift to backend2 and backend3. The system heals itself.

2. Zero-Downtime Deployment (The "Rolling Update" Scenario)

Problem: You need to deploy a new version of your code. Restarting the server drops active connections and causes downtime. Solution: Gracefully remove a server from rotation, update it, and add it back. Test It:

1. Run the loop from above.
2. Tell the load balancer to drain traffic from backend1:

   curl -X POST -d '{"url": "http://backend1:80"}' http://localhost:8081/admin/remove

3. Observation: Traffic stops going to backend1 (check the Dashboard).
4. "Update" the backend (in this case, just imagine you updated it).
5. Add it back:

   curl -X POST -d '{"url": "http://backend1:80", "weight": 1}' http://localhost:8081/admin/add

6. Observation: Traffic starts flowing to backend1 again. Zero errors were served to users during this process.

3. Resilience (The "Cascading Failure" Scenario)

Problem: A database slows down, causing one backend to timeout. If you keep sending requests, they pile up, consume all resources, and crash the entire system (Cascading Failure). Solution: The Circuit Breaker "trips" after a threshold of failures, instantly rejecting requests to that backend to give it time to recover. Test It:

1. This is harder to simulate with whoami, but conceptually:
2. If backend1 starts returning 500 errors (simulated by stopping it), the Circuit Breaker will open.
3. The load balancer stops wasting time trying to connect to it and immediately serves from healthy backends.

4. Scalability (The "Viral Traffic" Scenario)

Problem: Your app goes viral. One server crashes under the load. Solution: You can add more servers dynamically. Test It:

1. Spin up a new backend container manually (advanced Docker usage).
2. Register it via the Admin API: curl ... /admin/add.
3. The load balancer immediately starts using the new capacity without a restart.

Configuration

The load balancer is configured via config.yaml:

server:
  port: 8080
  admin_port: 8081

strategy: "round-robin" # Options: round-robin, least-connections, adaptive

backends:
  - url: "http://backend1:80"
    weight: 1
  - url: "http://backend2:80"
    weight: 1

API Reference

Admin API (Port 8081)

Add Backend

POST /admin/add

{
    "url": "http://localhost:8085",
    "weight": 1
}

Remove Backend

POST /admin/remove

{
    "url": "http://localhost:8085"
}

Metrics API (Port 8081)

Get Metrics

GET /metrics Returns JSON array of backend stats:

[
    {
        "url": "http://localhost:8082",
        "alive": true,
        "active_conns": 5,
        "latency": 15000000,
        "fails": 0
    }
]

Testing

Load Testing

You can use k6 or ab to test the load balancer.

Example k6 script:

import http from 'k6/http';
import { sleep } from 'k6';

export default function () {
  http.get('http://localhost:8080');
  sleep(1);
}

Architecture

The system is composed of several modular components:

• Router: Handles backend selection strategies.
• Proxy: Forwards requests and handles errors.
• Health: Manages active health checks.
• Circuit: Implements circuit breaker logic.
• RateLimit: Manages client rate limits.
• API: Exposes admin and metrics endpoints.