telecom-platform/docs/OPERATIONS.md

Operations

Deployment, monitoring, performance tuning, and production operations for the TaaS Platform.

Table of Contents

1. Docker Deployment
2. Kubernetes Deployment
3. Production Considerations
4. Performance & Optimization
5. Monitoring & Alerting
6. CI/CD
7. Backup & Disaster Recovery

Docker Deployment

Build

make docker-build
# Or individually
docker build -f deployments/docker/api-server.Dockerfile -t taas-api-server .
docker build -f deployments/docker/charging-engine.Dockerfile -t taas-charging-engine .
docker build -f deployments/docker/web-dashboard.Dockerfile -t taas-web-dashboard .

Run

docker-compose up -d
docker-compose logs -f
docker-compose down          # Stop
docker-compose down -v       # Stop + remove volumes

Troubleshooting

# MongoDB
docker-compose ps mongodb
docker-compose logs mongodb
docker exec -it taas-mongodb mongosh

# Redis
docker exec -it taas-redis redis-cli ping

Kubernetes Deployment

Prerequisites

• Kubernetes v1.28+
• kubectl configured
• Persistent storage provisioner
• LoadBalancer or Ingress controller

Deploy

kubectl create namespace taas-platform
kubectl apply -f deployments/kubernetes/
kubectl get pods -n taas-platform

Access

kubectl get svc api-server -n taas-platform
kubectl port-forward -n taas-platform svc/api-server 8000:8000

Scale

kubectl scale deployment api-server -n taas-platform --replicas=5
kubectl autoscale deployment api-server -n taas-platform \
  --min=3 --max=10 --cpu-percent=70

Rolling Updates

kubectl set image deployment/api-server \
  api-server=ghcr.io/nutcas3/telecom-platform/api-server:v2.0.0 \
  -n taas-platform
kubectl rollout status deployment/api-server -n taas-platform
kubectl rollout undo deployment/api-server -n taas-platform   # rollback

Logs & Events

kubectl logs -f deployment/api-server -n taas-platform
kubectl describe pod <pod-name> -n taas-platform
kubectl get events -n taas-platform --sort-by='.lastTimestamp'

Production Considerations

High Availability

PostgreSQL / MongoDB replica sets:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb
spec:
  replicas: 3
  serviceName: mongodb
  # See MongoDB Operator docs for full config

Redis Cluster:

helm install redis bitnami/redis-cluster \
  --namespace taas-platform \
  --set cluster.nodes=6 \
  --set cluster.replicas=1

Secrets Management

kubectl create secret generic db-secret \
  --from-literal=username=admin \
  --from-literal=password='STRONG_PASSWORD' \
  -n taas-platform

kubectl create secret generic api-keys \
  --from-literal=jwt-secret="$(openssl rand -base64 32)" \
  -n taas-platform

For production, prefer HashiCorp Vault or AWS Secrets Manager over raw K8s secrets.

Network Policies

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-server-policy
  namespace: taas-platform
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: web-dashboard
      ports:
        - protocol: TCP
          port: 8000
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: mongodb
      ports:
        - protocol: TCP
          port: 27017

Resource Limits

Service	CPU Request	CPU Limit	Mem Request	Mem Limit
API Server	100m	500m	128Mi	512Mi
Charging Engine	100m	1000m	128Mi	512Mi
Packet Gateway	250m	2000m	256Mi	1Gi
MongoDB	250m	1000m	512Mi	2Gi
PostgreSQL	250m	1000m	512Mi	2Gi
Redis	100m	500m	256Mi	2Gi

Performance & Optimization

Rate Limiting

In-memory sliding window with per-IP tracking. Configured via middleware.RateLimitByEndpoint() in @/Users/nutcase/Downloads/telecom-platform-starter/apps/api-server/internal/middleware/ratelimit.go.

Tier	Limit/min	Burst	Applies To
Auth	10	5	/v1/auth/login, /v1/auth/register
Read	200	50	/v1/services, /v1/monitoring/metrics
Admin	50	10	/v1/users, /v1/config
Default	100	20	Everything else

Caching

Layered in-memory LRU cache with TTL (@/Users/nutcase/Downloads/telecom-platform-starter/apps/api-server/internal/cache/cache.go):

Namespace	TTL	Max Items	Use
default	5 min	1000	General purpose
users	10 min	500	User data
services	2 min	200	Service info
metrics	1 min	1000	Monitoring data

cache := cache.GetDefaultCache()
cache.Set("key", value, 5*time.Minute)
value, found := cache.Get("key")

The cached database wrapper (@/Users/nutcase/Downloads/telecom-platform-starter/apps/api-server/internal/database/cached.go) provides CachedFind, CachedFirst, InvalidateCache.

Connection Pooling

Tuned in @/Users/nutcase/Downloads/telecom-platform-starter/apps/api-server/internal/database/database.go:50-54:

sqlDB.SetMaxIdleConns(25)                   // minimum idle
sqlDB.SetMaxOpenConns(100)                  // maximum open
sqlDB.SetConnMaxLifetime(time.Hour)
sqlDB.SetConnMaxIdleTime(30 * time.Minute)

db.PoolStats() exposes live pool metrics for dashboards and alerts.

Performance Middleware

Tracks response times, error rates, and slow requests (>100ms threshold). Adds:

• X-Response-Time header
• X-Request-ID correlation ID
• Security headers (CSP, HSTS, X-Frame-Options)
• Request size limit (10MB) and timeout (30s)

Metrics endpoint: GET /metrics/performance

Performance Targets

Metric	Target
API response time (p50)	<50ms
API response time (p99)	<100ms
Uptime	99.9%
Error rate	<1%
Test coverage	>80%

Load Testing

brew install k6
k6 run scripts/loadtest.js

Sample script:

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },
    { duration: '5m', target: 100 },
    { duration: '2m', target: 200 },
    { duration: '5m', target: 200 },
    { duration: '2m', target: 0 },
  ],
};

export default function () {
  const res = http.get('http://api-server:8000/health');
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(1);
}

Monitoring & Alerting

Health Checks

Three probe endpoints (@/Users/nutcase/Downloads/telecom-platform-starter/apps/api-server/internal/monitoring/health.go):

Endpoint	Purpose
/health	Aggregated component health
/ready	Kubernetes readiness probe
/live	Kubernetes liveness probe

Monitored components: database, Prometheus, Kubernetes, system (memory, goroutines, GC).

Alert Rules

Default rules in @/Users/nutcase/Downloads/telecom-platform-starter/apps/api-server/internal/monitoring/alerting.go:

Rule	Threshold	Severity
System Health	Degraded/Unhealthy	Warning
High Memory	>80%	Warning
High Goroutines	>1000	Warning
DB Connection Utilization	>80%	Warning

Alert Notifiers

• WebSocket — real-time dashboard
• Log — structured logs
• Composite — multiple channels simultaneously

Prometheus Metrics

Key exported metrics:

• api_requests_total — total request count
• esim_creations_total — eSIMs created
• data_usage_bytes_total — cumulative data usage
• charging_checks_duration_seconds — credit check latency histogram
• packet_gateway_packets_total — packets processed

Prometheus + Grafana Install

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

WebSocket Real-Time Events

GET /ws streams:

• Service status updates
• System metrics
• Alert notifications

CI/CD

GitHub Actions

Two workflows in .github/workflows/:

ci.yml — runs on every push/PR:

• Go tests with PostgreSQL service container
• go vet and golangci-lint
• Web dashboard lint + typecheck + build
• Docker image builds (on main)
• Trivy security scan

deploy.yml — runs on main pushes and version tags:

• Build + push images to GHCR
• Deploy to staging (main branch)
• Deploy to production (version tags)

Manual trigger via workflow_dispatch with environment input.

Local Testing

# Go tests with coverage
cd apps/api-server
go test ./... -v -race -coverprofile=coverage.out
go tool cover -html=coverage.out

# Benchmarks
go test -bench=. ./internal/middleware/
go test -bench=. ./internal/monitoring/

# Linting
golangci-lint run --timeout 5m
go vet ./...
go fix ./...

Backup & Disaster Recovery

MongoDB Backup (CronJob)

kubectl create cronjob mongodb-backup \
  --image=mongo:7.0 \
  --schedule="0 2 * * *" \
  -- mongodump --uri=$MONGODB_URI --out=/backup

Redis Persistence

Enable AOF for durability:

kubectl edit configmap redis-config -n taas-platform
# Set: appendonly yes

Recovery Objectives

• RTO (Recovery Time Objective): <1 hour
• RPO (Recovery Point Objective): <15 minutes

Recovery Procedure

1. Spin up new Kubernetes cluster
2. Restore database from latest backup (cross-region replicated)
3. Restore Redis from RDB snapshot or rebuild from DB
4. Deploy all services from Git (kubectl apply -f deployments/kubernetes/)
5. Update DNS to new cluster
6. Run make verify to confirm functionality

Backup Locations

• MongoDB/PostgreSQL: S3 bucket with cross-region replication
• Redis: S3 RDB snapshots
• Configuration: Git repository
• Secrets: HashiCorp Vault / AWS Secrets Manager

Cost Optimization

• Use spot/preemptible instances for non-critical services
• Enable autoscaling to match load
• Reserved instances for predictable workloads
• Store old backups in S3 Glacier

Sample monthly costs (AWS, base infrastructure):

• 3× t3.medium (API Server): $90
• 2× t3.medium (Charging Engine): $60
• 1× t3.large (MongoDB): $75
• 1× t3.small (Redis): $19
• Load Balancer: $25
• Total: ~$270/month + data transfer

Related Docs

• GETTING_STARTED.md — Setup and API reference
• ARCHITECTURE.md — System design, RBAC, security