mirror of
https://github.com/andwati/telecom-platform.git
synced 2026-10-20 11:54:25 +00:00
nutcas3
f071605a47
- Add README.md with architecture overview covering api-server (3 replicas), charging-engine (2 replicas), packet-gateway (2 replicas), web-dashboard (2 replicas), postgres, redis, monitoring stack (Prometheus, Grafana, InfluxDB, Alertmanager) - Add prerequisites section with kubectl/Docker/Helm requirements, Kubernetes 1.25+, 4 CPU/8GB RAM minimum, 50GB+ storage - Add quick start guide with deployment
410 lines
11 KiB
Markdown
410 lines
11 KiB
Markdown
# Telecom Platform Kubernetes Deployment
|
|
|
|
This directory contains the complete Kubernetes deployment configuration for the Telecom Platform, including all applications and monitoring stack.
|
|
|
|
## Architecture Overview
|
|
|
|
```
|
|
telecom-platform Namespace
|
|
api-server (3 replicas) - Main GraphQL API server
|
|
charging-engine (2 replicas) - Rust charging engine
|
|
packet-gateway (2 replicas) - eBPF packet gateway
|
|
web-dashboard (2 replicas) - Next.js admin dashboard
|
|
postgres (1 replica) - PostgreSQL database
|
|
redis (1 replica) - Redis cache
|
|
postgres-exporter (1 replica) - PostgreSQL metrics
|
|
redis-exporter (1 replica) - Redis metrics
|
|
|
|
telecom-monitoring Namespace
|
|
prometheus (1 replica) - Metrics collection
|
|
grafana (1 replica) - Visualization dashboards
|
|
influxdb (1 replica) - Time-series storage
|
|
alertmanager (1 replica) - Alert management
|
|
node-exporter (1 replica) - System metrics
|
|
```
|
|
|
|
## Prerequisites
|
|
|
|
### Required Tools
|
|
- **kubectl** - Kubernetes CLI
|
|
- **Docker** - Container runtime
|
|
- **Helm** (optional) - For advanced deployments
|
|
|
|
### Kubernetes Requirements
|
|
- **Kubernetes 1.25+** - For modern API support
|
|
- **Ingress Controller** - nginx-ingress recommended
|
|
- **Persistent Storage** - 50GB+ total for databases and monitoring
|
|
- **LoadBalancer** - For external service access (or NodePort)
|
|
|
|
### Resource Requirements
|
|
- **CPU**: 4+ cores minimum
|
|
- **Memory**: 8GB+ minimum
|
|
- **Storage**: 50GB+ persistent storage
|
|
|
|
## Quick Start
|
|
|
|
### 1. Deploy Everything
|
|
```bash
|
|
cd deployments/kubernetes
|
|
./deploy.sh
|
|
```
|
|
|
|
### 2. Access Services
|
|
After deployment, access services via:
|
|
- **Web Dashboard**: http://localhost:3000
|
|
- **API Server**: http://localhost:8000
|
|
- **Grafana**: http://localhost:3001 (admin/admin123)
|
|
- **Prometheus**: http://localhost:9090
|
|
|
|
### 3. Port Forwarding (if needed)
|
|
```bash
|
|
# Applications
|
|
kubectl port-forward svc/api-server-service 8000:8000 -n telecom-platform
|
|
kubectl port-forward svc/web-dashboard-service 3000:3000 -n telecom-platform
|
|
|
|
# Monitoring
|
|
kubectl port-forward svc/grafana 3001:3000 -n telecom-monitoring
|
|
kubectl port-forward svc/prometheus 9090:9090 -n telecom-monitoring
|
|
```
|
|
|
|
## Deployment Components
|
|
|
|
### Namespaces
|
|
- **telecom-platform** - Main application services
|
|
- **telecom-monitoring** - Monitoring and observability
|
|
|
|
### Applications
|
|
|
|
#### API Server
|
|
- **Replicas**: 3 (auto-scalable to 10)
|
|
- **Ports**: 8000 (HTTP), 9090 (Metrics)
|
|
- **Resources**: 256Mi-512Mi memory, 250m-500m CPU
|
|
- **Features**: GraphQL API, REST endpoints, metrics export
|
|
|
|
#### Charging Engine
|
|
- **Replicas**: 2 (auto-scalable to 6)
|
|
- **Port**: 3001 (HTTP + Metrics)
|
|
- **Resources**: 128Mi-256Mi memory, 100m-200m CPU
|
|
- **Features**: Real-time credit control, usage tracking
|
|
|
|
#### Packet Gateway
|
|
- **Replicas**: 2 (auto-scalable to 4)
|
|
- **Port**: 9000 (Metrics)
|
|
- **Resources**: 256Mi-512Mi memory, 250m-500m CPU
|
|
- **Features**: eBPF packet processing, network isolation
|
|
- **Privileges**: NET_ADMIN, SYS_ADMIN for eBPF
|
|
|
|
#### Web Dashboard
|
|
- **Replicas**: 2 (auto-scalable to 6)
|
|
- **Port**: 3000 (HTTP)
|
|
- **Resources**: 128Mi-256Mi memory, 100m-200m CPU
|
|
- **Features**: React admin interface, real-time updates
|
|
|
|
### Databases
|
|
|
|
#### PostgreSQL
|
|
- **Version**: 15
|
|
- **Storage**: 10Gi persistent volume
|
|
- **Features**: Primary database, metrics export
|
|
- **Exporter**: postgres-exporter on port 9187
|
|
|
|
#### Redis
|
|
- **Version**: 7-alpine
|
|
- **Storage**: 256Mi memory limit
|
|
- **Features**: Caching, session storage, real-time data
|
|
- **Exporter**: redis-exporter on port 9121
|
|
|
|
### Monitoring Stack
|
|
|
|
#### Prometheus
|
|
- **Version**: v2.45.0
|
|
- **Storage**: 20Gi persistent volume
|
|
- **Port**: 9090
|
|
- **Features**: Metrics collection, alerting rules
|
|
- **Targets**: Auto-discovery of all services
|
|
|
|
#### Grafana
|
|
- **Version**: 10.2.0
|
|
- **Storage**: 10Gi persistent volume
|
|
- **Port**: 3000
|
|
- **Features**: Dashboards, alerting, user management
|
|
- **Data Sources**: Prometheus, InfluxDB
|
|
|
|
#### InfluxDB
|
|
- **Version**: 2.7
|
|
- **Storage**: 20Gi persistent volume
|
|
- **Port**: 8086
|
|
- **Features**: Time-series data, long-term storage
|
|
- **Organization**: telecom, Bucket: telecom
|
|
|
|
#### Alertmanager
|
|
- **Version**: v0.25.0
|
|
- **Storage**: 5Gi persistent volume
|
|
- **Port**: 9093
|
|
- **Features**: Alert routing, notifications, silencing
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
Key configuration in `configmaps.yaml`:
|
|
- Database connections
|
|
- Redis configuration
|
|
- Monitoring endpoints
|
|
- Service URLs
|
|
|
|
### Secrets
|
|
Sensitive data in `secrets.yaml`:
|
|
- Database passwords
|
|
- API keys
|
|
- Authentication tokens
|
|
- TLS certificates
|
|
|
|
### Customization
|
|
To customize the deployment:
|
|
|
|
1. **Modify ConfigMaps**: Update `configmaps.yaml`
|
|
2. **Update Secrets**: Modify `secrets.yaml` with real values
|
|
3. **Adjust Resources**: Update resource limits in deployment files
|
|
4. **Change Replicas**: Modify replica counts
|
|
5. **Update Ingress**: Configure hostnames and TLS
|
|
|
|
## Auto-scaling
|
|
|
|
### Horizontal Pod Autoscalers (HPA)
|
|
All applications have HPA configured:
|
|
- **API Server**: 3-10 replicas, 70% CPU, 80% memory
|
|
- **Charging Engine**: 2-6 replicas, 70% CPU, 80% memory
|
|
- **Packet Gateway**: 2-4 replicas, 75% CPU, 85% memory
|
|
- **Web Dashboard**: 2-6 replicas, 70% CPU, 80% memory
|
|
|
|
### Scaling Behavior
|
|
- **Scale Up**: 50% increase, 60s stabilization
|
|
- **Scale Down**: 10% decrease, 300s stabilization
|
|
|
|
## Ingress Configuration
|
|
|
|
### Hostnames
|
|
Default ingress configuration:
|
|
- **API**: api.telecom.example.com
|
|
- **Dashboard**: dashboard.telecom.example.com
|
|
- **Grafana**: grafana.telecom.example.com
|
|
- **Prometheus**: prometheus.telecom.example.com
|
|
|
|
### TLS
|
|
- Automatic TLS with cert-manager
|
|
- Custom certificates supported
|
|
- HTTP to HTTPS redirection
|
|
|
|
### Rate Limiting
|
|
- 100 requests per minute per IP
|
|
- Configurable per service
|
|
|
|
## Monitoring and Alerting
|
|
|
|
### Prometheus Metrics
|
|
All services export metrics on dedicated endpoints:
|
|
- HTTP request metrics
|
|
- Business metrics (subscribers, usage)
|
|
- System metrics (CPU, memory)
|
|
- Custom application metrics
|
|
|
|
### Grafana Dashboards
|
|
Pre-configured dashboards:
|
|
- **Telecom Platform Overview** - System-wide metrics
|
|
- **Subscriber Metrics** - Detailed subscriber analytics
|
|
- **Infrastructure** - Resource utilization
|
|
- **Applications** - Service-specific metrics
|
|
|
|
### Alerting Rules
|
|
Critical alerts configured:
|
|
- High error rates (>10%)
|
|
- Service down events
|
|
- Resource exhaustion
|
|
- Database connection failures
|
|
|
|
### Alert Routing
|
|
- **Critical**: Email + webhook notifications
|
|
- **Warning**: Webhook notifications only
|
|
- **Silencing**: Configurable maintenance windows
|
|
|
|
## Storage
|
|
|
|
### Persistent Volumes
|
|
- **PostgreSQL**: 10Gi, ReadWriteOnce
|
|
- **Redis**: emptyDir (in-memory)
|
|
- **Prometheus**: 20Gi, ReadWriteOnce
|
|
- **Grafana**: 10Gi, ReadWriteOnce
|
|
- **InfluxDB**: 20Gi, ReadWriteOnce
|
|
- **Alertmanager**: 5Gi, ReadWriteOnce
|
|
|
|
### Storage Classes
|
|
Uses default storage class. Customize for your environment:
|
|
- AWS: gp2/gp3
|
|
- GCP: standard-ssd
|
|
- Azure: Premium SSD
|
|
- On-prem: Local SSD/NVMe
|
|
|
|
## Security
|
|
|
|
### Network Policies
|
|
Default policies allow:
|
|
- Intra-namespace communication
|
|
- Cross-namespace for monitoring
|
|
- External access via Ingress only
|
|
|
|
### RBAC
|
|
Service accounts with minimal permissions:
|
|
- Database access only for applications
|
|
- Monitoring access for Prometheus
|
|
- ConfigMap/Secret access as needed
|
|
|
|
### Secrets Management
|
|
- Kubernetes native secrets
|
|
- Encrypted at rest
|
|
- Environment variable injection
|
|
- Rotation policies recommended
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
#### Pods Not Starting
|
|
```bash
|
|
# Check pod status
|
|
kubectl get pods -n telecom-platform -o wide
|
|
|
|
# Check pod logs
|
|
kubectl logs -f deployment/api-server -n telecom-platform
|
|
|
|
# Describe pod for detailed info
|
|
kubectl describe pod <pod-name> -n telecom-platform
|
|
```
|
|
|
|
#### Service Not Accessible
|
|
```bash
|
|
# Check service endpoints
|
|
kubectl get endpoints -n telecom-platform
|
|
|
|
# Test service connectivity
|
|
kubectl exec -it <pod-name> -n telecom-platform -- curl http://api-server-service:8000/health
|
|
```
|
|
|
|
#### Monitoring Issues
|
|
```bash
|
|
# Check Prometheus targets
|
|
kubectl port-forward svc/prometheus 9090:9090 -n telecom-monitoring
|
|
# Visit http://localhost:9090/targets
|
|
|
|
# Check Grafana data sources
|
|
kubectl port-forward svc/grafana 3001:3000 -n telecom-monitoring
|
|
# Visit http://localhost:3001/datasources
|
|
```
|
|
|
|
#### Storage Issues
|
|
```bash
|
|
# Check PVC status
|
|
kubectl get pvc -n telecom-platform
|
|
kubectl get pvc -n telecom-monitoring
|
|
|
|
# Check storage class
|
|
kubectl get storageclass
|
|
```
|
|
|
|
### Debug Commands
|
|
```bash
|
|
# Full cluster status
|
|
kubectl get all -n telecom-platform
|
|
kubectl get all -n telecom-monitoring
|
|
|
|
# Resource usage
|
|
kubectl top pods -n telecom-platform
|
|
kubectl top nodes
|
|
|
|
# Events
|
|
kubectl get events -n telecom-platform --sort-by='.lastTimestamp'
|
|
kubectl get events -n telecom-monitoring --sort-by='.lastTimestamp'
|
|
```
|
|
|
|
## Maintenance
|
|
|
|
### Updates
|
|
```bash
|
|
# Update images
|
|
kubectl set image deployment/api-server api-server=telecom/api-server:v2.0.0 -n telecom-platform
|
|
|
|
# Rolling restart
|
|
kubectl rollout restart deployment/api-server -n telecom-platform
|
|
|
|
# Check rollout status
|
|
kubectl rollout status deployment/api-server -n telecom-platform
|
|
```
|
|
|
|
### Backup
|
|
```bash
|
|
# Backup databases
|
|
kubectl exec -it postgres-0 -n telecom-platform -- pg_dump telecom_platform > backup.sql
|
|
|
|
# Backup configurations
|
|
kubectl get all -n telecom-platform -o yaml > platform-backup.yaml
|
|
kubectl get all -n telecom-monitoring -o yaml > monitoring-backup.yaml
|
|
```
|
|
|
|
### Cleanup
|
|
```bash
|
|
# Remove everything
|
|
./deploy.sh cleanup
|
|
|
|
# Remove specific components
|
|
kubectl delete -f manifests/applications.yaml
|
|
kubectl delete -f manifests/monitoring.yaml
|
|
```
|
|
|
|
## Production Considerations
|
|
|
|
### High Availability
|
|
- Use multiple replicas for stateless services
|
|
- Configure database replication
|
|
- Implement backup and disaster recovery
|
|
- Use anti-affinity rules for pod distribution
|
|
|
|
### Performance
|
|
- Enable resource limits and requests
|
|
- Use appropriate storage classes
|
|
- Configure network policies
|
|
- Monitor and optimize based on metrics
|
|
|
|
### Security
|
|
- Enable Pod Security Policies
|
|
- Use network segmentation
|
|
- Implement secret rotation
|
|
- Regular security scanning
|
|
|
|
### Compliance
|
|
- Enable audit logging
|
|
- Implement data retention policies
|
|
- Use encrypted storage
|
|
- Regular compliance checks
|
|
|
|
## Next Steps
|
|
|
|
1. **Customize Configuration**: Update for your environment
|
|
2. **Set Up DNS**: Configure hostnames in your DNS
|
|
3. **Configure TLS**: Update certificates for production
|
|
4. **Set Up Monitoring**: Configure alert recipients
|
|
5. **Implement Backup**: Set up automated backups
|
|
6. **Performance Tuning**: Optimize based on actual usage
|
|
|
|
## Support
|
|
|
|
For deployment issues:
|
|
1. Check the troubleshooting section
|
|
2. Review pod logs and events
|
|
3. Verify resource availability
|
|
4. Test with smaller deployments first
|
|
|
|
For application issues:
|
|
1. Check the application documentation
|
|
2. Review service health endpoints
|
|
3. Verify database connectivity
|
|
4. Check monitoring dashboards
|