From b7c93e24a5d2a029791c65ab1bf405f8a389ac82 Mon Sep 17 00:00:00 2001 From: nutcas3 Date: Fri, 17 Apr 2026 17:41:31 +0300 Subject: [PATCH] feat: Add comprehensive monitoring stack documentation with architecture, setup, and troubleshooting guide - Add monitoring/README.md with complete observability stack documentation - Add architecture diagram showing Prometheus, Grafana, InfluxDB data flow - Add component descriptions for Prometheus exporter, time-series storage, Grafana dashboards, Docker Compose services - Add quick start guide with service access URLs and API server metrics setup - Add configuration section with environment variables, --- monitoring/README.md | 335 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 335 insertions(+) create mode 100644 monitoring/README.md diff --git a/monitoring/README.md b/monitoring/README.md new file mode 100644 index 0000000..61e686e --- /dev/null +++ b/monitoring/README.md @@ -0,0 +1,335 @@ +# Telecom Platform Monitoring Stack + +This directory contains the complete monitoring and observability setup for the Telecom Platform, including Prometheus metrics collection, Grafana dashboards, and InfluxDB time-series storage. + +## Architecture + +``` +Telecom Platform Services + | + | Metrics (Prometheus format) + v ++-----------------+ +| Prometheus | <- Scrapes metrics from all services ++-----------------+ + | + | Query API + v ++-----------------+ +| Grafana | <- Visualization and alerting ++-----------------+ + | + | Long-term storage + v ++-----------------+ +| InfluxDB | <- Time-series data storage ++-----------------+ +``` + +## Components + +### 1. Prometheus Metrics Exporter + +**Location**: `apps/api-server/internal/metrics/prometheus.go` + +**Features**: +- HTTP request metrics (rate, duration, status codes) +- Subscriber metrics (total, active, suspended) +- Usage metrics (data, voice, SMS) +- Charging metrics (credit balance, transactions) +- System metrics (sessions, uptime, database connections) +- Chaos engineering metrics +- eBPF packet gateway metrics + +**Key Metrics**: +- `telecom_http_requests_total` - Total HTTP requests +- `telecom_subscribers_total` - Total subscriber count +- `telecom_data_usage_bytes_total` - Data usage in bytes +- `telecom_credit_balance` - Subscriber credit balances +- `telecom_active_sessions` - Active session count +- `telecom_packets_processed_total` - eBPF packet processing + +### 2. Advanced Time-Series Storage + +**Location**: `apps/api-server/internal/metrics/timeseries.go` + +**Features**: +- InfluxDB integration for long-term storage +- High-resolution usage metrics +- Historical data analysis +- Complex queries and aggregations +- Top user analytics + +**Data Types**: +- Usage metrics (data up/down, voice, SMS) +- System metrics (CPU, memory, network) +- Charging metrics (transactions, balances) +- Packet processing metrics + +### 3. Grafana Dashboards + +**Location**: `monitoring/grafana/dashboards/` + +#### Dashboard: Telecom Platform Overview +- Total subscribers and active sessions +- Data usage rates +- HTTP request rates +- Subscriber status breakdown +- Payment transaction rates + +#### Dashboard: Telecom Subscriber Metrics +- Top 10 data users +- Subscriber credit balances +- Low balance alerts +- Data usage distribution +- Voice & SMS usage rates + +### 4. Docker Compose Setup + +**Location**: `monitoring/docker-compose.yml` + +**Services**: +- **Prometheus** (port 9090) - Metrics collection and storage +- **Grafana** (port 3001) - Visualization and dashboards +- **InfluxDB** (port 8086) - Time-series data storage +- **Redis** (port 6379) - Caching and session storage +- **Node Exporter** (port 9100) - System metrics +- **Alertmanager** (port 9093) - Alert management + +## Quick Start + +### 1. Start Monitoring Stack + +```bash +cd monitoring +docker-compose up -d +``` + +### 2. Access Services + +- **Prometheus**: http://localhost:9090 +- **Grafana**: http://localhost:3001 (admin/admin) +- **InfluxDB**: http://localhost:8086 + +### 3. Start API Server with Metrics + +```bash +cd apps/api-server +go run cmd/server.go +``` + +The API server will: +- Expose metrics on `http://localhost:9090/metrics` +- Store time-series data in InfluxDB (if configured) +- Update Prometheus metrics every 30 seconds + +## Configuration + +### Environment Variables + +```bash +# API Server +METRICS_PORT=9090 + +# InfluxDB (optional) +INFLUXDB_URL=http://localhost:8086 +INFLUXDB_TOKEN=your-token +INFLUXDB_ORG=telecom +INFLUXDB_BUCKET=telecom +``` + +### Prometheus Configuration + +**Location**: `monitoring/prometheus/prometheus.yml` + +Key scrape targets: +- `telecom-api-server` - Main API metrics +- `telecom-charging-engine` - Rust charging engine metrics +- `telecom-packet-gateway` - eBPF packet gateway metrics + +### Grafana Data Sources + +**Location**: `monitoring/grafana/provisioning/datasources/datasources.yml` + +- **Prometheus** - Primary metrics source +- **InfluxDB** - Time-series analytics + +## Metrics Collection + +### Automatic Collection + +The metrics collector automatically gathers: +1. **HTTP Metrics** - Request rate, duration, status codes +2. **Business Metrics** - Subscriber counts, usage data +3. **System Metrics** - Database connections, Redis operations +4. **Chaos Metrics** - Experiment results and status + +### Custom Metrics + +Add custom metrics in your service code: + +```go +// Record custom usage metrics +metricsCollector.RecordDataUsage("subscriber123", "up", 1024) +metricsCollector.UpdateCreditBalance("subscriber123", 25.50) + +// Record chaos experiment +metricsCollector.RecordChaosExperiment("latency", "completed") +``` + +### Time-Series Storage + +Store detailed metrics in InfluxDB: + +```go +usageMetrics := metrics.UsageMetrics{ + SubscriberID: "subscriber123", + IMSI: "208930000000001", + DataUp: 1024 * 1024, // 1MB + DataDown: 2 * 1024 * 1024, // 2MB + VoiceSeconds: 300, + SMSCount: 5, + Timestamp: time.Now(), +} + +err := timeSeriesStorage.StoreUsageMetrics(ctx, usageMetrics) +``` + +## Alerting + +### Prometheus Alert Rules + +Create alert rules in `monitoring/prometheus/alert_rules.yml`: + +```yaml +groups: + - name: telecom_alerts + rules: + - alert: HighErrorRate + expr: rate(telecom_http_requests_total{status_code=~"5.."}[5m]) > 0.1 + for: 5m + labels: + severity: critical + annotations: + summary: "High error rate detected" + + - alert: LowCreditBalance + expr: telecom_credit_balance < 5 + for: 2m + labels: + severity: warning + annotations: + summary: "Subscriber has low credit balance" +``` + +### Grafana Alerting + +Configure alerts in Grafana dashboards: +- Low balance alerts +- High error rate alerts +- System resource alerts +- Usage threshold alerts + +## Advanced Features + +### 1. Distributed Tracing + +Integration with Jaeger for request tracing: +- Trace HTTP requests across services +- Track GraphQL query performance +- Monitor database query performance + +### 2. Log Aggregation + +ELK stack integration for log aggregation: +- Centralized log collection +- Log parsing and indexing +- Log-based alerting + +### 3. Business Intelligence + +Advanced analytics with InfluxDB: +- Customer usage patterns +- Revenue analytics +- Network performance trends +- Capacity planning + +## Troubleshooting + +### Common Issues + +1. **Metrics not appearing in Prometheus** + - Check if metrics endpoint is accessible: `curl http://localhost:9090/metrics` + - Verify Prometheus configuration targets + - Check network connectivity + +2. **Grafana dashboards not loading** + - Verify Grafana can reach Prometheus: `curl http://prometheus:9090/api/v1/query?query=up` + - Check data source configuration + - Verify dashboard JSON syntax + +3. **InfluxDB connection issues** + - Verify InfluxDB is running: `curl http://localhost:8086/ping` + - Check authentication token + - Verify bucket and organization names + +### Debug Commands + +```bash +# Check Prometheus targets +curl http://localhost:9090/api/v1/targets + +# Query specific metric +curl "http://localhost:9090/api/v1/query?query=telecom_subscribers_total" + +# Check InfluxDB health +curl http://localhost:8086/health + +# View Grafana data sources +curl -u admin:admin http://localhost:3001/api/datasources +``` + +## Performance Considerations + +### Metrics Collection Overhead +- Prometheus scraping adds minimal overhead (~1-2ms per request) +- InfluxDB writes are batched for efficiency +- Time-series storage is optional and can be disabled + +### Storage Requirements +- Prometheus: ~1-2GB per month for medium load +- InfluxDB: Depends on retention period and data volume +- Grafana: Minimal storage for dashboards + +### Scaling +- Prometheus can handle ~10k metrics per second +- InfluxDB scales horizontally with clustering +- Grafana supports multiple data sources + +## Security + +### Authentication +- Grafana: Configure LDAP/OAuth integration +- InfluxDB: Use authentication tokens +- Prometheus: Use basic auth or TLS + +### Network Security +- Run monitoring stack in isolated network +- Use TLS for all communications +- Implement firewall rules for metric endpoints + +## Next Steps + +1. **Custom Dashboards** - Create domain-specific dashboards +2. **Advanced Alerting** - Implement predictive alerting +3. **ML Integration** - Add anomaly detection +4. **Automation** - Auto-scaling based on metrics +5. **Compliance** - Audit logging and retention policies + +## Support + +For monitoring-related issues: +1. Check service logs: `docker-compose logs [service]` +2. Verify configuration files +3. Test connectivity between services +4. Check resource usage and limits