EA Consent Issuer – Admin / DevOps Guide
Quick Navigation
This operations guide covers EA Consent Issuer deployment and administration in 8 sections:
- Getting Started (1-2): Overview and prerequisites
- Setup & Configuration (3-4): Docker deployment and YAML/secrets configuration
- Operations (5-7): Health monitoring, security controls, and logging
- Maintenance (8): Troubleshooting and incident response
New to deploying EA Consent Issuer? Start with sections 1-4 for initial deployment, then reference sections 5-8 for ongoing operations and troubleshooting.
Table of Contents
- EA Consent Issuer – Admin / DevOps Guide
- Quick Navigation
- 1. Overview
- 2. Prerequisites
- 3. Deployment
- 4. Configuration
- 5. Health Monitoring
- 6. Authorization & Security
- 7. Logs & Metrics
- 8. Troubleshooting
1. Overview
The EA Consent Issuer is a Docker application that provides an HTTP API to mint EasyAccess Consent credentials (W3C VCs v2.0) and wrap them in Trust Block credentials (also W3C VCs v2.0) for sharing into the Privacy Network.
From an operations perspective:
- Runtime: Docker container deployed to your private infrastructure
- Architecture: Two HTTP servers (API on port 8080, monitoring on port 8081)
- Processing: Accepts simplified consent JSON, validates schema, generates signed JWT credentials
- Output: Trust Block JWTs containing nested EA Consent VCs
- Integration: Runs behind your internal API gateway; forwards credentials to EA Consent Delta Writer
Trust Block Architecture (Nested JWT):
- Outer JWT (Trust Block): Wraps the EA Consent VC and documents privacy operations
- Inner JWT (EA Consent VC): The original consent credential
Both JWTs are independently signed and can be verified separately.
Key Characteristics:
- Stateless service (no persistent storage)
- JSON structured logging to stdout
- Graceful shutdown with 30-second timeout
- Private signing keys required at startup
- Minimal resource footprint (~256MB RAM)
2. Prerequisites
Required
- Docker Engine: Version 20.10 or later (or compatible container runtime like Podman)
- Network Access:
- Two available TCP ports (default: 8080 for API, 8081 for monitoring)
- Private network deployment recommended (not publicly exposed)
- Signing Keys:
- RSA (2048-bit minimum, 4096-bit default) for RS256
- ECDSA P-256 for ES256
- Unencrypted PEM format (PKCS#1, PKCS#8, or SEC1)
- Configuration File: YAML configuration file with issuer metadata
- Storage: Service does not read or write application data.
- Memory: Recommended minimum 256MB RAM
Recommended
- Private Network/VPC Deployment: Service should not be publicly exposed
- Secret Management: Supports CLI, ENVIONMENT variables, and AWS Secrets Manage for signing keys and bearer token.
- Health Check Integration: Configure container health checks using
/healthendpoint
3. Deployment
The EA Consent Issuer runs as a Docker container in your private infrastructure. It is designed to operate behind an API gateway or reverse proxy and should never be directly exposed to the public internet.
Docker Image Registry
The service is published to Docker Hub:
# Pull the latest image
docker pull webshield/ea-consent-issuer:1
Available Tags:
1- Most recent release- Semantic version tags (e.g.,
1.3.0) will be available in future releases
Basic Docker Deployment
Minimal deployment example:
docker run -d \
--name ea-consent-issuer \
-p 8080:8080 \
-p 8081:8081 \
-v /path/to/config.yaml:/app/config.yaml:ro \
-e SIGN_KEY_ID="production-key-2024-01" \
-e SIGN_KEY_PEM="$(cat /path/to/private-key.pem)" \
-e CLIENT_BEARER_TOKEN="fake_token" \
-e PUBLIC_ID_HASH_SALT="your-random-salt-32-bytes-minimum" \
webshield/ea-consent-issuer:1 \
-config /app/config.yaml
Deployment with AWS Secrets Manager:
docker run -d \
--name ea-consent-issuer \
-p 8080:8080 \
-p 8081:8081 \
-v /path/to/config.yaml:/app/config.yaml:ro \
-e SIGN_KEY_ID="aws_sm//production/ea-consent-issuer/sign-key-id" \
-e SIGN_KEY_PEM="aws_sm//production/ea-consent-issuer/signing-key#AWSCURRENT" \
-e CLIENT_BEARER_TOKEN="aws_sm//production/ea-consent-issuer/bearer-token" \
-e PUBLIC_ID_HASH_SALT="aws_sm//production/ea-consent-issuer/public-id-hash-salt" \
-e AWS_ACCESS_KID="${aws_kid}" \
-e AWS_ACCESS_SECRET="${aws_secret}" \
-e AWS_REGION="${aws_region}" \
webshield/ea-consent-issuer:1 \
-config /app/config.yaml
Network Configuration
Port Exposure:
- 8080 (API): Expose to your internal API gateway only (NOT directly to internet)
- 8081 (Monitoring): Internal monitoring only (should NOT be exposed externally)
Verification Steps
After deployment, verify the service is healthy:
1. Verify health endpoint:
curl http://localhost:8081/ea-consent-issuer/health
# Expected: {"status":"healthy"}
2. Check system status:
curl http://localhost:8081/ea-consent-issuer/status | jq
3. Verify API endpoint accessibility:
curl -X POST http://localhost:8080/ea-consent-issuer/api/v2/consents \
-H "Authorization: Bearer <your-token-here>" \
-H "Content-Type: application/json" \
-d '{
"subject": {"id": "test-user"},
"consent": {
"agreed": true,
"summary_html": "Test consent",
"details_html": "<p>Test consent details</p>",
"contains_ppn_consent": true
},
"evidence": [
{
"type": "AuthenticationEvidence",
"id_token": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.test.sig",
"verifies": ["email"]
}
]
}'
5. Check logs for successful startup:
Expected log entry:
{"time":"2025-11-14T15:30:00Z","level":"INFO","msg":"ea-consent-issuer started successfully","service":"EA-Consent-Issuer","event":"Service-Started"}
4. Configuration
The service is configured via a YAML file and runtime secrets (signing keys).
Configuration File Example
server:
api_port: 8080 # Port for API endpoints
monitor_port: 8081 # Port for monitoring endpoints (health and status)
debug: false # Enable debug logging AND allow ?debug=true query parameter (default: false)
custom_uri_prefix: "" # Optional customer-specific prefix for all routes (default: empty)
issuer:
url: https://consent.example.com # Base URL of the issuer (used in credential IDs)
name: Example Corp Consent Service # Human-readable issuer name
subject_handle:
scope: "global" # Scope identifier for subject handle generation (REQUIRED)
Configuration file location:
- Specify with
--configflag:--config /custom/path/config.yaml - Mount via volume:
-v /host/path/config.yaml:/app/config.yaml:ro
Secrets
The service requires signing key and client bearer token to be provided via environment variables or CLI flags. These secrets can be direct values or references to AWS Secrets Manager.
| Variable | Description | Required | Example |
|---|---|---|---|
SIGN_KEY_ID |
Key identifier (kid) for the signing key used in JWT headers | ✅ Required | production-key-2024-01 |
SIGN_KEY_PEM |
Private RSA/ECDSA/EdDSA PEM for signing consent VCs | ✅ Required | $(cat /path/to/private-key.pem) |
CLIENT_BEARER_TOKEN |
Bearer Token that must be passed by authorized client | ✅ Required | fake-token |
PUBLIC_ID_HASH_SALT |
Salt for creating privacy-preserving subject handles (minimum 32 bytes). CRITICAL: Must remain constant across service restarts - changing it breaks correlation of historical consents | ✅ Required | your-random-salt-32-bytes-minimum |
AWS_ACCESS_KID |
AWS Access Key ID | ⚠️ Required if using AWS SM | AKIAIOSFODNN7EXAMPLE |
AWS_ACCESS_SECRET |
AWS Secret Access Key | ⚠️ Required if using AWS SM | wJalrXUtnFEMI/K7MDENG/... |
AWS_REGION |
AWS Region for Secrets Manager | ⚠️ Required if using AWS SM | us-east-1 |
Notes:
- Use environment variables, secret files, or secret management systems
- Keys must be in valid PEM format (PKCS#1, PKCS#8, or SEC1)
- Supported algorithms: RSA (RS256, RS384, RS512), ECDSA (ES256, ES384, ES512), EdDSA (EdDSA)
Using AWS Secrets Manager (AWS SM) (Optional)
Instead of providing secret values directly via environment variables, you can reference secrets stored in AWS Secrets Manager using the aws_sm/ prefix. The service automatically detects when AWS Secrets Manager references are used and initializes AWS credentials accordingly.
Format: aws_sm//path/to/secret#version
- The path follows your AWS Secrets Manager naming convention
- Version is optional:
#AWSCURRENT(default),#AWSPREVIOUS, or a specific version ID
Example - Direct values (default):
export SIGN_KEY_ID="production-key-2024-01"
export SIGN_KEY_PEM="$(cat /path/to/private-key.pem)"
export CLIENT_BEARER_TOKEN="your-secret-token"
export PUBLIC_ID_HASH_SALT="your-random-salt-32-bytes-minimum"
Example - AWS Secrets Manager references:
# Reference secrets in AWS Secrets Manager
export SIGN_KEY_ID="aws_sm//production/ea-consent-issuer/sign-key-id"
export SIGN_KEY_PEM="aws_sm//production/ea-consent-issuer/signing-key#AWSCURRENT"
export CLIENT_BEARER_TOKEN="aws_sm//production/ea-consent-issuer/bearer-token"
export PUBLIC_ID_HASH_SALT="aws_sm//production/ea-consent-issuer/public-id-hash-salt"
# AWS credentials (required when using aws_sm/ prefix)
export AWS_ACCESS_KID="AKIAIOSFODNN7EXAMPLE"
export AWS_ACCESS_SECRET="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
export AWS_REGION="us-east-1"
Mixed Configuration: You can use direct values for some secrets and AWS Secrets Manager for others. The service will detect any aws_sm/ prefix and require AWS credentials.
IAM Roles: When running on AWS infrastructure (EC2, ECS, Lambda), you can use IAM roles instead of explicit AWS credentials. Configure the AWS_ACCESS_KID and AWS_ACCESS_SECRET environment variables with the appropriate IAM role credentials, or use the default credential chain provided by the AWS SDK.
Configuration Field Reference
Server Configuration
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
api_port |
int | No | 8080 | TCP port for API endpoints |
monitor_port |
int | No | 8081 | TCP port for monitoring endpoints (health and status) |
debug |
bool | No | false | Enable debug-level server-side logging (HTTP requests, internal operations) AND acts as a global guard for ?debug=true query parameter. When false, the ?debug=true query parameter is ignored (no debug data in responses). When true, enables both server logging and allows ?debug=true to include debug data in API responses. |
custom_uri_prefix |
string | No | ”” | Optional customer-specific prefix prepended to all routes (e.g., /custom1 → /custom1/ea-consent-issuer/...) |
Issuer Configuration
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
url |
string | Yes | - | Base URL of the credential issuer (used in credential IDs, subject IDs) |
name |
string | Yes | - | Human-readable name of the issuer (used in W3C Verifiable Credentials) |
Subject Handle Configuration
| Field | Type | Required | Default | Description |
|---|---|---|---|---|
scope |
string | Yes | - | Scope identifier for subject handle generation (e.g., “global”, “tenant-123”). Used to create privacy-preserving correlation handles that enable consent correlation across services without exposing actual identity |
Signing Keys Configuration
The service requires a private key to sign W3C Verifiable Credentials in JWT format. Signing keys are provided via CLI flags or environment variables (never in config files).
Supported Key Types
- RSA (2048-bit minimum, 4096-bit default) for RS256
- ECDSA P-256 for ES256
- Unencrypted PEM format (PKCS#1, PKCS#8, or SEC1)
Key Generation Examples
Generate RSA Key (2048-bit):
openssl genrsa -out signing-key.pem 2048
Generate RSA Key (4096-bit, recommended for production):
openssl genrsa -out signing-key.pem 4096
Generate ECDSA Key (P-256):
openssl ecparam -name prime256v1 -genkey -noout -out signing-key.pem
Key Rotation Procedure
To rotate signing keys without downtime:
- Generate new private key using one of the examples above
- Update environment variables with new key ID and PEM:
export SIGN_KEY_ID="production-key-2025-01" export SIGN_KEY_PEM="$(cat /path/to/new-signing-key.pem)" - Restart the service (graceful shutdown handles in-flight requests)
- Verify service startup and key validation in logs:
docker logs ea-consent-issuer | grep "Signing-Keys-Validated" - Keep old keys available for verifying previously issued credentials:
- The EA Consent Delta Writer fetches public keys from this service’s
/api/v2/keysendpoint - Delta Sharing stores all historical public keys in the
ea_consent_verification_keystable - Old keys are never deleted from Delta Sharing, ensuring historical credentials remain verifiable
- Do NOT delete old private keys until all credentials signed with them have expired
- The EA Consent Delta Writer fetches public keys from this service’s
- Publish new public key to verification endpoints:
- New key automatically appears in
/api/v2/keysafter service restart - Post the new key to the EA Consent Delta Writer verification keys endpoint
- Delta Sharing clients can verify both new and historical credentials
- New key automatically appears in
Recommended rotation frequency: Every 90-180 days for production environments.
Configuration Examples
Production Deployment
server:
api_port: 8080
monitor_port: 8081
debug: false # Disable debug mode in production
issuer:
url: https://consent.acmehealth.com
name: Acme Health Consent Service
subject_handle:
scope: "global"
Signing keys provided via environment:
export SIGN_KEY_ID="acme-prod-2024-11"
export SIGN_KEY_PEM="$(cat /secrets/production-signing-key.pem)"
Development with Debug Logging
server:
api_port: 8080
monitor_port: 8081
debug: true # Enable debug mode for development
issuer:
url: https://localhost:8080
name: Dev Consent Issuer
subject_handle:
scope: "dev"
5. Health Monitoring
The EA Consent Issuer provides dedicated monitoring endpoints for health checks, readiness probes, and runtime statistics.
Available Monitoring Endpoints
All monitoring endpoints run on the monitoring server (default port 8081, separate from API port 8080).
| Endpoint | Method | Purpose | Use Case |
|---|---|---|---|
/ea-consent-issuer/health |
GET | Quick health check | Container health checks, liveness probes |
/ea-consent-issuer/status |
GET | Detailed system status | Debugging, runtime statistics, error tracking |
Note: If custom_uri_prefix is configured, it applies to monitoring endpoints too (e.g., /custom1/ea-consent-issuer/health).
Health Check Endpoint
Endpoint: GET /ea-consent-issuer/health
Purpose: Quick boolean health check for container orchestration.
Example Request:
curl http://localhost:8081/ea-consent-issuer/health
Response (200 OK):
{
"status": "healthy"
}
Response Codes:
200 OK- Service is healthy and accepting requests5xx- Service is unhealthy or unavailable
Status Endpoint
Endpoint: GET /ea-consent-issuer/status
Purpose: Comprehensive system information including runtime statistics, error counts, and resource usage.
Example Request:
curl http://localhost:8081/ea-consent-issuer/status | jq
Response (200 OK):
{
"status": "healthy",
"system_info": {
"process_id": 12345,
"system_hostname": "ea-consent-issuer-7f8d9c-xyz",
"number_cpu": 8,
"start_mem_alloc": 2097152,
"current_mem_alloc": 5242880,
"num_go_routines": 12
},
"statistics": {
"server_error_stats": {
"server_internal_error_count": 0,
"bad_request_error_count": 3,
"unauthorized_error_count": 0,
"not_implemented_error_count": 0
}
}
}
Response Fields:
| Field | Description |
|---|---|
status |
Overall service health status (e.g., “healthy”) |
system_info.process_id |
Operating system process ID |
system_info.system_hostname |
Container/system hostname |
system_info.number_cpu |
Number of available CPUs |
system_info.start_mem_alloc |
Memory allocated at service start (bytes) |
system_info.current_mem_alloc |
Current memory allocation (bytes) |
system_info.num_go_routines |
Number of active goroutines |
statistics.server_error_stats.* |
Cumulative error counts by type |
6. Authorization & Security
The EA Consent Issuer is designed to operate within a private network or VPC behind an API gateway or reverse proxy.
Security Model
| Layer | Implementation | Status |
|---|---|---|
| Network Isolation | Private VPC / API Gateway | ✅ Required |
| API Authentication | Built-in CLIENT_BEARER_TOKEN validation | ✅ Implemented |
| JWT Signing | Private key signing of credentials | ✅ Implemented |
| TLS/HTTPS | Reverse proxy or gateway | ✅ Recommended |
| Rate Limiting | Gateway or reverse proxy | ⚙️ Optional |
| JWT signing | All trust blocks and EA Consents are cryptographically signed (prevents tampering) | ✅ Implemented |
API Authentication
- Protected API endpoints require bearer token authentication via
CLIENT_BEARER_TOKEN(configured in Secrets section). - Configured via environment variable at container startup
- Unauthenticated requests to protected endpoints are rejected with 401 Unauthorized
Protected vs Public Endpoints:
- Protected (require bearer token):
POST /consents,POST /consent-batches - Public (no auth required):
GET /metadata,GET /keys,GET /verification-keys - Monitoring (no auth, internal only):
/health,/status
Request Format:
curl -X POST http://localhost:8080/ea-consent-issuer/api/v2/consents \
-H "Authorization: Bearer YOUR_CLIENT_BEARER_TOKEN" \
-H "Content-Type: application/json" \
-d '{...}'
Additional Gateway Security (Optional):
- API Gateway can provide additional layers: mTLS, IP allowlisting, OAuth2, rate limiting
- TLS/HTTPS termination should be handled by reverse proxy or API gateway
- Monitoring port (8081) should be accessible only to internal monitoring systems
JWT Signing and Verification
Signing Process:
- Service receives consent input via API
- Generates EA Consent VC and Trust Block as JWTs
- Signs both JWTs with private key (configured via
SIGN_KEY_PEM) - Returns signed Trust Block to client
Verification Process (downstream services):
- Fetch public keys from
/api/v2/keys(JWKS format) - Verify Trust Block JWT signature using public key
- Extract EA Consent VC from Trust Block
- Verify EA Consent VC JWT signature
Public Key Distribution:
- Direct verification: Services can fetch keys directly from this service’s
/api/v2/keysendpoint - Via Delta Sharing: The EA Consent Delta Writer fetches public keys from
/api/v2/verification-keysand stores them in Delta Sharing’sea_consent_verification_keystable - Historical keys: Delta Sharing maintains all historical public keys, allowing verification of credentials issued with rotated keys
- Client access: Delta Sharing clients query historical keys by
issuerandkid(key ID) to verify any credential regardless of when it was issued
Example: Fetching verification keys:
curl http://localhost:8080/ea-consent-issuer/api/v2/keys
Response:
{
"keys": [
{
"kty": "RSA",
"kid": "production-key-2024-01",
"use": "sig",
"alg": "RS256",
"n": "xGOr_H3N...",
"e": "AQAB"
}
]
}
Security Checklist for Operators
Before deploying to production, ensure:
- Deploy in private subnet (no public IP exposure)
- Configure CLIENT_BEARER_TOKEN (strong, unique token via environment variable or secret manager)
- Enable HTTPS/TLS at API gateway (TLS 1.2+ recommended)
- Restrict ingress to trusted IP ranges (firewall rules or security groups)
- Use strong signing keys (RSA 4096-bit or ECDSA P-384 minimum)
- Store secrets securely (AWS Secrets Manager, HashiCorp Vault, or Azure Key Vault)
- Rotate signing keys every 90-180 days (test rotation procedure first)
- Restrict monitoring port (8081) access (internal monitoring systems only)
- Forward logs to centralized logging (ELK, Splunk, CloudWatch, or Datadog)
- Disable debug mode in production (
server.debug: false) - Set up alerting (error rates, memory leaks, health check failures)
- Test graceful shutdown and restart (verify in-flight request handling)
- Document incident response procedures (key compromise, service outage)
Security Roadmap
| Feature | Current Status | Future Enhancements |
|---|---|---|
| Network Isolation | ✅ Private VPC recommended | - |
| API Authentication | ✅ CLIENT_BEARER_TOKEN validation | JWT token validation, OAuth2 |
| JWT Signing | ✅ RSA/ECDSA/EdDSA credentials | - |
| TLS/HTTPS | ⚙️ Gateway-managed (recommended) | Optional built-in TLS |
| Rate Limiting | ⚙️ Gateway-managed (optional) | Built-in rate limiting |
| Audit Logging | ✅ Structured JSON logs | Enhanced audit events |
7. Logs & Metrics
The EA Consent Issuer outputs structured JSON logs to stdout for centralized log aggregation and analysis.
Log Output Format
All logs are written to stdout in JSON format, making them easy to parse and index.
Example log entry:
{
"time": "2025-11-14T15:30:00Z",
"level": "INFO",
"msg": "ea-consent-issuer started successfully",
"service": "EA-Consent-Issuer",
"event": "Service-Started",
"filename": "cmd/server/main.go",
"version": "1.3.0"
}
Standard Fields:
| Field | Description | Example |
|---|---|---|
time |
ISO 8601 timestamp | 2025-11-14T15:30:00Z |
level |
Log level (DEBUG, INFO, WARN, ERROR) | INFO |
msg |
Human-readable message | Service started successfully |
service |
Service name (always “EA-Consent-Issuer”) | EA-Consent-Issuer |
event |
Structured event name for filtering | Service-Started |
filename |
Source file that generated the log | cmd/server/main.go |
version |
Service version | 1.3.0 |
Log Levels
| Level | When Used | Recommended Action |
|---|---|---|
DEBUG |
Only when server.debug: true - verbose operational details |
Normal in development; disable in production |
INFO |
Normal operations, startup, requests, configuration loaded | Monitor for service lifecycle events |
WARN |
Degraded operation, retries, non-critical issues | Investigate if frequent |
ERROR |
Failures, exceptions, critical issues | Alert and investigate immediately |
Key Log Events
| Event Name | Level | When Logged | Description |
|---|---|---|---|
Service-Started |
INFO | Startup | Service successfully started |
Configuration-Loaded |
INFO | Startup | Configuration file loaded successfully |
Signing-Keys-Validated |
INFO | Startup | Signing keys validated successfully |
Add-API-HTTPHandler |
INFO | Startup | API route registered |
Add-Health-HTTPHandler |
INFO | Startup | Monitoring route registered |
Server-Starting |
INFO | Startup | HTTP server starting |
HTTP-Request |
DEBUG | Each request | HTTP request details (requires debug: true) |
Shutdown-Signal-Received |
INFO | Shutdown | SIGINT/SIGTERM received |
Server-Shutdown |
INFO | Shutdown | Server shutting down gracefully |
All-Servers-Stopped |
INFO | Shutdown | All servers stopped successfully |
Config-Load-Error |
ERROR | Startup | Failed to load configuration |
Services-Start-Error |
ERROR | Startup | Failed to initialize application |
API-Server-Failed |
ERROR | Runtime | API server error |
Health-Server-Failed |
ERROR | Runtime | Health server error |
Server-Shutdown-Error |
ERROR | Shutdown | Error during shutdown |
Accessing Logs
Docker (local development):
# Follow logs in real-time
docker logs -f ea-consent-issuer
# View last 100 lines
docker logs ea-consent-issuer --tail 100
# Filter for errors only
docker logs ea-consent-issuer | jq 'select(.level=="ERROR")'
# Watch for specific events
docker logs -f ea-consent-issuer | jq 'select(.event=="Service-Started")'
# Find logs for a specific request ID
docker logs ea-consent-issuer | jq 'select(.request_id=="f47ac10b-58cc-4372-a567-0e02b2c3d479")'
Kubernetes:
# Follow logs
kubectl logs -f deployment/ea-consent-issuer
# View logs from specific pod
kubectl logs ea-consent-issuer-7f8d9c-xyz
# Filter for errors
kubectl logs deployment/ea-consent-issuer | jq 'select(.level=="ERROR")'
Debug Mode Logging
When server.debug: true is set in configuration, the service logs additional DEBUG-level events:
HTTP Request Logging (each request):
{
"time": "2025-11-14T15:30:01Z",
"level": "DEBUG",
"msg": "http request",
"service": "EA-Consent-Issuer",
"event": "HTTP-Request",
"filename": "httphandlers/middleware/logging.go",
"method": "POST",
"path": "/ea-consent-issuer/api/v2/consents",
"status": 201,
"duration_ms": 15,
"remote_addr": "192.168.1.100:54239",
"user_agent": "curl/8.1.2",
"content_type": "application/json",
"content_length": 512,
"query": "debug=true",
"referer": "https://example.com"
}
⚠️ Production Warning: Debug mode significantly increases log volume and may log sensitive information. Only enable temporarily for troubleshooting.
Log Aggregation
Centralized Logging Integration:
The service outputs JSON logs to stdout, making it easy to integrate with log aggregation platforms:
Metrics Tracking
Operators can derive metrics from logs and the /status endpoint:
| Metric | Source | Query/Extraction |
|---|---|---|
| Request Count | DEBUG logs | Count HTTP-Request events |
| Request Duration | DEBUG logs | Extract duration_ms field from HTTP-Request events |
| Error Rate | /status endpoint |
server_error_stats.server_internal_error_count / total requests |
| 4xx Error Count | /status endpoint |
server_error_stats.bad_request_error_count |
| 5xx Error Count | /status endpoint |
server_error_stats.server_internal_error_count |
| Memory Usage | /status endpoint |
system_info.current_mem_alloc |
| Goroutine Count | /status endpoint |
system_info.num_go_routines |
| Uptime | Logs | Time since Service-Started event |
8. Troubleshooting
This section documents common operational issues, symptoms, causes, and resolution steps.
Common Issues
| Symptom | Likely Cause | Resolution |
|---|---|---|
| Container exits immediately | Missing or invalid signing keys | Verify SIGN_KEY_ID and SIGN_KEY_PEM environment variables are set. Check logs for Services-Start-Error event. See Missing Signing Keys. |
/health endpoint returns 500 or times out |
Service failed to initialize or dependencies unavailable | Check logs for startup errors (Config-Load-Error, Services-Start-Error). Verify configuration file is valid YAML. Ensure signing keys are in valid PEM format. |
422 Unprocessable Entity errors |
Invalid JSON request or schema validation failure | Validate request payload structure. Check error.description field for specific validation failures (e.g., missing summary_html). See developer guide for schema details. |
400 Bad Request errors |
Malformed JSON or incorrect Content-Type header | Ensure Content-Type: application/json header is set. Validate JSON syntax (use jq or JSON validator). |
| Port already in use | Another process is using port 8080 or 8081 | Change api_port or monitor_port in config, or stop conflicting service: lsof -ti:8080 \| xargs kill |
| Configuration file not found | Invalid config file path or missing volume mount | Verify config file exists at specified path. Check Docker volume mount: -v /host/path/config.yaml:/app/config.yaml:ro |
| High memory or CPU usage | Large batch size or debug mode enabled | Reduce batch size (max 50 items). Disable debug mode (server.debug: false) for production. Monitor /status endpoint for resource metrics. |
| Logs not appearing | Logs sent to stderr or container stopped | Verify container is running: docker ps. Check Docker log driver configuration. Use docker logs <container> to view stdout/stderr. |
| JWT signature verification fails (downstream) | Public key mismatch or key rotation not synced | Ensure downstream services fetch latest keys from /api/v2/keys. Verify kid in JWT header matches published key. Check key rotation timing. |