Troubleshooting Guide

​

Installation Failed

1. Check prerequisites are installed:
   go version
   docker --version
   docker compose version
   psql --version
   redis-cli --version
   
2. Ensure proper permissions:
   # macOS/Linux
   mkdir -p ~/rapida-data/assets
   chmod -R 755 ~/rapida-data
   
   # Linux with Docker group
   sudo usermod -aG docker $USER
   newgrp docker
   
3. Run setup command:
   make setup-local
   

​

Database Connection Failed

• Error messages mentioning connection refused
• Services failing to start
• Logs showing database errors

1. Verify PostgreSQL is running:
   # macOS
   brew services list | grep postgres
   
   # Linux
   sudo systemctl status postgresql
   
   # Or check directly
   psql -h localhost -U rapida_user -d web_db -c "SELECT 1;"
   
2. Check database credentials:
   # Verify user exists
   psql postgres -c "\du"
   
   # Verify databases exist
   psql postgres -c "\l"
   
3. Check connection settings in .env:
   # Docker
   grep POSTGRES docker/web-api/.web.env
   
   # Manual setup
   grep POSTGRES env/.web.env
   
4. Restart PostgreSQL:
   # macOS
   brew services restart postgresql
   
   # Linux
   sudo systemctl restart postgresql
   
   # Docker
   make down-db
   make up-db
   
5. Check PostgreSQL logs:
   # Docker
   docker logs postgres
   
   # macOS
   tail -f /usr/local/var/log/postgres.log
   
   # Linux
   sudo tail -f /var/log/postgresql/postgresql.log
   

​

Redis Connection Issues

• Cache operations failing
• Session errors
• Redis connection refused messages

1. Verify Redis is running:
   redis-cli ping
   # Should return: PONG
   
2. Start Redis:
   # macOS
   brew services start redis
   
   # Linux
   sudo systemctl start redis-server
   
   # Docker
   make up-redis
   
3. Check Redis logs:
   # Docker
   docker logs redis
   
   # Linux
   sudo tail -f /var/log/redis/redis-server.log
   
4. Test Redis connection:
   redis-cli -h localhost -p 6379
   > ping
   PONG
   > exit
   

​

OpenSearch Won’t Start

• Container immediately exits
• curl: (7) Failed to connect to localhost port 9200
• Out of memory errors

1. Check resource limits:
   # OpenSearch requires 512MB minimum
   # Increase Docker memory allocation
   # Docker Desktop -> Preferences -> Resources -> Memory: 8GB
   
   # Or reduce allocation in docker-compose.yml:
   OPENSEARCH_JAVA_OPTS: "-Xms256m -Xmx256m"
   
2. Check logs:
   docker logs opensearch
   
3. Reset OpenSearch:
   # Docker
   make down-opensearch
   docker volume rm $(docker volume ls | grep opensearch | awk '{print $2}')
   make up-opensearch
   
   # Or manually
   rm -rf ~/rapida-data/assets/opensearch/*
   make up-opensearch
   
4. Verify health:
   # Check cluster health
   curl -s http://localhost:9200/_cluster/health | jq .
   
   # Expected: "status": "green" or "yellow"
   

​

Service Issues

​

Service Won’t Start

• Container exits after a few seconds
• Status shows “Exited (1)”
• Error logs in docker logs

1. Check service logs:
   # Docker
   docker logs web-api
   
   # Manual setup
   # Check terminal where service was running
   
2. Check port availability:
   # Find process using port
   lsof -i :9001
   
   # Kill process or change port
   kill -9 <PID>
   # Or edit docker-compose.yml/config
   
3. Verify database is ready:
   # PostgreSQL must be running first
   psql -h localhost -U rapida_user -d web_db -c "SELECT 1;"
   
   # If not ready, wait and retry
   make down-web
   sleep 10
   make up-web
   
4. Check configuration:
   # Verify environment variables
   docker exec web-api env | grep POSTGRES
   
   # Check config syntax
   cat docker/web-api/.web.env
   

​

Services Crashing After Startup

• Services in “Exited” state
• Intermittent crashes
• Error logs showing timeouts or connection issues

1. Check resource constraints:
   # Monitor resource usage
   docker stats
   
   # If 100% CPU or memory, increase limits or optimize queries
   
2. Check database connections:
   # Too many connections?
   psql -h localhost -U rapida_user -d web_db
   SELECT count(*) FROM pg_stat_activity;
   
   # Increase max connections
   ALTER SYSTEM SET max_connections = 200;
   
3. Review logs for patterns:
   docker logs web-api --tail=100 -f
   
   # Look for patterns in error messages
   

​

High Memory Usage

1. Check which service is consuming memory:
   docker stats
   
2. For OpenSearch (common culprit):
   # Reduce memory allocation
   # Edit docker-compose.yml
   OPENSEARCH_JAVA_OPTS: "-Xms256m -Xmx256m"
   
   # Restart
   docker compose restart opensearch
   
3. For PostgreSQL:
   # Check for slow queries
   psql -h localhost -U rapida_user -d web_db
   SELECT query, calls, total_time FROM pg_stat_statements
   ORDER BY total_time DESC LIMIT 10;
   
4. General troubleshooting:
   # Restart affected service
   docker compose restart web-api
   
   # Clean up unused data
   docker system prune -a
   

​

Network Issues

​

Cannot Access Services

• curl: (7) Failed to connect
• Connection refused
• Services ping but don’t respond

1. Verify services are running:
   docker ps | grep rapida
   # All services should show "Up"
   
2. Check port bindings:
   # Verify ports are mapped
   docker port web-api
   # Should show: 9001/tcp -> 0.0.0.0:9001
   
3. Test network connectivity:
   # From host
   curl http://localhost:9001/readiness/
   
   # If fails, check if service is responsive
   docker exec web-api curl http://localhost:9001/readiness/
   
4. Check firewall:
   # macOS
   sudo lsof -i :9001
   
   # Linux
   sudo netstat -tlnp | grep 9001
   
5. Verify DNS resolution:
   # Check service name resolution in Docker
   docker exec web-api ping postgres
   docker exec web-api ping redis
   docker exec web-api ping opensearch
   

​

Services Can’t Communicate

• Service logs showing “host not found”
• Connection timeouts between services
• Integration failures

1. Verify network:
   # Check if services on same network
   docker network ls
   docker network inspect api-network
   
2. Test service-to-service connectivity:
   # From web-api container
   docker exec web-api curl http://assistant-api:9007/readiness/
   
   # Check DNS resolution
   docker exec web-api nslookup assistant-api
   
3. Check Docker Compose:
   # Ensure services are on api-network
   grep "networks:" docker-compose.yml
   grep "api-network" docker-compose.yml
   
4. Recreate network:
   docker compose down
   docker network rm api-network
   docker compose up -d
   

​

Data Issues

​

Lost Data After Restart

1. Verify volumes are properly mounted:
   # Check volume mounts
   docker inspect postgres | grep -A 5 Mounts
   
   # Should show volume mount to ~/rapida-data/assets/db
   
2. Check Docker Compose volume configuration:
   # Verify volumes section
   grep -A 20 "volumes:" docker-compose.yml | head -20
   
3. Ensure data directory exists:
   ls -la ~/rapida-data/assets/
   # Should show: db/, redis/, opensearch/ directories
   
4. Backup and restore data:
   # Backup PostgreSQL
   docker compose exec postgres pg_dump -U rapida_user web_db > backup.sql
   
   # Restore
   docker compose exec -T postgres psql -U rapida_user web_db < backup.sql
   

​

Database Corruption

• Strange SQL errors
• Data appearing corrupted
• Ability to read but not write

1. Check database integrity:
   psql -h localhost -U rapida_user -d web_db
   SELECT pg_database.datname, pg_database_size(pg_database.datname)
   FROM pg_database;
   
2. VACUUM and ANALYZE:
   psql -h localhost -U rapida_user -d web_db
   VACUUM FULL;
   ANALYZE;
   
3. Rebuild indexes:
   psql -h localhost -U rapida_user -d web_db
   REINDEX DATABASE web_db;
   
4. Full restoration:
   # Stop services
   make down-all
   
   # Backup current data
   cp -r ~/rapida-data/assets/db ~/rapida-data/assets/db.backup
   
   # Reset database
   rm -rf ~/rapida-data/assets/db/*
   
   # Start and restore
   make up-db
   # Wait for initialization
   docker compose exec -T postgres psql -U rapida_user web_db < backup.sql
   

​

Performance Issues

​

Slow API Responses

1. Check database performance:
   psql -h localhost -U rapida_user -d web_db
   SELECT query, calls, total_time FROM pg_stat_statements
   ORDER BY total_time DESC LIMIT 10;
   
2. Monitor resource usage:
   docker stats
   
   # If CPU/memory at limit, need to optimize or increase
   
3. Check query plans:
   psql -h localhost -U rapida_user -d web_db
   EXPLAIN ANALYZE SELECT * FROM conversations WHERE id = '123';
   
4. Create missing indexes:
   # Common queries to index:
   CREATE INDEX idx_conversations_user_id ON conversations(user_id);
   CREATE INDEX idx_messages_conversation_id ON messages(conversation_id);
   
5. Reduce logging verbosity:
   # Change LOG_LEVEL from debug to info
   docker compose restart web-api
   

​

High Database Connection Usage

1. Monitor connections:
   psql -h localhost -U rapida_user -d web_db
   SELECT datname, count(*) FROM pg_stat_activity GROUP BY datname;
   
2. Increase connection limit:
   POSTGRES__MAX_OPEN_CONNECTION=20
   POSTGRES__MAX_IDEAL_CONNECTION=10
   
3. Kill idle connections:
   psql -h localhost -U rapida_user -d web_db
   SELECT pg_terminate_backend(pid) FROM pg_stat_activity
   WHERE state = 'idle' AND query_start < now() - interval '1 hour';
   

​

Slow Search/Analytics

1. Check OpenSearch status:
   curl http://localhost:9200/_cluster/health
   
2. Monitor task queue:
   curl http://localhost:9200/_tasks?detailed
   
3. Optimize index settings:
   # Reduce refresh frequency
   curl -X PUT "localhost:9200/assistant-conversations/_settings" \
     -H 'Content-Type: application/json' \
     -d '{"settings": {"refresh_interval": "30s"}}'
   
4. Delete old indices:
   # Remove indices older than 30 days
   curl -X DELETE "localhost:9200/logs-2024-01-*"
   

​

Debugging

​

Enable Debug Logging

# Update .env file
LOG_LEVEL=debug

# Restart service
docker compose restart web-api

# Monitor logs
docker logs -f web-api

​

Check Service Configuration

# Verify service has correct config
docker exec web-api env | grep POSTGRES
docker exec web-api env | grep REDIS

# Check if service can reach dependencies
docker exec web-api curl http://postgres:5432 2>&1
docker exec web-api curl http://redis:6379 2>&1

​

View Network Details

# List all networks
docker network ls

# Inspect api-network
docker network inspect api-network

# View network traffic (if netcat installed)
docker exec web-api netcat -zv redis 6379

​

Health Checks

# Readiness check (service ready to accept traffic)
curl -s http://localhost:9001/readiness/ | jq .

# Liveness check
curl -s http://localhost:9001/healthz/ | jq .

​

Getting Help

​

Collect Debug Information

# System info
uname -a
docker --version
docker compose version

# Service status
docker ps -a
docker network ls

# Recent logs
docker logs web-api --tail=50 > web-api.log
docker logs postgres --tail=50 > postgres.log
docker logs redis --tail=50 > redis.log

# Resource usage
docker stats --no-stream > stats.log

# Configuration
cat env/.web.env > config-web.log

​

Report Issues

1. Steps to reproduce
2. Error message/logs
3. System information
4. What you tried to fix it
5. Output of debug commands above

​

Contact Support

• GitHub Issues: https://github.com/rapidaai/voice-ai/issues
• Documentation: https://rapida.ai/docs
• Community: https://github.com/rapidaai/voice-ai/discussions
• Email: prashant@rapida.ai

​

Quick Reference

​

Common Commands

# Status
make ps-all
docker ps

# Logs
make logs-all
docker logs -f web-api

# Restart services
make restart-all
docker compose restart

# Clean up
make clean
docker compose down -v

# Database
make shell-db
psql -h localhost -U rapida_user -d web_db

# Redis
redis-cli -h localhost -p 6379

# Verify services
curl http://localhost:9001/readiness/

​

Next Steps

1. Installation - Installation guide
2. Configuration - Config-related issues