Troubleshooting¶

Common issues and solutions when deploying with fujin.

Quick Diagnostic Commands ¶

When something goes wrong, start with these:

# Check service status
fujin app status

# View recent logs
fujin app logs

# Check deployment history
fujin audit

# Verify configuration
fujin show env
fujin show caddy
fujin show units

# Check server connectivity
ssh user@your-server.com

Deployment Issues ¶

“Connection refused” or “Permission denied”¶

Symptoms: - Cannot SSH to server - fujin commands fail with connection errors

Solutions:

Check SSH connection:

# Test SSH manually
ssh user@your-server.com

# Check if key is loaded
ssh-add -l

# Add key if needed
ssh-add ~/.ssh/id_rsa

Verify fujin.toml configuration:

[[hosts]]
user = "deploy"  # Correct username?
domain_name = "example.com"  # Correct domain?
key_filename = "~/.ssh/id_rsa"  # Correct key path?

Check server firewall:

ssh user@server sudo ufw status
# If SSH blocked:
sudo ufw allow 22/tcp

“Build failed” errors ¶

Symptoms: - Deployment fails during build phase - “Command not found” errors

Solutions:

Check build_command:

# Test build locally
uv build

# Check pyproject.toml is valid
cat pyproject.toml

Verify dependencies:

# Sync dependencies
uv sync

# Check if all required packages are listed
grep dependencies pyproject.toml

Check Python version:

# In fujin.toml
python_version = "3.12"  # Matches your local version?

“Upload failed” or checksum mismatch ¶

Symptoms: - Bundle upload succeeds but verification fails - Checksum errors

Solutions:

Retry deployment:
```
fujin deploy
```
Check disk space on server:
```
ssh user@server df -h
```

Network issues - try with smaller timeout:

# Increase SSH timeout
ssh user@server 'cat > /dev/null'  # Keep connection alive

Application Won’t Start ¶

Services fail to start after deployment ¶

Symptoms: - Deployment succeeds but app doesn’t respond - Services show as “failed” or “inactive”

Diagnosis:

# Check status
fujin app status

# View logs
fujin app logs web

# Check systemd status directly
ssh user@server systemctl --user status yourapp.service

Common Causes:

Missing environment variables:

# Check what's configured
fujin show env

# Verify all required vars are set
fujin show env --plain | grep -E "SECRET_KEY|DATABASE_URL"

Database connection issues:
```
# Test database connection
ssh user@server psql -h localhost -U dbuser -d dbname -c 'SELECT 1;'
```
If this fails: - Check database is running: sudo systemctl status postgresql - Verify credentials in .env file - Check database exists: psql -l

Port/socket already in use:

# Check what's using the port
ssh user@server lsof -i :8000

# Or for socket
ssh user@server ls -la /run/yourapp/

Permission issues:

# Check app directory permissions
ssh user@server ls -la ~/apps/yourapp

# Fix if needed
ssh user@server chmod -R 755 ~/apps/yourapp

“ModuleNotFoundError” or import errors ¶

Symptoms: - Python can’t find installed packages - Import errors in logs

Solutions:

Check virtual environment:

# Verify venv exists
ssh user@server ls ~/apps/yourapp/.venv

# Check installed packages
ssh user@server ~/apps/yourapp/.venv/bin/pip list

Reinstall dependencies:

# SSH to server
ssh user@server

# Navigate to app
cd ~/apps/yourapp

# Reinstall
.venv/bin/pip install -r requirements.txt

Check requirements.txt:

# Verify requirements.txt was included in deployment
ssh user@server cat ~/apps/yourapp/requirements.txt

“Gunicorn bind failed” errors ¶

Symptoms: - Gunicorn can’t bind to socket/port - “Address already in use” errors

Solutions:

Check socket directory exists:

# Create runtime directory if missing
ssh user@server mkdir -p /run/yourapp
ssh user@server chown $USER:$USER /run/yourapp

Kill stale processes:

# Find processes using the socket
ssh user@server lsof /run/yourapp/yourapp.sock

# Kill if needed
ssh user@server pkill -f gunicorn

# Restart service
fujin app restart web

Use systemd socket activation:

# In fujin.toml
[processes.web]
command = ".venv/bin/gunicorn ..."
socket = true  # Let systemd manage the socket

Web Server Issues ¶

“502 Bad Gateway” errors ¶

Symptoms: - Caddy returns 502 - Site unreachable but Caddy is running

Diagnosis:

# Check if application is running
fujin app status

# Check Caddy logs
ssh user@server sudo journalctl -u caddy -n 50

# Check upstream connection
fujin show caddy  # Verify upstream setting

Solutions:

Application not running:
```
fujin app start web
```

Wrong socket path:

# Check actual socket location
ssh user@server ls -la /run/yourapp/

# Update fujin.toml if needed
[webserver]
upstream = "unix//run/yourapp/yourapp.sock"  # Match actual path

Permissions on socket:

Fujin automatically handles socket permissions by:

Adding UMask=0002 to service files (makes sockets group-writable)
Adding the caddy user to your app’s group during deployment

If you’re experiencing permission issues:

# Verify caddy is in the app group
ssh user@server groups caddy
# Should show: caddy : caddy {app_name}

# If missing, add it manually
ssh user@server sudo usermod -aG {app_name} caddy
ssh user@server sudo systemctl restart caddy

# Check socket permissions
ssh user@server ls -la /run/{app_name}/
# Socket should be: srwxrwxr-x (group-writable)

Static files not loading (404 errors)¶

Symptoms: - CSS/JS/images return 404 - Django admin has no styling

Diagnosis:

# Check if files were collected
ssh user@server ls -la /var/www/yourapp/static/

# Check Caddy config
fujin show caddy

Solutions:

Static files not collected:

# Run collectstatic
fujin app exec collectstatic --no-input

# Or redeploy (release_command should collect)
fujin deploy

Wrong static directory in Caddy:

# In fujin.toml
[webserver.statics]
"/static/*" = "/var/www/{app_name}/static/"  # Check path is correct

Permission issues:

# Fix permissions
ssh user@server sudo chown -R $USER:www-data /var/www/yourapp
ssh user@server sudo chmod -R 755 /var/www/yourapp

SSL certificate issues ¶

Symptoms: - HTTPS not working - “Certificate error” in browser - Caddy won’t start

Diagnosis:

# Check Caddy status
ssh user@server sudo systemctl status caddy

# Check Caddy logs
ssh user@server sudo journalctl -u caddy -n 100

Solutions:

Domain not pointing to server:

# Check DNS
dig your-domain.com

# Should return your server's IP

Port 443 blocked:

# Check firewall
ssh user@server sudo ufw status

# Allow HTTPS if blocked
ssh user@server sudo ufw allow 443/tcp

Caddy config error:

# Test Caddy config
ssh user@server sudo caddy validate --config /etc/caddy/Caddyfile

# Check generated Caddyfile
fujin show caddy

Worker/Background Task Issues ¶

Celery workers not processing tasks ¶

Symptoms: - Tasks stuck in pending state - No worker logs

Diagnosis:

# Check worker status
fujin app status | grep worker

# Check worker logs
fujin app logs worker

# Test Celery connection
fujin server exec "celery -A yourapp inspect ping"

Solutions:

Workers not running:
```
fujin app start worker
```

Redis/RabbitMQ not running:

# Check Redis
ssh user@server sudo systemctl status redis

# Start if stopped
ssh user@server sudo systemctl start redis

Wrong broker URL:

fujin show env | grep CELERY_BROKER_URL

# Should match your Redis/RabbitMQ setup

Task not registered:

Check that Celery autodiscover is configured:
```
# celery.py
app.autodiscover_tasks()
```

Beat scheduler not running tasks ¶

Symptoms: - Scheduled tasks don’t run - Beat logs show no activity

Solutions:

Check beat is running:
```
fujin app status | grep beat
```
Check beat schedule:
```
fujin app logs beat
```
Restart beat:
```
fujin app restart beat
```

Timer-based tasks not running ¶

Symptoms: - Systemd timer exists but task doesn’t run

Diagnosis:

# Check timer status
fujin app logs healthcheck.timer

# Check when it will run next
ssh user@server systemctl --user list-timers

Solutions:

Timer not enabled:

ssh user@server systemctl --user enable yourapp-healthcheck.timer
ssh user@server systemctl --user start yourapp-healthcheck.timer

Check timer configuration:
```
fujin show healthcheck.timer
```

Configuration Issues ¶

“Invalid configuration” errors ¶

Symptoms: - fujin commands fail with validation errors - TOML parsing errors

Solutions:

Validate TOML syntax:

# Use Python to check
python3 -c "import tomllib; tomllib.load(open('fujin.toml', 'rb'))"

Common TOML mistakes:

# Wrong: Using quotes for table names
[host]  # Should be [[hosts]]

# Wrong: Mixing inline and regular tables
[processes]
web = { command = "..." }
[processes.worker]  # Can't mix inline and block tables

# Correct:
[processes.web]
command = "..."
[processes.worker]
command = "..."

Check for required fields:

# These are required
app = "myapp"  # App name
[[hosts]]  # At least one host
domain_name = "example.com"  # Domain required
user = "deploy"  # User required

Secrets not resolving ¶

Symptoms: - Environment variables show as $variable instead of values - “Secret not found” errors

Diagnosis:

# Check if secrets are redacted (good)
fujin show env

# Check actual values (careful!)
fujin show env --plain

Solutions:

Secret manager not authenticated:

# Bitwarden
bw login
bw unlock
export BW_SESSION="..."

# Or use password env
export BW_PASSWORD="your-password"

Wrong secret name:

# List secrets
bw list items

# Check exact name matches
# In .env: SECRET_KEY=$my-secret-key
# In Bitwarden: Item name must be exactly "my-secret-key"

Secret manager CLI not installed:

# Install Bitwarden CLI
# See: https://bitwarden.com/help/cli/

Getting Help ¶

If you’re still stuck:

Check logs carefully:

fujin app logs -n 200  # Last 200 lines

Search GitHub issues:

https://github.com/falcopackages/fujin/issues
Create a minimal reproduction:
- Fresh Django project
- Simplest fujin.toml
- Document exact steps to reproduce
Report the issue:

Include:
- fujin version (fujin --version)
- OS (uname -a)
- fujin.toml (sanitized)
- Full error output
- Relevant logs