at-container-registry/docs/TROUBLESHOOTING.md

ATCR Troubleshooting Guide

This document provides troubleshooting guidance for common ATCR deployment and operational issues.

OAuth Authentication Failures

JWT Timestamp Validation Errors

Symptom:

error: invalid_client
error_description: Validation of "client_assertion" failed: "iat" claim timestamp check failed (it should be in the past)

Root Cause: The AppView server's system clock is ahead of the PDS server's clock. When the AppView generates a JWT for OAuth client authentication (confidential client mode), the "iat" (issued at) claim appears to be in the future from the PDS's perspective.

Diagnosis:

1. Check AppView system time:

date -u
timedatectl status

2. Check if NTP is active and synchronized:

timedatectl show-timesync --all

3. Compare AppView time with PDS time (if accessible):

# On AppView
date +%s

# On PDS (or via HTTP headers)
curl -I https://your-pds.example.com | grep -i date

4. Check AppView logs for clock information (logged at startup):

docker logs atcr-appview 2>&1 | grep "Configured confidential OAuth client"

Example log output (the startup message logs key_id and key_path only; system time is not included):

level=INFO msg="Configured confidential OAuth client" key_id=did:key:z... key_path=/var/lib/atcr/oauth/client.key

To verify clock skew separately, compare the output of date +%s on the AppView host with the Date: header returned by the PDS (see step 3 above).

Solution:

1. Enable NTP synchronization (recommended):

   On most Linux systems using systemd:

   # Enable and start systemd-timesyncd
   sudo timedatectl set-ntp true
   
   # Verify NTP is active
   timedatectl status
   

   Expected output:

   System clock synchronized: yes
   NTP service: active
   

2. Alternative: Use chrony (if systemd-timesyncd is not available):

   # Install chrony
   sudo apt-get install chrony  # Debian/Ubuntu
   sudo yum install chrony       # RHEL/CentOS
   
   # Enable and start chronyd
   sudo systemctl enable chronyd
   sudo systemctl start chronyd
   
   # Check sync status
   chronyc tracking
   

3. Force immediate sync:

   # systemd-timesyncd
   sudo systemctl restart systemd-timesyncd
   
   # Or with chrony
   sudo chronyc makestep
   

4. In Docker/Kubernetes environments:

   The container inherits the host's system clock, so fix NTP on the host machine:

   # On Docker host
   sudo timedatectl set-ntp true
   
   # Restart AppView container to pick up correct time
   docker restart atcr-appview
   

5. Verify clock skew is resolved:

   # Should show clock offset < 1 second
   timedatectl timesync-status
   

Acceptable Clock Skew:

• Most OAuth implementations tolerate ±30-60 seconds of clock skew
• DPoP proof validation is typically stricter (±10 seconds)
• Aim for < 1 second skew for reliable operation

Prevention:

• Configure NTP synchronization in your infrastructure-as-code (Terraform, Ansible, etc.)
• Monitor clock skew in production (e.g., Prometheus node_exporter includes clock metrics)
• Use managed container platforms (ECS, GKE, AKS) that handle NTP automatically

---

DPoP Nonce Mismatch Errors

Symptom:

error: use_dpop_nonce
error_description: DPoP "nonce" mismatch

Repeated multiple times, potentially followed by:

error: server_error
error_description: Server error

Root Cause: DPoP (Demonstrating Proof-of-Possession) requires a server-provided nonce for replay protection. These errors typically occur when:

1. Multiple concurrent requests create a DPoP nonce race condition
2. Clock skew causes DPoP proof timestamps to fail validation
3. PDS session state becomes corrupted after repeated failures

Diagnosis:

1. Check if errors occur during concurrent operations:

# During docker push with multiple layers
docker logs atcr-appview 2>&1 | grep "use_dpop_nonce" | wc -l

2. Check for clock skew (see section above):

timedatectl status

3. Look for session lock acquisition in logs (requires debug-level logging):

ATCR_LOG_LEVEL=debug docker logs atcr-appview 2>&1 | grep "Acquired session lock"

Solution:

1. If caused by clock skew: Fix NTP synchronization (see section above)

2. If caused by session corruption:

   # The AppView will automatically delete corrupted sessions
   # User just needs to re-authenticate
   docker login atcr.io
   

3. If persistent despite clock sync:

   • Check PDS health and logs (may be a PDS-side issue)
   • Verify network connectivity between AppView and PDS
   • Check if PDS supports latest OAuth/DPoP specifications

What ATCR does automatically:

• Per-DID locking prevents concurrent DPoP nonce races
• Indigo library automatically retries with fresh nonces
• Sessions are auto-deleted after repeated failures
• Service token cache prevents excessive PDS requests

Prevention:

• Ensure reliable NTP synchronization
• Use a stable, well-maintained PDS implementation
• Monitor AppView error rates for DPoP-related issues

---

OAuth Session Not Found

Symptom:

error: failed to get OAuth session: no session found for DID

Root Cause:

• User has never authenticated via OAuth
• OAuth session was deleted due to corruption or expiry
• Database migration cleared sessions

Solution:

1. User re-authenticates via OAuth flow:

   docker login atcr.io
   # Or for web UI: visit https://atcr.io/login
   

2. If using app passwords (legacy), verify token is cached:

   # Check if app-password token exists
   docker logout atcr.io
   docker login atcr.io -u your.handle -p your-app-password
   

---

AppView Deployment Issues

Client Metadata URL Not Accessible

Symptom:

error: unauthorized_client
error_description: Client metadata endpoint returned 404

Root Cause: PDS cannot fetch OAuth client metadata from {ATCR_SERVER_BASE_URL}/client-metadata.json

Diagnosis:

1. Verify client metadata endpoint is accessible:

   curl https://your-atcr-instance.com/client-metadata.json
   

2. Check AppView logs for startup errors:

   docker logs atcr-appview 2>&1 | grep "client-metadata"
   

3. Verify ATCR_SERVER_BASE_URL is set correctly:

   echo $ATCR_SERVER_BASE_URL
   

Solution:

1. Ensure ATCR_SERVER_BASE_URL matches your public URL:

   export ATCR_SERVER_BASE_URL=https://atcr.example.com
   

2. Verify reverse proxy (nginx, Caddy, etc.) routes /.well-known/* and /client-metadata.json:

   location / {
       proxy_pass http://localhost:5000;
       proxy_set_header Host $host;
       proxy_set_header X-Forwarded-Proto $scheme;
   }
   

3. Check firewall rules allow inbound HTTPS:

   sudo ufw status
   sudo iptables -L -n | grep 443
   

---

Hold Service Issues

Blob Storage Connectivity

Symptom:

error: failed to upload blob: connection refused

Diagnosis:

1. Check hold service logs:

   docker logs atcr-hold 2>&1 | grep -i error
   

2. Verify S3 credentials are correct:

   # Test S3 access
   aws s3 ls s3://your-bucket --endpoint-url=$S3_ENDPOINT
   

3. Check hold configuration:

   env | grep -E "(S3_|AWS_|STORAGE_)"
   

Solution:

1. Verify environment variables in hold service:

   export AWS_ACCESS_KEY_ID=your-key
   export AWS_SECRET_ACCESS_KEY=your-secret
   export S3_BUCKET=your-bucket
   export S3_ENDPOINT=https://s3.us-west-2.amazonaws.com
   

2. Test S3 connectivity from hold container:

   docker exec atcr-hold curl -v $S3_ENDPOINT
   

3. Check S3 bucket permissions (requires PutObject, GetObject, DeleteObject)

---

Hold Dropped by the Relay

Symptom:

Push and pull counts on the AppView go stale (up to 24 hours behind), or the hold's records stop appearing anywhere on the network, while the hold itself serves pushes and pulls normally and logs no errors at all.

Diagnosis:

curl -s "https://relay1.us-west.bsky.network/xrpc/com.atproto.sync.getHostStatus?hostname=hold.example.com"

"status":"offline" means the relay gave up dialing your hold and, because the hold only sends requestCrawl at boot, it will never retry on its own.

Solution:

Request a crawl from the hold admin panel's Relays tab, or restart the hold.

Before doing either, confirm the deployed build contains commit ca539b1. Older builds can crash during the firehose replay that a reconnecting relay triggers.

Full write-up, including the 2026-08-20 fleet outage, the diagnosis procedure, and a deferred proposal for automatic recovery: hold.md, "Relay Subscription".

---

Performance Issues

High Database Lock Contention

Symptom: Slow Docker push/pull operations, high CPU usage on AppView

Diagnosis:

1. Check SQLite database size:

   ls -lh /var/lib/atcr/ui.db
   

2. Look for long-running queries:

   docker logs atcr-appview 2>&1 | grep "database is locked"
   

Solution:

1. For production, enable libSQL embedded-replica mode to offload reads to a remote sync target (Turso cloud or self-hosted libsql-server):

   export ATCR_UI_LIBSQL_SYNC_URL=libsql://your-db.turso.io
   export ATCR_UI_LIBSQL_AUTH_TOKEN=your-auth-token
   export ATCR_UI_LIBSQL_SYNC_INTERVAL=60s   # optional, default 60s
   

   Or in YAML (config-appview.yaml):

   ui:
     libsql_sync_url: libsql://your-db.turso.io
     libsql_auth_token: your-auth-token
     libsql_sync_interval: 60s
   

   PostgreSQL is not supported; the AppView uses SQLite/libSQL only.

2. Vacuum the database to reclaim space:

   sqlite3 /var/lib/atcr/ui.db "VACUUM;"
   

---

Logging and Debugging

Enable Debug Logging

Set log level to debug for detailed troubleshooting:

export ATCR_LOG_LEVEL=debug
docker restart atcr-appview

Useful Log Queries

OAuth token exchange errors:

docker logs atcr-appview 2>&1 | grep "OAuth callback failed"

Service token request failures:

docker logs atcr-appview 2>&1 | grep "OAuth authentication failed during service token request"

Clock diagnostics (confidential OAuth client startup):

docker logs atcr-appview 2>&1 | grep "Configured confidential OAuth client"

DPoP nonce issues:

docker logs atcr-appview 2>&1 | grep -E "(use_dpop_nonce|DPoP)"

Health Checks

AppView health:

curl http://localhost:5000/v2/
# Should return: {"errors":[{"code":"UNAUTHORIZED",...}]}

Hold service health:

curl http://localhost:8080/.well-known/did.json
# Should return DID document

---

Getting Help

If issues persist after following this guide:

1. Check Issues: https://tangled.org/evan.jarrett.net/at-container-registry/issues
2. Collect logs: Include output from docker logs for AppView and Hold services
3. Include diagnostics:
   • timedatectl status output
   • AppView version: docker exec atcr-appview cat /VERSION (if available)
   • PDS version and implementation (Bluesky PDS, other)
4. File an issue with reproducible steps

---

Common Error Reference

Error Code	Component	Common Cause	Fix
invalid_client (iat timestamp)	OAuth	Clock skew	Enable NTP sync
use_dpop_nonce	OAuth/DPoP	Concurrent requests or clock skew	Fix NTP, wait for auto-retry
server_error (500)	PDS	PDS internal error	Check PDS logs
invalid_grant	OAuth	Expired auth code	Retry OAuth flow
unauthorized_client	OAuth	Client metadata unreachable	Check ATCR_SERVER_BASE_URL and firewall
RecordNotFound	ATProto	Manifest doesn't exist	Verify repository name
Connection refused	Hold/S3	Network/credentials	Check S3 config and connectivity