SSH & SQL Bastion

SSH Bastion Gateway

Secure SSH gateway with OAuth 2.0 device code authentication, interactive command shell, SFTP relay, port forwarding, SQL bastion, session recording, and certificate-based SSH proxying

Overview

The bastion runtime provides a secure SSH gateway with OAuth 2.0 Device Authorization Grant (RFC 8628) authentication. It offers:

• Interactive command shell (REPL) with bash-style features (tab completion, syntax

  highlighting, history search with Ctrl+R, bang expansion)

• OAuth 2.0 Device Code authentication with QR code display (no passwords or SSH keys)
• SSH certificate-based proxying to destination hosts via ephemeral Ed25519 certificates

  signed by the bastion's User CA

• Port forwarding with rule-based ACL (local -L, remote -R, dynamic -D SOCKS proxy)
• SQL bastion for secure database access (MySQL, PostgreSQL) with 7-layer query validation
• SFTP subsystem relay for secure file transfer through the bastion (virtual filesystem,

  per-host connections with ephemeral certificates, audit logging)

• Session recording in asciinema v2 format for audit and compliance
• Geo-IP and time-based access restrictions (inheritable from [service] config)
• Comprehensive DoS protection (connection, auth, session, and command rate limiting)
• Distributed session storage for cluster-wide visibility and admin tools
• Background OIDC token and userinfo refresh with automatic session termination

  on user disable

• SSRF protection via configurable CIDR blocklists
• Extensible command plugin system with group-based per-command authorization
• Built-in Snake game (‘snake’ command) that keeps sessions alive during port forwarding

The bastion integrates with the cluster for distributed session management and certificate operations.

Config

Core configuration under [bastion]:

[bastion]

  enabled = true                       # Enable SSH bastion service
  port = 2222                          # SSH listen port (default: 2222)
  network_interface = "eth0"           # Network interface to bind (default: eth0)
  host_key_salt = "..."                # 32+ char entropy for deterministic key generation
  required_groups = ["bastion_admins"] # Groups required for bastion access (empty = any user)
  device_code_timeout = "5m"           # Device code expiry (default: 5m)
  idle_timeout = "30m"                 # Session idle timeout (default: 30m)
  max_session_duration = "24h"         # Maximum session duration (default: 24h)
  blocked_cidrs = []                   # CIDR blocklist for SSRF protection (empty = allow all)

  # SSH Certificate Authority
  ca_user_comment = "hexon-user-ca@hexon.io"   # User CA identifier
  ca_host_comment = "hexon-host-ca@hexon.io"   # Host CA identifier
  ca_threshold = false                     # FROST threshold Ed25519 for User CA (requires cluster_mode)

  # Shell Features
  enable_syntax_highlighting = true    # Real-time command coloring (default: true)
  enable_autosuggestions = false        # Autosuggestion from history (default: false)
  cursor_style = "default"             # Cursor: default, blinking_block, steady_block,
                                       # blinking_underline, steady_underline, blinking_bar, steady_bar

  # Command History
  history_enabled = true               # Enable history tracking (default: true)
  history_case_insensitive = false     # Case-insensitive Ctrl+R search (default: false)
  history_ignore_dups = true           # Ignore consecutive duplicates (default: true)
  history_ignore_space = false         # Ignore commands starting with space (default: false)
  history_bang_expansion = true        # Enable bang (!) expansion (default: true)

  # Session Recording
  enable_recording = false             # Enable asciinema v2 recording (default: false)
  recording_path = "/var/log/hexon/bastion"  # Recording storage directory
  recording_retention_days = 90        # Retention before cleanup (default: 90, 0 = forever)

  # Token/UserInfo Refresh
  refresh_access_token = "45m"         # OIDC access token refresh interval (default: 45m)
  refresh_user_info = "5m"             # UserInfo refresh interval (default: 5m)

  # Connection Timeouts
  ssh_connection_timeout = "30s"       # SSH TCP + handshake timeout (default: 30s)
  port_forward_timeout = "10s"         # Port forward connection timeout (default: 10s)
  sql_connection_timeout = "10s"       # SQL connection + ping timeout (default: 10s)

Geo-IP and time-based restrictions (falls back to [service] if not set in [bastion]):

[bastion]

  geo_enabled = true                      # Enable bastion-specific geo restrictions
  geo_allow_countries = ["US", "CA"]      # Country allowlist
  geo_deny_countries = ["CN", "RU"]       # Country denylist
  geo_allow_asn = ["AS15169"]             # ASN allowlist
  geo_deny_asn = ["AS12345"]              # ASN denylist
  geo_bypass_cidr = ["10.0.0.0/8"]        # CIDR ranges bypassing geo checks
  time_enabled = true                     # Enable bastion-specific time restrictions
  time_timezone = "America/New_York"      # Default timezone
  time_allow_days = ["Mon","Tue","Wed","Thu","Fri"]
  time_allow_hours = "09:00-17:00"
  time_deny_message = "SSH access not permitted outside business hours"
  time_bypass_cidr = ["10.0.0.0/8"]

SFTP subsystem relay:

[bastion.sftp]

  enabled = false                          # Enable SFTP subsystem relay (default: false)
  required_groups = ["sftp_users"]         # Groups required for SFTP access (empty = any bastion user)
  idle_timeout = "10m"                     # Idle timeout for SFTP sessions (default: 10m)

SFTP paths use /principal@hostname[:port]/remote/path format. Connect with:

  sftp -P 2222 bastion.example.com

The virtual root ’/’ lists all principal@hostname combinations from SSH cert rules matching the user’s groups (e.g., root@db-server, deploy@web-server, gedo@db-server). An explicit principal is required — no auto-selection. Navigate using the entries shown in the virtual root listing.

When navigating to a host without a subpath (cd /root@db-server), the session starts in the remote user’s home directory (e.g., /root), matching standard SFTP behavior. The pwd command reflects the full virtual path: /root@db-server/root.

Connections to remote hosts use ephemeral Ed25519 certificates signed by the bastion User CA (same as SSH proxying). Host key verification is non-interactive: only CA-signed or previously approved keys are accepted. Unknown hosts must be approved via interactive SSH first. Connector site routing is supported for remote destinations.

Port forwarding ACL:

[bastion.port_forwarding_acl]

  enabled = true                         # Enable port forwarding
  required_groups = []                   # Groups required for any forwarding
  default_allow = false                  # Deny unmatched requests (DANGEROUS if true)
  dynamic_acl = true                     # Per-user ACL from directory
  max_forwards_per_session = 10          # Concurrent forward limit
  idle_timeout = "15m"                   # Port forward idle timeout
  allowed_destinations = ["localhost:*", "*.hexon.private:22"]

  [[bastion.port_forwarding_acl.rules]]
  name = "database-access"
  groups = ["developers", "dba"]
  destinations = ["db*.hexon.private", "10.10.0.0/24"]
  allowed_ports = ["3306", "5432"]       # Single ports, ranges ("8080-8090"), or "*"
  allowed_types = ["local"]              # local, remote, dynamic
  source_cidrs = []                      # Source IP restriction
  max_forwards_per_session = 5           # Override global limit

SSH certificate configuration:

[bastion.ssh_cert]

  enabled = true
  cert_ttl = "1m"                        # Certificate validity (default: 1m)
  source_cidrs = []                      # Source IP restriction on cert
  permit_pty = true
  permit_port_forwarding = false
  permit_agent_forwarding = false

  [[bastion.ssh_cert.rules]]
  name = "admin-root-access"
  groups = ["admins"]
  destinations = ["*"]
  principals = ["root", "admin"]
  cert_ttl = "15m"                       # Override global TTL
  permit_port_forwarding = true

  [[bastion.ssh_cert.rules]]
  name = "remote-dc-access"
  groups = ["devops"]
  destinations = ["*.internal"]
  site = "prod-dc-a8f3c1"               # Route SSH through connector tunnel
  principals = ["deploy"]

DoS protection:

[bastion.dos_protection]

  max_global_connections = 1000          # Total concurrent SSH connections
  max_connections_per_ip = 5             # Concurrent connections per IP
  connection_rate_limit = "5/1m"         # New connection rate per IP
  auth_rate_limit = "10/1m"              # Auth attempts per IP
  auth_failure_threshold = 5             # Failures before ban
  auth_failure_ban_duration = "1h"       # Ban duration
  max_sessions_per_user = 5              # Sessions per user
  max_sessions_per_ip = 3                # Sessions per IP
  max_total_sessions = 1000              # Total sessions cluster-wide
  session_creation_rate = "1/5s"         # Session creation rate per IP
  command_rate_limit = "100/1m"          # Commands per minute per session
  max_command_length = 4096              # Max command length in bytes
  max_history_size = 1000                # Max history entries per session
  qr_generation_rate = "1/10s"           # QR code generation rate per IP

SQL bastion:

[sql_bastion]

  enabled = true

  [[sql_bastion.sites]]
  name = "mysql-hexon"
  type = "mysql"                         # mysql or postgres
  host = "mysql.hexon.private"
  port = 3306
  user = "hexon"
  password = "${MYSQL_PASSWORD}"         # Env var expansion
  database = "hexon"
  ssl = true
  skip_tls = false                       # Skip server cert verification (self-signed certs)
  ssl_ca = "/path/to/ca.crt"
  ssl_cert = "/path/to/client.crt"       # Mutual TLS (optional)
  ssl_key = "/path/to/client.key"

  [[sql_bastion.sites]]
  name = "postgres-remote"
  type = "postgres"
  host = "pg.internal"
  port = 5432
  user = "app"
  password = "${PG_PASSWORD}"
  database = "production"
  site = "prod-dc-a8f3c1"               # Route through connector tunnel

    [[sql_bastion.sites.acls]]
    acl = "bastion_mysql"
    groups = ["developers"]
    readonly = true
    query_timeout = "5s"
    query_max_limit = 100
    allowed_tables = ["*"]
    forbidden_tables = ["user_credentials", "api_keys"]

      [sql_bastion.sites.acls.rate_limit]
      queries_per_minute = 60
      burst_allowed = 10

      [sql_bastion.sites.acls.query_limits]
      max_complexity_score = 100
      max_execution_time = "30s"
      max_rows = 10000

Hot-reloadable: history settings, DoS protection limits, timeouts, geo/time restrictions, port forwarding ACL rules, SQL ACL settings. Cold (restart required): bastion.enabled, port, network_interface, host_key_salt.

Security

Authentication:

• OAuth 2.0 Device Authorization Grant (RFC 8628) — no passwords or SSH keys to manage
• SSH password and public key authentication are explicitly DISABLED (rejected)
• Keyboard-interactive enabled only for SSH protocol setup; actual auth is device code flow
• Group-based access control via required_groups (LDAP/directory integration)
• Background OIDC token refresh terminates sessions when users are disabled (within 5m)

SSH Certificate Authority:

• Two CAs: User CA (signs user certs) and Host CA (signs host keys)
• User CA: deterministically derived from host_key_salt (default) or threshold FROST

  Ed25519 (ca_threshold=true) — when threshold mode is enabled, the User CA key is
  generated via distributed key generation (FROST DKG), not derived from any seed.
  Fail-closed: no SSH user certs until FROST DKG completes. Host CA stays salt-derived.

• Ephemeral certificates for SSH proxying (default TTL: 1 minute)
• Certificate extensions: permit-pty, permit-port-forwarding, permit-agent-forwarding
• Critical options: source-address restriction, force-command restriction
• Multiple matching rules merge with most-privileged-wins semantics
• Private keys zeroed on bastion shutdown

Geo-IP restrictions (checked at TCP layer before SSH handshake):

• Blocked clients get silent TCP close (no information leakage)
• Fail-closed: check errors result in connection denial
• Localhost bypass for web shell (127.0.0.1 and ::1)

Time-based restrictions (checked after SSH session start):

• Blocked users see configurable denial message before graceful close
• Fail-closed: check errors result in access denial
• Localhost bypass for web shell

Port forwarding security:

• All forwarding hostnames resolved via DNS module with DNSSEC validation
• DNSSEC failure blocks the forward (no system DNS fallback for security)
• Rule-based ACL with first-match-wins evaluation
• Source CIDR restrictions limit forward origination
• Per-session forward limits prevent resource exhaustion

SSRF protection:

• Configurable CIDR blocklist for SSH connections and port forwards
• Default open (bastion is a security tool); explicit blocklist required
• Invalid IPs blocked by default (fail-safe)
• Applied after DNS resolution, before connection attempt

DoS protection layers:

• Connection: global limit, per-IP limit, connection rate limit
• Authentication: auth rate limit, failure tracking with auto-ban
• Session: global/per-user/per-IP limits, creation rate limit
• Command: per-session rate limit, command length truncation

SQL bastion security:

• Multi-layer query validation pipeline (rate limiting, readonly enforcement,

  database access control, pattern blocking, table restrictions, complexity scoring,
  LIMIT enforcement)

• Dangerous operation blocking: file I/O, privilege escalation, bypass attempts,

  and system catalog access are blocked before query execution

• SQL-aware parsing: properly handles comments, quoted strings, and dialect-specific

  syntax for both MySQL and PostgreSQL

• Readonly enforcement at multiple levels with SQL-aware analysis
• Table access control with extraction from all common SQL statement types
• TLS 1.2+ enforced, mutual TLS support, skip_tls for self-signed certs
• Passwords expanded from environment variables, never logged
• All queries logged with unique UUIDs for audit correlation

Troubleshooting

Common symptoms and diagnostic steps:

Connection refused or timeout:

• Bastion not enabled: ‘config show bastion’ to verify enabled=true
• Wrong port: verify port setting (default: 2222)
• Network interface binding: check network_interface matches actual interface
• Listener not started: ‘logs search bastion’ for initialization errors
• Firewall blocking: ‘firewall rules’ and ‘firewall check <username>’
• Global connection limit reached: ‘bastion metrics’ shows connection_limit_hits
• Per-IP limit: check dos_protection.max_connections_per_ip (default: 5)
• Rate limited: ‘ratelimit stats’ for ban status
• Start with: ‘diagnose user <username>’ for cross-subsystem check

Authentication failures:

• Device code timeout: increase device_code_timeout (default: 5m)
• Group access denied: ‘directory user <username>’ to check group membership
• Auth rate limited: check dos_protection.auth_rate_limit (default: 10/1m)
• IP banned from failures: ‘ratelimit stats’ to check bans
• OIDC provider unreachable: ‘auth health’ and ‘net http <oidc-url>’
• Device code polling slow_down errors: check polling tolerance window in logs

Session disconnects unexpectedly:

• Idle timeout: check idle_timeout (default: 30m). Users with port forwards can run

  'snake' to keep the session alive while monitoring forward traffic.

• Max session duration: check max_session_duration (default: 24h)
• Token refresh failure: user may have been disabled in directory
• UserInfo refresh failure: check refresh_user_info interval (default: 5m)
• Check: ‘sessions list —user=<username>’ and ‘bastion sessions’
• Session cleanup: runs every 5 minutes, check logs for eviction reasons

Geo-IP or time-based blocks:

• Geo check: ‘geo lookup <ip>’ and ‘geo check <ip>’
• Time check: ‘geo timecheck <ip>’
• Verify config priority: [bastion] overrides [service] when geo_enabled/time_enabled=true
• Localhost bypass: 127.0.0.1 and ::1 always skip both checks

Port forwarding failures:

• ACL denied: ‘bastion forwards’ to see active forwards
• Rule evaluation: check port_forwarding_acl.rules order (first match wins)
• DNS resolution: ‘dns test <hostname>’ for DNSSEC validation
• SSRF blocked: check blocked_cidrs configuration
• Forward limit: check max_forwards_per_session (default: 10)
• Idle timeout: forwards close after port_forwarding_acl.idle_timeout (default: 15m)

SSH proxying (ssh command) failures:

• No matching rules: check ssh_cert.rules for user’s groups and destination
• Multiple principals prompt: expected when multiple rules match
• Certificate expired: cert_ttl too short for slow connections (default: 1m)
• DNSSEC validation: ‘dns test <hostname>’ — failure blocks connection
• Host key trust: target server needs User CA in TrustedUserCAKeys
• SSRF blocked: resolved IP matched blocked_cidrs

SFTP failures:

• SFTP not enabled: ‘config show bastion’ to verify sftp.enabled=true
• Group access denied: check sftp.required_groups and bastion required_groups
• No destinations listed: check ssh_cert.rules match user’s groups
• “no principal specified”: user navigated to /hostname without principal. Use

  principal@hostname format shown in the virtual root listing (ls /)

• Host key rejected: “unknown host key” means user must SSH interactively first to approve
• Connection timeout to remote host: ‘net tcp <host:port>’ to test connectivity
• Certificate issues: same troubleshooting as SSH proxying (cert_ttl, CA trust)
• Connector site routing: check ssh_cert.rules site field for remote destinations
• Connection dropped after file transfer: check sftp.idle_timeout (default: 10m)
• Audit: SFTP operations always logged as audit class (controlled by [telemetry] audit)
• Authentication: keyboard-interactive prompt shows auth URL. If SFTP client doesn’t

  display it, lazy auth shows the URL in error messages on first operation

SQL bastion issues:

• Connection failed: ‘net tcp <host:port> —tls’ to test database connectivity
• Query denied: check ACL readonly mode, forbidden_tables, query patterns
• Complexity exceeded: simplify query (reduce JOINs, subqueries, or nested operations)
• Rate limited: check acl.rate_limit.queries_per_minute
• TLS errors: verify ssl_ca, ssl_cert, ssl_key paths and permissions
• Site selection: ‘bastion sql’ to list configured sites

Session recording issues:

• Recordings not created: verify enable_recording=true
• Permission denied: check recording_path is writable by bastion process
• Disk space: recordings are ~1-5 MB/hour, check available storage
• Playback: use ‘asciinema play <file>.cast’ or web player

Interpreting tool output:

  'bastion sessions':
    Normal: Active sessions show User, IP, Duration, Commands executed
    Idle: Long Duration with low command count — user may have left terminal open
    Action: Suspicious session → 'sessions show <id>' for details

  'bastion metrics':
    Normal: connection_limit_hits=0, auth_failure_bans=0
    Under attack: High connection_limit_hits or auth_failure_bans — DoS protection active
    Action: High auth failures → 'ratelimit stats' for banned IPs

  'bastion forwards':
    Normal: Active port forwards show User, Type (local/remote/dynamic), Destination
    High count: Many forwards per user → check max_forwards_per_session limit
    Action: Suspicious forward → check destination against blocked_cidrs and ACL rules

Internal queries

The SQL bastion service account issues standard schema introspection queries for browsing.

MySQL / MariaDB:

  - Database listing
  - Table listing and filtering
  - Column metadata with types, nullability, defaults, and key info

PostgreSQL:

  - Database listing (excludes template databases)
  - Table listing with schema filtering (excludes system schemas)
  - Column metadata with types, nullability, defaults, and primary key detection

All queries use parameterized inputs. The service account should have read-only privileges on system catalogs and the user databases it manages.

Relationships

Module dependencies and interactions:

• listener: Bastion processes SSH connections on the configured port.
• authentication: OAuth 2.0 Device Authorization Grant for login. Background OIDC

  token refresh terminates sessions when users are disabled.

• sessions: Distributed session storage for cluster-wide visibility and admin tools.
• directory: Group membership lookup for access control, per-command authorization,

  and port forwarding ACL evaluation.

• dns: Hostname resolution with DNSSEC validation for SSH connections and port

  forwarding. DNSSEC failure blocks connections (no fallback for security).

• geoaccess: Geo-IP restriction at TCP layer. Uses [bastion] config with fallback

  to [service]. Silent close on block.

• timeaccess: Time-based restriction at SSH session start. Uses [bastion] config

  with fallback to [service]. Shows denial message before close.

• ratelimit: Connection, auth, session, and command rate limiting with auto-ban.
• firewall: Network-level access rules applied before bastion connections.
• certificates: SSH CA key management. User CA signs ephemeral certificates for

  SSH proxying. Host CA signs bastion host key for client verification.

• metrics: Prometheus metrics for connections, auth, sessions, commands, and DoS.