Skip to main content
Version: v3.8.0

Edgelet troubleshooting

Common issues when running Edgelet on Edgelet nodes.

Daemon won't start​

Symptoms: systemctl start edgelet fails; no listening socket on :54321.

Checks:

  1. Validate config paths and permissions:

    ls -la /etc/edgelet/
    edgelet system info

  2. Review journal logs:

    sudo journalctl -u edgelet -n 100 --no-pager
    tail -f /var/log/edgelet/daemon-startup.log

  3. Confirm containerEngine is valid on this platform:

    edgelet system version -o json | jq '{allowedEngines, containerEngine}'
    grep containerEngine /etc/edgelet/config.yaml

    Linux allows edgelet, docker, or podman. Darwin and windows allow docker or podman only.

  4. Check disk space:

    df -h /var/lib/edgelet /var/lib/edgelet-containerd

    If usage grows after deleting microservices, orphaned volume data may remain under /var/lib/edgelet/volumes/data/ until prune runs. See Volumes.

Cgroup delegation (embedded edgelet engine)​

Symptoms: daemon fails at startup with controller cpu is not available; RunPodSandbox errors; nested Docker deploy fails immediately.

Checks:

  1. Confirm cgroup mode and driver:

    edgelet system status -o json | jq '{cgroupMode,cgroupDriver,cgroupNested,cgroupDelegatedControllers}'

  2. On cgroup v2 hosts, verify delegated controllers in the edgelet cgroup:

    grep -E '^(cpu|memory|pids)' /sys/fs/cgroup/cgroup.controllers
    cat /sys/fs/cgroup/cgroup.subtree_control

  3. Hybrid v1+v2: edgelet logs a warning and prefers unified v2. Migrate the host to pure cgroup v2 when possible.

  4. Nested edgelet container in Docker: --privileged is required:

    docker run -d --name edgelet --privileged \
    -v /var/lib/edgelet:/var/lib/edgelet \
    -v /etc/edgelet:/etc/edgelet \
    ghcr.io/datasance/edgelet:1.0.0

    Without --privileged, cpu/memory/pids controllers are not delegated and CRI cannot create sandboxes.

  5. Non-systemd init (openrc, sysvinit, s6): edgelet uses the cgroupfs driver automatically. Ensure /sys/fs/cgroup is writable and controllers are mounted.

  6. LXC/VM machine root (/.lxc, OpenRC on OrbStack Alpine): error mentions edgelet-cgroup-prep or empty /.lxc/cgroup.controllers after cold boot:

    rc-status | grep edgelet-cgroup-prep
    cat /proc/self/cgroup
    cat /sys/fs/cgroup/.lxc/cgroup.controllers

    Reinstall edgelet so install.sh registers edgelet-cgroup-prep in sysinit, then reboot:

    sudo ./install.sh --version=1.0.0
    sudo reboot

  7. RunPodSandbox / sd-bus call: Invalid unit name or type: crun with SystemdCgroup=true (manual override or old build). Reinstall the current edgelet build. Generated config must have SystemdCgroup = false for crun.

    edgelet system status -o json | jq '{cgroupDriver,cgroupContainerdPath}'
    grep -E '^\[cgroup\]|path =|SystemdCgroup' /var/lib/edgelet-containerd/config.toml

Control vs data plane restart​

Symptoms: Microservices disappear after systemctl restart edgelet; unexpected downtime during agent OTA.

Checks:

  1. Identify which unit you restarted:

    systemctl status edgelet edgelet-containerd --no-pager

  2. docker/podman: control restart should not drain microservices (shutdownPolicy=leave-running default):

    grep shutdownPolicy /etc/edgelet/config.yaml
    edgelet system status -o json | jq '."runtime.shutdownPolicy", ."runtime.agentPhase"'
    docker ps --filter label=iofog.org/microservice-uuid

  3. Embedded split: restart edgelet only for control OTA; restart edgelet-containerd only when the fat/runtime bundle changed (expect microservice stop + reconcile).

  4. Cold engine change: changing containerEngine always recreates microservices. This is expected.

  5. Legacy unit on VM: if systemctl cat edgelet-containerd shows PartOf=edgelet.service, reinstall packaging and run systemctl daemon-reload.

Containerd socket (edgelet engine)​

Symptoms: connection refused to /run/edgelet/containerd.sock; microservices stuck in pull/create.

Checks:

  1. Verify socket exists:

    ls -la /run/edgelet/containerd.sock

  2. Check embedded containerd logs in daemon journal output.

  3. Shim load errors such as address: no such file usually mean stale runtime task metadata. Restart the data plane first:

    systemctl restart edgelet-containerd.service
    sleep 3
    systemctl restart edgelet.service

  4. Look for orphan overlay mounts blocking cleanup:

    mount | grep edgelet

EdgeletAPI 401 / auth failures​

Symptoms: CLI exits 3 (Unauthorized); curl returns 401.

Checks:

  1. Token file present and readable:

    sudo ls -la /etc/edgelet/edgelet-api

  2. CLI uses correct CA:

    edgelet auth whoami
    edgelet auth whoami -o json

  3. After provisioning, ensure you are not sending an unsigned bootstrap JWT.

  4. Regenerate PKI only when directed. Mismatched CA breaks existing CLI sessions.

See EdgeletAPI v1.

CLI connectivity (exit 10)​

Symptoms: Error[DAEMON_UNAVAILABLE]; exit code 10.

Checks:

  1. Daemon running:

    systemctl is-active edgelet
    edgelet system status

  2. Socket override - if using --socket, path must match /run/edgelet/edgelet.sock.

  3. Firewall - EdgeletAPI listens on localhost/TLS; remote access is not the default operator path.

External Docker / Podman​

Symptoms: Cannot connect to Docker daemon; containers not starting.

sudo systemctl status docker    # or podman.socket
ls -la /var/run/docker.sock
grep containerEngineUrl /etc/edgelet/config.yaml

Ensure the configured containerEngineUrl matches the running engine socket. See Container engines.

Controller connectivity​

Symptoms: connectionToController not ok in edgelet system status.

edgelet system status -o json | jq .connectionToController
curl -v "$(grep controllerUrl /etc/edgelet/config.yaml | awk '{print $2}')/health"
edgelet config cert <base64-controller-cert>   # if TLS verification fails

Microservice exec​

Symptoms: Error[EXEC_START_TIMEOUT]; attach fails with Error[NOT_FOUND]; Controller exec works but local ms exec fails (or vice versa).

Checks:

  1. Confirm the microservice container is running:

    edgelet ms inspect <uuid|namespace.name>
    edgelet ms ls

  2. Retry after a start timeout. POST waits up to 15 seconds for the shell:

    edgelet ms exec <uuid> -- /bin/sh

    See Exec sessions for the multi-session model.

  3. Concurrent sessions: local CLI exec is unlimited per microservice. Controller exec is capped at 5 per microservice on Controller v3.8.0. Multiple local sessions should not block each other.

  4. Orphan containerd exec (embedded engine): if a prior exec crashed without cleanup, retry after the process manager orphan sweep or restart the microservice:

    edgelet ms restart <uuid>

  5. Controller exec only: verify Edgelet node connectivity and that Controller v3.8.0 is deployed. Status should list active Controller session ids:

    edgelet system status -o json | jq '.connectionToController'

  6. execEnabled: Edgelet no longer gates exec on this flag. Session poll drives Controller exec. Do not expect toggling execEnabled to fix attach issues.

Collecting diagnostics​

sudo journalctl -u edgelet --since "1 hour ago" > edgelet-journal.log
edgelet system status -o json > edgelet-status.json
edgelet system info -o json > edgelet-info.json
edgelet system version -o json > edgelet-version.json

Do not share /etc/edgelet/edgelet-api or private keys in support tickets.

See also​

Group 3See anything wrong with the document? Help us improve it!

Edit this page on GitHub!