Skip to main content
Version: v3.9.0

Observe and troubleshoot

Day-2 diagnostics combine EdgeOps Console visibility, potctl inspection commands, and on-host Edgelet tools. Use this page as a hub; deep daemon and cgroup topics stay in Edgelet troubleshooting.

Workload container logs and shells: Logs and exec. Edgelet daemon logs and a shell on the node: Node logs and exec.

Prerequisites​

  • Console sign-in or potctl connect with a subject that can get and list the resources you inspect.
  • For exec and logs, RBAC that allows microservice or agent session endpoints (same WebSocket API for Console and CLI).

EdgeOps Console​

Dashboard and inventory​

Start on the dashboard for ECN-wide health, then drill into Workloads, Nodes, or Network when a symptom is scoped to one area. See Operate overview.

Events (audit trail)​

Open Events to filter Controller API audit records by type, endpoint, status, actor, and time range. Export or prune old events when retention grows large.

Retention is configured on the control plane (spec.events.retentionDays). See Control plane (Configure).

See Console features - Events and REST API explorer.

Logs and exec in the browser​

From Workloads or Nodes:

  • Stream microservice container logs over WebSocket (same transport as the CLI logs command)
  • Open exec sessions inside a microservice or on an Edgelet node

Controller v3.8.0+ uses a multi-session exec model (default five concurrent exec sessions per microservice). Details: Exec sessions, Applications and microservices.

REST API explorer​

The embedded OpenAPI explorer calls /api/v3/... with your Console auth context. Use it when a resource has no dedicated form editor yet. Reference: Controller API.

CLI inspection loop​

Microservice runtime state​

potctl get microservices
potctl describe microservice APP_NAME/MSVC_NAME

On platform train v3.9.0, describe output includes read-only status such as podId, restartCount, lastError, and lastErrorAt. See Microservices (Configure).

When lastError mentions models, knowledge, or volume mounts, walk the attach checklist on Configuration and Microservices - models, knowledge, and runtime classes.

Logs and exec​

potctl logs microservice APP_NAME/MSVC_NAME
potctl logs microservice APP_NAME/MSVC_NAME --tail 200 --follow=false
potctl exec microservice APP_NAME/MSVC_NAME

If logs are empty, confirm get microservices shows RUNNING and retry after rebuild microservice.

Edgelet nodes​

potctl get agents
potctl describe agent NODE_NAME

On the host, use edgelet system status, edgelet ms ls, and edgelet ms logs. See Edgelet CLI and Edgelet logging. Daemon logs (logs agent) and a shell on the node (exec agent) are on Node logs and exec.

Control plane and system microservices​

potctl describe controlplane
potctl get controllers
potctl describe controller CONTROLLER_NAME
potctl get system-microservices

Prefer Reconcile platform before restarting Controller-owned system microservices ad hoc.

Symptom → where to look​

SymptomFirst steps
Microservice not RUNNINGdescribe microservice → lastError; node describe agent; image pull and Registries
NATS connect or ACL errorsMessage bus; user rule subjects; redeploy after rule change
service.local connection failuresNetwork; bridge port from get services; node tags
Console or CLI 403Access control. Check the RoleBinding for your OIDC user or group.
Edgelet CLI exit 10 (daemon unavailable)Edgelet troubleshooting - CLI connectivity
Edgelet API 401Edgelet troubleshooting - EdgeletAPI auth
Cgroup or sandbox errorsEdgelet troubleshooting - Cgroup delegation
Exec timeout or session limitExec sessions, Troubleshooting - Microservice exec

Platform maintenance signals​

TaskCommand or doc
Retry service hub provisioningreconcile service - Network
Retry agent provisioningreconcile agent - Edgelet nodes
Prune dangling images on a nodeprune agent - Image pruning
Controller train upgrade issuesControl plane, What's New

Developing and debugging microservices​

Application authors often combine logs, exec, and local SDK debugging. See Developing microservices - debugging and Applications and microservices.

Tutorial with NATS and services observability: Acme Smart Plant.