Debugging APIs

We currently have the device send log information to the controller using the log API, and this can be useful when debugging issues in EVE.

However, in some cases it is useful to also be able to inspect the current state. That state could be the state maintained by the EVE microservices (e.g., the AppInstanceStatus maintained by zedmanager), or it could be external state such as the iptables or ps process output.

This proposal specifies how a well-defined set of such information can be retrieved by the controller.

Background and motivation

We currently deliver the logs from the EVE microservices to the controller, plus specific information relating to the device and instance status and metrics. However, two issues makes it harder to use those logs than those on the device, the first being that they are consolidated from all the agents, and the second being that the logs are for the lifetime of the device (split in IMGA and IMGB logs) and in most cases one cares about what happened after the last reboot.

In addition, the current state of the device is easier to determine by examining /var/run on the device, and looking at things like the output of ps or xl list.

Finally, there are implementation internal aspects (such as iptables -L, ip rule show, ip route show) which are useful when debugging issues.

Proposal

We already have the logging API as a flexible and scaleable way to deliver information from the device to the controller, with the appropriate retry/retransmission logic in EVE. Its only constraint is that a single log item must be smaller than the maximum size configured in the web server running on the controller.

We also have a flexible way to extend the configuration using the ConfigItem message in the configuration; a string key plus a string value, which is used for timer and policy settings.

Last but not least we have a way to send commands such as the RebootCmd using eventual consistency by having a counter to ensure that a command is executed at least once.

Combining those we can add support for additional debug commands by defining a ConfigItem key string for each, where the value is a number. When the device receives such a ConfigItem it checks if the number is different than what it last processed for that particular key, and if it is the device performs the operation and the output is sent to the log API.

Initial set of keys/commands

Command	Reported information	Potential use
ps	ps output	Look for hung processes
du	du -a /persist	Track down disk usage
du.<subdir>	du -a /persist.<subdir>	E.g., du.log, du.IMGA
state	All of /var/run content	Snapshot for all agents and object
state.<agent>	/var/run/<agent>	Snapshot for one agent
state.<agent>.<type>	/var/run/<agent>/<type>	For agent and type
state.obj.<key>	/var/run///<key>	E.g., look for an instance UUID
config	/config except any *.key.pem	Looking for stale files
persist.<subdir>	ls /persist/<subdir>	Looking for stale files or missing certs
lspci	Alpine lspci output	Check if pci controllers match model
lsusb	Alpine lsusb output	Check if any USB devices connected
iptables	Iptables -t filter; iptables -t raw; iptables -t nat, all with -L -nv	Check if iptables are wrong + counters
route	ip route show
route.X	ip route show table X
rule	ip rule show

Considerations for adding future commands

For security reasons any command should be of a fixed function; no command should ever allow arbitrary execution of e.g., shell commands. Furthermore, when defining new commands one needs to take care to not expose any secret information from the device, such as the content of running edge container objects, or credentials for datastore access.

Currently none of the defined commands alter the state of the device, and if there is a desire to alter the state (e.g., purge certain directories to recover from low on disk space) it would make sense to explore alternative approaches than this basic fire-and-forget approach.

Implementation notes

The device will retain counter Y value for command string X, in similar ways as it retains a rebootCount and uuidtonum persistently across reboots.

This could be in /persist/status/zedagent/KeyToNum/X.json

When zedagent receives config items from the controller it will compare the counter Y with what is recorded, and if it is different than it will send the requested output to the log API. It makes sense for the log output to include the command string and counter value.