EVE remote debug improvements

Motivation

Document here describes new debug enhancements required in EVE. At a high level, we need the ability to request any EVE running node to perform certain

operations (eg: get output of a command, do clear operation on a given directory) and return the output/status of execution back to Zedcloud. Such ability in EVE

can be helpful when debugging EVE nodes in problem state and also for automation scripts to get status from devices without a direct SSH connection.

 

The document linked above describes at a high level the mechanism for sending such commands to device and lists a set of useful debug commands. It is still

not very clear how the result of command execution should be sent back to Zedcloud. Output sent from EVE node should be in a form useful for both logging and 

be convenient for processing with other software (eg: cloud code for storing in DB or integration tests for checking status/counters).

 

EVE response to Zedcloud

Two possible ways of sending command response back to zedcloud:

  1. Split command output into multiple lines (carriage return delimited split) and log each line similar to any other log. Command id (random number that user sends
    along with the command) will be logged as part of each line. These logs will reach Kibana using the exiting logging mechanism.
    Pros:
     - Relatively easy to implement.
     - Loss of data due to buffer size limitations (per line logged) is highly unlikely.

    Cons:
     - Since the response not stored as object in cloud, it cannot be queried & retrieved.
     - Difficult to interpret for both human users and software.
     - No possibility of time series data analysis if required later.

  2. Create a new API endpoint and new proto message for sending command output response to Zedcloud.
    message CommandResponse {
        CommandId            int;  // Random/running number coming from user
        FirstFrag                  bool;
        MoreFrag                bool;
        FragData                 string;
    }
    Depending on the output size from command and the data size imposed by API, multiple messages can be created in EVE and sent to Zedcloud.
    Zedcloud would only store the last unfragmented/fully re-assembled command response.
    EVE not makes sure that it does not interleaves messages corresponding to different commands at the same time.

    Pros:
    - Easy to consume for both human users and software. Easy to query for last response from cloud.
    - Integration tests can query for specific command outputs without needing direct SSH connection to EVE node.

    Cons:
    - Takes relatively more effort on the EVE side.
    - Zedcloud will need software changes to support this.

 

Please comment if there are other ways of doing this and any suggestions to the methods above.