EVE Guest Agent

Owned by Vitaliy Kuznetsov
Oct 06, 2022

Motivation

Based on user stories for Snapshots in EVE, we also have a need to create consistent snapshots through the VM.
The user should be able to create snapshots both through the controller and through the VM. And if to solve the problem where it is necessary to create snapshots through the controller, it would be enough for us to use Qemu-guest-agent, then in the case when the user wants to create snapshots from the VM, we need a utility that will allow us to do this. The main requirement in this case is to provide the user with a clear interface and the ability to work with snapshots on the VM side.

Possible User stories for using eve-guest-agent to work with snapshots:

  1. As a VM user, must specify the hard drive for which I would like to make a snapshot during operations. Since this is the only information that I have.
    For example: 
    eve-guest-agent snapshot create /dev/sdc
    eve-guest-agent snapshot delete "uuid"
    eve-guest-agent snapshot rollback "snap_name" /dev/sdc
  2. As a VM user, I would like to receive a correct and understandable error with a description of the problem if any of the operations failed (for example, when EVE ran out of free space) or receive confirmation that the snapshot was successfully created.

  3. As a VM user, would like to get a complete list of available snapshots and information about them for all logical volumes that only my VM uses. For example by calling cmd:
    eve-guest-agent snapshot list
    and get something like:

    Date & timeSnapshot nameSnapshot UUIDSizeTarget disk
    18.01.2023 18:00
    Weekly Backup
    123123j1sadad1231231
    16Gb
    /dev/sdc
    ...
    ...
    ...
    ...
    ...
    20.12.2022 16:30
    Gold backup
    2342t34ewdasr3234398
    38Gb
    /dev/sda

It should be noted here that if the user wants to create a consistent (and this is 98% of our cases) snapshot or perform a rollback operation, in the case when this is done through eve-guest-agent, the user will have to take care of running applications himself, terminate them or transfer to the backup state. And after the successful completion of the operation, return the applications to their normal state.

Create snapshots via command from the controller using eve-guest-agent

If the user wants to create a consistent snapshot or perform a rollback operation via a command received from the controller, then the user must create custom shell scripts on the VM before and after the snapshot to run before and after the snapshot is captured. These pre and post-scripts will be used for operations such as:

  1. Suspend applications or operating system processes running on a virtual machine that is writing data to persistent disk.
  2. Clear disk buffers. For example, MySQL has a FLUSH statement. Use whatever tool is available for your application.
  3. Other process with applications on VM...

Important:

The user must store their scripts on the virtual machine in the /etc/eve/snapshots/ directory. The full path of the prescript should be /etc/eve/snapshots/pre.sh and the full path of your postscript should be /etc/eve/snapshots/post.sh.

In turn, EVE will send a command to the VM before taking actions on snapshots, which will run these scripts before and after the snapshot operation. If any of the scripts fail, EVE will abort the operation and return the error to the controller.

Requirements for the eve-guest-agent utility

  1. Use without settings. Downloaded/installed and ran/called the command;
  2. Work in the background and autorun (required to execute commands received by EVE from the controller);
  3. Ability to display information in JSON;
  4. Support for the required commands for working with snapshots (VM/Controller);
  5. The user should receive information about snapshots only for his VM.

Creating own client will also allow you to develop additional functionality in the future. For example, collect the necessary information from the VM.

Basic design and technology

Directly, the very implementation of the interaction between EVE and the VM will be implemented in a way similar to qemu-guest-agent, through vhost-vsock-pci:

Briefly about what will be done:

  • In EVE will create a message structure for communication between eve-guest-agent and EVE.
  • When creating each VM, EVE will assign it a unique CID from 2 to 9000, by which EVE will identify the VM by commands coming from it on the socket.
  • On the EVE side, the appropriate functionality will be written to work with the eve-guest-agent and a new agent that will listen on the main socket.
  • eve-guest-agent will be written from scratch, where it will be based on a ready-made golang package for working with vhost-vsock-pci. This will allow you to implement the ability to run the agent without any settings, as well as meet all the specified requirements.
  • This solution will not require extending the current API described in Snapshot EVE to support commands received from the controller.
*The plan itself for implementing the functionality for working with snapshots is presented in another proposal.

Interaction process

Next, the blocks with steps will be described on which the implementation will be built for different use cases.

The steps that describe the process of working with snapshots through the controller are as follows:

  1. EVE receive a command from the controller to create or rollback a snapshot;
  2. EVE sends a command to eve-guest-agent to put the application into create snapshot/backup state;
  3. eve-guest-agent run the /etc/eve/snapshots/pre.sh shell script;
  4. eve-guest-agent will send information to EVE about the results of running /etc/eve/snapshots/pre.sh;
  5. EVE checking the state of the file system in the VM;
  6. EVE checking sure the file system is working normally;
  7. EVE flush RAM to disk and freeze the file system in the VM;
  8. EVE checking sure the file system in the VM is frozen;
  9. EVE create a new snapshot or rollback to an existing snapshot;
  10. After completing the command for snapshot, EVE send the command to unfreeze the file system in the VM;
  11. Make sure that the file system is unfrozen and works normally;
  12. EVE sends a command to eve-guest-agent to put the application into a normal state;
  13. eve-guest-agent run the /etc/eve/snapshots/post.sh shell script;
  14. eve-guest-agent will send information to EVE about the results of running /etc/eve/snapshots/post.sh;
  15. VM goes back to normal updating of their volume;
  16. EVE sends up-to-date information to the controller.

*It is also worth noting that when the user creates a snapshot through the controller, the VM can be turned off at that moment, in which case the process of working with snapshots will look much simpler.

The steps that describe the process of working with snapshots through the VM are as follows:

  1. User on VM prepares applications for snapshot creation by transferring them to backup mode or by terminating their execution;
  2. Runs a command in eve-guest-agent (eve-guest-agent snapshot create /dev/sdc);
  3. EVE receives the command, checks the environmental conditions, and makes any necessary free-space calculations;
  4. EVE checking the state of the file system in the VM;
  5. EVE checking sure the file system is working normally;
  6. EVE flush RAM to disk and freeze the file system in the VM;
  7. EVE checking sure the file system in the VM is frozen;
  8. EVE create a new snapshot or rollback to an existing snapshot;
  9. After completing the command for snapshot, EVE send the command to unfreeze the file system in the VM;
  10. EVE checks sure that the file system is unfrozen and works normally;
  11. EVE sends up-to-date information to the eve-guest-agent on VM;
  12. EVE sends up-to-date information to the controller;
  13. The user can verify that the snapshot was created by calling the command eve-guest-agent snapshot list.


Perhaps at some steps additional checks of states will be required, but these are already features of the development itself. In any case, the steps will only advance if all the previous steps have been completed successfully and the state of the system on the VM and EVE itself is normal and allows you to work with snapshots.

References

Snapshots in EVE