...
When the EdgeView is started based on the configuration on the controller, it can be a single instance or multiple instance session. The multi-instance case is used when there is a need for multiple users to access the device or applications at the same time. For the multi-instance session, the user needs to supply the 'instance ID' when issuing EdgeView command, for example with the above 'edgeview.sh' script, an instance number is needed:
edgeview.sh -inst 2 route
The above command uses the 'instance 2' for EdgeView session to the device to query for the routes on the system.
...
The 'addhost' command adds one entry of specified hostname to IP address mapping into the EdgeView container's '/etc/hosts' for. This can be useful if there is no DNS entry for the hostname and the user knows the static mapping. An example:
edgeview.sh addhost/zedcontrol.local.zededa.net/32.165.49.119
...127.0.0.1 localhost::1 localhost ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allrouters
32.165.49.119 zedcontrol.local.zededa.net
This '/etc/hosts' is in the EdgeView container, and not in EVE device's host and not in other EVE containers.
...
nslookup[/<ip or name>] - display domain name and dns server information e.g. nslookup/www.amazon.com -- display DNS information on www.amazon.com nslookup/8.8.8.8 -- display DNS information on address 8.8.8.8
...
showcerts[/<url>][/proxy-addr:proxy-port] - display TLS connection certificates of server side
e.g. showcerts/zedcloud.local.zededa.net -- display TLS certificates from the controller
showcerts/zedcloud.local.zededa.net/10.10.1.128:3128 -- display controller TLS certificates through a proxy server
...
The bellow example displays the 'showcerts' without option, the server url is 'zedcloud.local.zededa.net', and the proxy is '31.198.61.228:3128'. The certificates from the peer are belong to the server or the controller, not to the proxy server. This may help the toubleshooting to determine if the proxy is a passthrough or a MiTM type.
...
=== Network: <peercerts> ===
url: zedcloud.local.zededa.net/31.198.61.228:3128(0) Certificate:
Data:
Version: 3
...
Not Before: 2022-04-11 18:19:37 +0000 UTCNot After: 2023-04-21 18:19:37 +0000 UTC
Subject: CN=zedcloud.local.zededa.net,O=Zededa Inc.,L=San Jose,ST=California,C=US
...
trace[/<ip or name>] - traceroute to www.google.com and zedcloud server, or to specified ip or name, 10 hops limite.g. trace -- traceroute to www.google.com and to zedcloud servertrace/www.microsoft.com -- run traceroute to www.microsoft.com
The 'trace' command uses the 'traceroute' utility of Linux and returns the hop-by-hop result if available. It uses two-queries per hop (useful in ECMP) and it is limited to a maximum of 10 hops. The option can be an IP address or domain name.
...
- zedagent statsinterface: eth0Success: 1853 Last Success: 2022-08-12 18:45:00.027419055 +0000 UTC
https://zedcloud.local.zededa.net/api/v2/edgedevice/id/37df4d43-6d3e-4369-a455-9a189b1426bb/config Recv (KBytes): 7, Sent 307120, SentMsg: 880, TLS resume: 880, Total Time(sec): 108
https://zedcloud.local.zededa.net/api/v2/edgedevice/id/37df4d43-6d3e-4369-a455-9a189b1426bb/info Recv (KBytes): 0, Sent 219684, SentMsg: 82, TLS resume: 82, Total Time(sec): 9
...
The EdgeView system related commands (similar to network commands, but less related to TCP/IP) are the items printed from the '-h' output:
[configitem cat cp datastore dmesg download du hw lastreboot ls model newlog pci ps cipher usb techsupport top volume]
Configitem
configitem - display the device configitem settings, highlight the non-default values
...
debug.default.loglevel: info
debug.enable.usb: false; default true
Cat
cat/<path to filename> - to display the content of a file
...
collectinfo - collect the device information using collect-info.sh and download a compressed file in tar.gz format
Collect-Info will instruct the EVE device to run the 'collect-info.sh' script and gather various device status, network information, logs and system information. After the collection is finished on the device, the EdgeView will download the compressed file in .tar.gz format. Since the collection task takes time, it may take a couple of minutes or longer. It then prints out the .tar.gz filename and size. It then starts to transfer the .tar.gz file from the device to the laptop. The file will be saved at the location '/tmp/download' directory on the laptop. An example of collectinfo output:
transfer file: name eve-info-edgeview-v29-2024-10-22-18-27-17.tar.gz, size 40875906
[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++] 100%
transfer rate in 5.74 Mbps
file eve-info-edgeview-v29-2024-10-22-18-27-17.tar.gz size 40875906
Cp
For the details of 'cp' command, see section Copy File Command.
...
The 'dmesg' command is to display the system log in kernel memory. The log severity Error and above is printed in Red, and Warn is printed in Pink, and rest of them in normal text color.
E.g.:
edgeview.sh dmesg
[ 1.063566] sd 0:0:1:0: [sda] Attached SCSI disk
[ 1.064425] sd 0:0:1:0: Attached scsi generic sg0 type 0
[ 1.065545] Rounding down aligned max_sectors from 4294967295 to 4294967288
[ 1.066723] db_root: cannot open: /etc/target
[ 1.067668] tun: Universal TUN/TAP device driver, 1.6
[ 1.159820] VMware vmxnet3 virtual NIC driver - version 1.5.0.0-k-NAPI
[ 1.174923] i8042: Warning: Keylock active
Download
download - display the download config and status during downloading operation and url stats since reboot
...
For example, in the TPM cert info:
- TPMmgr Edgenode Certs:
40d54a918e2057350b38ba916a93f3a1:
...
The 'usb' command uses the 'lsusb' utility to display the device USB information.
Tar
tar/<path to directory> - to generate a tarfile of the directory
e.g. tar//persist/agentdebug -- download the tarfile persist.agentdebug.<time>.tar of that directory
The 'tar' command generates a tar file from the directory on a remote device with the given path. It will deposit the tar file in the mounted directory on the user's laptop for downloading files. This command allows the directory with data up to 512 MBytes. Certain directories may have user sensitive data and can not be tarred, e.g. '/persist/vault', '/persist/clear' and '/run/domainmgr/cloudinit'. The files with file name ends with '.key.pem' and '.key' will not be included in the tar file.
...
The 'log' command is used to search for log entries for the device and applications. Even though the logs are normally uploaded to the controller side, the users of the enterprise may not have the capability to search in the cloud. The application logs depending on the setting, it can be configured not to upload, then the only way to view them is through this 'log search' in EdgeView if needed.
log/<search string> [-time <start>-<end>] [-json] [-type <app|dev>] - display log with search-string, default is now to 30 mins ago
...
log/certificate -time 2021-08-15T23:15:29Z-2021-08-15T22:45:00Z -json -- display log during the specified time in RFC3339 format which contains 'certificate' in json format
log/copy-logfiles -time 2022-02-15T22:25:00Z-2022-02-15T22:40:00Z -- 'copy-logfiles' is reserved usage, to download all logfiles in the specified time duration, maximum time frame is 30 minutes
...
All the services supports 'pub' command:
[baseosmgr domainmgr downloader edgeview global loguploader msrv newlogd nim nodeagent tpmmgr vaultmgr volumemgr watcher zedagent zedclient zedkube zedmanager zedrouter zfsmanager]
the 'pub/<service-name>' will display all the data items of the service. For instance, for 'domainmgr' service, it has the data for 'AssignableAdapters', 'Capabilities', 'CipherMetrics', 'DomainMetric', 'HostMemory' and "processMetric'. The user can supply a string for the display of only that particular data-structure, e.g. only for 'DomainMetric', that display with 3 applications on the device:
...
If the user wants to log into the applications on the EVE device. There are multiple ways to do that using EdgeView. If the application runs SSHd inside, the user can use SSH:
first use 'edgeview.sh app' to find out the IP address of the applications, for example two applications and their interface IPs are 10.1.0.130 and 192.168.1.100
setup the TCP channel for ssh into both applications: edgeview.sh tcp/10.1.0.130:22/192.168.1.100:22
tcp mapping locally listening 2 ports to remote:0.0.0.0:9001 -> 10.1.0.130:22
0.0.0.0:9002 -> 192.168.1.100:22
assume both applications, has username of 'testing', open two more terminals on your laptop, launch the ssh session on first app with 'ssh testing@localhost -p 9001'; and in another terminal, launch another ssh session with 'ssh testing@localhost -p 9002'
In the above example, the ssh client session targets the laptop with port number 9001 and 9002, which will be virtually connecting to the remote applications 10.1.0.130:22 and 192.168.1.100:22. EdgeView handles all the TCP packets stitching and relaying.
If the application is a VM on the EVE device, instead of using SSH, the user can use VNC to connect to the application's console using EdgeView. Assume the two applications as above:
first use 'edgeview.sh app' to find out the VNC display ID of the applications. For example, the VNC IDs are 4 and 5 for the applications.
setup TCP channel into applications consoles by: edgeview.sh tcp/localhost:5904/localhost:5905
tcp mapping locally listening 2 ports to remote:0.0.0.0:9001 -> localhost:5904
0.0.0.0:9002 -> localhost:5905
open two VNC viewers on the laptop, enter the VNC endpoint for one: 'localhost:9001' and the other 'localhost:9002'.
The reason the 'tcp' command has the option to 'localhost:590x' is that the VNC service is maintained by the EVE Dom0 side 127.0.0.1 with port numbers 590x for application console access.
...
The applications may have TCP services other than SSH, for example, it may have normal HTTP service. EdgeView TCP channel can be used to access those services by specifying the related ports. Here is an example of 'fledge' IoT application. It services 3 different TCP ports, 80, 8081 and 4840. Port 80 is for initial browser connection; after connecting, there is a page to setup browser to another port 8081 usually. The port 4840 is for accessing the 'TCP binary' data for IoT applications, it requires a special OPC UA client software to access. Here is an example to access the 'fledge' application on EVE device:
first use 'edgeview.sh app' to find out the 'fledge' interface IP address, in this case it's 10.1.0.3
- app: fledge-app, appNum: 1
app uuid 9ce27b08-47e0-4568-a56e-b6dd0e4514cd
== bridge: bn1, nbu1x1, 10.1.0.3, 00:16:3e:00:01:01
setup the TCP channel for the 'fledge' endpoints: edgeview.sh tcp/10.1.0.3:80/10.1.0.3:8081/10.1.0.3:4840
tcp mapping locally listening 3 ports to remote:0.0.0.0:9001 -> 10.1.0.3:80
0.0.0.0:9002 -> 10.1.0.3:8081
0.0.0.0:9003 -> 10.1.0.3:4840
open a web browser to 'http://localhost:9001', and set up the page for browser switching to endpoint 'localhost:9002' for HTTP service. To access the OPC data, on the MacOS, the user can download the 'prosys-opc-us-client' and set the remote endpoint to 'localhost:9003'.
Access TCP Services of External Hosts
...
This is also similar to the above application access, but with Dom0 side of the IP addresses and ports. For example, TCP access to the 'meta data' services for internal bridges:
first use 'edgeview.sh socket' to find out which ports Dom0 side listening on
tcp LISTEN 0 1 0.0.0.0:5904 0.0.0.0:* users:(("qemu-system-x86",pid=4192,fd=24))
tcp LISTEN 0 1024 10.1.0.1:80 0.0.0.0:* users:(("zedbox",pid=1903,fd=378))
tcp LISTEN 0 1024 10.3.0.1:80 0.0.0.0:* users:(("zedbox",pid=1903,fd=347))
tcp LISTEN 0 1 0.0.0.0:5905 0.0.0.0:* users:(("qemu-system-x86",pid=4112,fd=24))
setup TCP channel to both bridge's 'meta data' servers: edgeview.sh tcp/10.1.0.1:80/10.3.0.1:80
tcp mapping locally listening 2 ports to remote:0.0.0.0:9001 -> 10.1.0.1:80
0.0.0.0:9002 -> 10.3.0.1:80
open a web browser with 'http://localhost:9001/eve/v1/network.json' to get the external ip address on the device, it returns below in this example:
{"caller-ip":"10.1.0.1:57152","external-ipv4":"192.168.86.36","hostname":""}
...
For example, the application with interface IP address of 10.1.0.2 is listening on TCP port 6443 as a HTTPs service for kubernetes API service.
setup TCP proxy command: edgeview.sh tcp/proxy
tcp mapping locally listening 1 ports to remote:
0.0.0.0:9001 -> proxy
assume the user has downloaded the 'kubeConfig' file on a local laptop, a 'kubernetes' management software can be used to point to the proxy server of 'localhost:9001'. The kubeConfig remote API server address in this case is the real remote IP address: 10.1.0.2, and that is inside the certificate the API server uses.
The reason this 'proxy' is part of the TCP command is that the 'proxy' service is only for one port 9001 here, the others can still be used for regular TCP channels for SSH service and others.
...
The 'techsupport' command is to gather most of the EdgeView other command output and save into a compressed file then uploading to the user's laptop. This is similar to some router vendor's "show tech-support" command on devices. The command includes the above mentioned EdgeView commands:
Network: (route, arp, if, acl, connectivity, url, socket, app, mdns, nslookup/google.com, trace/8.8.8.8, wireless, flow)
System: (hw, model, pci, usb, lastreboot, newlog, volume, app, datastore, cipher, dmesg, configitem)
Pubsub: (nim, domainmgr, nodeagent, baseosmgr, tpmmgr, global, vaultmgr, volumemgr, zedagent, zedmanager, zedrouter, zedclient, fsmanager, edgeview, watcher)
...
Similar to 'cp' and 'log/copy-logfiles', the 'techsupport' requires the docker volume of directory '/download' be mounted in the user's local system. This command takes a while to run (about 60 seconds). Here is an example of the output:
edgeview.sh techsupport
++++++++++++++++++++++++++++++++++++++++++++++++++++++++
...
zcat < /tmp/download/techsupport-20220816202445.gz | head -50
- Show Tech-Support -
Device IPs: [192.168.86.36]; Endpoint IP 167.12.91.124:40839UUID: 07ab2c40-408a-4bc2-b9f2-8ca94235074fController: zedcloud.local.zededa.netEVE-OS release 0.0.0-master-ca6084b9-2022-08-16.20.07-kvm-amd64, IMGAEdge-View Ver: 0.8.2, JWT expires at 2022-08-23T01:35:56Z2022-08-16T20:24:45Z(UTC), uptime 474 (sec) = 0 days
- network info -
=== Network: <route> ===
- ip rule:
1000: table 510, ip rule 1000: from all to all table 510
routes in table: 510{Ifindex: 10 Dst: <nil> Src: <nil> Gw: <nil> Flags: [] Table: 510 Realm: 0}
...
07ab2c40-408a-4bc2-b9f2-8ca94235074f:~# cd /run/edgeview/
07ab2c40-408a-4bc2-b9f2-8ca94235074f:/run/edgeview# ls
07ab2c40-408a-4bc2-b9f2-8ca94235074f:/run/edgeview# touch run-techsupport
07ab2c40-408a-4bc2-b9f2-8ca94235074f:/run/edgeview# ls -lttotal 92-rw-r--r-- 1 root root 92917 Aug 22 22:07 techsupport-20220822220657.gz
07ab2c40-408a-4bc2-b9f2-8ca94235074f:/run/edgeview#
Then copy the file '/run/edgeview/techsupport-20220822220657.gz' onto a USB disk.
Kubernetes Command
...
Assume the EVE device hypervisor type is 'Kubevirt', the device's EVE image name has the suffix like 'kubevirt-amd64', use the TCP command 'kube' to launch the kubernetes setup:
~ edgeview.sh tcp/kube
Edgeview is in multi-instance mode, use '-inst 1-3', try '-inst 1' here
tcp mapping locally listening 1 ports to remote:
0.0.0.0:9001 -> kube
....
transfer file: name kube-config.yaml, size 2265
[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++] 100%
transfer rate in 1.28 Mbps
file kube-config.yaml size 2265
This command 'tcp/kube' sets up the secure connection to the EVE device kubernetes cluster, and downloaded at the sametime the special 'kube-config.yaml' file to be used for the 'kubectl' for that cluster of the EVE device. This file is downloaded at the location '/tmp/download/kube-config.yaml' on user's laptop.
Go to another terminal on user's laptop, and make an alias, e.g. "alias k='kubectl --kubeconfig=/tmp/download/kube-config.yaml'" and save to your shell configuration file. Then the user can use the 'kubectl':
☁ ~ k get pod -n eve-kube-app
NAME READY STATUS RESTARTS AGE
alpine-nohyper-app-6d3d1-8t4r5 1/1 Running 0 10h
virt-launcher-bionic-ubuntu-vm-099a55jhd2-zw6ps 2/2 Running 0 10h
virt-launcher-my-container-vmapp-25351thrld-fkv9j 2/2 Running 0 10h
☁ ~ k get vmi -n eve-kube-app
NAME AGE PHASE IP NODENAME READY
bionic-ubuntu-vm-099a55jhd2 10h Running plex1-7050m True
my-container-vmapp-25351thrld 10h Running plex1-7050m True
☁ ~
In the above, EVE device uses namespace 'eve-kube-app' for the virtualized applications.
...
With the above 'tcp/kube' setup, users can also use the 'virtctl' tool on the laptop to inspect the VMI (VM instance of kubevirt). the 'virtctl' tool needs to be downloaded and installed on user's laptop. Make an alias similar to the 'kubectl' above, e.g. "alias v='virtctl --kubeconfig=/tmp/download/kube-config.yaml'" and save to user's shell configuration file. Then the user can issue the 'virtctl':
v vnc bionic-ubuntu-vm-099a55jhd2 -n eve-kube-app
v console bionic-ubuntu-vm-099a55jhd2 -n eve-kube-app
The first command will launch a local 'VNC Viewer' (assume user has already installed a VNC viewer on the laptop) on user's laptop to connect directly into the VMI bionic-ubuntu-vm-099a55jhd2 in namespace 'eve-kube-app', and the second command will enter the 'console' mode to get into the same VMI.
...