ThunderX2 Pod 2

Overview

The ThunderX2 Pod 1 consists of 3 Arm®v8 based Gigabyte R281-T91 servers and 1 Ampere HR330A jumphost server with the following specifications per server:



Gigabyte R281-T91

Ampere HR330A



Gigabyte R281-T91

Ampere HR330A

CPU

2x 28 core 2.0 GHz Marvell TX2 Processors

1x 32 core 3.3 GHz Ampere ARMv8 64-bit Processor

RAM

4x 32GB RDIMM-Tx2 

8x 16GB DDR4-DIMM

Storage

1x 480GB SSD
1x 1TB HDD

1x 480GB SSD

Networking

2x 10Gbe SFP+ (not connected)
2x 1/10Gbe BASE-T (1 connected)
2x 40Gbe QSFP (coming soon)
1x IPMI / Lights-out Management

2x 10Gbe SFP+ (connected)

1x 1Gbe BASE-T (not connected)

1x IPMI / Lights-out Management

Usage & Users

This information can now be found on the Akraino dashboard. Check the Shared Community Lab page for more information.

Server Access

For the meantime when you request access to the Akraino Pods at UNH-IOL we will ask that you send us your public ssh key so we can add it to the root users of those machines.

If you wish to have your own user we can get that setup, but it is much quicker and easier if you just stick with the root user

IPMI Access

When booking a pod, the Akraino dashboard will provide IPMI credentials via email.

You can access an IPMI interface like the example below:

thunder2-gigabyte1-ipmi.akr.iol.unh.edu

Then enter the username and password provided from the file in /opt and now you can manage the system on your and will even be able to add your own public keys in the event of a reinstall of the machine.

If you have any issues with any of the IPMI interfaces please email akraino-lab@iol.unh.edu and we will assist you in anyway we can.

Networking

The networking within the pod is currently provided by a 10Gbe BASE-T switch, which only supports static configurations (i.e. no SDN control).  This switch will be replaced with an Edgecore 7816-64X that is currently on order, along with 40Gbe NICs to be installed on each server.

Network

IP Network

VLAN ID

Description

Network

IP Network

VLAN ID

Description

IPMI / Management

10.11.17.0/24

GW: 10.11.17.1

3017

Connections to all IPMI interfaces on each server and the switch management interface

Public

10.11.18.0/24

GW: 10.11.18.1

3018

Public network (able to reach the Internet), available to each server.



Servers and Switches

The 3 Gigabyte R281-T91 have 3 sets of dual port interfaces:

  • Mellanox Technologies MT27700 on PCI-e expansion slot - two ports on PCI addresses 0000:91:00.0 and 0000:91:00.0 (enp145s0f0 and enp14s0f1)

  • QLogic Corp. FastLinQ QL41000 Series integrated on the main board - two ports on PCI addresses 0000:0c:00.0 and 0000:0c:00.1 (enp12s0f0 and enp12s0f1 or eno1 and eno2)

  • Intel 10G X550T on PCI-e expansion slot - two ports on PCI address 0000:05:00.0 and 0000:05:00.1 (enp5s0f0 and enp5s0f1); not connected at the moment

In the table below, the switch ports for the 3 gigabyte servers are in the following order:

  • Mellanox MT27700 port 1

  • Mellanox MT27700 port 0

  • QLogic FaslinQ QL41000 port 0

  • QLogic FaslinQ QL41000 port 1



The ampere-jumphost2 has the following network interfaces:

  • Mellanox MT27710 on PCI-e expansion slot - two  ports at PCI addresses 0000:01:00.0 and 0000:01:00.1 (enp1s0f0 and enp1s0f1)

  • Intel Gigabit I210 on PCI-e expansion slowt - one port at PCI address 0002:01:00.0 (enP2p1s0); not connected at the moment

In the table below, the switch ports are in the following order:

  • MT27710 port 1

  • MT27710 port 0



Server Name

IPMI Address

IPMI VLAN ID

Public Network Address

Public Network VLAN ID

Switch Port(s)

Serial Connection

OS Installed

Server Name

IPMI Address

IPMI VLAN ID

Public Network Address

Public Network VLAN ID

Switch Port(s)

Serial Connection

OS Installed

thunder2-gigabyte1

10.11.17.11

3017

10.11.18.11

3018

gigabyte-edgecore2: left 40Bbe Port 1, right 40Gbe Port 2, left 10Gbe Port 7 Breakout 1,  right 10Gbe Port 7 Breakout 2

Async 1/6

CentOS 7.6

thunder2-gigabyte2

10.11.17.12

3017

10.11.18.12

3018

gigabyte-edgecore2: left 40Bbe Port 3, right 40Gbe Port 4, left 10Gbe Port 7 Breakout 3, right 10Gbe Port 7 Breakout 4

Async 1/7

Ubuntu 18.04

thunder2-gigabyte3

10.11.17.13

3017

10.11.18.13

3018

gigabyte-edgecore2: left 40Bbe Port 5, right 40Gbe Port 6, left 10Gbe Port 9 Breakout 1,  right 10Gbe Port 9 Breakout 2

Async 1/8

Ubuntu 18.04

thunder2-jumphost

10.11.17.14

3017

10.11.18.14

3018

gigabyte-edgecore2: left 10 Gbe Port 9 breakout 3, right 10 Gbe Port 9 breakout 4

N/A

Ubuntu 18.04



Switch Name

Mgmt Address

VLAN ID

Serial Connection

OS Installed

Switch Name

Mgmt Address

VLAN ID

Serial Connection

OS Installed

thunder2-egdecore

10.11.17.5

3017

Async 1/2

SONiC