Tutorial for deploying and configuring VMware HCX in both on-premises and VMware Cloud on AWS with service mesh creation and L2 extension

Deploying HCX (VMware Hybrid Cloud Extensions) is considered to be complex and difficult by most. It doesn’t help that it’s usually one of those things you’d only do once so it doesn’t pay to spend a lot of effort to learn. However, as with everything it’s not hard once you know how to do it. This video aims to show how to deploy HCX both in VMC (VMware Cloud on AWS) and in the on-premises DC or lab.

It uses both the method of creating the service mesh over the internet as well as how to create it over a private connection, like DX (AWS Direct Connect) or a VPN.

A VPN cannot be used for L2 Extension if it is terminated on the VMC SDDC. In this tutorial I’ll use a VPN which is terminated on an AWS TGW which is in turn peered with a VTGW connected to the SDDC we’re attaching to.

Video chapters

  1. Switching vCenter to private IP and deploying HCX Cloud in VMC: https://youtu.be/ho2DY-TP-SA?t=43
  2. Initial SDDC firewall configuration: https://youtu.be/ho2DY-TP-SA?t=97
  3. Switching HCX to private IP and adding HCX firewall rules: https://youtu.be/ho2DY-TP-SA?t=405
  4. Downloading and deploying HCX for the on-prem DC side: https://youtu.be/ho2DY-TP-SA?t=585
  5. Adding HCX license, linking on-prem HCX with vCenter: https://youtu.be/ho2DY-TP-SA?t=740
  6. HCX site pairing between HCX Connector and HCX Cloud: https://youtu.be/ho2DY-TP-SA?t=959
  7. Creating HCX Network and Compute profiles: https://youtu.be/ho2DY-TP-SA?t=1011
  8. Choice: Deploy service mesh over public IP or private IP: https://youtu.be/ho2DY-TP-SA?t=1374
  9. Deploy service mesh over public IP: https://youtu.be/ho2DY-TP-SA?t=1399
  10. Live migrating a VM to AWS: https://youtu.be/ho2DY-TP-SA?t=1679
  11. Deploy service mesh over private IP (DX, VPN to TGW): https://youtu.be/ho2DY-TP-SA?t=1789

Some architecture diagrams for reference

Connecting all over the public internet is one method
The best performance may be had over a dedicated DX Private VIF to the SDDC
Separating the management traffic over a VPN while doing the L2 Extension over the internet is a bit of a hybrid
For the setup used in the tutorial I use a VPN to a TGW which is peered with a VTGW

Mikrotik VPN to AWS VPC

Quick (?) steps for connecting a Mikrotik router in an on-premises lab or DC to an AWS VPC using a VPN. All commands done over AWS CLI and Mikrotik CLI.

Note: The values for tunnel IP addresses and secrets etc. can be found in your VPN configuration file (downloaded later). Please don’t use the ones in this guide or an IT fairy will jump to her death from a VAX system in some remote DC. The values used here are already invalid as the resources have been deleted by the time of writing. Do think of the fairies though.

Architecture diagram

In this case the Mikrotik is not directly attached to the internet. It goes via an ISP router. If your setup is the same, please configure port forwarding for ESP, UDP port 500 and UDP port 4500 from the ISP public interface to the Mikrotik router as per the diagram.

If the Mikrotik is directly attached to the internet please open the firewall ports accordingly for ESP and UDP 500 / 4500.

AWS-side configuration

Creating the VGW (Virtual Private Gateway but called vpn-gateway on the CLI). I used 65011 here for the AWS-side ASN but feel free to use something different as long as it is supported

jonas@frantic-aerobics:~$ aws ec2 create-vpn-gateway --type ipsec.1 --amazon-side-asn 65011 | jq
  "VpnGateway": {
    "State": "available",
    "Type": "ipsec.1",
    "VpcAttachments": [],
    "VpnGatewayId": "<your-vgw-id>",
    "AmazonSideAsn": 65011

Verify the ID of the AWS VPC you want to connect to

jonas@frantic-aerobics:~$ aws ec2 describe-vpcs | jq
  "Vpcs": [
      "CidrBlock": "",
      "DhcpOptionsId": "dopt-d9bcfeb0",
      "State": "available",
      "VpcId": "<your-vpc-id>",
      "OwnerId": "111222333444555",
      "InstanceTenancy": "default",
      "CidrBlockAssociationSet": [
          "AssociationId": "vpc-cidr-assoc-fdf9af94",
          "CidrBlock": "",
          "CidrBlockState": {
            "State": "associated"
      "IsDefault": true

Attach VGW to VPC

jonas@frantic-aerobics:~$ aws ec2 attach-vpn-gateway --vpn-gateway-id <your-vgw-id> --vpc-id <your-vpc-id> | jq
  "VpcAttachment": {
    "State": "attaching",
    "VpcId": "<your-vpc-id>"

Verify that attachment is successful

jonas@frantic-aerobics:~$ aws ec2 describe-vpn-gateways --vpn-gateway-id <your-vgw-id> | jq
  "VpnGateways": [
      "State": "available",
      "Type": "ipsec.1",
      "VpcAttachments": [
          "State": "attached",
          "VpcId": "<your-vpc-id>"
      "VpnGatewayId": "<your-vgw-id>",
      "AmazonSideAsn": 65011,
      "Tags": []

Create the CGW (register your public IP in AWS basically). I used 65010 here for the on-prem ASN but feel free to use something different as long as it is supported

jonas@frantic-aerobics:~$ curl icanhazip.com
jonas@frantic-aerobics:~$ aws ec2 create-customer-gateway --type ipsec.1 --public-ip <your-onprem-public-ip> --bgp-asn 65010 | jq
  "CustomerGateway": {
    "BgpAsn": "65010",
    "CustomerGatewayId": "<your-cgw-id>",
    "IpAddress": "<your-onprem-public-ip>",
    "State": "available",
    "Type": "ipsec.1",
    "Tags": []

Create the VPN connection

jonas@frantic-aerobics:~$ aws ec2 create-vpn-connection --type ipsec.1 --customer-gateway-id <your-cgw-id> --vpn-gateway-id <your-vgw-id>
    "VpnConnection": {
        "CustomerGatewayConfiguration": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<vpn_connection id=\"<your-vpn-connection-id>\">\n  <cus
..... <shortened for brevity>
                    "OutsideIpAddress": "",
                    "TunnelInsideCidr": "",
                    "PreSharedKey": "<tunnel-1-secret-or-key>"
        "Routes": [],
        "Tags": []

Download the router configuration from the AWS console. Navigate to VPC and select Site-to-site VPN connection on the left-hand list. Pick the connection we just created and download the config as a text file

That’s it. The AWS side is done for now. We’ll need to add return routes from the VPC to the on-prem networks later but for now we can continue on to the Mikrotik configuration

Mikrotik configuration

Open the downloaded router configuration text file and SSH to the Mikrotik router. I use RouterOS 6.49.6 for this guide (latest at time of writing). An AWS VPN uses two tunnels. We have to configure both but will disable one of them later. Mikrotik doesn’t support dual active tunnels to AWS.

Create the IP addresses for the VPN tunnels. Search from the top of the file and look for “Customer gateway Inside Address”. The first 169.254.x.x IP will be for Tunnel 0. A second IP will be listed further down for Tunnel 1. We use a /30 subnet mask for the tunnel IPs.

Use your router outside interface. Mine is “sfp-sfpplus1” for this example

[admin@MikroTik] > ip address add address= interface=sfp-sfpplus1
[admin@MikroTik] > ip address add address= interface=sfp-sfpplus1
[admin@MikroTik] >
[admin@MikroTik] > ip address print
Flags: X - disabled, I - invalid, D - dynamic
 #   ADDRESS            NETWORK         INTERFACE
 0   ;;; defconf     bridge
 1       vl420
 2       vl701
 3       vl702
 4       vl800
 5       vl703
 6 D     sfp-sfpplus1
 7  sfp-sfpplus1
 8  sfp-sfpplus1
[admin@MikroTik] >

Add the IPsec peers

[admin@MikroTik] > ip ipsec peer add address= local-address= name=AWS-VPN-Peer-0
[admin@MikroTik] > ip ipsec peer add address= local-address= name=AWS-VPN-Peer-1

Add the IPsec identities (secrets for the two tunnels)

[admin@MikroTik] > ip ipsec identity add peer=AWS-VPN-Peer-0 secret=<tunnel-0-secret-or-key>
[admin@MikroTik] > ip ipsec identity add peer=AWS-VPN-Peer-1 secret=<tunnel-1-secret-or-key>

Add new or update the default IPsec profile and proposal

[admin@MikroTik] > ip ipsec profile set [ find default=yes ] dh-group=modp1024 dpd-interval=10s dpd-maximum-failures=3 enc-algorithm=aes-128 lifetime=8h
[admin@MikroTik] >
[admin@MikroTik] > ip ipsec proposal set [ find default=yes ] enc-algorithm=aes-128 lifetime=1h
[admin@MikroTik] >

Update the BGP instance settings

[admin@MikroTik] > routing bgp instance set default as=65010 redistribute-connected=yes redistribute-static=yes router-id=<your-onprem-public-ip>

Add the VPN tunnel BGP Peers (one will be disabled later)

[admin@MikroTik] > routing bgp peer add hold-time=30s keepalive-time=10s name=BGP-AWS-VPN-Peer-0 remote-address= remote-as=65011
[admin@MikroTik] > routing bgp peer add hold-time=30s keepalive-time=10s name=BGP-AWS-VPN-Peer-1 remote-address= remote-as=65011
[admin@MikroTik] >

Add any networks you wish to advertise to the VPC over the VPN

[admin@MikroTik] > routing bgp network add network=
[admin@MikroTik] > routing bgp network add network=
[admin@MikroTik] > routing bgp network add network=
[admin@MikroTik] > routing bgp network add network=
[admin@MikroTik] >

Set the firewall rules. One for the VPN tunnel CIDR range and one for the VPC CIDR ( in this example)

[admin@MikroTik] > ip firewall nat add action=accept chain=srcnat dst-address=
[admin@MikroTik] > ip firewall nat add action=accept chain=srcnat dst-address=

View the NAT rules

[admin@MikroTik] > ip firewall nat print
Flags: X - disabled, I - invalid, D - dynamic
 0    chain=srcnat action=masquerade out-interface-list=WAN

 1    chain=srcnat action=accept dst-address=

 2    chain=srcnat action=accept dst-address=
[admin@MikroTik] >

This won’t do. The WAN rule need to come last. Change the order using the “move” command

[admin@MikroTik] > ip firewall nat move 1 0
[admin@MikroTik] > ip firewall nat print
Flags: X - disabled, I - invalid, D - dynamic
 0    chain=srcnat action=accept dst-address=

 1    chain=srcnat action=masquerade out-interface-list=WAN

 2    chain=srcnat action=accept dst-address=
[admin@MikroTik] > ip firewall nat move 2 1
[admin@MikroTik] > ip firewall nat print
Flags: X - disabled, I - invalid, D - dynamic
 0    chain=srcnat action=accept dst-address=

 1    chain=srcnat action=accept dst-address=

 2    chain=srcnat action=masquerade out-interface-list=WAN
[admin@MikroTik] >

Create IPsec policies for the two VPN tunnels

[admin@MikroTik] > ip ipsec policy add dst-address= src-address= proposal=default peer=AWS-VPN-Peer-0 tunnel=yes
[admin@MikroTik] > ip ipsec policy add dst-address= src-address= proposal=default peer=AWS-VPN-Peer-1 tunnel=yes

Now the tunnel status should have changed to up. Verify from the AWS CLI

jonas@frantic-aerobics:~$ aws ec2 describe-vpn-connections | jq

Disable one of the tunnels

[admin@MikroTik] > routing bgp peer print
Flags: X - disabled, E - established
 0 E default     65011
 1 E default     65011
[admin@MikroTik] >
[admin@MikroTik] > routing bgp peer disable numbers=1
[admin@MikroTik] >
[admin@MikroTik] > routing bgp peer print
Flags: X - disabled, E - established
 0 E default     65011
 1 X default     65011
[admin@MikroTik] >

Add the final IPsec policy for the VPC network CIDR. Be sure to pick the tunnel Peer (0 or 1) which is still up.

[admin@MikroTik] > ip ipsec policy add dst-address= src-address= proposal=default peer=AWS-VPN-Peer-0 tunnel=yes
[admin@MikroTik] >

That’s it. Good job. The Mikrotik is now fully configured. All that is left is to add a return route to the on-premises networks from the VPC

Access the routing table for your VPC subnet and add return routes pointing to your VGW

Configuration complete. Time to test with a ping (be sure your security group for your EC2 instances have the correct ports open of course)

All works perfectly fine. Enjoy your new VPN!

Migrate VMware VMs from an on-prem DC to VMware Cloud on AWS (VMC) using Veeam Backup and Replication

When migrating from an on-premises DC to VMware Cloud on AWS it is usually recommended to use Hybrid Cloud Extension (HCX) from VMware. However, in some cases the IT team managing the on-prem DC is already using Veeam for backup and want to use their solution also for the migration.

They may also prefer Veeam over HCX as HCX often requires professional services assistance for setup and migration planning. In addition, since HCX is primarily a tool for migrations, the customer is unlikely to have had experience setting it up in the past and while it is an excellent tool there is a learning curve to get started.

Migrating with Veeam vs. Migrating with HCX

Veeam Backup & RecoveryVMware Hybrid Cloud Extension (HCX)
Licensed (non-free) solutionFree with VMware Cloud on AWS
Arguably easy to set up and configureArguably challenging to set up and configure
Can do offline migrations of VMs, single or in bulkCan do online migrations (no downtime), offline migrations, bulk migrations and online migrations in bulk (RAV), etc.
Can not do L2 extensionCan do L2 extension of VLANs if they are connected to a vDS
Can be used for backup of VMs after they have been migratedIs primarily used for migration. Does not have backup functionality
Support for migrating from older on-prem vSphere environmentsAt time of writing, full support for on-prem vSphere 6.5 or newer. Limited support for vSphere 6.0 up to March 12th 2023

What we are building

This guide covers installing and configuring a single Veeam Backup and Recovery installation in the on-prem VMware environment and linking it to both vCenter on-prem as well as in VMware Cloud on AWS. Finally we do an offline migration of a VM to the cloud to prove it that it works.


The guide assumes the following is already set up and available

  • On-premises vSphere environment with admin access (7.0 used in this example)
  • Windows Server VM to be used for Veeam install
    • Min spec here
    • Windows Server 2019 was used for this guide
    • Note: I initially used 2 vCPU, 4GB RAM and 60 GB HDD for my Veeam VM but during the first migration the entire thing stalled and wouldn’t finish. After changing to 4 vCPU, 32Gb RAM and 170 GB HDD the migration finished quickly and with no errors. Recommend to assign as much resources as is practical to the Veeam VM to facilitate and speed up the migration
  • One VMware Cloud on AWS (VMC) Software Defined Datacenter (SDDC)
  • Private IP connectivity to the VMC SDDC
    • Use Direct Connect (DX) or VPN but it must be private IP connectivity or it won’t work
    • For this setup I used a VPN to a TGW, then a peering to a VMware Transit Connect (VTGW) which had an attachment to the SDDC, but any private connectivity setup will be OK
  • A test VM to use for migration

Downloading and installing Veeam

Unless you already have a licensed copy, sign up for a trial license and then download Veeam Backup and Recovery from here. Version used in this guide.

In your on-premises vSphere environment, create or select a Windows Server VM to use for the Veeam installation. The VM spec used for this install are as follows:

Run the install with default settings (next, next, next, etc.)

Register the on-prem vCenter in Veeam

Navigate to “Inventory” at the bottom left, then “Virtual Infrastructure” and click “Add Server” to register the on-prem vCenter server

Listing VMs in the on-prem vSphere environment after the vCenter server has been registered in the Veeam Backup & Recovery console

Switching on-prem connectivity to VMware Cloud on AWS SDDC to use private IP addresses

For this setup there is a VPN from the on-premises DC to the SDDC (via a TGW and VTGW in this case) but the SDDC FQDN is still configured to return the public IP address. Let’s verify by pinging the FQDN

Switching the SDDC to return the private IP is easy. In the VMware Cloud on AWS web console, navigate to “Settings” and flip the IP to return from public to private

Ping the vCenter FQDN again to verify that private IP is returned by DNS and that we can ping it successfully over the VPN

All looks good. The private IP is returned. Time to register the VMware Cloud on AWS vCenter instance in the Veeam console

Registering the VMC vCenter instance with Veeam

Just use the same method as used when adding the on-premises vCenter server: Navigate to “Inventory” at the bottom left, then “Virtual Infrastructure” and click “Add Server” to register the on-prem vCenter server with Veeam

Note: If the SDDC vCenter had not been switched to use a private IP there will be an error in listing the data stores. Subsequently when migrating a VM the target data store won’t be listed and the migration can’t be started

After adding the VMware Cloud on AWS SDDC vCenter the resource pools will be visible in the Veeam console

Now both vSphere environments are registered. Time to migrate a VM to the cloud!

Migrating a VM to VMware Cloud on AWS

Below is both a video and a series of screenshots describing the migration / replication job creation for the VM.

Creating some test files on the source VM to be migrated

Navigate to “Inventory” using the bottom left menu, click the on-premises vCenter server / Cluster and locate a VM to migrate in the on-premises DC VM inventory. Right-click the VM to migrate and create a replication job

When selecting the target for the replication, be sure to expand the VMware cloud on AWS cluster and select one of the ESXi servers. Selecting the cluster is not enough to list up the required resources, like storage volumes

Since VMC is a managed environment we want to select the customer-side of the storage, folder and resource pools

If you checked the box for remapping the network is even possible to select a target VLAN for the VM to be connected to on the cloud side!

Select to start the “Run the job when I click finish” and move to the “Home” tab to view the “Running jobs”

The migration of the test VM finished in less than 9 minutes

In the vCenter client for VMware Cloud on AWS we can verify that the replicated VM is present

After logging in and listing the files we can verify that the VM is not only working but also have the test files present in the home directory

Thank you for reading! Hopefully this has provided an easy-to-understand summary of the steps required for a successful migration / replication of VMs to VMC using Veeam

Creating an Amazon AMI2 Linux VM in vSphere for use as a golden image in Terraform deployments

With CentOS being less than attractive to use now when Red Hat has changed how it is updated, the Amazon AMI2 Linux distribution can be an excellent alternative.

However, when deploying an Amazon AMI2 on vSphere for the first time there are a few hoops to jump through. This video shows how to create a golden image and deploy it with Terraform in less than 15 minutes

VMware home lab: 6 months with the new setup

In spring of 2021 I wanted a proper VMware lab setup at home. The primary reason was, and still is, having an environment in which to learn and experiment with the latest VMware and AWS solutions. I strongly believe that actual hands-on experience is the gateway to real knowledge, despite how well the documentation may be written.

To that end I went about listing up what would be needed to make this dream of a home lab come true. The lack of space meant that the setup would end up in my bedroom and therefore needed to be quiet. That removed most 2nd hand enterprise servers from the list. Possibly with the exception of the VRTX chassis from Dell, which I would still REALLY want for a home lab, but it’s way to expensive – even 2nd hand.


  • As compatible with the VMware HCL as possible (as-is or via Flings)
  • Quiet (no enterprise servers)
  • Energy efficient
  • Not too big (another nail in the coffin for full-depth 19″ servers)
  • Reasonable performance
  • Ability to run vSAN
  • 10Gbps networking

Server hardware

Initially I considered the Intel NUCs and Skull / Ghost Canyon mini-PCs as these are very popular among home-lab enthusiasts. However, the 10Gbps requirement necessitated a PCIe slot and the models supporting this from Intel are very expensive.

The SuperMicro E300-9D was also on the list but they too tend to get expensive and a bit hard to get on short notice where I live.

Therefore, going with a custom build sounded more and more in line with what would work for this setup. In the end I settled on the below. The list contain all the parts used for the ESXi nodes, minus the network cards which are listed separately in the networking section below.

PartBrandCost (JPY)
MoboASRock Intel H410M-ITX/ac I219V12,980link
CPUCore i5 10400 BOX (6c w. graphics)20,290link
RAMTEAM DDR4 2666Mhz PC4-21300 (2×32)33780link
m.2 cacheWD Black 500Gb SSD M.2-2280 SN7509,580link
2.5″ driveSanDisk 2.5″ SSD Ultra 3D 1TB13,110link
PSUThermaltake Smart 500W -STANDARD4,756link
CaseCooler Master H100 Mini Tower7,023link

Mainboard and case

The choice of mainboard came down to the onboard network chipset. It had to be possible to run the ESXi installer and it won’t work if it can’t find the network. Initially I only had the onboard NIC and no 10Gbps cards. Unfortunately the release of vSphere version 7.x restricted the hardware support significantly. This time I was going to make an AMD build, but most of their mainboards come with Realtek onboard NICs and they are no longer recognized by the ESXi installer. Another consideration was size and expansion options. An ITX formfactor meant that the size of the PC case could be reduced while still having a PCIe slot for a 10Gbps NIC.

The Cooler Master H100 case has a single big fan which makes it pretty quiet. Its small size also makes it an ideal case for this small-footprint lab environment. It even comes with LEDs in the fan which are hooked up to the reset button on the case to switch between colors (or to turn it off completely).


Due to the onboard NIC support the build was restricted to an Intel CPU. Gen 11 had been released but Gen 10 CPUs were still perfectly fine and could be had for less money. Obviously, there was no plan to add a discreet GPU so the CPU also had to come with built-in graphics. The Core i5 10400 seemed to meet all criteria while having a good cost / performance balance.


The little ASRock H410M-ITX/ac mainboard supports up to 64Gb of RAM and I filled it up from the start. One can never have too much RAM. With three nodes we get a total of 192Gb which will be sufficient for most tasks. Likely there will come a day later when a single workload (looking at you NSX-T!!) will require more. This is the only area which I feel could become a limitation soon. For that day I’ll likely have to add a box with more memory specifically for covering that workload.


A vSAN environment was one of the goals for the lab and with an NVME PCIe SSD as the cache tier and a 2.5″ drive as the capacity tier this was accomplished. It was a bit scary ordering these parts without knowing if they would be recognized in vCenter as usable for vSAN, but in the end there was no issue at all. They were all recognized immediately and could be assigned to the vSAN storage pool.

For the actual ESXi install I was going to use a USB disk initially but ended up re-using some old 2.5″ and 3.5″ spinning rust drives for the hypervisor install. These are not part of the cost calculation above as I just used whatever was laying around at home. The cost of these is negligible though.

Performance of the vSAN cluster isn’t too bad for using consumer hardware 🙂

Network hardware

To ensure vSAN performance and to support the 10Gbps internet router uplink a 10Gbps managed switch was required. Copper ports become very expensive so SFP+ would be the way to go. Mikrotik has a good 8+1 port switch / router in their CRS309-1G-8S+IN model. In the end this was a good fit for the home lab because not only does it have 8x 10Gbps SFP+ ports, it is also fanless and the software support several advanced features, like BGP.

I’m still happy with the choice 6 months later. It’s a great switch but it took a while to get used to it. Most of us probably come from a Cisco or Juniper background. The configuration for the Mikrotik is completely different and won’t be intuitive for the majority of users.


On the server side I wanted something which would be guaranteed to work with ESXi, so a 10Gbps card which is on the HCL was a must. Intel has a lot of cards on the list and their X520 series can be found pretty easily. In the end I got three X520-DP2 (dual port) cards and they have worked perfectly so far.

There is also a 1Gbps managed Dell x1026p switch to allow for additional networking options with NSX-T. With the Mikrotik 10Gbps switch there the Dell switch is more an addition for corner cases. It does help when attaching other devices which doesn’t support 10Gbps though.

The Mikrotik has a permanent VPN connection to an AWS Transit Gateway and from there to various VPCs and sometimes the odd VMware Cloud on AWS SDDC.

Installation media etc.

These servers still require custom installation media to be created for the installation to work. Primarily for the onboard Intel networking and the USB network Fling. An explanation for how to create custom media can be found here.

vCenter is hosted on an NFS share from a separate server. This is done so it could be on shared storage for the cluster while simultaneously being separate from the vSAN while the environment is being built.

ESXi is installed over PXE to allow for fully automated installations.


That’s it – a fully functional VMware lab. Quiet and with reasonably high performance. Also, RGB LEDs adds at least 20% extra performance – a bit like red paint on a sports car 😉