IPアドレスを変更せずにEC2からNC2への移行とAWSの2つのリージョン間でのDRフェイルオーバーを実現

一部のお客様は、AWSでリージョンをまたいだディザスタリカバリ(DR)を必要としていますが、別のリージョンにフェイルオーバーする際にIPアドレスが変更されるという課題に直面することがよくあります。この変更により、DRポリシーで保護されているインスタンス上で実行されているサービスへの外部アクセスが中断される可能性があります。

Nutanix Cloud Clusters (NC2) は、この課題に対応するためのDR機能をビルトインしており、リージョン間でフェイルオーバーする際にもIPアドレスを一貫して維持することができます。さらに、NC2ではCPUのオーバープロビジョニングが可能であるため、NC2への移行後にコンピュートコストを削減できる可能性もあります。ただし、DR中にIPアドレスを保持できるのは、すでにNutanixクラスター上に存在しているワークロードのみであるため、EC2インスタンスは先に移行しておく必要があります。

無料の移行ツールであるNutanix Moveを使用すれば、Amazon EC2からNutanix Cloud Clustersにワークロードを移行することができます。ただし、現時点ではIP保持をサポートしていません。このブログでは、移行中に一貫したIPアドレスを維持するための工夫を紹介しますが、MACアドレスが変更されることには注意してください。その後、NC2はNutanix DRソリューションの一部としてリージョン間フェイルオーバー中にIPアドレスを保持します。それでは始めましょう!

ソリューションアーキテクチャ

この例では、オンプレミスのデータセンター(DC)、DRポリシーでカバーするAWS VPC(別のリージョンにフェイルオーバーしてもIPアドレスが変更されないようにする)にあるEC2インスタンス、そして2つのNC2クラスター(東京リージョンと大阪リージョン)を使用します。この例では、AWSの東京リージョン(ap-northeast-1)をプライマリリージョン、大阪リージョン(ap-northeast-3)をディザスタリカバリの場所として使用します。

オンプレミス環境との接続をDirect Connectで示していますが、このソリューションのテストはすべて各リージョンのTGWに接続されたS2S VPNで行われています。DR間の接続は、クロスリージョンVPCピアリングまたは2つのTransit Gateway (TGW) のピアリングを使用して行うことができます。

ネットワーキング

移行されたEC2インスタンスのIPアドレスを保持するために、Flow Virtual Networking (FVN) を使用して、NC2上にオーバーレイのNATなしオーバーレイネットワークを作成します。このネットワークのCIDR範囲は、EC2インスタンスが接続されている元のサブネットの範囲と一致させます。このオーバーレイネットワークは、東京および大阪のNC2クラスターの両方に作成されます。これにより、VMがフェイルオーバーされても同じCIDR範囲を持つネットワークに接続できます。

オンプレミスDCがこれらのVMにアクセスできるようにするために、プロセス全体でルートテーブルを変更します。これにより、移行されたEC2インスタンスの場所に関係なく、それらを指すルートを維持できます。

Terraform / Open Tofu を使用したVPCとTGWの自動作成

このソリューションを試してみたい場合、VPCやTGW、ルーティングをデプロイするためのTerraform / Open Tofuテンプレートを以下で入手できます:

https://github.com/jonas-werner/aws-dual-region-peered-tgw-with-vpcs-for-nc2-dr

リージョン間接続のためのピアリングタイプの選択

一般的に、以下のように考えられます:

  • VPCピアリング: データ転送コストがやや高いものの、直接的な接続のシンプルさが求められる低トラフィックのシナリオに最適です。
  • TGWピアリング: 高トラフィック環境や複雑なアーキテクチャに適しています。集中管理や低いデータ転送料金が、TGWアタッチメントの追加コストを上回ります。なお、トラフィックは2つのTGWを通過しますが、ピアリングインターフェイスではデータ転送料金は発生しないため、データは1回だけ課金されます(東京リージョンではおおよそ0.02セント/GiB)。

IP保持のための回避策

冒頭で述べたように、Nutanix Move仮想アプライアンスはEC2からNC2への移行に非常に優れていますが、現時点では移行されたワークロードのIPアドレスを保持する機能はありません。この課題を回避するために、以下の手順を実施します:

  1. 移行前
    AWS Systems Manager (SSM) を使用して、移行対象のインスタンス上でPowerShellまたはBashスクリプトを実行します。このスクリプトは、EC2インスタンスのID、ホスト名、およびローカルIPアドレスを取得し、その情報をDynamoDBテーブルに保存します。
  2. 移行中
    Nutanix Moveを使用して、EC2からNC2にインスタンスを移行します。この間にIPアドレスは変更されますが、移行先はEC2インスタンスが接続されていた元のネットワークのCIDR範囲と一致するFVNオーバーレイネットワークです。
  3. 移行後
    DynamoDBに保存した情報をテンプレートとして使用するPythonスクリプトを実行します。このスクリプトは、Nutanix Prism Central APIに接続し、既存のネットワークインターフェイスを削除し、正しい(元の)IPアドレスを持つ新しいインターフェイスを各インスタンスに追加します。

これらの手順を経て、インスタンスがNC2に移行された後は、東京と大阪のリージョン間でのDR設定が簡単に行えるようになります。

このブログに使われているスクリプトは以下の GitHubページからダウンロードできます:

https://github.com/jonas-werner/EC2-to-NC2-with-IP-preservation/tree/main

ステップ1: EC2インスタンスのIPアドレスを取得

このステップでは、EC2からNC2への移行の準備を行います。ネットワーク、ワークロード、および172.30.1.0/24ネットワークへのルートの初期状態は、以下の図の赤い線で示されています。

最初に、EC2インスタンスに関する情報を収集し、その情報をDynamoDBに保存します。効率を重視して、SSM Run Commandを使用してPowerShellスクリプトを実行します。これにより、WindowsおよびLinuxワークロードの両方を1回または2回の操作で簡単に処理できます。この例では、単一のWindows Server 2019 EC2インスタンスをテスト対象として使用します。

まず、この情報を保持するためのDynamoDBテーブルを作成します。このテーブルには特別な要件はなく、SSMがスクリプトを実行する際にアクセスできれば十分です。もちろん、SSMコマンドを実行する際に使用するIAMロールにDynamoDBへのアクセス権を付与する必要があります。そのため、以下のような権限を標準のSSMロールに追加します:

{
    "Sid": "AllowDynamoDBAccess",
    "Effect": "Allow",
    "Action": [
        "dynamodb:PutItem"
    ],
    "Resource": "arn:aws:dynamodb:<your-aws-region>:<your-aws-account>:table/<dynamodb-table-name>"
},

インスタンス名をメタデータから収集するために、EC2コンソールで「インスタンスメタデータにタグを許可」設定を有効化します。この設定は、移行後にNC2でインスタンスを検索する際に「Name」タグをキーとして使用するために重要です。他の方法(たとえばインスタンス名自体を使用する)も可能ですが、このケースではEC2の「Name」タグを使用します。これは、移行後も同じタグがNC2に表示されるためです。

SSM Run Commandを使用して、インスタンス上でスクリプトを実行します。以下のコマンド例を参照してください:

スクリプトの実行後、Windows EC2インスタンスのエントリがDynamoDBに表示されます。これには、インスタンスID、ホスト名、およびIPアドレス(例: 172.30.1.34)が含まれます。このIPアドレスは保持したいアドレスです。

次に、EC2 から NC2 への移行を実行します。

ステップ2: EC2からNC2への移行

次に、EC2からNC2への移行を実施します。この移行では、Nutanix MoveをNC2クラスター上に展開済みである必要があります。また、FVNオーバーレイネットワークを作成しており、そのCIDRはEC2インスタンスが接続されていた元のサブネットと一致していますが、DHCPの範囲は現在そのサブネットで使用されているIPアドレスを避けるよう設定されています。

Moveには、NC2クラスターとAWS環境が移行元および移行先として設定されています。

「Missing Permissions」という警告が表示される場合がありますが、これはAWS IAMポリシーで、EC2への移行を許可していないためです。しかし、EC2からの移行のみを行う場合、この警告は無視して構いません。必要なIAMポリシーの詳細は、Moveのマニュアルをご確認ください。

移行後、VMは異なるIPアドレスを持つようになります(移行先のFVNサブネットのDHCP範囲から取得されます)。

次のステップで、Pythonスクリプトを使用して元のIPアドレスに戻す処理を行います。

ステップ3: EC2インスタンスが元々持っていたIPアドレスに戻す

次に、Pythonスクリプトを実行して、DynamoDBに保存されたインスタンス名を参照し、それをNC2のVM名と照合します。その後、Prism Central APIを使用して既存のネットワークインターフェイスを削除し、新しいネットワークインターフェイスを追加します。この新しいインターフェイスには、元の静的IPアドレスが設定されます。

このスクリプトはGitHubからダウンロードできます。スクリプトを実行するには、Prism Centralのユーザー名とパスワードを環境変数としてエクスポートしてください。また、Prism CentralのIPアドレス、使用するサブネット名、AWSリージョン、DynamoDBテーブル名をお使いの環境に合わせて更新してください。

スクリプトを実行すると、VMが元のIPアドレスを取得したことを確認できます。ただし、このプロセスではNICが置き換えられるため、IPアドレスは同じですが、MACアドレスは変更されています。

EC2からNC2への移行後の東京リージョンでのルーティング

VMがNC2上に存在するようになったため、トラフィックが元のEC2インスタンスではなく、このVMに向かうようルーティングを更新する必要があります。これは、EC2 VPCをTGWから切断し、NC2 VPCを指すようにTGWに静的ルートを追加することで実現します。このサブネットはすでにDXGWの「許可されたプレフィックス」として存在するはずなので、この部分は変更する必要はありません。

赤で強調されたアタッチメントは、172.30.1.0/24サブネットへのアクティブルートを示しており、現在はNC2 VPCを指すように変更されています。このサブネットはFVNのNo-NATサブネットであるため、NC2 VPCのルートテーブルに表示されます。

移行作業のまとめ

これでEC2インスタンスはNC2に移行されました。IPアドレスは保持されており、AWSとオンプレミスDC間のルーティングが更新されたため、オンプレミスのユーザーは、通常通り移行されたインスタンスにアクセスできます。実際、移行のメンテナンスウィンドウ、NC2でのVMの電源オン、およびルーティングの切り替えを除けば、これらのユーザーは元のEC2インスタンスが別のプラットフォームで動作するようになったことに気付くことはほとんどありません

東京と大阪のNC2クラスター間でのDR構成

ここまでで、東京リージョンのNC2クラスターにEC2インスタンスを移行し、IPアドレスを保持した状態でオンプレミス環境と通信できるようになりました。次に行うのは、東京と大阪の2つのNutanixクラスター間で災害復旧(DR)構成を設定することです。DRはNutanixの標準機能として組み込まれているため、この設定は非常に簡単です。Prism Centralインスタンスをリンクし、大阪側でもFVNオーバーレイネットワークを作成して、フェイルオーバー後も同じCIDR範囲を使用できるようにします。

災害復旧機能を有効にした後、Prism Centralを使用してDRプランを簡単に作成できます。

DRプランを作成する際、東京ネットワーク上のVMが大阪DRサイト上の対応するネットワークにフェイルオーバーするように設定します。

最後に、東京のNC2クラスターから大阪のNC2クラスターにVMをフェイルオーバーします。

フェイルオーバー後、VMが大阪で正常に起動していることを確認できるだけでなく、期待通りにIPアドレスが保持されていることも確認できます。

東京から大阪へのルーティング更新

東京から大阪へのフェイルオーバーが完了したら、172.30.1.0/24ネットワークを指すルートを更新し、大阪を指すようにTGWの設定を変更します。これにより、以下のようなネットワーク構成になります。

大阪TGWでは、172.30.0.0/16ネットワークを大阪のNC2 VPCに向けた静的ルートを作成します。

また、東京TGWの静的ルートも更新し、ローカルのNC2 VPCを指すルートを大阪へのピアリング接続に変更します。

結果とまとめ

これらのルーティング変更が適用されることで、オンプレミスのデータセンターからのユーザーは、同じIPアドレスを使用して同じVMにアクセスし続けることが可能になります。この一貫性は、EC2からNC2への移行、および東京リージョンから大阪リージョンへの災害復旧計画に基づいたフェイルオーバー後も維持されます。

このソリューションにより、AWS上で動作していたワークロードがNC2上で動作するようになり、その後もIPアドレスを変更することなく運用を継続できます。これにより、ユーザーにとっての影響を最小限に抑えつつ、DR計画を実現できます。

ぜひこのソリューションを試してみてください。また、この種のソリューションに興味がある場合は、Nutanixの担当者にお問い合わせください。お読みいただきありがとうございました!

リンク

Migrate from EC2 to NC2 and perform DR failover between two AWS regions without changing IP addresses

Some customers require cross-region disaster recovery (DR) in AWS but often face the challenge of changing IP addresses during a failover to another region. This change can disrupt external access to services running on instances covered by the DR policy.

Nutanix Cloud Clusters (NC2) address this challenge with built-in DR functionality that ensures IP addresses remain consistent during failovers between regions. Bonus: It is possible to over-provision CPU on NC2, so it may actually be possible to save on compute costs after the migration to NC2. However it can only retain IP addresses during DR for workloads which are already residing on a Nutanix cluster, so we have to migrate the EC2 instances first.

The free Nutanix Move migration tool can migrate workloads from Amazon EC2 to Nutanix Cloud Clusters, though it currently lacks support for IP retention. In this blog we use some creative workarounds to maintain consistent IPs throughout the migration, although note that MAC addresses will change. NC2 then retains the IPs during regional failovers as part of the Nutanix DR solution. Let’s dive in!

Solution architecture

In this case we have an on-premises datacenter (DC), an AWS VPC with EC2 instances which we want to have covered by a DR policy (so they can fail over to another region without changing IP addresses) and finally two NC2 clusters – one in the primary region and one in a separate region for DR purposes. We use Tokyo (ap-northeast-1) as the primary AWS region and Osaka (ap-northeast-3) as the disaster recovery location in this example.

Overview of solution architecture. Click to embiggen.

We illustrate connectivity to the on-premises environment by using Direct Connect. Note that all the testing of this solution has been done with S2S VPN attached to the TGW’s in each region. Peering between the two DR locations can be done by using cross-region VPC peering or peering of two Transit Gateways (TGW).

Networking

To retain the IP addresses of the migrated EC2 instances we use Flow Virtual Networking (FVN) to create overlay no-NAT overlay networks on NC2 with a CIDR range which matches that of the original subnet the EC2 instances are connected to. We create this overlay network in both Tokyo and Osaka NC2 clusters so that we can later fail over the VMs and have them attach to a network with the same CIDR range.

To ensure the on-premises DC is able to access the VMs we modify the route tables throughout the process. That way we maintain routes which point to the migrated EC2 instances, regardless of where they are located.

Automating the VPC and TGW creation with Terraform / Open Tofu

In the case you’d like to try this out yourself, the Terraform / Open Tofu templates for deploying the VPC’s, TGW’s and the routing for these can be found on GitHub here:

https://github.com/jonas-werner/aws-dual-region-peered-tgw-with-vpcs-for-nc2-dr

When to use which peering type for inter-region connectivity

Generally it can be said that VPC Peering is better for lower traffic scenarios or when the simplicity of direct peering is desirable, despite slightly higher data transfer costs incurred for VPC peering.

TGW Peering is more cost-efficient for high-traffic environments or complex architectures, where the centralized management and lower data transfer rates outweigh the additional costs per TGW attachment. Note that although traffic passes through two TGW’s, the peering interface doesn’t incur data transfer charges so the data is only charged once (roughly 0.02 cents / GiB in the Tokyo region).

The workaround for IP retention

As mentioned in the introduction, while the Nutanix Move virtual appliance is very capable at migrating from EC2 to NC2, it is at time of writing unable to retain the IP addresses of the workloads it migrates. To work around this we do the following:

  1. Prior to the migration we use AWS Systems Manager (SSM) to run a PowerShell or Bash script on the instances to be migrated. The script captures the EC2 instances ID, hostname and local IP address and stores that information into a DynamoDB table for use later
  2. We perform the migration from EC2 to NC2 using Nutanix Move. The IP address will change although we migrate the instance to a Flow Virtual Networking (FVN) overlay network with the same CIDR range as the original network the EC2 instances are connected to.
  3. We run a Python script which uses the DymamoDB information as a template and then connects to the Nutanix Prism Central API. It then removes the existing network interface and adds a new one with the correct (original) IP address to each of the migrated instances.

Once the instances are migrated to NC2 the process of configuring DR between Tokyo and Osaka regions is trivial.

You can download the PowerShell and Python scripts used in this blog on GitHub:

https://github.com/jonas-werner/EC2-to-NC2-with-IP-preservation/tree/main

Step 1: Capture IP addresses of the EC2 instances

In this step we prepare for the migration from EC2 to NC2. The initial state of the network, the workloads and the route to the 172.30.1.0/24 network is as illustrated by the red line in the below diagram.

To start with we gather information about the EC2 instances and store that info in DynamoDB. In the name of efficiency we use the SSM Run command to execute the PowerShell script. This makes it easy to get this done in a single go (or two “goes” if we do both Windows and Linux workloads). We test with a single Windows 2019 Server EC2 instance in this example.

First create a DynamoDB table to hold this information. Nothing special is required for this table as long as it is accessible to SSM as it runs the script. We need to give the IAM role used when running SSM commands access to DynamoDB of course, so we add the following permissions to the standard SSM role:

        {
            "Sid": "AllowDynamoDBAccess",
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem"
            ],
            "Resource": "arn:aws:dynamodb:<your-aws-region>:<your-aws-account>:table/<dynamodb-table-name>"
        },

In order to collect the instance name from meta data we enable the “Allow tags in instance metadata” setting in the EC2 console. This is important as we will use the “Name” tag in EC2 as the Key to look up the instance in NC2 post-migration. Of course other methods could be used – most obviously the name of the instance itself. However in this case we use the EC2 name tag, as this is also how the VM will show up in NC2 post migration.

The we execute the script on our instances through the SSM Run command as follows

After execution we can see an entry for our Windows EC2 instance showing its instance ID, hostname and IP address: 172.30.1.34. This is the IP we want to retain.

That’s all for this section. Next we perform the migration from EC2 to NC2.

Step 2: Migrating from EC2 to NC2

For the migration we have deployed Nutanix Move on the NC2 cluster. We have also created an FVN overlay no-NAT network with the same CIDR as the subnet the EC2 instance is connected to, although the DHCP range is set to avoid any of the IPs currently used by instances on that subnet.

Move has the NC2 cluster and the AWS environment added in as migration sources / targets.

It complains about “missing permissions” but this is because we have only given it permission to migrate FROM EC2, not TO EC2. Since that is all we want to do, this is fine. Please refer to the Move manual for details on the AWS IAM policy required depending on your use case.

Post migration the VM will have a different IP address (taken from the DHCP range on the FVN subnet it is connected to).

We use a Python script in the next section to revert the IP address to what it was while running as an EC2 instance.

Step 3: Revert the IP address to match what the EC2 instance had originally

Now we execute a Python script which will look up the instance name in DynamoDB, match it with the VM name in NC2 and then remove and re-create the network interface using the Prism Central API. The new interface will have the original IP configured as a static address.

The script can be downloaded from GitHub here. Please export the Prism Central username and password as environment variables to run the script. Also update the Prism Central IP and the subnet name to match the one used in your environment as well as the AWS region and the DynamoDB table name.

After running the script we can now verify that the VM has received its original IP address. Note that since we have replaced the NIC in this process, the IP is the same as before, but the MAC address will have changed.

Routing after migration from EC2 to NC2 in Tokyo

Now that the VM exists on NC2 we need to update our routing to ensure that traffic is directed to this VM and not the original EC2 instance (which has now been shut down by Move after the migration).

To do this we disconnect the EC2 VPC from the TGW and add the subnet as a static route in the TGW, this time pointing to the NC2 VPC rather than the EC2 VPC. The subnet should already exist as an “Allowed prefix” on the DXGW, so that part can be left as-is.

The attachments highlighted in red shows the active route to the 172.30.1.0/24 subnet, which has now been changed to point to the NC2 VPC. Since the subnet is a FVN no-NAT subnet it will show up in the NC2 VPC route table.

Wrapping up the migration part

Now our EC2 instance has been migrated to NC2. Its IP address is intact and since we have updated the routing between AWS and the on-premises DC, the on-prem users can access the migrated instances just like they normally would. In fact, apart from the maintenance window for the migration, VM power-up on NC2 and the routing switch, they are unlikely to notice that their former EC2 instance is now running on another platform.

Configuring DR between the Tokyo and Osaka NC2 clusters

At this point all we have left to do is set up the DR configuration between the two Nutanix clusters in Tokyo and Osaka. Since DR is a built-in component, this is very straight forward. We link the two Prism Central instances and of course create the FVN overlay network on the Osaka side as well to ensure we can keep the same CIDR range also after failover.

After enabling Disaster Recovery we can easily create the DR plan through Prism Central

When we create the DR plan we set the VMs on the Tokyo network to fail over to its equivalent on the Osaka DR site

Finally we proceed to fail over our VM from NC2 in Tokyo to NC2 in Osaka

After failing over we can confirm that the VM is not only powered up in Osaka, but that it has also retained the IP address, as expected.

Updating the routing to point to Osaka rather than Tokyo

After failing over from Tokyo to Osaka we need to also update the routes pointing to the 172.30.1.0/24 network by modifying the TGW in Osaka. From a diagram perspective it will look like follows.

On the Osaka TGW we create a static route to the 172.30.0.0/16 network pointing to the Osaka NC2 VPC

We also update the static route on the Tokyo TGW which points to the local NC2 VPC and instead set it to point to the peering connection to Osaka

Results and wrap-up

With these routing changes implemented it is now possible for users on the on-premises DC to continue to access the very same VMs with the very same IP addresses. This possible is even after those workloads have been migrated from EC2 to NC2 in Tokyo and then further having been failed over with a DR plan from Tokyo region to Osaka region.

Hope that was helpful! Please reach out to your local Nutanix representative for discussions if this type of solution is of interest. Thank you for reading!

Links

Change NC2 bare-metal node type while the cluster is running

Normally during long car races the drivers enter the pit do change to fresh tires, fill up fuel etc. Metaphorically speaking, Nutanix Cloud Clusters on AWS can change the entire car – without the need for a pit stop. The driver can just keep driving while the bare metal is replaced underneath, as if nothing has happened with the exception of getting more power. All the benefits – none of the downtime. And, it can be done with a single command through the Nutanix MCM portal.

Introduction

In this example we swap out i3.metal nodes to more powerful i4i.metal while the cluster is running. The starting point is a cluster with three i3.metal nodes and the end state is the same cluster, but now with three i4i.metal nodes. The change is seamless for the workloads running on top of NC2. Apart from a few packets dropped during the network change they experience no disruption.

Starting point

We start out with a plain NC2 on AWS cluster with three i3.metal nodes. In addition to the basic cluster components we have also opted to deploy Prism Central and Flow overlay networking.

Multiple VMs are running on the NC2 cluster. To monitor their health we start a continuous ping which statistics can be evaluated after the cluster nodes have been replaced.

On the networking side we have set up No-NAT networking with Flow and as such the subnet the test VM is attached to is accessible also from the native AWS VPC. In this case we are pinging the NC2 test VM from an EC2 instance in a separate AWS VPC.

Updating the Cluster Capacity settings in the Nutanix MCM portal

The management portal for NC2 allows for easy updates to the cluster capacity and configuration. We highlight our cluster and navigate to Cluster Capacity where the node types and the number of nodes can be changed.

A few clicks later we have added three new i4i nodes to our original configuration of three i3 nodes and we have also set the number of i3 nodes to zero. This way we get three new nodes of a more powerful configuration added and after all data has been transferred over, the old cluster nodes will be removed and billing for them stopped.

The task has now been accepted by the MCM portal and is being executed in the background. VMs running on NC2 continue working as usual, unaware of the big changes to the system which are under way.

EC2 bare-metal changes as seen from the AWS console

In the AWS console it is possible to witness the process of the i4i.metal nodes being added, i3 and i4i nodes running at the same time while the cluster shifts to run on the new nodes and finally the decommissioning of the i3.metal nodes.

From a networking perspective: The i3.metal ENI which was the active point of North-South communication for the cluster, and therefore part of the AWS VPC route table, has been shifted to an ENI on one of the new i4i.metal hosts post migration.

Result

The node swap has completed without a hitch and without any need of input from the IT administrator managing the NC2 cluster – well, apart from initiating the change at the start. The entire process took just under one hour to complete:

More importantly, the workloads have experienced just a blip in network connectivity and no downtime or reboots.

The Linux VM which we started pinging at the beginning of the blog post is still up and the pings are still getting through. Throughout the hour-long change a total of 3381 pings were sent. 26 of these were lost (0% loss).

The uptime command on the Linux host also show that there was no rebooting of VMs involved.

Conclusion

This was an example showing of how quick and easy it is to migrate from one EC2 bare metal instance type to another when using Nutanix Cloud Clusters on AWS. For more information, please visit the Nutanix Cloud Clusters page below:

https://www.nutanix.com/products/nutanix-cloud-clusters

How to set up NAT and No-NAT networking with NC2 on AWS

Nutanix Cloud Clusters (NC2) on AWS support native AWS networking but it is also possible to deploy Flow overlay networking as part of the cluster creation process. Flow overlay networking enables many powerful features. One of those is the ability to create completely new Flow VPCs with subnets using entirely different CIDR ranges than the native AWS VPC. This gives great flexibility in handling networking for Virtual Machines (VMs) running on NC2 on AWS.

When Flow overlay networking is used, VMs can communicate with the outside world in a few different ways. One is through NAT:ed networking, where the overlay subnet which the VMs are connected to is internal to NC2 only and is not visible outside the cluster from the AWS native side.

The other is to use No-NAT. In that case the overlay subnet which the VMs are connecting to is added to the native AWS VPC routing table. Thereby it is possible for entities in the native AWS VPC, or elsewhere, to access VMs on NC2. This is despite those VMs being connected to overlay networks with CIDR ranges which doesn’t exist on the AWS native side.

Architecture

In this example, two Flow overlay VPC’s are created with one subnet each. One is the NAT:ed network with a CIDR range of 10.20.0.0/24 and the other is a No-NAT network with a CIDR range of 10.30.0.0/24.

The neighboring VPC with CIDR 10.70.0.0/16 is connected to the VPC holding the NC2 cluster via an Amazon Transit Gateway (TGW). A Windows EC2 instance will be used to verify connectivity once routing has been set up for the No-NAT network.

Flow Transit VPC

To handle North-South connectivity for the workloads on NC2 there is a Flow Transit VPC deployed as part of the cluster creation. It is a special VPC in that while it handles external connectivity for VMs on NC2, those VMs don’t connect to it directly. Instead separate Flow VPC’s are created for VM connectivity and those VPCs are in turn attached to the Flow Transit VPC.

NAT network connectivity

In this section a new Flow VPC called “VPC A” is created and attached to the Flow Transit VPC. VPC A will be used for VMs which use NAT:ed communication with the outside world.

In Prism Central on NC2 on AWS, navigate to “Network and Security” and create a new VPC.

The VPC is given the name “VPC A” and for “External Connectivity” the already existing “overlay-external-subnet-nat” subnet is used. Check the box for using this subnet as the next hop / default route for all outbound traffic.

The VPC is now created but we also need a subnet for VMs to attach to. Click on “VPC A” and then “Create Subnet” from the “Subnet” tab. In this case the CIDR range 10.20.0.0/24 is used.

Now the NAT network configuration is complete. We can test connectivity by creating a new VM and attaching it to the new “VPC A – Subnet 1” NAT network.

In Prism Central, navigate to “VM” and create a new VM

Once the VM is created, power it on and verify that it can connect to the outside world.

In this case we get an IP of “10.20.0.16” and can verify North-South connectivity by pinging a Google DNS server.

This concludes the NAT portion of the setup.

No-NAT network connectivity

For No-NAT connectivity, the steps are very similar to those just performed for the NAT:ed portion. However, there are two additional steps to perform. The first is to create a new subnet in the Flow Transit VPC specifically for No-NAT connectivity. The second is to add the Flow overlay network CIDR range that we want to use for No-NAT as an Externally Routable Prefix, or ERP, to both the Flow Transit VPC and the Flow VPC we will create for No-NAT connectivity. Please read on to see how to set this up.

First we create a new subnet in the Flow Transit VPC and call it “overlay-external-subnet-nonat”. Navigate to “Network and Security”, select “Virtual Private Clouds” and click on the “transit-vpc”.

Go to the “Subnets” tab and select “Create Subnet”. Give it a name, like “overlay-external-subnet-nonat” and for IP address management, add any CIDR range which isn’t used elsewhere in your organization. This CIDR range will be used internally in NC2 but will not be routable or visible outside the cluster.

The “transit-vpc” will now have two subnets, like the below. The “overlay-external-subnet-nat” CIDR range will depend on the AWS native subnet NC2 has been deployed into.

Next, create a new VPC, called “VPC B” in this example and for external connectivity use the newly created “overlay-external-subnet-nonat”. Note that “NAT-ed” is set to “No”.

We also create a new subnet in that VPC so VMs have something to connect to. In this case we use a CIDR range of “10.30.0.0/24”.

Finally, the most important point is to add the CIDR range of the newly created subnet to both the Flow Transit VPC and to VPC B as an ERP or Externally Routable Prefix.

Select “VPC B” and click “Update” to add “10.30.0.0/24” as the ERP

Next, do the same to the “transit-vpc”

That is all we need to do on the configuration side. Congratulations!

Verify that the new No-NAT network shows up in the native AWS route table

In the AWS console, go to the default route table of the VPC which NC2 is deployed into and verify that the “10.30.0.0/24” network has been automatically added to the route table. It will be pointing to the ENI (Elastic Network Interface) of the currently active NC2 bare-metal node.

Verifying No-NAT routing and connectivity

For testing purposes, deploy a VM in NC2 and attach it to the newly created No-NAT network. In this case the VM received an IP of “10.30.0.20”.

Next we add a static route to 10.30.0.0/24 in the TGW route table, marked as step 1 in the diagram below.

In the AWS console, navigate to “Transit Gateway route tables”, select the route table for the TGW used to provide connectivity between the two AWS native VPCs and add a static route for “10.30.0.0/24” pointing to the VPC attachment for the VPC holding the NC2 cluster.

Finally, we add a static route for “10.30.0.0/24” to the route table of the native AWS VPC with CIDR “10.70.0.0/16” to enable the EC2 instances there to communicate with the test VM running on NC2. The route here will be pointing to the TGW attachment since the TGW will be handling the communication between the NC2 cluster VMs and the attached AWS native VPC.

This concludes the routing configuration. As the final step we can verify connectivity by connecting from the AWS native Windows EC2 instance to the Ubuntu test VM on NC2 on AWS (with IP 10.30.0.20″) using SSH.

As shown in the screenshot, there is no problem to access the NC2 VM from an AWS EC2 instance, despite the NC2 VM being attached to a Flow overlay network which doesn’t exist outside NC2. The No-NAT configuration makes connectivity from the outside world possible.

L2 extension from on-prem VMware cluster to Nutanix Cloud Clusters (NC2) on AWS using Cisco CSR1000V as on-prem VTEP

This guide has been written in cooperation with Steve Loh, Advisory Solutions Architect, Network & Security at Nutanix in Singapore, gentleman extraordinaire and master of the Cisco Dark Arts.

Introduction

Organizations frequently choose to extend on-premises networks to the cloud as part of retaining connectivity between virtual machines (VMs) during migrations. This is called L2 or Layer 2 extension. With an extended network, VMs connected to the same subnet can communicate as usual even when some of them have been migrated to the cloud and others are still on-premises awaiting migration.

L2 extension is sometimes used on a more permanent basis when the same network segment contain VMs to be migrated as well as appliances which must remain on-prem. If the organization doesn’t want to change the IP addresses of any of these entities and still need part of them migrated, they might chose to maintain L2 extension of some subnets indefinitely. However, this is generally considered a risk and is not recommended.

Architecture diagram

This is a graphical representation of the network stretch from on-prem VMware to Nutanix Cloud Clusters in AWS which is covered in this blog post. Other configurations are also possible, especially when Nutanix is also deployed on-prem.

Video of process

For those preferring watching a demo video of this process rather than reading the blog, please refer to the below.

Limitations and considerations

While L2 extension is a useful tool for migrations, please keep the following points in mind when deciding whether or not to utilize this feature:

  • L2 extension will complicate routing and thereby also complicate troubleshooting in case there are issues
  • L2 extension may introduce additional network latency. This takes the shape of trombone routing where traffic need to go from the cloud via a gateway on the on-premises side and then back to the cloud again. Nutanix Flow Policy Based Routing (PBR) may be used to alleviate this.
  • If routing is set to go via one default gateway either on-premises or in the cloud, if the network connecting the on-premises DC with the cloud environment has downtime, the VMs on the side without the default gateway will no longer be able to communicate with networks other than their own
  • The Nutanix L2 extension gateway appliance does not support redundant configurations at time of writing
  • A Nutanix L2 extension gateway can support up to five network extensions
  • A single Prism Central instance can support up to five Nutanix L2 extension gateways
  • Always keep MTU sizes in mind when configuring L2 extension to avoid unnecessary packet fragmentation. MTU settings can be configured when extending a network.
  • Even though VMs are connected to an extended network, if the current version of Move is used for migration, VM IP addresses will not be retained. A Move release in the near future will enable IP retention when migrating from VMware ESXi to NC2 on AWS.

Types of L2 extension

Various methods of extending a network exist. This blog will cover one of these cases – on-premises VMware with Cisco CSR1000V as VTEP to Nutanix Cloud Clusters (NC2) on AWS with a Nutanix gateway appliance.

On-premises VLANs or Flow overlay networks can be extended using Nutanix GW appliances to Flow overlay networks in NC2. It is also possible to extend using Nutanix VPN appliances in case the network underlay is not secure (directly over the internet). Finally, when the on-premises environment does not run Nutanix, using a virtual or physical router with VXLAN and VTEP capabilities is possible. This blog focuses on the last use case as it is a commonly discussed topic among customers considering NC2 and L2 extension.

Routing

When extending networks, the default gateway location and routing to and from VMs on an extended network become important to understand. Customers used to extending networks with VMware HCX or NSX Autonomous Edge are familiar with the concept of trombone routing over a default gateway located on-premises. With Nutanix it is possible to use Policy Based Routing (PBR) to control how routing should be performed for different networks. In many ways, Nutanix PBR offers more detailed tuning of routes than can be done with VMware MON in HCX.

A key difference between extending networks with HCX vs. with Nutanix is that with HCX the extended network appears as a single entity, although it exists on both sides (on-prem and cloud). The default gateway would generally be on-prem and both DHCP and DNS traffic would be handled by on-prem network entities, regardless if a VM was on-prem or in the cloud.

For L2 extension with Nutanix, things work a bit differently. The on-prem network will be manually recreated as an overlay network on NC2, with the same default gateway as the on-prem network but with a different DHCP range. The on-prem and cloud networks are then connected through a Nutanix GW appliance deployed as a VM in Prism Central.

Prerequisites

This guide assumes that an on-premises VMware vSphere 7.x environment and an NC2 version 6.8.1 cluster are already present. It also assumes that the on-prem and NC2 environments are connected over a L3 routed network, like a site-to-site (S2S) VPN or DirectConnect (DX). The two environments have full IP reachability and can ping each other.

In this case we are extending a VLAN which has been configured as a port group with VLAN tagging on standard vSwitches on the ESXi hosts.

Overview of steps

  1. Recreate the network to be extended using Flow overlay networking on NC2
  2. Deploy the Nutanix gateway appliance on NC2
  3. Deploy the Cisco CSR1000V in the on-premises VMware cluster
  4. Enable Promiscuous mode and Forged transmits on the vSwitch portgroup of the VLAN to be extended
  5. Register the CSR1000V routable IP address as a Remote Gateway in NC2
  6. Configure the CSR1000V IP leg on the VLAN to be extended and set up VNI and other settings required to extend the network
  7. Extend the network from NC2 with the CSR1000V as the on-prem VTEP

In the demo video included in this post we also perform a migration of a VM from on-prem to NC2 and verify connectivity with ICMP.

Step 1: Recreate the network to be extended using Flow overlay networking on NC2

Access Prism Central, navigate to Network and Security. Select “Create VPC” to add a new Nutanix Flow VPC to hold the subnet we want to extend.

After the VPC has been created, go to Subnets and create a subnet in the newly created VPC with settings matching the network which will be extended. In this case we create “VPC-C” and a subnet with a CIDR of “10.42.3.0/24”. The default gateway is configured to be the same as on-prem but the DHCP range is set to not overlap.

Step 2: Deploy the Nutanix gateway appliance on NC2

In Prism Central, navigate to “Network and Security” and select “Connectivity”. From here click “Create Gateway” and select “Local” to create the gateway on the NC2 side.

Add a name, set the Gateway Attachment to VPC and select the VPC which was just created in the previous steps.

For Gateway Service, select VTEP and allow NC2 to automatically assign a Floating IP from the AWS VPC CIDR. This IP will be accessible from the on-prem environment and will be used as the anchor point for the L2E when configuring the CSR1000V in a later step

Note that a new VM (the gateway appliance) will automatically be deployed on the NC2 cluster by Prism Central.

Step 3: Deploy the Cisco CSR1000V in the on-premises VMware cluster

Deploy the Cisco appliance on the VMware cluster and select the first network interface to connect to the routable underlay network (IP connectivity to NC2) and the second and third interfaces to connect into the port group of the VLAN to be extended.

In this case VL420 is routable underlay network and VL423 the VLAN to be extended

Configure an IP address on the management network and make note of it as we will use it in a subsequent step. In this case we use “10.42.0.106” as the management IP address on VL420.

Step 4: Enable Promiscuous mode and Forged transmits on the vSwitch portgroup of the VLAN to be extended

In order to pass traffic from the on-premises network to the NC2 network it is necessary to enable Promiscuous mode and Forged transmits on the vSwitch port group on the VMware cluster. In this case we are using standard vSwitches.

Step 5: Register the CSR1000V routable IP address as a Remote Gateway in NC2

We need to create a representation of the on-premises CSR1000V appliance in NC2 so that we can refer to it when extending the network in a later step. This is essentially just a matter of adding in the IP address as a “Remote Gateway”.

In Prism Central, navigate to “Network and Security”, select “Connectivity” and “Create Gateway”. Select “Remote” and add the details for the on-prem Cisco appliance. Give it a name, select “VTEP” as the “Gateway Service” and add the IP address. Let the VxLAN port remain as “4789”.

Step 6: Configure the CSR1000V IP leg on the VLAN to be extended and set up VNI and other settings required to extend the network

In this step we do the configuration of the CSR1000V over SSH. To enable SSH you may need to to the following through the console for the virtual appliance first.

Enable SSH

en
conf t
username cisco password Password1!
line vty 0 4
login local
transport input ssh
end

Now when SSH is available, SSH to the appliance as the user “cisco” with password “Password1!” and complete the remaining configurations.

Configure interface in VLAN to be extended

Configure the 2nd interface to be a leg into the VLAN to be extended by giving it an IP address and enabling the interface

CSR1000V3#
CSR1000V3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
CSR1000V3(config)#
CSR1000V3(config)#int gi2
CSR1000V3(config-if)#
CSR1000V3(config-if)#ip address 10.42.3.2 255.255.255.0
CSR1000V3(config-if)#no shut

Configure NVE 1 and the VNI to be used + link with the NC2 gateway IP

For ingress-replication, use the Floating IP from the AWS VPC CIDR range which was assigned to the gateway appliance after deploying on NC2.

CSR1000V3#
CSR1000V3#conf term
Enter configuration commands, one per line.  End with CNTL/Z.
CSR1000V3(config)#
CSR1000V3(config)#
CSR1000V3(config)#int NVE 1
CSR1000V3(config-if)#no shutdown
CSR1000V3(config-if)#source-interface gigabitEthernet 1
CSR1000V3(config-if)#member vni 4300
CSR1000V3(config-if-nve-vni)#ingress-replication 10.70.177.192
CSR1000V3(config-if-nve-vni)#
CSR1000V3(config-if-nve-vni)#end
CSR1000V3(config-if)#end

Configure bridge domain and L2E via the 3rd interface (Gi3)

CSR1000V3#
CSR1000V3#conf t
Enter configuration commands, one per line. End with CNTL/Z.
CSR1000V3(config)#bridge-dom
CSR1000V3(config)#bridge-domain 12
CSR1000V3(config-bdomain)#member VNI 4300
CSR1000V3(config-bdomain)#member gigabitEthernet 3 service-instance 1
CSR1000V3(config-bdomain-efp)#end

CSR1000V3#conf t
Enter configuration commands, one per line. End with CNTL/Z.
CSR1000V3(config)#int
CSR1000V3(config)#interface giga
CSR1000V3(config)#interface gigabitEthernet 3
CSR1000V3(config-if)#no shut

CSR1000V3(config-if)#
CSR1000V3(config-if)#service instance 1 ethernet
CSR1000V3(config-if-srv)# encapsulation untagged
CSR1000V3(config-if-srv)#no shut
CSR1000V3(config-if-srv)#end
CSR1000V3#
CSR1000V3#

Configure a default route

We set the default route to go over the default gateway for the underlay we use to connect to AWS and NC2 on AWS

CSR1000V3#
CSR1000V3#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
CSR1000V3(config)#
CSR1000V3(config)#
CSR1000V3(config)#ip route 0.0.0.0 0.0.0.0 10.42.0.1
CSR1000V3(config)#
CSR1000V3(config)#
CSR1000V3(config)#end

Step 7: Extend the network from NC2 with the CSR1000V as the on-prem VTEP

In Prism Central, navigate to “Network and Security” and select “Subnets”. From here we will create the network extension.

Click the subnet to be extended and then select “Extend”

If we were extending a network between two Nutanix clusters we would select “Across Availability Zones” but in this case we extend from a pure VMware environment and a 3rd party (Cisco) appliance, so we select “To A Third-Party Data Center”.

Select the CSR1000V as the remote VTEP gateway and the VPC which contains the subnet we want to extend.

For “Local IP Address”, enter a free IP in the subnet to be extended. This will be the leg the Nutanix gateway appliance extends into that subnet.

Also set the VxLAN VNI we used when configuring the CSR1000V earlier. Adjust the MTU to ensure there is no packet fragmentation. The default is 1392 but this will vary depending on the connectivity provider between on-prem and the AWS cloud.

Configuration of the Layer 2 extension is now complete. In the following section we verify connectivity from on-prem to the cloud using the newly extended network.

Verifying connectivity

As a first step it’s good to ping the IP address in the extended network which is assigned to the Nutnix Gateway appliance. We can verify that the “Local IP Address” is configured for the gateway VM by navigating to “VM” in Prism Central and checking that “10.42.3.3” shows up as an IP for the gateway

Pinging this IP address from a VM in the on-premises VMware environment shows that it can reach the gateway appliance across the extended network without problems. The local VM has an IP of 10.42.3.99, in the same VLAN which has been extended. Latency is about 5ms across the S2S VPN + L2 extension.

As a next step I have migrated a VM using Move from the on-prem VMware environment to NC2. After migration it was assigned an IP of “10.42.3.106” as per the screenshot below

Pinging this VM from on-prem also works just fine

Conclusion

That concludes the walkthrough of configuring L2 extension from on-premises VMware with Cisco CSR1000V to Nutanix Cloud Clusters (NC2). Hopefully this was helpful.

For reference, please refer to the below links to Nutanix and Cisco pages about L2 extension