Accessing Resources in other VPCs
Depending upon the size of our organization, we may have more than one or two VPCs, or have a VPC deployed in more than one region. There are many cases for multiple VPCs:
- segregates applications based upon some criteria, like function, security level, data classification, etc.;
- use different VPCs for the various business units;
- service deployment in multiple regions; or
- for different stages in the development process.
If you have worked with AWS for any length of time, you know that every AWS account comes with a default VPC in every supported region. Every default VPC uses the same CIDR block, 172.31.0.0/16. This is important for later in the article.
However, we may find the CIDR block doesn’t fit our needs, or we want to implement additional VPCs for specific reasons, some of which are mentioned above. (My article, Creating a Custom VPC in AWS Using CloudFormation, dives into the network components needed when creating a custom VPC and illustrates the CloudFormation elements needed to accomplish the task.
Regardless of the specific reason for multiple VPCs, sometimes we need to connect them together so they can exchange information. There are two methods for achieving this connectivity in AWS. The first is VPC Peering, and the second is the Transit Gateway. This article is going to discuss using VPC peering.
What is VPC Peering?
Peering allows traffic from VPC-A to be sent to VPC-B, based upon the network address of the specific resource. Basically, VPC peering is just a fancy way of saying we are establishing a route between two or more VPCs.
Here is a two VPC peering example.
Once configured, the resources in both VPCs will be able to access and share resources in the other VPC.
It starts to get a little more complicated when there are more than two VPCs in the peering arrangement.
VPC peering does not support transitive peering. Transitive peering means if VPC-A is connected to VPC-B and VPC-B is connected to VPC-C, VPC-A resources can communicate with resources in VPC-B, and communicate with resources in VPC-C, as traffic is routed from VPC-A to VPC-C via VPC-B.
This networking arrangement is not supported in the VPC peering model. Consequently, peering using a mesh topology is required to create peering between multiple VPCs.
The requirement for a mesh topology makes VPC peering with large numbers of VPCs difficult to set up and manage over time. As we can see, our four VPCs require a total of 6 peering connections to establish the mesh.
Even though it is referred to a mesh, each peering connection is still only point to point. If there is a failure between one of the peering connections, the traffic cannot be routed across a different connection.
As the need to include more and more VPC peers grows, so does the needs for improved reliability. Transit Gateway is the solution when there are large numbers of peers or when we just cannot tolerate a failure in any of the peering connections.
VPC Peering is almost impossible to set up in AWS using the default VPC, as each default VPC uses the exact same address space, 172.31.0.0/16. You can augment the address space associated with your VPC, but then you have to remember not to use the default subsets. Therefore it is best to create a custom VPC because chances are the default VPC doesn’t exactly meet your needs in some way for your peering arrangement.
If you have taken any AWS networking training, you also know there is a VPC router which we do not have the ability to interact with directly, but which is responsible for all of the traffic routing in and out of our VPC. I only mention it here because, as you would expect, it has a role to play in peering.
Configuring your VPC peering arrangement requires several steps:
- Have two VPCs which do not have overlapping CIDR blocks.
- Create a peering connection request between the two VPCs.
- The owner of the remote VPC approves the peering request.
- Configure routing on both VPCs.
- Adjust affected security groups.
Let’s walk through the steps involved using the AWS Console and discuss the specifics associated with each of these steps. Afterward, we can review the specific AWS CLI commands needed to accomplish the same steps either using the CLI or the AWS-Shell.
Peering within the Same AWS Account
As we saw earlier, we need two VPCs to establish a peering arrangement. If we don’t have two VPCs, we need to create at least one, which I am not going to cover here. Here is an illustration of our two VPCs and their associated network configuration.
If we look in the AWS Console, we can see the two VPCs are in our account on the VPC Dashboard.
The VPC where the request is made from is known as the requester VPC and the peer is called the accepter VPC. This is because one VPC makes the request to the other VPC, which must accept the request. When both the requester and accepter are in the same AWS account, and the VPC peering request is created using CloudFormation, CloudFormation will attempt to accept the request automatically.
To create the peering request in the AWS Console, navigate to the VPC Dashboard, and click on VPC Peering Connections. Unless you already have a peering connection, the Console will show a message indicating there are no peering connections. To create a peering request, click on the Create Peering Connection button. The Create Peering Connection form is displayed. You are prompted to enter a name for the connection, the requester, and accepter VPCs, along with the account and region the accepter VPC is located in.
In this example, I am creating a peering connection between two VPCs in my account, and in the same region. If you select to peer with a VPC in a different account or region, the form will expand to provide those fields.
Complete these additional fields as needed. If you choose to peer with a VPC in a different region or in a different account, you will need to know the VPC Identifier for the accepter VPC; the AWS Console will not give you a list you can select from. We will look at peering between accounts in more detail later in this article.
After submitting the peering request, you will see a message indicating the peering request has been submitted. When we look at the VPC Dashboard, we can indeed see the peering connection request. Notice, that even though the request was for our own account, it’s status is pending acceptance. Before we can continue with configuring the peering connection, we have to accept the request. Select the peering connection to accept (or reject), and choose the appropriate option under the Actions menu.
A new dialog is displayed where you confirm the acceptance of the peering request. Once accepted, you are prompted to update the routing tables to facilitate the routing of traffic between the two peered VPCs.
Peering between AWS Accounts
We have seen configuring a peering connection in the same account. But, how is going cross-account different? I have several accounts built inside an AWS Organization. Both accounts are part of that organization; otherwise, they are separate accounts, with their own VPCs. Let’s create a VPC peering connection across accounts and regions.
VPC A, which is in Account A, will request a peering connection to VPC B in Account B. Both VPCs are in us-east-1.
Using the same form as before, we complete the Peering Connection Request form in the AWS Console.
Like before, we receive notification the peering request has been created, and the account owner for the peered VPC must accept the request. We can see the status of the peering request in the AWS Console.
We can see two peering connections, one active and one pending acceptance. We can see the account numbers are different for the request which is pending acceptance. This is where there is a breakdown in my opinion. As the owner of the accepter account as well, I received no notification of the peering request.
This way, the owners of both accounts would have to talk to each other to establish the agreement to peer and verify no constraints will impact the arrangement. However, if I am the owner of both accounts, and the peering request is being done by someone with the authority to create a peering request (remember the principle of least privilege!), then why can’t I just accept the peering request? Again, it basically is a mechanism to ensure you understand who is connecting to your VPC, and no constraints have been violated.
It may be possible to create an event using Amazon EventBridge to detect the VPC peering request and process it using AWS Lambda. However, if peering is so prominent, then it is more likely you will need Transit Gateway.
Like we did with the single account peering connection, the accepter either accepts or rejects the request. If the request is accepted, then routing must be configured on both ends of the connection. If the connection is rejected, a confirmation dialog is displayed, and the connection shows as rejected in the console.
Once the peering request has been rejected, a new request must be created and approved.
Updating the Route Tables
Even though we have established a connection between the VPC peers, no data will traverse the connection until routing has been set up. This behavior is just like what we would see had these two VPCs been physical buildings and we installed a point to point leased line (like a 45 Mbps fiber connection).
Configuring the route tables is something you should know how to do already, and is handled in the same manner as you would with an internet or NAT gateway. The difference is the target. When we add a route for the VPC peering connection, we go to the VPC Dashboard, choose Route Tables, and then select the route table for the requester VPC. Now it is time to add the route.
Selecting Edit Routes allows us to enter the CIDR range for the target VPC, just as we would do other services. In this case, we choose Peering Connection as the target, which displays the available peering connections.
Select the desired peering connection, and click on the save routes button. Our route to the other VPC has been created on the requester side of the peering connection. The same change must be made on the accepter VPC’s route table for information to get back to the accepter.
We now have a functioning VPC connection. This does not mean however resources in the accepter VPC can communicate with resources in the requester VPC. This depends upon how stringent the rules are for the VPC Network Access Control List (NACL) on the receiving side of the peering connection and the security group configuration for the specific resources.
Let’s assume for a moment that the accepter VPC has an EC2 instance in its VPC, and the requester VPC also has an EC2 instance. Both EC2 instances have a security group rule only allowing traffic from the respective VPC CIDR block range.
Because a peering connection is not a NAT connection, but a route, the EC2 in the requester VPC will not be able to establish an SSH connection to the EC2 in the accepter VPC (assuming SSH is allowed to the instance). To provide the ability for the requester EC2 to login to the accepter EC2 using SSH, we have to add or modify the security group to allow this connectivity.
Updating Security Groups
Creating and modifying security groups should also not be a new topic for you. Here is our current configuration:
- The requester EC2 allows SSH from a single internet IP (it has a public IP) and any IP in the VPC CIDR block.
- The requester EC2 allows outbound on all ports.
- The accepter EC2 allows SSH only from IP addresses in the VPC CIDR range (it does not have a public IP).
Because the security group on the accepter-EC2 is configured to only accept connections from the VPC CIDR block, we should expect our ssh connection to fail.
$ ssh -i key.pem firstname.lastname@example.org
ssh: connect to host 192.168.1.133 port 22: Connection timed out
This is the expected result given the state of the security group configuration on the accepter-EC2. To allow this connectivity, we need to add the requester VPC CIDR range to the security group used by accepter-EC2.
Now, our security group has two inbound SSH rules. One allowing inbound ssh from the VPC CIDR, and one for the VPC Peer CIDR. Let’s try our ssh connection again.
$ ssh -i key.pem email@example.com
__| __|_ )
_| ( / Amazon Linux 2 AMI
We now have connectivity from the requester-EC2 to the accepter-EC2. We will have to change all of the affected security groups to allow for the peer network to access the desired resources.
I more or less covered this process while verifying the security group. The fastest way to validate the peering connection and routing is to attempt an ssh connection between the two peer networks. Once you have validated ssh, it is time to move on to the other VPC resources you wish to share.
As we have proven the VPC peering connection is operational, any resource protected by a security group will need modification to allow for the peer VPC to access the resource.
Using CloudFormation for your Peering Request
Using CloudFormation to create your peering requests is, for the most part, the same as using the AWS Console. There is one exception: if you are creating a peering connection within the same account as where the CloudFormation stack is being executed, CloudFormation will attempt to perform the acceptance of the peering request. If the request can be accepted, then the stack will complete successfully.
Because routing can’t be configured until the peering connection is created and accepted, it is advisable to make these two tasks separate events in CloudFormation, especially since you need to update the route tables in both VPCs. I have created two CloudFormation stacks: one which takes all the parameters requested to create the peering request and another which takes the VPC and route table identifiers and performs the routing updates.
Using the AWS CLI
It is also possible to create the peering connection using the CLI (or aws-shell) with the command
aws ec2 create-vpc-peering-connection and supplying the same information as in the Console request. Like in the console, this is a two-step process. The acceptor can accept or reject the peering request in the Console, or by using the
aws ec2 accept-vpc-peering-connection or the
aws ec2 reject-vpc-peering-connection commands as appropriate.
When Things Get Complicated
When you start to have a lot of peering connections between multiple VPCs, in multiple regions, across multiple AWS accounts, managing the connections starts to get rather difficult. When it starts to get difficult, it is time to start looking at moving the existing peering connections to Transit Gateway and managing the connections there.
The ability to expand connectivity to use resources in another VPC, whether it belongs to you or not, can be crucial to the operation of your services. It can assist in reducing costs, such as providing access to a Relational Database Service (RDS) instance in a different VPC to access the same information instead of creating a second RDS instance and incurring twice the cost, along with data replication charges (the latency of the data transfer notwithstanding).
Sometimes we need to allow resources in different VPCs to exchange data based upon the architecture of the service. Regardless of your specific need, using VPC peering is a great way of extending your VPC design.
About the Author
Chris is a highly-skilled Information Technology, AWS Cloud, Training and Security Professional bringing cloud, security, training, and process engineering leadership to simplify and deliver high-quality products. He is the co-author of seven books and author of more than 70 articles and book chapters in technical, management, and information security publications. His extensive technology, information security, and training experience make him a key resource who can help companies through technical challenges.
This article is Copyright © 2020, Chris Hare.
#AWS #VPC #Networking