One Week In: How likely are you to recover released EIPs?

/One Week In: How likely are you to recover released EIPs?

One week in

After getting in touch with AWS support and getting the usual limit of 5 EIPs increased. We were ready to allocate some addresses. Now I took a look in the console and the CLI and couldn’t see an obvious way to recover an address. Let’s take a look at the original article. There it is. “Customers can recover their EIPs via CLI by using the allocate-address command and specifying the IP address using the –address parameter.”

Right now it’s a CLI(API) only option and you need the –address option. Let’s see if I can see that with auto-completion:

$ aws ec2 allocate-address --
--ca-bundle --color --endpoint-url --no-sign-request --query
--cli-connect-timeout --debug --generate-cli-skeleton --no-verify-ssl --region
--cli-input-json --domain --no-dry-run --output --version
--cli-read-timeout --dry-run --no-paginate --profile

Doesn’t look like it. On my Mac I use brew to manage many of my tools so a quick brew upgrade awscli later:

$ aws ec2 allocate-address --
--address                --cli-read-timeout       --dry-run                --no-paginate            --profile
--ca-bundle              --color                  --endpoint-url           --no-sign-request        --query
--cli-connect-timeout    --debug                  --generate-cli-skeleton  --no-verify-ssl          --region
--cli-input-json         --domain                 --no-dry-run             --output                 --version

That looks better.

Just for fun, lets find the commit for this.

For anyone who is familiar with the CLI, it is build using python and so relies on boto and specifically botocore.

There are a few other things going on in this commit, but if you look through you can see changes to AllocateAddress sections, and specifically the addition of:

+        "Address":{
+          "shape":"String",
+          "documentation":"

[EC2-VPC] The Elastic IP address to recover.

"
+        },

So with all the tools at hand; lets allocate our 21 addresses:

for ((i=0; i<21; i++)); do aws ec2 allocate-address; done

This was completed around 1700 on 28 August 2017.

As per set out in our method, we will now wait about (I’m only being semi-scientific here) one day.

Getting a listing of all our IP address can be achieved with AWS CLI and JQ. JQ is a neat tool that helps you manipulate JSON.

aws ec2 describe-addresses | jq -r '.Addresses[] | if has("AssociationId") then empty else .PublicIp end'

I’m doing this, as this account has some test hosts with EIP associations already, and I don’t want to disturb those people.

Our 24 hours is up.

Let’s release all our addresses:

for allocid in $(aws ec2 describe-addresses | jq -r '.Addresses[] | if has("AssociationId") then empty else .AllocationId end')
do
  aws ec2 release-address --allocation-id "${allocid}"
done

Now we need to wait one hour and try and recover our first IPs.

One Hour

An hour has past, so let’s try and recover our first 3 address.

We now need to add the –address option on to the allocate-address sub-command:

$ aws ec2 allocate-address --address 13.210.63.105
{
    "PublicIp": "13.210.63.105",
    "Domain": "vpc",
    "AllocationId": "eipalloc-c6a166a2"
}
$ aws ec2 allocate-address --address 13.54.121.67
{
    "PublicIp": "13.54.121.67",
    "Domain": "vpc",
    "AllocationId": "eipalloc-c7a166a3"
}
2:~$ aws ec2 allocate-address --address 54.66.204.43
{
    "PublicIp": "54.66.204.43",
    "Domain": "vpc",
    "AllocationId": "eipalloc-aca067c8"
}

Wow, look at that. All 3 were able to be rescued. Looks like we are off to a good start.
Now let’s release these 3, and try again in 12 hours.

12 Hours

Now we are about 12 hours after we rescued our first 3 addresses. Let’s try those original 3 plus the next 3 and see how we go.

$ for ip in 13.210.63.105 13.54.121.67 54.66.204.43 13.54.114.110 52.64.88.154 13.54.253.167
do
  aws ec2 allocate-address --address "${ip}"
done
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.210.63.105' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.54.121.67' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '54.66.204.43' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.54.114.110' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '52.64.88.154' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.54.253.167' does not exist or is not available.

Oh no! All IP addresses are now unable to be rescued.

In another 12 hours we’ll try again.

24 Hours

Well here we are at hour 24 (or there about). Let’s try and recovery our original 1 hour, 12 hour and the 24 hour IP addresses.

$ for ip in 13.210.63.105 13.54.121.67 54.66.204.43 13.54.114.110 52.64.88.154 13.54.253.167 52.62.84.127 54.252.170.82 52.63.208.57
do
  aws ec2 allocate-address --address "${ip}"
done
 
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.210.63.105' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.54.121.67' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '54.66.204.43' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.54.114.110' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '52.64.88.154' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.54.253.167' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '52.62.84.127' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '54.252.170.82' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '52.63.208.57' does not exist or is not available.

Oh no! Again we are unable to recover any IP addresses. While it’s too early to draw any conclusions, I think the advice given in the AWS blog post is very true: “The sooner customers try to recover their EIPs, the better are the chances of recovering them.”

We won’t stop though. Let’s keep going and see how we get on with our 48 hour addresses.

48 Hours

$ for ip in 13.210.63.105 13.54.121.67 54.66.204.43 13.54.114.110 52.64.88.154 13.54.253.167 52.62.84.127 54.252.170.82 52.63.208.57 54.66.172.149 54.66.163.238 54.153.216.37
> do
> aws ec2 allocate-address --address "${ip}"
> done
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.210.63.105' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.54.121.67' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '54.66.204.43' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.54.114.110' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '52.64.88.154' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '13.54.253.167' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '52.62.84.127' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '54.252.170.82' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '52.63.208.57' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '54.66.172.149' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '54.66.163.238' does not exist or is not available.
 
An error occurred (InvalidAddress.NotFound) when calling the AllocateAddress operation: The Elastic IP address '54.153.216.37' does not exist or is not available.

Oh well, I think you pretty much have to catch those IP addresses really quickly.

Raw data

EIP 1 Hour 12 Hours 24 Hours 48 Hours 1 Week 2 Weeks 1 Month Time Assignment
13.210.63.105 (successful) (failed) (failed) (failed) 1 hour
13.54.121.67 (successful) (failed) (failed) (failed) 1 hour
54.66.204.43 (successful) (failed) (failed) (failed) 1 hour
13.54.114.110 (failed) (failed) (failed) 12 hours
52.64.88.154 (failed) (failed) (failed) 12 hours
13.54.253.167 (failed) (failed) (failed) 12 hours
52.62.84.127 (failed) (failed) 24 hours
54.252.170.82 (failed) (failed) 24 hours
52.63.208.57 (failed) (failed) 24 hours
54.66.172.149 (failed) 48 hours
54.66.163.238 (failed) 48 hours
54.153.216.37 (failed) 48 hours
52.62.57.147 1 week
13.55.140.79 1 week
54.66.172.8 1 week
52.65.29.139 2 weeks
13.55.223.46 2 weeks
54.153.133.124 2 weeks
52.63.129.155 1 month
52.64.128.86 1 month
52.65.211.221 1 month

Part 2 of our Testing the new EIP Recovery technical series.

By | 2017-09-04T04:56:35+00:00 September 1st, 2017|AWS, Technical Blog|