Update NGINX Upstreams with an AWS Lambda Function

Using an AWS Lambda function to update a Route53 DNS record periodically to allow for dynamic updates to NGINX's upstream server list.

Davy Hua

3 minute read

Problem Statement: A subset of our microservices uses the gRPC http/2 protocol. The problem was due to AWS’ lack of direct end-to-end support in all the various types of load balancers available. Since our stack runs in DC/OS, we utilize Marathon-LB to provide load balancing for the service endpoints.

Due to this limitation, we have to bypass AWS ELB by using NGINX’s new gRPC support to provide reverse proxy into to our Marathon-LB instances, which leaves wide open an issue where one or more of the upstream servers’s IP could change at any given time by our auto scaling group. This would then require manual updates to NGINX conf file each time that happens.

I wanted to automate this such that the upstream server can reference a DNS name with multiple A records. These A records would be updated each time any of the Marathon-LB node changes.

Possible Solutions:

  1. Put a script on cron in ALL of the NGINX instances to read the node changes and grab the IPs, update the upstream conf block dynamically, then reload NGINX. Very hacky.
  2. Run a container which accomplishes step #1 but that still leaves the issue of needing to get the updated conf to the NGINX instances. We can run the container natively on the NGINX instances, but that’s just more configuration and overhead. Again, very hacky.
  3. Use a Lambda function to perform this task in detecting the IP changes to these instances and updating the route53 DNS record. NGINX would just point to this DNS record and no subsequent updates to its conf needed.

There are probably more options, but I feel going Serverless was the best choice in this scenario.

Implementation: How I got it done.

  • I created an IAM Role with the following policy:
 {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ec2:DescribeInstances",
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "route53:GetChange",
                "route53:ChangeResourceRecordSets"
            ],
            "Resource": [
                "arn:aws:route53:::hostedzone/*",
                "arn:aws:route53:::change/*"
            ]
        }
    ]
}

This role grants the Lambda function the ability to describe the instances based on the tags, ability to update route53 record on all hosted zones. You can add more granular control to the hostedzone by specifying a subset of the zones instead.

import boto3

def lambda_handler(event,context):
    message = ''

    stack = event['stack_name']
    if (stack == "<my_stack_1>"):
        dns_name = '<my_full_dns_name_1>'
        host_id = '<my_hostedzone_id_1>'
        message = list_nodes_by_tag_value("<my_keypair1>",stack)        
    elif (stack == "<my_stack_2>"):
        dns_name = '<my_full_dns_name_2>'
        host_id = '<my_hostedzone_id_2'
        message = list_nodes_by_tag_value("<my_keypair1>",stack)        
    else:
        dns_name = ''
        host_id = ''
        message = ''

    nginx = boto3.client('route53')
    response = nginx.change_resource_record_sets(
        HostedZoneId=host_id,
        ChangeBatch={
            'Changes': [
                {
                    'Action': 'UPSERT',
                    'ResourceRecordSet': {
                        'Name': dns_name,
                        'Type': 'A',
                        'TTL': 60,
                        'ResourceRecords': [
                            // three IPs, add more if needed
                            {
                                'Value': message[0]
                            },
                            {
                                'Value': message[1]
                            },
                            {
                                'Value': message[2]
                            }
                        ]
                    }
                }
            ]
        }
    )
    return {
        'message': message
    }

def my_route53():
    client = boto3.client('route53')
    response = client.list_hosted_zones()
    print(response)

def list_nodes_by_tag_value(mykeypair, myvalue):
 
    ec2client = boto3.client('ec2')
 
    response = ec2client.describe_instances(
        Filters=[
            {
                'Name': 'tag:'+mykeypair,
                'Values': [myvalue]
            },
            {
                'Name': 'instance-state-name',
                'Values': ['running']
            }
        ]
    )
    instancelist = []
    for reservation in (response["Reservations"]):
        for instance in reservation["Instances"]:
            instancelist.append(instance["PublicIpAddress"])
    return instancelist
  • Create a CloudWatch rule to invoke this Lambda function every 5 minutes. Call the function with the following JSON Constant: {"stack_name":"<my_stack_name>"}

  • Bonus points: We can update the Lambda function to use an internal DNS name with PrivateIpAddress A records if we enable VPC peering.

comments powered by Disqus