Problem:

You want to run awscli commands from within Lambda function but the Lambda execution environment doesn’t provide pre-installed AWS CLI and it can’t be installed using pip package. What options do we have?

 

Solution:

You can create a Lambda Layer that contains all the necessary dependencies and make it available to the function. First, install AWS CLI in a local virtual environment, package AWS CLI and all its dependencies to a zip file and create a Lambda Layer.

You will need Python3, pip, zip and virtualenv on your machine.

Note: These instructions are tested on WSL on Windows.

Verify whether you already have these on your machine, else install them-

/mnt/c/tmp/awscli$ python --version
Python 3.6.9
/mnt/c/tmp/awscli$ pip --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
/mnt/c/tmp/awscli$ virtualenv --version
15.1.0

 

–> Open the WSL terminal and run the following commands in the terminal.

mkdir awscli

cd awscli

export PYTHON_VERSION=`python3 -c 'import sys; version=sys.version_info[:3]; print("{0}.{1}".format(*version))'`

export VIRTUAL_ENV_DIR="awscli-venv"

export LAMBDA_LAYER_DIR="lambda-layer"

export ZIP_FILE_NAME="lambda-layer.zip"

 

–> Run following commands to create an isolated virtual environment using vitualenv and activate it-

mkdir ${VIRTUAL_ENV_DIR}

virtualenv ${VIRTUAL_ENV_DIR} -p $(which python3)

cd ${VIRTUAL_ENV_DIR}/bin/

source activate

Now your prompt will look like below (it will have awscli-venv at the beginning)-

(awscli-venv) /mnt/c/tmp/awscli/awscli-venv/bin$

 

—> Now install AWS CLI and its dependencies

pip3 install awscli

 

–> The AWS CLI executable is the file named aws and its first line provides the path to the Python interpreter. In a virtual environment, it will have a path local to the environment: #!/PATH_TO_WORKING_DIR/awscli/awscli-venv/bin/python3

This path needs to be replaced with the path to the Python interpreter in the Lambda execution environment, which is #!/var/lang/bin/python.

You can open aws in a text editor and replace it manually or run this command to replace it-

sed -i "1s/.*/\#\!\/var\/lang\/bin\/python/" aws

 

–> Now deactivate virtualenv

(awscli-venv) /mnt/c/tmp/awscli/awscli-venv/bin$ deactivate

 

–> Run following commands to package AWS CLI and its dependencies in a zip file-

cd ../..

mkdir ${LAMBDA_LAYER_DIR}

cd ${LAMBDA_LAYER_DIR}

cp ../${VIRTUAL_ENV_DIR}/bin/aws .

cp -r ../${VIRTUAL_ENV_DIR}/lib/python${PYTHON_VERSION}/site-packages/* .

zip -r ../${ZIP_FILE_NAME} *

 

–> As package zip file is created, delete the temp directories which were created-

cd ..

rm -r ${VIRTUAL_ENV_DIR}

rm -r ${LAMBDA_LAYER_DIR}

 

–> Now go to the Lambda console, add Lambda layer following the wizard to create a layer. Upload lambda-layer.zip and select python3.6 as runtime-

 

–> Create a Lambda function using Python 3.6 which is used for packaging AWS CLI. Once the function is created, in Designer, click on Layers, click Add layer and select the custom layer created-

 

–> Paste the following code and execute the function by clicking the Test button

import subprocess
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def run_command(command):
    command_list = command.split(' ')
    #print(command_list)
    try:
        logger.info("Running shell command: \"{}\"".format(command))
        result = subprocess.run(command_list, stderr=subprocess.STDOUT, stdout=subprocess.PIPE);
        #print(result)
        logger.info("Command output:\n---\n{}\n---".format(result.stdout.decode('UTF-8')))
    except Exception as e:
        logger.error("Exception: {}".format(e))
        return False

    return result

def lambda_handler(event, context):
    
    run_command('/opt/aws --version')
    run_command('/opt/aws s3 ls')

Note: You have to provide the /opt/aws path to invoke the cli.

Information from following post used to setup this lambda layer –