Scenario:

You have created kubernetes cluster using kops. Using private topology the nodes are created in private subnets. You created bastion server in public subnet to allow users to access the cluster. kops will create bastion server and a bastion-elb (elastic load balancer) in front to fit. Now you would like users to access the bastion server with their own ssh key pair (private & public key).

Problem:

This requires getting public keys of the users and adding them to authorized_keys on bastion server. There are many reasons why this is not very effective method (especially from maintenance point).

Solution:

You can use Amazon EC2 Instance Connect which provides a simple and secure way to connect to your instances using Secure Shell (SSH). Benefits of using EC2-instance-connect:

  1. With EC2 Instance Connect, you use AWS Identity and Access Management (IAM) policies and principals to control SSH access to your instances, removing the need to share and manage SSH keys.
  2. All connection requests using EC2 Instance Connect are logged to AWS CloudTrail so that you can audit connection requests.
  3. User provided ssh public key is retained for 60 seconds only.

How does it work

When you connect to an instance using EC2 Instance Connect, the Instance Connect API pushes a one-time-use SSH public key to the instance metadata where it remains for 60 seconds. An IAM policy attached to your IAM user authorizes your IAM user to push the public key to the instance metadata. The SSH daemon uses AuthorizedKeysCommand and AuthorizedKeysCommandUser, which are configured when Instance Connect is installed, to look up the public key from the instance metadata for authentication, and connects you to the instance.

Limitation:

Only following Linux distributions are supported:

  • Amazon Linux 2 (any version)
  • Ubuntu 16.04 or later

Steps to configure EC2-instance-connect

-> Install Ec2-instance-connect on Linux Bastions server

sudo yum install ec2-instance-connect

-> To  grant an IAM user permission fro EC2 instance connect, create this policy document

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "ec2-instance-connect:SendSSHPublicKey",
        "Resource": [
            "arn:aws:ec2:region:account-id:instance/i-1234567890abcdef0"
        ],
        "Condition": {
            "StringEquals": {
                "ec2:osuser": "${aws:username}"
            }
        }
      },
      {
        "Effect": "Allow",
        "Action": "ec2:DescribeInstances",
        "Resource": "*"
      }
    ]
}

-> In the above policy replace your bastion server instance-id. When user submits the public key for the bastion server, IAM will verify that the user has this policy with the right instance and the correct “ec2:osuser” AWS user provided. It will make sure that the user is able to send the public key to only his user account.

-> Now from your laptop using AWS command you can send public key to bastion server and then use PuTTY or ssh client to connect to the bastion server.

Note: you should have AWS CLI version 1.18 or later.

-> Generate new SSH private and public keys, my_rsa_key and my_rsa_key.pub, using the following command: $ ssh-keygen -t rsa -f my_rsa_key

-> Push your SSH public key to the instance.

$ aws ec2-instance-connect send-ssh-public-key \
--instance-id <replace your Bastion server instance ID here> \ --availability-zone us-west-2b \ --instance-os-user <replace your AWS user here>  \ --ssh-public-key file://my_rsa_key.pub

Note: You should create user in Bastion server with same name as AWS user.

-> Use PuTTy or ssh client and connect to the Bastion server using private key.

You can access the audit logs in Audit trail for event source “SendSSHPublicKey”.

You can see these links for more details –

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Connect-using-EC2-Instance-Connect.html

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/monitor-with-cloudtrail.html#ec2-instance-connect-cloudtrail

Benefits of this method:

  • It may appear that this process is already explained by AWS. Actually it is little different. Here we are not using linux os user ‘ec2-user’. Every individual user will have their own login to bastion.
  • The user in bastion is same as user in AWS IAM that way the user can upload the public key only to his/her own user account.
  • The bastion server need not to be opened to public. It will allow traffic from bastion-elb which will be open for office network traffic only.
  • Deleting/disabling the user from AWS IAM will stop access for the user in bastion server. No more headaches for removing the user from bastion as well.
  • Using ‘ec2-user’ account for all users is not safe but with this method user data is private to himself.