AWS Permission denied (publickey,gssapi-keyex,gssapi-with-mic) EC2. No access to console

I hope to get some help over here. I know this issue is all over the internet, but I think I have a different kind of situation.

My setup is the following:

  • Amazon Linux EC2
  • The instance has its VPC
  • The VPC has its internet gateway
  • To enable access I created a TCP Target Group, which is assigned to an Elastic Load Balancer with a 22 port listener.

I had to create the load balancer because it is an abandoned infrastructure by a previous engineer and I had to know how to regain access to the EC2 without affecting the load balancer.

In order to do that, I created a new Load Balancer to gain SSH access.

Once I achieved to get connected to the instance through the load balancer DNS, I received this error, having permission denied.

These are the logs using ssh verbose:

soto@LAPTOP-VT374I5C:~/.ssh$ ssh -vvv -i soto.pem  ec2-user@ipfs-api-6fffb2e30dd6fff6.elb.us-east-1.amazonaws.com -p 22
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/soto/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "ipfs-api-6fffb2e30dd6fff6.elb.us-east-1.amazonaws.com" port 22
debug2: ssh_connect_direct
debug1: Connecting to ipfs-api-6fffb2e30dd6fff6.elb.us-east-1.amazonaws.com [23.20.173.43] port 22.
debug1: Connection established.
debug1: identity file soto.pem type -1
debug1: identity file soto.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to ipfs-api-6fffb2e30dd6fff6.elb.us-east-1.amazonaws.com:22 as 'ec2-user'
debug3: hostkeys_foreach: reading file "/home/soto/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/soto/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from ipfs-api-6fffb2e30dd6fff6.elb.us-east-1.amazonaws.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:Q7K81dtMhKJBZUTLXi5ixEAXRIifIlNqy2O1F2Ckwcs
debug3: hostkeys_foreach: reading file "/home/soto/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/soto/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from ipfs-api-6fffb2e30dd6fff6.elb.us-east-1.amazonaws.com
debug3: hostkeys_foreach: reading file "/home/soto/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/soto/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from 23.20.173.43
debug1: Host 'ipfs-api-6fffb2e30dd6fff6.elb.us-east-1.amazonaws.com' is known and matches the ECDSA host key.
debug1: Found key in /home/soto/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: soto.pem  explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)


debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)


debug2: we did not send a packet, disable method
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: soto.pem
Load key "soto.pem": Permission denied
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
ec2-user@ipfs-api-6fffb2e30dd6fff6.elb.us-east-1.amazonaws.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Things I have already tried:

  • Add a username to the ssh connection, which in this case, since it is an Amazon Linux 2, my username us ec2-user.

  • Modified the sshd_config file, adding the line PasswordAuthentication yes and restarting the sshd service.

  • Tried putty in windows instead of ubuntu, which gave me this error:

enter image description here

So I am really stuck, I am starting to think that something is wrong with my AWS configuration.

Also, I followed the permissions instructions regarding chmod 400 to my pem file, so permissions are enough to send the pem.

Thanks a lot in advance for taking a look at this complicated situation, I am more than happy to give more and explained information if need.

Thanks a lot.

PD: I can't access authorized_keys on the server-side, since I only have the AWS console as the primary and only access.

1 answer

  • answered 2020-10-16 05:24 James Shapiro

    You should not be using a load balancer as a bastion host. I would suggest launching a bastion host into a public subnet in your VPC, SSH-ing into the bastion host, then using the bastion host to SSH into your instance. You'll need to configure your security groups to allow this access.