Otdfbt

I need to login to a host via ssh, get a keyfile, close the connection and then use the key to login without password on several machines.

The underlying problem is that I need to perform tests on a couple of servers via ssh. I have already successfully written code using paramiko that does that with regular login. But I just cannot figure out how ssh works.

This is what I have done to set it all up. I create keys on the client and copy them to a server:

[me@localhost ~]$ ssh 192.168.122.33
me@192.168.122.33's password: 
Last login: Mon Apr 29 09:26:15 2019 from gateway
[me@test33 ~]$ exit
logout
Connection to 192.168.122.33 closed.
[me@localhost ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/me/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/me/.ssh/id_rsa.
Your public key has been saved in /home/me/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:CV4HoJfwABDD15AoGDtZdTgAlEWZJfP8cFW+18lk9Gg me@localhost.localdomain
The key's randomart image is:
+---[RSA 2048]----+
 Removed
+----[SHA256]-----+

[me@localhost ~]$ ssh-copy-id 192.168.122.33
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 2 key(s) remain to be installed -- if you are prompted now it is to install the new keys
me@192.168.122.33's password: 

Number of key(s) added: 2

Now try logging into the machine, with: "ssh '192.168.122.33'"
and check to make sure that only the key(s) you wanted were added.

[me@localhost ~]$ ssh 192.168.122.33
Last login: Mon Apr 29 09:27:26 2019 from gateway
[me@test33 ~]$ 

Works as expected so far. Now I can login without being prompted for password. But now there are things I don't understand. I started with deleting the files id_rsa and id_rsa.pub from the client:

[me@test33 ~]$ exit
logout
Connection to 192.168.122.33 closed.
[me@localhost ~]$ cd .ssh/
[me@localhost .ssh]$ ls
id_rsa id_rsa.pub known_hosts
[me@localhost .ssh]$ mkdir ../backup
[me@localhost .ssh]$ mv id* ../backup

Now, I expected to NOT being able to login without a password, but that's obviously wrong.

[me@localhost .ssh]$ ssh 192.168.122.33
Last login: Mon Apr 29 09:29:20 2019 from gateway
[me@test33 ~]$ 

So I suspected that the keys are cached somewhere, so I tried to restart the ssh daemon.

[me@localhost .ssh]$ sudo service sshd restart
[sudo] password for me: 
Redirecting to /bin/systemctl restart sshd.service
[me@localhost .ssh]$ ssh 192.168.122.33
Last login: Mon Apr 29 09:38:23 2019 from gateway
[me@test33 ~]$ 

Did not work either. Sorry for being unclear, but there is probably something conceptual I just don't understand. What I want to have done right now is a procedure to configure one server in a way so that I can login to it with username and password and fetch a keyfile. Then I want to copy that configuration to a number of other servers. Lastly, I want to use my client to fetch the keyfile from one of the servers and use it to login and run various commands on the servers. For security reasons I also need to store the keyfile in ram, and probably encrypt it somehow, but that's a later issue.

I will write everything in Python later, but I need to solve this to start coding. This pseudopython describes what I'm looking for:

hosts=['192.168.122.' + str(i) for i in range(1,10)]
key=fetch_key(hosts[0]) # Or any of the servers.
for host in hosts: # This part is already done. I just need to get the key.
 execute_test(host, key)

The servers are running CentOS 7

I posted this at SO first, but it was OT, but by request from a commenter, here is the output from ssh -vv

[me@localhost ~]$ ssh -vv 192.168.122.33
OpenSSH_7.8p1, OpenSSL 1.1.1 FIPS 11 Sep 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.122.33 is address
debug2: ssh_connect_direct
debug1: Connecting to 192.168.122.33 [192.168.122.33] port 22.
debug1: Connection established.
debug1: identity file /home/me/.ssh/id_rsa type -1
debug1: identity file /home/me/.ssh/id_rsa-cert type -1
debug1: identity file /home/me/.ssh/id_dsa type -1
debug1: identity file /home/me/.ssh/id_dsa-cert type -1
debug1: identity file /home/me/.ssh/id_ecdsa type -1
debug1: identity file /home/me/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/me/.ssh/id_ed25519 type -1
debug1: identity file /home/me/.ssh/id_ed25519-cert type -1
debug1: identity file /home/me/.ssh/id_xmss type -1
debug1: identity file /home/me/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 6 setting O_NONBLOCK
debug1: Authenticating to 192.168.122.33:22 as 'me'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,blowfish-cbc,cast128-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:tDHtYxD9gjd6WxS82Y5unZRfSTA42S0BWPQB244fP2s
debug1: Host '192.168.122.33' is known and matches the ECDSA host key.
debug1: Found key in /home/me/.ssh/known_hosts:1
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug2: key: me@localhost.localdomain (0x5620eac46f80), agent
debug2: key: me@localhost.localdomain (0x5620eac4a830), agent
debug2: key: /home/me/.ssh/id_rsa ((nil))
debug2: key: /home/me/.ssh/id_dsa ((nil))
debug2: key: /home/me/.ssh/id_ecdsa ((nil))
debug2: key: /home/me/.ssh/id_ed25519 ((nil))
debug2: key: /home/me/.ssh/id_xmss ((nil))
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KCM:)


debug1: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KCM:)


debug2: we did not send a packet, disable method
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:a3sjstTdNHguzTkUQ7u/1ExNPPijAkXbNYl7CbrwCOQ me@localhost.localdomain
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg rsa-sha2-256 blen 279
debug2: input_userauth_pk_ok: fp SHA256:a3sjstTdNHguzTkUQ7u/1ExNPPijAkXbNYl7CbrwCOQ
debug1: Authentication succeeded (publickey).
Authenticated to 192.168.122.33 ([192.168.122.33]:22).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 6 setting TCP_NODELAY
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug1: Sending env XMODIFIERS = @im=ibus
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Mon Apr 29 10:24:40 2019 from gateway
[me@test33 ~]$ 

Your ssh key was held in memory by your ssh agent. Deleting it won't prevent the agent from continuing to use it. You would need to stop the agent (usually by logging out of your workstation) first.