Otdfbt

SUMMARY

I have ansible and ansible AWX installed on same AWS EC2 instance and I am trying to deploy build on another app AWS EC2 instance. I am able to run the playbook via ansible-playbook command,however, get Failed to connect to the host via ssh when using AWX

• AWX inventory has just that host added along with ansible_user :
  xxxxx which is the user I want to deploy with


• Ansible/AWX host has the xxxxx user setup with password less access to the target host which also has the same user (public keys copied)


• I also tried creating this user in aws_task and aws_web containers and copied the public keys to target host


• 
  

  Running the ssh command AWS runs while gathering information fails

  
  
  

  
  

  ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=xxxxx -o ConnectTimeout=10 -o ControlPath=/tmp/awx_108_dKOwxm/cp/61735cd9d3 hostname '/bin/sh -c '"'"'echo ~xxxxx && sleep 0'"'"''

  
  

  
  

Output:

debug3: muxserver_listen: temporary control path /tmp/awx_72_VGFOPt/cp/61735cd9d3.16cjnTwr0hSE1mXc
bind: No such file or directory
unix_listener: cannot bind to path: /tmp/awx_72_VGFOPt/cp/61735cd9d3.16cjnTwr0hSE1mXc

• 
  

  Running the command after leaving some configuration options - ControlPersist,ControlMaster,ssh works -

  
  
  

  
  

  ssh -vvv -C -o StrictHostKeyChecking=no -o 'IdentityFile="/home/xxxxx/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=xxxxx -o ConnectTimeout=10 hostname

  
  

  
  

Questions

1. Is this a permission issue while creating the ControlPath? The aws_web and aws_task container both have /tmp folder with 777 permission


2. Which configuration (ansible.cfg) does AWX pick - from the aws_task/aws_web containers or from the host on which it is deployed. I have also tried to set ssh_args = -C (removing the ControlMaster and ControlPersist) at all the three places in ansible.cfg and restarted the AWS containers, without any luck


3. Why it is not working with Control Params

Additional Information

Ansible Version

ansible 2.7.1
 config file = /etc/ansible/ansible.cfg
 configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
 ansible python module location = /usr/lib/python2.7/site-packages/ansible
 executable location = /bin/ansible
 python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

Configuration

ANSIBLE_SSH_ARGS(/etc/ansible/ansible.cfg) = -C

OS / ENVIRONMENT

Ansible and AWX host OS -> CentOS Linux release 7.5.1804 (Core)

Target host OS -> CentOS Linux release 7.5.1804 (Core)

AWX Version: 2.1.0

AWX installed via -> https://github.com/ansible/awx/blob/devel/INSTALL.md#docker-or-docker-compose

STEPS TO REPRODUCE

1) Created Project in AWX with SCM type git.

2) Created inventory in AWX adding the host with variable ansible_user: xxxxx

3) Created job template specifying job type run and selecting project and inventory created in 1 and 2. Provided the extra variable build: in the template as required in the playbook

PlayBook

https://github.com/sandeepdharembra/playbooks/blob/master/installnodejs.yml

ACTUAL RESULTS while running the job template

ansible-playbook 2.7.1
 config file = /etc/ansible/ansible.cfg
 configured module search path = [u'/var/lib/awx/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
 ansible python module location = /usr/lib/python2.7/site-packages/ansible
 executable location = /usr/bin/ansible-playbook
 python version = 2.7.5 (default, Jul 13 2018, 13:06:57) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Parsed /tmp/awx_108_dKOwxm/tmpo_yCMg inventory source with script plugin
Loading callback plugin awx_display of type stdout, v2.0 from /usr/lib/python2.7/site-packages/awx/lib/awx_display_callback/module.pyc
1 plays in installnodejs.yml
PLAY [all] *********************************************************************
META: ran handlers
TASK [Created new build directory] *********************************************
task path: /var/lib/awx/projects/_6__festaging/installnodejs.yml:10
<hostname> ESTABLISH SSH CONNECTION FOR USER: xxxxx
<hostname> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=xxxxx -o ConnectTimeout=10 -o ControlPath=/tmp/awx_108_dKOwxm/cp/61735cd9d3 hostname '/bin/sh -c '"'"'echo ~xxxxx && sleep 0'"'"''
<hostname> (255, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 58: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug1: Control socket "/tmp/awx_108_dKOwxm/cp/61735cd9d3" does not existrndebug2: resolving "hostname" port 22rndebug2: ssh_connect_direct: needpriv 0rndebug1: Connecting to hostname [hostname] port 22.rndebug2: fd 3 setting O_NONBLOCKrndebug1: fd 3 clearing O_NONBLOCKrndebug1: Connection established.rndebug3: timeout: 9997 ms remain after connectrndebug1: permanently_set_uid: 0/0rndebug1: key_load_public: No such file or directoryrndebug1: identity file /root/.ssh/id_rsa type -1rndebug1: key_load_public: No such file or directoryrndebug1: identity file /root/.ssh/id_rsa-cert type -1rndebug1: key_load_public: No such file or directoryrndebug1: identity file /root/.ssh/id_dsa type -1rndeb…
fatal: [hostname]: UNREACHABLE! => {
 "changed": false, 
 "msg": "Failed to connect to the host via ssh: OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017rndebug1: Reading configuration data /etc/ssh/ssh_configrndebug1: /etc/ssh/ssh_config line 58: Applying options for *rndebug1: auto-mux: Trying existing masterrndebug1: Control socket "/tmp/awx_108_dKOwxm/cp/61735cd9d3" does not existrndebug2: resolving "hostname" port 22rndebug2: ssh_connect_direct: needpriv 0rndebug1: Connecting to hostname [172.16.5.91] port 22.rndebug2: fd 3 setting O_NONBLOCKrndebug1: fd 3 clearing O_NONBLOCKrndebug1: Connection established.rndebug3: timeout: 9997 ms remain after connectrndebug1: permanently_set_uid: 0/0rndebug1: key_load_public: No such file or directoryrndebug1: identity file /root/.ssh/id_rsa type -1rndebug1: key_load_public: No such file or directoryrndebug1: identity file /root/.ssh/id_rsa-cert ty…
PLAY RECAP *********************************************************************
hostname : ok=0 changed=0 unreachable=1 failed=0 

I was finally able to find a solution to this issue -

The job template I created had the credentials not assigned. I created the credentials with which I wanted to access the target host (user present on target host), assigned the private key and added these credentials to the job template.
Job was now able to run (ssh on the remote host)