2021-03-15T18:42:46.247 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-15T18:42:46.250 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2021-03-15_17:46:49-rados-octopus-distro-basic-smithi/5967658 --name yuriw-2021-03-15_17:46:49-rados-octopus-distro-basic-smithi --verbose --owner scheduled_yuriw@teuthology --description rados/singleton-nomsgr/{all/ceph-kvstore-tool rados supported-random-distro$/{rhel_latest}} -- /home/teuthworker/archive/yuriw-2021-03-15_17:46:49-rados-octopus-distro-basic-smithi/5967658/orig.config.yaml 2021-03-15T18:42:46.284 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-03-15T18:42:46.340 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-03-15_17:46:49-rados-octopus-distro-basic-smithi/5967658 branch: octopus description: rados/singleton-nomsgr/{all/ceph-kvstore-tool rados supported-random-distro$/{rhel_latest}} email: ceph-qa@ceph.io first_in_suite: false job_id: '5967658' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2021-03-15_17:46:49-rados-octopus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: rhel os_version: '8.2' overrides: admin_socket: branch: octopus ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: debug ms: 1 debug osd: 20 osd debug verify cached snaps: true osd debug verify missing on start: true osd op queue: debug_random osd op queue cut off: debug_random log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - overall HEALTH_ - \(POOL_APP_NOT_ENABLED\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) pre-mgr-commands: - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force sha1: 4db20a7386a7816fbf53ee7139cdb4ca22d9e1de ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4db20a7386a7816fbf53ee7139cdb4ca22d9e1de selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: octopus sha1: 4db20a7386a7816fbf53ee7139cdb4ca22d9e1de owner: scheduled_yuriw@teuthology priority: 79 repo: https://github.com/ceph/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 4db20a7386a7816fbf53ee7139cdb4ca22d9e1de sleep_before_teardown: 0 suite: rados suite_branch: octopus suite_path: /home/teuthworker/src/github.com_ceph_ceph_4db20a7386a7816fbf53ee7139cdb4ca22d9e1de/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: 4db20a7386a7816fbf53ee7139cdb4ca22d9e1de targets: smithi188.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+zCIxzFOUAabcE9V73eUlDWg4SWm4K/aIh619iK0f1JlkeVIdk4knWocnWRsFd6FeYNBSRkhFLi+NcaXEmbJzeEji9h2KyjWhvqI9uYq4XuRK9X3UUDz2D0p1wcuvp8mwp6CLpXZCECltmpZNhxx/gaKB7EQ6ovOxstZJQAC0oB+yG5B46zc3TjV/WjKNIzxtvxE6qSB/pwiHpr9dGs5yVdF23qwQWrkEFopEDO3BTQFXDhwH4TUV/2EpNaEtyXIGDSDiJgtPR74HvKPWJMufd+da0PXQVpaN+5DClUVOF/HJg9zuket2JUh41J/lMB1Dg9Q19ZBPfpL+NBb2kNUv tasks: - install: null - ceph: null - workunit: clients: all: - cephtool/test_kvstore_tool.sh teuthology_branch: master teuthology_sha1: b96569170f15eae4604f361990ea65737b28dff1 timestamp: 2021-03-15_17:46:49 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2137 2021-03-15T18:42:46.341 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_4db20a7386a7816fbf53ee7139cdb4ca22d9e1de/qa; will attempt to use it 2021-03-15T18:42:46.346 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_4db20a7386a7816fbf53ee7139cdb4ca22d9e1de/qa/tasks 2021-03-15T18:42:46.346 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-03-15T18:42:46.348 INFO:teuthology.task.internal:Checking packages... 2021-03-15T18:42:46.371 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash '4db20a7386a7816fbf53ee7139cdb4ca22d9e1de' 2021-03-15T18:42:46.371 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-15T18:42:46.372 INFO:teuthology.packaging:ref: None 2021-03-15T18:42:46.372 INFO:teuthology.packaging:tag: None 2021-03-15T18:42:46.372 INFO:teuthology.packaging:branch: octopus 2021-03-15T18:42:46.373 INFO:teuthology.packaging:sha1: 4db20a7386a7816fbf53ee7139cdb4ca22d9e1de 2021-03-15T18:42:46.373 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=octopus 2021-03-15T18:42:46.569 INFO:teuthology.task.internal:Found packages for ceph version 15.2.9-145.g4db20a73 2021-03-15T18:42:46.570 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-03-15T18:42:46.588 INFO:teuthology.task.internal:no buildpackages task found 2021-03-15T18:42:46.589 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-03-15T18:42:46.676 INFO:teuthology.task.internal:Saving configuration 2021-03-15T18:42:46.689 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-03-15T18:42:46.725 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-15T18:42:46.744 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-15 18:37:33.776228', 'description': '/home/teuthworker/archive/yuriw-2021-03-15_17:46:49-rados-octopus-distro-basic-smithi/5967660', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHnjENUYT3xNvQC1sjw03prWhvh1KH4eodk4Iho4LsZrKVBFNrmT1uJBfRyWwSRG1RXqRtofIHCYo15sTm7qwfA=', 'os_version': '7.7', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'rhel', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi188.front.sepia.ceph.com'} 2021-03-15T18:42:46.745 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-03-15T18:42:46.792 INFO:teuthology.task.internal:roles: ubuntu@smithi188.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2021-03-15T18:42:46.793 INFO:teuthology.run_tasks:Running task console_log... 2021-03-15T18:42:47.113 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5ea4655840>, signals=[15]) 2021-03-15T18:42:47.114 INFO:teuthology.run_tasks:Running task internal.connect... 2021-03-15T18:42:47.154 INFO:teuthology.task.internal:Opening connections... 2021-03-15T18:42:47.154 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi188.front.sepia.ceph.com 2021-03-15T18:42:47.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T18:42:47.230 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-03-15T18:42:47.240 DEBUG:teuthology.orchestra.run.smithi188:> uname -m 2021-03-15T18:42:47.482 INFO:teuthology.orchestra.run.smithi188.stdout:x86_64 2021-03-15T18:42:47.483 DEBUG:teuthology.orchestra.run.smithi188:> cat /etc/os-release 2021-03-15T18:42:47.524 INFO:teuthology.orchestra.run.smithi188.stdout:NAME="Red Hat Enterprise Linux Server" 2021-03-15T18:42:47.524 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION="7.7 (Maipo)" 2021-03-15T18:42:47.524 INFO:teuthology.orchestra.run.smithi188.stdout:ID="rhel" 2021-03-15T18:42:47.524 INFO:teuthology.orchestra.run.smithi188.stdout:ID_LIKE="fedora" 2021-03-15T18:42:47.525 INFO:teuthology.orchestra.run.smithi188.stdout:VARIANT="Server" 2021-03-15T18:42:47.525 INFO:teuthology.orchestra.run.smithi188.stdout:VARIANT_ID="server" 2021-03-15T18:42:47.525 INFO:teuthology.orchestra.run.smithi188.stdout:VERSION_ID="7.7" 2021-03-15T18:42:47.525 INFO:teuthology.orchestra.run.smithi188.stdout:PRETTY_NAME="Employee SKU" 2021-03-15T18:42:47.526 INFO:teuthology.orchestra.run.smithi188.stdout:ANSI_COLOR="0;31" 2021-03-15T18:42:47.526 INFO:teuthology.orchestra.run.smithi188.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:7.7:GA:server" 2021-03-15T18:42:47.526 INFO:teuthology.orchestra.run.smithi188.stdout:HOME_URL="https://www.redhat.com/" 2021-03-15T18:42:47.526 INFO:teuthology.orchestra.run.smithi188.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-15T18:42:47.526 INFO:teuthology.orchestra.run.smithi188.stdout: 2021-03-15T18:42:47.527 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" 2021-03-15T18:42:47.527 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=7.7 2021-03-15T18:42:47.528 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-15T18:42:47.528 INFO:teuthology.orchestra.run.smithi188.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7.7" 2021-03-15T18:42:47.529 INFO:teuthology.lock.ops:Updating smithi188.front.sepia.ceph.com on lock server 2021-03-15T18:42:47.547 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-03-15T18:42:47.561 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-03-15T18:42:47.736 INFO:teuthology.task.internal:Checking for old test directory... 2021-03-15T18:42:47.737 DEBUG:teuthology.orchestra.run.smithi188:> test '!' -e /home/ubuntu/cephtest 2021-03-15T18:42:47.777 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-03-15T18:42:47.799 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-03-15T18:42:47.800 DEBUG:teuthology.orchestra.run.smithi188:> test -z $(ls -A /var/lib/ceph) 2021-03-15T18:42:47.874 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-03-15T18:42:47.905 INFO:teuthology.run_tasks:Running task kernel... 2021-03-15T18:42:47.952 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2021-03-15T18:42:47.952 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-03-15T18:42:47.953 DEBUG:teuthology.orchestra.run.smithi188:> uname -r 2021-03-15T18:42:47.993 INFO:teuthology.orchestra.run.smithi188.stdout:3.10.0-1062.9.1.el7.x86_64 2021-03-15T18:42:47.994 INFO:teuthology.task.kernel:Running kernel on smithi188: 3.10.0-1062.9.1.el7.x86_64 2021-03-15T18:42:47.994 DEBUG:teuthology.orchestra.run.smithi188:> sudo yum install -y kernel 2021-03-15T18:42:49.036 INFO:teuthology.orchestra.run.smithi188.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-03-15T18:42:49.037 INFO:teuthology.orchestra.run.smithi188.stdout: : disabled-repos, subscription-manager 2021-03-15T18:42:49.250 INFO:teuthology.orchestra.run.smithi188.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-03-15T18:42:49.299 INFO:teuthology.orchestra.run.smithi188.stdout:Determining fastest mirrors 2021-03-15T18:42:49.388 INFO:teuthology.orchestra.run.smithi188.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-03-15T18:42:50.852 INFO:teuthology.orchestra.run.smithi188.stdout:Nothing to do 2021-03-15T18:42:50.871 DEBUG:teuthology.orchestra.run.smithi188:> echo no | sudo yum reinstall kernel || true 2021-03-15T18:42:51.069 INFO:teuthology.orchestra.run.smithi188.stdout:Loaded plugins: fastestmirror, langpacks, priorities, product-id, search- 2021-03-15T18:42:51.070 INFO:teuthology.orchestra.run.smithi188.stdout: : disabled-repos, subscription-manager 2021-03-15T18:42:51.133 INFO:teuthology.orchestra.run.smithi188.stdout:This system is not registered with an entitlement server. You can use subscription-manager to register. 2021-03-15T18:42:51.210 INFO:teuthology.orchestra.run.smithi188.stderr:Skipping the running kernel: kernel-3.10.0-1062.9.1.el7.x86_64 2021-03-15T18:42:51.232 INFO:teuthology.orchestra.run.smithi188.stdout:Loading mirror speeds from cached hostfile 2021-03-15T18:42:51.233 INFO:teuthology.orchestra.run.smithi188.stdout: * epel: download-cc-rdu01.fedoraproject.org 2021-03-15T18:42:51.237 INFO:teuthology.orchestra.run.smithi188.stdout:Installed package kernel-3.10.0-1062.el7.x86_64 (from anaconda) not available. 2021-03-15T18:42:51.237 INFO:teuthology.orchestra.run.smithi188.stderr:Error: Nothing to do 2021-03-15T18:42:51.256 DEBUG:teuthology.orchestra.run.smithi188:> rpm -q kernel --last | head -n 1 2021-03-15T18:42:51.456 INFO:teuthology.orchestra.run.smithi188.stdout:kernel-3.10.0-1062.9.1.el7.x86_64 Wed 11 Dec 2019 03:55:23 PM UTC 2021-03-15T18:42:51.457 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.9.1.el7.x86_64 2021-03-15T18:42:51.458 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-15T18:42:51.458 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-03-15T18:42:51.458 DEBUG:teuthology.orchestra.run.smithi188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-15T18:42:51.523 INFO:teuthology.orchestra.run.smithi188.stdout:ttyS1 2021-03-15T18:42:51.536 INFO:teuthology.run_tasks:Running task internal.base... 2021-03-15T18:42:51.549 INFO:teuthology.task.internal:Creating test directory... 2021-03-15T18:42:51.549 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-15T18:42:51.591 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-03-15T18:42:51.604 INFO:teuthology.run_tasks:Running task internal.archive... 2021-03-15T18:42:51.721 INFO:teuthology.task.internal:Creating archive directory... 2021-03-15T18:42:51.721 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-15T18:42:52.065 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-03-15T18:42:52.079 INFO:teuthology.task.internal:Enabling coredump saving... 2021-03-15T18:42:52.080 DEBUG:teuthology.orchestra.run.smithi188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-15T18:42:52.137 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-15T18:42:52.150 INFO:teuthology.orchestra.run.smithi188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-15T18:42:52.151 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-03-15T18:42:52.246 INFO:teuthology.task.internal:Configuring sudo... 2021-03-15T18:42:52.247 DEBUG:teuthology.orchestra.run.smithi188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-15T18:42:52.300 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-03-15T18:42:52.313 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-03-15T18:42:52.313 DEBUG:teuthology.orchestra.run.smithi188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-15T18:42:52.355 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-15T18:42:52.452 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-15T18:42:52.555 DEBUG:teuthology.orchestra.run.smithi188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-15T18:42:52.617 DEBUG:teuthology.orchestra.run.smithi188:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-15T18:42:52.705 DEBUG:teuthology.orchestra.run.smithi188:> set -ex 2021-03-15T18:42:52.707 DEBUG:teuthology.orchestra.run.smithi188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-15T18:42:52.782 DEBUG:teuthology.orchestra.run.smithi188:> sudo service rsyslog restart 2021-03-15T18:42:52.903 INFO:teuthology.orchestra.run.smithi188.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-03-15T18:42:52.929 INFO:teuthology.run_tasks:Running task internal.timer... 2021-03-15T18:42:52.937 INFO:teuthology.task.internal:Starting timer... 2021-03-15T18:42:52.937 INFO:teuthology.run_tasks:Running task pcp... 2021-03-15T18:42:53.058 INFO:teuthology.run_tasks:Running task selinux... 2021-03-15T18:42:53.108 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-03-15T18:42:53.109 DEBUG:teuthology.orchestra.run.smithi188:> sudo service auditd rotate 2021-03-15T18:42:53.184 INFO:teuthology.orchestra.run.smithi188.stdout:Rotating logs: [ OK ] 2021-03-15T18:42:53.186 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-03-15T18:42:53.186 DEBUG:teuthology.orchestra.run.smithi188:> /usr/sbin/getenforce 2021-03-15T18:42:53.237 INFO:teuthology.orchestra.run.smithi188.stdout:Permissive 2021-03-15T18:42:53.237 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi188.front.sepia.ceph.com': 'permissive'} 2021-03-15T18:42:53.238 DEBUG:teuthology.orchestra.run.smithi188:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-03-15T18:42:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T18:42:53.291 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-03-15T18:42:53.291 DEBUG:teuthology.orchestra.run.smithi188:> sudo /usr/sbin/setenforce permissive 2021-03-15T18:42:53.346 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-03-15T18:42:53.357 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-03-15T18:42:53.358 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-03-15T18:42:53.375 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-03-15T18:42:53.377 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-03-15T18:42:53.809 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-03-15T18:42:54.662 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-03-15T18:42:56.808 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-03-15T18:42:56.839 INFO:teuthology.task.ansible.out:Monday 15 March 2021 18:42:56 +0000 (0:00:00.047) 0:00:00.047 ********** 2021-03-15T18:42:58.275 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi188.front.sepia.ceph.com] 2021-03-15T18:42:58.339 INFO:teuthology.task.ansible.out:Monday 15 March 2021 18:42:58 +0000 (0:00:01.500) 0:00:01.548 ********** 2021-03-15T18:42:58.662 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-03-15T18:42:58.664 INFO:teuthology.task.ansible.out:ok: [smithi188.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-03-15T18:42:58.686 INFO:teuthology.task.ansible.out:Monday 15 March 2021 18:42:58 +0000 (0:00:00.346) 0:00:01.894 ********** 2021-03-15T18:44:39.502 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** fatal: [smithi188.front.sepia.ceph.com]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"smithi188.front.sepia.ceph.com\". Make sure this host can be reached over ssh: ssh: connect to host smithi188.front.sepia.ceph.com port 22: No route to host\r\n", "unreachable": true} 2021-03-15T18:44:39.513 INFO:teuthology.task.ansible.out: NO MORE HOSTS LEFT ************************************************************* 2021-03-15T18:44:39.522 INFO:teuthology.task.ansible.out: PLAY RECAP ********************************************************************* smithi188.front.sepia.ceph.com : ok=2 changed=0 unreachable=1 failed=0 skipped=0 rescued=0 ignored=0 Monday 15 March 2021 18:44:39 +0000 (0:01:40.834) 0:01:42.729 ********** =============================================================================== ansible-managed : Create the ansible user. ---------------------------- 100.83s Gathering Facts --------------------------------------------------------- 1.50s ansible-managed : Create the sudo group. -------------------------------- 0.35s 2021-03-15T18:44:39.856 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2021-03-15_17:46:49-rados-octopus-distro-basic-smithi/5967658/ansible_failures.yaml 2021-03-15T18:44:39.857 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/ansible.py", line 421, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/ansible.py", line 263, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/ansible.py", line 290, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/ansible.py", line 315, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: {'smithi188.front.sepia.ceph.com': {'changed': False, 'msg': 'Data could not be sent to remote host "smithi188.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi188.front.sepia.ceph.com port 22: No route to host\r\n', 'unreachable': True}} 2021-03-15T18:44:39.960 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fdf32dc9f0d84e0787426fddc87ae1d8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/ansible.py", line 421, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/ansible.py", line 263, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/ansible.py", line 290, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/ansible.py", line 315, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: {'smithi188.front.sepia.ceph.com': {'changed': False, 'msg': 'Data could not be sent to remote host "smithi188.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi188.front.sepia.ceph.com port 22: No route to host\r\n', 'unreachable': True}} 2021-03-15T18:44:39.961 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-15T18:44:39.975 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-03-15T18:44:39.975 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-03-15T18:44:40.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T18:44:40.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T18:44:42.567 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2021-03-15T18:44:42.568 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 159, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 190, in archive_log args=full_cmd.format(audit_archive=audit_archive) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi188 was lost: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' 2021-03-15T18:44:42.575 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-03-15T18:44:42.586 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-03-15T18:44:42.629 INFO:teuthology.task.internal:Duration was 109.691329 seconds 2021-03-15T18:44:42.630 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-03-15T18:44:42.690 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-03-15T18:44:42.690 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T18:44:42.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T18:44:45.634 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2021-03-15T18:44:45.635 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/syslog.py", line 71, in syslog yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 159, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 190, in archive_log args=full_cmd.format(audit_archive=audit_archive) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi188 was lost: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/syslog.py", line 89, in syslog wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi188 was lost: 'sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart' 2021-03-15T18:44:45.644 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-03-15T18:44:45.655 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-03-15T18:44:45.656 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T18:44:45.657 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T18:44:48.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2021-03-15T18:44:48.707 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/__init__.py", line 428, in sudo yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 159, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 190, in archive_log args=full_cmd.format(audit_archive=audit_archive) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi188 was lost: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/__init__.py", line 433, in sudo path=sudoers_file, ext=backup_ext File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi188 was lost: 'sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers' 2021-03-15T18:44:48.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-03-15T18:44:48.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T18:44:48.729 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T18:44:51.778 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2021-03-15T18:44:51.779 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/__init__.py", line 463, in coredump yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 159, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 190, in archive_log args=full_cmd.format(audit_archive=audit_archive) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi188 was lost: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/__init__.py", line 479, in coredump wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi188 was lost: "sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump" 2021-03-15T18:44:51.787 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-03-15T18:44:51.808 INFO:teuthology.task.internal:Transferring archived files... 2021-03-15T18:44:51.810 DEBUG:teuthology.misc:Transferring archived files from smithi188:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2021-03-15_17:46:49-rados-octopus-distro-basic-smithi/5967658/remote/smithi188 2021-03-15T18:44:51.812 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T18:44:51.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T18:44:54.850 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2021-03-15T18:44:54.851 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/__init__.py", line 368, in archive yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 159, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 190, in archive_log args=full_cmd.format(audit_archive=audit_archive) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi188 was lost: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/__init__.py", line 394, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 743, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 461, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi188 was lost: 'sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- .' 2021-03-15T18:44:54.860 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-03-15T18:44:54.874 INFO:teuthology.task.internal:Not uploading archives. 2021-03-15T18:44:54.874 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-03-15T18:44:55.185 INFO:teuthology.task.internal:Tidying up after the test... 2021-03-15T18:44:55.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T18:44:55.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T18:44:57.922 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.188 2021-03-15T18:44:57.923 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/__init__.py", line 46, in base yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 159, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/selinux.py", line 190, in archive_log args=full_cmd.format(audit_archive=audit_archive) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi188 was lost: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/task/internal/__init__.py", line 56, in base wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 438, in run raise ConnectionLostError(command=quote(args), node=name) teuthology.exceptions.ConnectionLostError: SSH connection to smithi188 was lost: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2021-03-15T18:44:57.932 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-03-15T18:44:57.952 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: ConnectionLostError: SSH connection to smithi188 was lost: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' 2021-03-15T18:44:57.952 INFO:teuthology.nuke:Checking targets against current locks 2021-03-15T18:44:58.058 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run.py", line 409, in main run_tasks(tasks=config['tasks'], ctx=fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run_tasks.py", line 204, in run_tasks raise SystemExit(1) SystemExit: 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/nuke/__init__.py", line 250, in nuke if ctx.name not in lock['description']: TypeError: argument of type 'NoneType' is not iterable