2023-01-19T07:41:54.295 INFO:root:teuthology version: 0.0.1.dev20+g5abecab 2023-01-19T07:41:54.295 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_gabrioux@teuthology --verbose --name gabrioux-2023-01-19_06:14:52-orch:cephadm-wip-guits-testing-2023-01-18-2355-distro-default-smithi --description orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli} --archive /home/teuthworker/archive/gabrioux-2023-01-19_06:14:52-orch:cephadm-wip-guits-testing-2023-01-18-2355-distro-default-smithi/7129978 -- /home/teuthworker/archive/gabrioux-2023-01-19_06:14:52-orch:cephadm-wip-guits-testing-2023-01-18-2355-distro-default-smithi/7129978/orig.config.yaml 2023-01-19T07:41:54.324 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-01-19T07:41:54.393 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2023-01-19_06:14:52-orch:cephadm-wip-guits-testing-2023-01-18-2355-distro-default-smithi/7129978 branch: wip-guits-testing-2023-01-18-2355 description: orch:cephadm/workunits/{0-distro/rhel_8.6_container_tools_rhel8 agent/on mon_election/connectivity task/test_orch_cli} email: gabrioux@redhat.com first_in_suite: false job_id: '7129978' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2023-01-19_06:14:52-orch:cephadm-wip-guits-testing-2023-01-18-2355-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-guits-testing-2023-01-18-2355 ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 8c2873fed10920a9f11c8fd09e87e77c92bfa3c8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: flavor: default sha1: 8c2873fed10920a9f11c8fd09e87e77c92bfa3c8 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-guits-testing-2023-01-18-2355 sha1: 8c2873fed10920a9f11c8fd09e87e77c92bfa3c8 owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - osd.0 - osd.1 - osd.2 - mon.a - mgr.a - client.0 sha1: 8c2873fed10920a9f11c8fd09e87e77c92bfa3c8 sleep_before_teardown: 0 suite: orch:cephadm suite_branch: wip-guits-testing-2023-01-18-2355 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_8c2873fed10920a9f11c8fd09e87e77c92bfa3c8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 8c2873fed10920a9f11c8fd09e87e77c92bfa3c8 targets: smithi100.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: null - cephadm: null - cephadm.shell: host.a: - ceph orch apply mds a - cephfs_test_runner: modules: - tasks.cephadm_cases.test_cli teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5abecab36c9da4d70461d81781464cf5acaea292 timestamp: 2023-01-19_06:14:52 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.8687 2023-01-19T07:41:54.393 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_8c2873fed10920a9f11c8fd09e87e77c92bfa3c8/qa; will attempt to use it 2023-01-19T07:41:54.394 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_8c2873fed10920a9f11c8fd09e87e77c92bfa3c8/qa/tasks 2023-01-19T07:41:54.394 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-01-19T07:41:54.395 INFO:teuthology.task.internal:Checking packages... 2023-01-19T07:41:54.421 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '8c2873fed10920a9f11c8fd09e87e77c92bfa3c8' 2023-01-19T07:41:54.421 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-01-19T07:41:54.421 INFO:teuthology.packaging:ref: None 2023-01-19T07:41:54.421 INFO:teuthology.packaging:tag: None 2023-01-19T07:41:54.421 INFO:teuthology.packaging:branch: wip-guits-testing-2023-01-18-2355 2023-01-19T07:41:54.421 INFO:teuthology.packaging:sha1: 8c2873fed10920a9f11c8fd09e87e77c92bfa3c8 2023-01-19T07:41:54.422 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-guits-testing-2023-01-18-2355 2023-01-19T07:41:54.631 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-1773.g8c2873fe 2023-01-19T07:41:54.631 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-01-19T07:41:54.639 INFO:teuthology.task.internal:no buildpackages task found 2023-01-19T07:41:54.639 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-01-19T07:41:54.675 INFO:teuthology.task.internal:Saving configuration 2023-01-19T07:41:54.686 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-01-19T07:41:54.692 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-01-19T07:41:54.722 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2023-01-19_06:14:52-orch:cephadm-wip-guits-testing-2023-01-18-2355-distro-default-smithi/7129978', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-01-19 07:36:46.122430', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-01-19T07:41:54.722 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-01-19T07:41:54.728 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'mon.a', 'mgr.a', 'client.0'] 2023-01-19T07:41:54.728 INFO:teuthology.run_tasks:Running task console_log... 2023-01-19T07:41:54.834 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0b544e6f70>, signals=[15]) 2023-01-19T07:41:54.835 INFO:teuthology.run_tasks:Running task internal.connect... 2023-01-19T07:41:54.861 INFO:teuthology.task.internal:Opening connections... 2023-01-19T07:41:54.861 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2023-01-19T07:41:54.862 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-19T07:41:54.927 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-01-19T07:41:54.934 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2023-01-19T07:41:55.000 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2023-01-19T07:41:55.001 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2023-01-19T07:41:55.057 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="Red Hat Enterprise Linux" 2023-01-19T07:41:55.057 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="8.6 (Ootpa)" 2023-01-19T07:41:55.058 INFO:teuthology.orchestra.run.smithi100.stdout:ID="rhel" 2023-01-19T07:41:55.058 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="fedora" 2023-01-19T07:41:55.058 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="8.6" 2023-01-19T07:41:55.058 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el8" 2023-01-19T07:41:55.058 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-01-19T07:41:55.058 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2023-01-19T07:41:55.058 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-01-19T07:41:55.058 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://www.redhat.com/" 2023-01-19T07:41:55.058 INFO:teuthology.orchestra.run.smithi100.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-01-19T07:41:55.058 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-01-19T07:41:55.059 INFO:teuthology.orchestra.run.smithi100.stdout: 2023-01-19T07:41:55.059 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-01-19T07:41:55.059 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-01-19T07:41:55.059 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-01-19T07:41:55.059 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-01-19T07:41:55.060 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2023-01-19T07:41:55.080 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-01-19T07:41:55.088 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-01-19T07:41:55.148 INFO:teuthology.task.internal:Checking for old test directory... 2023-01-19T07:41:55.148 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2023-01-19T07:41:55.167 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-01-19T07:41:55.270 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-01-19T07:41:55.270 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2023-01-19T07:41:55.296 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-01-19T07:41:55.340 INFO:teuthology.run_tasks:Running task kernel... 2023-01-19T07:41:55.456 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-01-19T07:41:55.456 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-01-19T07:41:55.456 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2023-01-19T07:41:55.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-19T07:41:55.475 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2023-01-19T07:41:55.535 INFO:teuthology.orchestra.run.smithi100.stdout:4.18.0-372.9.1.el8.x86_64 2023-01-19T07:41:55.535 INFO:teuthology.task.kernel:Running kernel on smithi100: 4.18.0-372.9.1.el8.x86_64 2023-01-19T07:41:55.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum install -y kernel 2023-01-19T07:41:56.765 INFO:teuthology.orchestra.run.smithi100.stdout:Updating Subscription Management repositories. 2023-01-19T07:41:56.765 INFO:teuthology.orchestra.run.smithi100.stdout:Unable to read consumer identity 2023-01-19T07:41:58.180 INFO:teuthology.orchestra.run.smithi100.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-01-19T07:41:59.789 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 13 MB 00:01 2023-01-19T07:42:02.806 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 312 kB/s | 24 kB 00:00 2023-01-19T07:42:03.416 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-01-19T07:42:03.477 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2023-01-19T07:42:03.478 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2023-01-19T07:42:03.478 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2023-01-19T07:42:03.623 DEBUG:teuthology.orchestra.run.smithi100:> echo no | sudo yum reinstall kernel || true 2023-01-19T07:42:03.954 INFO:teuthology.orchestra.run.smithi100.stdout:Updating Subscription Management repositories. 2023-01-19T07:42:03.954 INFO:teuthology.orchestra.run.smithi100.stdout:Unable to read consumer identity 2023-01-19T07:42:04.208 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:02 ago on Thu 19 Jan 2023 07:42:02 AM UTC. 2023-01-19T07:42:04.249 INFO:teuthology.orchestra.run.smithi100.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-01-19T07:42:04.289 INFO:teuthology.orchestra.run.smithi100.stderr:Error: No packages marked for reinstall. 2023-01-19T07:42:04.335 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum reinstall -y kernel || true 2023-01-19T07:42:04.664 INFO:teuthology.orchestra.run.smithi100.stdout:Updating Subscription Management repositories. 2023-01-19T07:42:04.665 INFO:teuthology.orchestra.run.smithi100.stdout:Unable to read consumer identity 2023-01-19T07:42:04.914 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:02 ago on Thu 19 Jan 2023 07:42:02 AM UTC. 2023-01-19T07:42:04.954 INFO:teuthology.orchestra.run.smithi100.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-01-19T07:42:04.986 INFO:teuthology.orchestra.run.smithi100.stderr:Error: No packages marked for reinstall. 2023-01-19T07:42:05.032 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel --last | head -n 1 2023-01-19T07:42:05.111 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Thu 14 Jul 2022 06:23:21 PM UTC 2023-01-19T07:42:05.112 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-01-19T07:42:05.112 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-01-19T07:42:05.112 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-01-19T07:42:05.113 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-01-19T07:42:05.150 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2023-01-19T07:42:05.164 DEBUG:teuthology.parallel:result is None 2023-01-19T07:42:05.164 INFO:teuthology.run_tasks:Running task internal.base... 2023-01-19T07:42:05.174 INFO:teuthology.task.internal:Creating test directory... 2023-01-19T07:42:05.174 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-01-19T07:42:05.220 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-01-19T07:42:05.227 INFO:teuthology.run_tasks:Running task internal.archive... 2023-01-19T07:42:05.332 INFO:teuthology.task.internal:Creating archive directory... 2023-01-19T07:42:05.332 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-01-19T07:42:05.362 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-01-19T07:42:05.379 INFO:teuthology.task.internal:Enabling coredump saving... 2023-01-19T07:42:05.380 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-01-19T07:42:05.447 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-19T07:42:05.457 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-01-19T07:42:05.459 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-01-19T07:42:05.466 INFO:teuthology.task.internal:Configuring sudo... 2023-01-19T07:42:05.466 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-01-19T07:42:05.492 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-01-19T07:42:05.558 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-01-19T07:42:05.559 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-01-19T07:42:05.575 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-19T07:42:05.663 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-01-19T07:42:05.708 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-19T07:42:05.795 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-01-19T07:42:05.834 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2023-01-19T07:42:05.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-01-19T07:42:05.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2023-01-19T07:42:06.002 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-01-19T07:42:06.471 INFO:teuthology.run_tasks:Running task internal.timer... 2023-01-19T07:42:06.481 INFO:teuthology.task.internal:Starting timer... 2023-01-19T07:42:06.481 INFO:teuthology.run_tasks:Running task pcp... 2023-01-19T07:42:06.572 INFO:teuthology.run_tasks:Running task selinux... 2023-01-19T07:42:06.634 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-01-19T07:42:06.634 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2023-01-19T07:42:06.694 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2023-01-19T07:42:06.696 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-01-19T07:42:06.696 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2023-01-19T07:42:06.719 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2023-01-19T07:42:06.720 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi100.front.sepia.ceph.com': 'permissive'} 2023-01-19T07:42:06.720 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|comm="sss_cache"\|context=system_u:system_r:NetworkManager_dispatcher_t:s0\)' 2023-01-19T07:42:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-01-19T07:42:06.787 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-01-19T07:42:06.788 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2023-01-19T07:42:06.854 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-01-19T07:42:06.870 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-01-19T07:42:06.874 INFO:teuthology.repo_utils:Fetching main from origin 2023-01-19T07:42:06.983 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-01-19T07:42:06.998 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-01-19T07:42:06.999 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi100.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-01-19T07:52:26.916 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/gabrioux-2023-01-19_06:14:52-orch:cephadm-wip-guits-testing-2023-01-18-2355-distro-default-smithi/7129978/ansible_failures.yaml 2023-01-19T07:52:26.917 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/ansible.py", line 422, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/ansible.py", line 265, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/ansible.py", line 291, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/ansible.py", line 316, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: {'smithi100.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkkaMdv4WnWYBDeU5YMA1yM5WbFIiKpra1'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', 'delta': '0:00:01.308511', 'end': '2023-01-19 07:48:03.083038', 'failed': False, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkkaMdv4WnWYBDeU5YMA1yM5WbFIiKpra1'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': '', 'rc': 0, 'start': '2023-01-19 07:48:01.774527', 'stderr': '', 'stderr_lines': [], 'stdout': 'Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.', 'stdout_lines': ['Creating new GPT entries.', 'Warning: The kernel is still using the old partition table.', 'The new table will be used at the next reboot or after you', 'run partprobe(8) or kpartx(8)', 'GPT data structures destroyed! You may now partition the disk using fdisk or', 'other utilities.']}, {'_ansible_item_label': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-dBCeol-z2RH-Y8j1-xG16-Afi4-Q1iH-xKKuC3', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400WR400BGN', 'nvme-nvme.8086-50484654363230343030575234303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-dBCeol-z2RH-Y8j1-xG16-Afi4-Q1iH-xKKuC3', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400WR400BGN', 'nvme-nvme.8086-50484654363230343030575234303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi100.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkSSiN2rzYbuk30YAZQVRvlu1w1Hxq67NX'], 'labels': [], 'masters': [], 'uuids': ['6b4cd949-f25b-4592-9d33-8b97e5d86250']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkSSiN2rzYbuk30YAZQVRvlu1w1Hxq67NX'], 'labels': [], 'masters': [], 'uuids': ['6b4cd949-f25b-4592-9d33-8b97e5d86250']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi100.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkRfm2AJn3BWZmI6OOXZVpxcTcT0r8pSAL'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkRfm2AJn3BWZmI6OOXZVpxcTcT0r8pSAL'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi100.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9Ck3Nipbjri67TVYlIhjrktGnj4Se6MNdPB'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9Ck3Nipbjri67TVYlIhjrktGnj4Se6MNdPB'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi100.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QKL1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QKL1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QKL1', 'scsi-35000c50091e1a316', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QKL1', 'wwn-0x5000c50091e1a316'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QKL1-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QKL1-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QKL1-part1', 'scsi-35000c50091e1a316-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QKL1-part1', 'wwn-0x5000c50091e1a316-part1'], 'labels': [], 'masters': [], 'uuids': ['78886b72-f581-4b1d-8578-0d8161fd4ca1']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '78886b72-f581-4b1d-8578-0d8161fd4ca1'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W5QKL1', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c50091e1a316'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'item': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QKL1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QKL1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QKL1', 'scsi-35000c50091e1a316', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QKL1', 'wwn-0x5000c50091e1a316'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QKL1-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QKL1-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QKL1-part1', 'scsi-35000c50091e1a316-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QKL1-part1', 'wwn-0x5000c50091e1a316-part1'], 'labels': [], 'masters': [], 'uuids': ['78886b72-f581-4b1d-8578-0d8161fd4ca1']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '78886b72-f581-4b1d-8578-0d8161fd4ca1'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W5QKL1', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c50091e1a316'}}, 'skip_reason': 'Conditional result was False', 'skipped': True}, {'_ansible_item_label': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkYIFa4JgCAhvkwtZdrs4zzDrSR3eHcffe'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkYIFa4JgCAhvkwtZdrs4zzDrSR3eHcffe'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi100.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}]}} 2023-01-19T07:52:26.974 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ab6f24b46cb04d27bf6a4b31b1a3d072 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/ansible.py", line 422, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/ansible.py", line 265, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/ansible.py", line 291, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/ansible.py", line 316, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: {'smithi100.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkkaMdv4WnWYBDeU5YMA1yM5WbFIiKpra1'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', 'delta': '0:00:01.308511', 'end': '2023-01-19 07:48:03.083038', 'failed': False, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkkaMdv4WnWYBDeU5YMA1yM5WbFIiKpra1'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': '', 'rc': 0, 'start': '2023-01-19 07:48:01.774527', 'stderr': '', 'stderr_lines': [], 'stdout': 'Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.', 'stdout_lines': ['Creating new GPT entries.', 'Warning: The kernel is still using the old partition table.', 'The new table will be used at the next reboot or after you', 'run partprobe(8) or kpartx(8)', 'GPT data structures destroyed! You may now partition the disk using fdisk or', 'other utilities.']}, {'_ansible_item_label': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-dBCeol-z2RH-Y8j1-xG16-Afi4-Q1iH-xKKuC3', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400WR400BGN', 'nvme-nvme.8086-50484654363230343030575234303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-dBCeol-z2RH-Y8j1-xG16-Afi4-Q1iH-xKKuC3', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400WR400BGN', 'nvme-nvme.8086-50484654363230343030575234303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi100.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkSSiN2rzYbuk30YAZQVRvlu1w1Hxq67NX'], 'labels': [], 'masters': [], 'uuids': ['6b4cd949-f25b-4592-9d33-8b97e5d86250']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkSSiN2rzYbuk30YAZQVRvlu1w1Hxq67NX'], 'labels': [], 'masters': [], 'uuids': ['6b4cd949-f25b-4592-9d33-8b97e5d86250']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi100.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkRfm2AJn3BWZmI6OOXZVpxcTcT0r8pSAL'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkRfm2AJn3BWZmI6OOXZVpxcTcT0r8pSAL'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi100.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9Ck3Nipbjri67TVYlIhjrktGnj4Se6MNdPB'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9Ck3Nipbjri67TVYlIhjrktGnj4Se6MNdPB'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi100.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QKL1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QKL1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QKL1', 'scsi-35000c50091e1a316', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QKL1', 'wwn-0x5000c50091e1a316'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QKL1-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QKL1-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QKL1-part1', 'scsi-35000c50091e1a316-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QKL1-part1', 'wwn-0x5000c50091e1a316-part1'], 'labels': [], 'masters': [], 'uuids': ['78886b72-f581-4b1d-8578-0d8161fd4ca1']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '78886b72-f581-4b1d-8578-0d8161fd4ca1'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W5QKL1', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c50091e1a316'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'item': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QKL1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QKL1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QKL1', 'scsi-35000c50091e1a316', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QKL1', 'wwn-0x5000c50091e1a316'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QKL1-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QKL1-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QKL1-part1', 'scsi-35000c50091e1a316-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QKL1-part1', 'wwn-0x5000c50091e1a316-part1'], 'labels': [], 'masters': [], 'uuids': ['78886b72-f581-4b1d-8578-0d8161fd4ca1']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '78886b72-f581-4b1d-8578-0d8161fd4ca1'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W5QKL1', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c50091e1a316'}}, 'skip_reason': 'Conditional result was False', 'skipped': True}, {'_ansible_item_label': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkYIFa4JgCAhvkwtZdrs4zzDrSR3eHcffe'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-ufyEq3hgqmkP3VvANznZuqWvwNL8e9CkYIFa4JgCAhvkwtZdrs4zzDrSR3eHcffe'], 'labels': [], 'masters': [], 'uuids': []}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi100.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}]}} 2023-01-19T07:52:26.976 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-01-19T07:52:26.984 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-01-19T07:52:26.984 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-01-19T07:52:27.021 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-01-19T07:52:27.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-19T07:52:29.576 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2023-01-19T07:52:29.577 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 164, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 194, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi100 2023-01-19T07:52:29.581 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-01-19T07:52:29.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-01-19T07:52:29.601 INFO:teuthology.task.internal:Duration was 623.119848 seconds 2023-01-19T07:52:29.601 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-01-19T07:52:29.633 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-01-19T07:52:29.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-01-19T07:52:29.633 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-19T07:52:32.648 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2023-01-19T07:52:32.649 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 164, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 194, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi100 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi100 2023-01-19T07:52:32.655 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-01-19T07:52:32.663 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-01-19T07:52:32.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-01-19T07:52:32.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-19T07:52:35.720 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2023-01-19T07:52:35.721 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/internal/__init__.py", line 453, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 164, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 194, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi100 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/internal/__init__.py", line 456, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi100 2023-01-19T07:52:35.727 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-01-19T07:52:35.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-01-19T07:52:35.737 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-19T07:52:38.792 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2023-01-19T07:52:38.793 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/internal/__init__.py", line 489, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 164, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 194, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi100 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/internal/__init__.py", line 493, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi100 2023-01-19T07:52:38.799 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-01-19T07:52:38.808 INFO:teuthology.task.internal:Transferring archived files... 2023-01-19T07:52:38.809 DEBUG:teuthology.misc:Transferring archived files from smithi100:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2023-01-19_06:14:52-orch:cephadm-wip-guits-testing-2023-01-18-2355-distro-default-smithi/7129978/remote/smithi100 2023-01-19T07:52:38.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-01-19T07:52:38.810 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-19T07:52:41.864 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2023-01-19T07:52:41.865 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/internal/__init__.py", line 393, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 164, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 194, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi100 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/internal/__init__.py", line 419, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 664, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi100 2023-01-19T07:52:41.871 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-01-19T07:52:41.881 INFO:teuthology.task.internal:Not uploading archives. 2023-01-19T07:52:41.881 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-01-19T07:52:41.954 INFO:teuthology.task.internal:Tidying up after the test... 2023-01-19T07:52:41.954 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2023-01-19T07:52:41.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-01-19T07:52:44.936 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2023-01-19T07:52:44.937 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/internal/__init__.py", line 49, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 164, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/selinux.py", line 194, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi100 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/task/internal/__init__.py", line 55, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/orchestra/remote.py", line 524, in run raise Exception(f'Cannot connect to remote host {self.shortname}') Exception: Cannot connect to remote host smithi100 2023-01-19T07:52:44.943 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-01-19T07:52:44.952 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: Cannot connect to remote host smithi100 2023-01-19T07:52:44.952 INFO:teuthology.nuke:Checking targets against current locks 2023-01-19T07:52:44.984 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run.py", line 411, in main run_tasks(tasks=config['tasks'], ctx=fake_ctx) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run_tasks.py", line 216, in run_tasks raise SystemExit(1) SystemExit: 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/virtualenv/bin/teuthology", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run.py", line 414, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/run.py", line 271, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_teuthology_5abecab36c9da4d70461d81781464cf5acaea292/teuthology/nuke/__init__.py", line 246, in nuke if ctx.name and ctx.name not in status.get('description', ""): TypeError: argument of type 'NoneType' is not iterable