2021-03-14T14:02:25.331 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-14T14:02:25.444 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi003.front.sepia.ceph.com' reimaging 2021-03-14T14:02:25.445 INFO:teuthology.lock.ops:Updating [ubuntu@smithi003.front.sepia.ceph.com]: reset os type and version on server 2021-03-14T14:02:25.446 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2021-03-14T14:02:25.465 INFO:teuthology.lock.ops:Node 'ubuntu@smithi003.front.sepia.ceph.com' reimaging is complete 2021-03-14T14:02:25.466 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi068.front.sepia.ceph.com' reimaging 2021-03-14T14:02:25.466 INFO:teuthology.lock.ops:Updating [ubuntu@smithi068.front.sepia.ceph.com]: reset os type and version on server 2021-03-14T14:02:25.466 INFO:teuthology.lock.ops:Updating smithi068.front.sepia.ceph.com on lock server 2021-03-14T14:02:25.480 INFO:teuthology.lock.ops:Node 'ubuntu@smithi068.front.sepia.ceph.com' reimaging is complete 2021-03-14T14:02:25.664 INFO:teuthology.provision.fog.smithi003:Scheduling deploy of rhel 8.3 2021-03-14T14:02:25.674 INFO:teuthology.provision.fog.smithi068:Scheduling deploy of rhel 8.3 2021-03-14T14:02:26.031 INFO:teuthology.orchestra.console:Power off smithi068 2021-03-14T14:02:26.050 INFO:teuthology.orchestra.console:Power off smithi003 2021-03-14T14:02:26.196 INFO:teuthology.orchestra.console:Power off for smithi068 completed 2021-03-14T14:02:26.208 INFO:teuthology.orchestra.console:Power off for smithi003 completed 2021-03-14T14:02:26.297 INFO:teuthology.orchestra.console:Power on smithi068 2021-03-14T14:02:26.310 INFO:teuthology.orchestra.console:Power on smithi003 2021-03-14T14:02:34.674 INFO:teuthology.orchestra.console:Power on for smithi068 completed 2021-03-14T14:02:34.678 INFO:teuthology.orchestra.console:Power on for smithi003 completed 2021-03-14T14:02:34.775 INFO:teuthology.provision.fog.smithi068:Waiting for deploy to finish 2021-03-14T14:02:34.780 INFO:teuthology.provision.fog.smithi003:Waiting for deploy to finish 2021-03-14T14:06:51.598 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-14T14:06:51.943 INFO:teuthology.provision.fog.smithi003:Node is ready 2021-03-14T14:06:51.958 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003.front.sepia.ceph.com 2021-03-14T14:06:52.013 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3 smithi003.front.sepia.ceph.com smithi003 2021-03-14T14:06:52.340 INFO:teuthology.provision.fog.smithi003:Deploy complete! 2021-03-14T14:06:56.472 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-03-14T14:07:17.192 INFO:teuthology.provision.fog.smithi068:Node is ready 2021-03-14T14:07:17.211 INFO:teuthology.orchestra.run.smithi068.stdout:smithi068.front.sepia.ceph.com 2021-03-14T14:07:17.268 INFO:teuthology.orchestra.run.smithi068.stdout:172.21.15.68 smithi068.front.sepia.ceph.com smithi068 2021-03-14T14:07:17.630 INFO:teuthology.provision.fog.smithi068:Deploy complete! 2021-03-14T14:07:17.823 INFO:teuthology.lock.ops:Checking smithi003.front.sepia.ceph.com 2021-03-14T14:07:17.824 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-14T14:07:17.840 INFO:teuthology.lock.ops:Checking smithi068.front.sepia.ceph.com 2021-03-14T14:07:17.841 INFO:teuthology.lock.ops:New key found. Updating... 2021-03-14T14:07:17.856 INFO:teuthology.lock.ops:Updating [smithi003.front.sepia.ceph.com]: set os type and version on server 2021-03-14T14:07:17.856 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-14T14:07:18.157 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2021-03-14T14:07:18.212 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="Red Hat Enterprise Linux" 2021-03-14T14:07:18.212 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="8.3 (Ootpa)" 2021-03-14T14:07:18.212 INFO:teuthology.orchestra.run.smithi003.stdout:ID="rhel" 2021-03-14T14:07:18.213 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="fedora" 2021-03-14T14:07:18.213 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="8.3" 2021-03-14T14:07:18.213 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el8" 2021-03-14T14:07:18.213 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-14T14:07:18.214 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2021-03-14T14:07:18.214 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-14T14:07:18.214 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://www.redhat.com/" 2021-03-14T14:07:18.214 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-14T14:07:18.214 INFO:teuthology.orchestra.run.smithi003.stdout: 2021-03-14T14:07:18.215 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-14T14:07:18.215 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-14T14:07:18.215 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-14T14:07:18.215 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-14T14:07:18.216 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2021-03-14T14:07:18.233 INFO:teuthology.lock.ops:Updating [smithi068.front.sepia.ceph.com]: set os type and version on server 2021-03-14T14:07:18.234 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-14T14:07:18.523 INFO:teuthology.orchestra.run.smithi068.stdout:x86_64 2021-03-14T14:07:18.577 INFO:teuthology.orchestra.run.smithi068.stdout:NAME="Red Hat Enterprise Linux" 2021-03-14T14:07:18.577 INFO:teuthology.orchestra.run.smithi068.stdout:VERSION="8.3 (Ootpa)" 2021-03-14T14:07:18.577 INFO:teuthology.orchestra.run.smithi068.stdout:ID="rhel" 2021-03-14T14:07:18.578 INFO:teuthology.orchestra.run.smithi068.stdout:ID_LIKE="fedora" 2021-03-14T14:07:18.578 INFO:teuthology.orchestra.run.smithi068.stdout:VERSION_ID="8.3" 2021-03-14T14:07:18.578 INFO:teuthology.orchestra.run.smithi068.stdout:PLATFORM_ID="platform:el8" 2021-03-14T14:07:18.578 INFO:teuthology.orchestra.run.smithi068.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" 2021-03-14T14:07:18.579 INFO:teuthology.orchestra.run.smithi068.stdout:ANSI_COLOR="0;31" 2021-03-14T14:07:18.579 INFO:teuthology.orchestra.run.smithi068.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" 2021-03-14T14:07:18.579 INFO:teuthology.orchestra.run.smithi068.stdout:HOME_URL="https://www.redhat.com/" 2021-03-14T14:07:18.579 INFO:teuthology.orchestra.run.smithi068.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-03-14T14:07:18.579 INFO:teuthology.orchestra.run.smithi068.stdout: 2021-03-14T14:07:18.580 INFO:teuthology.orchestra.run.smithi068.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-03-14T14:07:18.580 INFO:teuthology.orchestra.run.smithi068.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 2021-03-14T14:07:18.580 INFO:teuthology.orchestra.run.smithi068.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-03-14T14:07:18.580 INFO:teuthology.orchestra.run.smithi068.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.3" 2021-03-14T14:07:18.581 INFO:teuthology.lock.ops:Updating smithi068.front.sepia.ceph.com on lock server 2021-03-14T14:07:18.669 INFO:teuthology.dispatcher.supervisor:Running job 5965024 2021-03-14T14:07:18.669 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/bin/teuthology -v --owner scheduled_kchai@teuthology --archive /home/teuthworker/archive/kchai-2021-03-14_09:40:23-rados-wip-kefu-testing-2021-03-14-1612-distro-basic-smithi/5965024 --name kchai-2021-03-14_09:40:23-rados-wip-kefu-testing-2021-03-14-1612-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/small-objects-localized} -- /home/teuthworker/archive/kchai-2021-03-14_09:40:23-rados-wip-kefu-testing-2021-03-14-1612-distro-basic-smithi/5965024/orig.config.yaml 2021-03-14T14:07:18.673 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/kchai-2021-03-14_09:40:23-rados-wip-kefu-testing-2021-03-14-1612-distro-basic-smithi/5965024 2021-03-14T14:07:18.674 INFO:teuthology.dispatcher.supervisor:Job PID: 24900 2021-03-14T14:07:18.675 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-03-15T02:08:09.023 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-03-15T02:08:09.215 INFO:teuthology.kill:Killing Pids: {24900} 2021-03-15T02:08:11.341 INFO:teuthology.kill:Nuking machines: ['smithi003', 'smithi068'] 2021-03-15T02:08:13.308 INFO:teuthology.kill:2021-03-15 02:08:13,307.307 INFO:teuthology.nuke:targets: 2021-03-15T02:08:13.310 INFO:teuthology.kill: smithi003.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJy6f2gGvQKf/tfsf9M83LsVOfrJdFZtqK+JnG/U5MBjypCym4zrAnVfbUInPAP/K3Sm5wiUQK7POVGApAxuB7NOVhLw6XEwpy3UIB+o7ZzZin8ZTyd0cK18uNCRMpxJLzYxAmhFBBE3WvBkyQiEk4QCVylv/cZ+X+EoRgEMnu0hl9n+d8MCEhHrba5TzeE3sjI6/LNjDPAhv0u4cRnc/EvDjeB8UV0XMcDhEgziYviHYVYZLDdgHMbLxehKcc/eW5fXsjsXb1t1XhisJhYfmNxJJe0qlJ/Zb8gSLEMD37PpQIH3PObQY3DD0jkH5mW4VtcLcoa42t9GSZ8OFH/bVHNzFjqbM/3s4jLsDTIcVN5BIsl/zNyiFgNDWil3qrIAH+CbuJ4tAJetQsb+rR07qrUWxXC4d0b3RGphfVQ7J9qN43MAHtMZfcusiVVOcQ1pm1kpc76xVUhIf9sxFKTSeLcGC28cfVUrgRCYJXd9D0LuNIrsV1VXtVColT8RJHzsk= 2021-03-15T02:08:13.311 INFO:teuthology.kill: smithi068.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj5IVVvVHT84fT+4CZ6aXVyzXakSABOnoGMKf0TFoml59cfGVUxSIZLMGVsEwPWetcuoTLy6mHrEXxmqiG6TVwb0xMVAr26CUME5p5hZiA1Pe5M3tnAFc8MUjHYLL2BRjaoOSkCB9ZRJQBNcOH9q1rf+f5tSUUX14ejvchrpB169l5LHKFzKyN10memF4hhK7/d1T6Np3si44KGOB/zIElFWpPCDaH/yDa3lR6IHdFuWgsVDIS7RqaxJMSGlCrQqmKZ+pd5GDzzjGzzUv79VAka+zsa4JYRqiH4IZzoG68YPsilpBaJmK+1ZEf5wcSZGd6kBrir+Q3/YKSxb7PUI3xAuwUJnCNiOpbt1psdmNrlau6vE3yGWwkphiooDsAsnatYYAQ86GqVBBzz+/qET/VS2az15lD0K2vZWkXTn4+lctSGQOmkKzKi0CXNbbXZN6qJV6fMGTKZkQaPUicDFqLAnNfKhZbBN15CJ2sJbKJqJmVqEq0pTWwBTAm1ZGH1mM= 2021-03-15T02:08:13.311 INFO:teuthology.kill:2021-03-15 02:08:13,308.308 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-15T02:08:13.312 INFO:teuthology.kill:2021-03-15 02:08:13,312.312 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-15T02:08:13.393 INFO:teuthology.kill:2021-03-15 02:08:13,393.393 INFO:teuthology.orchestra.console:Power off smithi003 2021-03-15T02:08:13.413 INFO:teuthology.kill:2021-03-15 02:08:13,412.412 INFO:teuthology.orchestra.console:Power off smithi068 2021-03-15T02:08:25.928 INFO:teuthology.kill:2021-03-15 02:08:25,927.927 INFO:teuthology.orchestra.console:Power off for smithi003 completed 2021-03-15T02:08:25.933 INFO:teuthology.kill:2021-03-15 02:08:25,933.933 INFO:teuthology.orchestra.console:Power off for smithi068 completed 2021-03-15T02:08:26.192 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-03-15T02:08:26.195 INFO:teuthology.task.internal:roles: ubuntu@smithi068.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-03-15T02:08:26.195 INFO:teuthology.misc:Compressing logs... 2021-03-15T02:08:26.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T02:09:26.241 ERROR:teuthology.dispatcher.supervisor:Could not save logs Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 231, in run_with_watchdog teuth_config.archive_base, job_config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/dispatcher/supervisor.py", line 289, in transfer_archives compress_logs(ctx, log_path) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/misc.py", line 1374, in compress_logs wait=False, File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/cluster.py", line 64, in return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/remote.py", line 215, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/teuthology/orchestra/run.py", line 434, in run transport = client.get_transport() AttributeError: 'NoneType' object has no attribute 'get_transport' 2021-03-15T02:09:26.312 INFO:teuthology.kill:No teuthology processes running 2021-03-15T02:09:28.359 INFO:teuthology.kill:Nuking machines: ['smithi003', 'smithi068'] 2021-03-15T02:09:29.675 INFO:teuthology.kill:2021-03-15 02:09:29,674.674 INFO:teuthology.nuke:targets: 2021-03-15T02:09:29.676 INFO:teuthology.kill: smithi003.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJy6f2gGvQKf/tfsf9M83LsVOfrJdFZtqK+JnG/U5MBjypCym4zrAnVfbUInPAP/K3Sm5wiUQK7POVGApAxuB7NOVhLw6XEwpy3UIB+o7ZzZin8ZTyd0cK18uNCRMpxJLzYxAmhFBBE3WvBkyQiEk4QCVylv/cZ+X+EoRgEMnu0hl9n+d8MCEhHrba5TzeE3sjI6/LNjDPAhv0u4cRnc/EvDjeB8UV0XMcDhEgziYviHYVYZLDdgHMbLxehKcc/eW5fXsjsXb1t1XhisJhYfmNxJJe0qlJ/Zb8gSLEMD37PpQIH3PObQY3DD0jkH5mW4VtcLcoa42t9GSZ8OFH/bVHNzFjqbM/3s4jLsDTIcVN5BIsl/zNyiFgNDWil3qrIAH+CbuJ4tAJetQsb+rR07qrUWxXC4d0b3RGphfVQ7J9qN43MAHtMZfcusiVVOcQ1pm1kpc76xVUhIf9sxFKTSeLcGC28cfVUrgRCYJXd9D0LuNIrsV1VXtVColT8RJHzsk= 2021-03-15T02:09:29.677 INFO:teuthology.kill: smithi068.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj5IVVvVHT84fT+4CZ6aXVyzXakSABOnoGMKf0TFoml59cfGVUxSIZLMGVsEwPWetcuoTLy6mHrEXxmqiG6TVwb0xMVAr26CUME5p5hZiA1Pe5M3tnAFc8MUjHYLL2BRjaoOSkCB9ZRJQBNcOH9q1rf+f5tSUUX14ejvchrpB169l5LHKFzKyN10memF4hhK7/d1T6Np3si44KGOB/zIElFWpPCDaH/yDa3lR6IHdFuWgsVDIS7RqaxJMSGlCrQqmKZ+pd5GDzzjGzzUv79VAka+zsa4JYRqiH4IZzoG68YPsilpBaJmK+1ZEf5wcSZGd6kBrir+Q3/YKSxb7PUI3xAuwUJnCNiOpbt1psdmNrlau6vE3yGWwkphiooDsAsnatYYAQ86GqVBBzz+/qET/VS2az15lD0K2vZWkXTn4+lctSGQOmkKzKi0CXNbbXZN6qJV6fMGTKZkQaPUicDFqLAnNfKhZbBN15CJ2sJbKJqJmVqEq0pTWwBTAm1ZGH1mM= 2021-03-15T02:09:29.697 INFO:teuthology.kill:2021-03-15 02:09:29,697.697 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-15T02:09:29.705 INFO:teuthology.kill:2021-03-15 02:09:29,705.705 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-15T02:09:29.771 INFO:teuthology.kill:2021-03-15 02:09:29,770.770 INFO:teuthology.orchestra.console:Power off smithi003 2021-03-15T02:09:29.812 INFO:teuthology.kill:2021-03-15 02:09:29,812.812 INFO:teuthology.orchestra.console:Power off smithi068 2021-03-15T02:09:29.955 INFO:teuthology.kill:2021-03-15 02:09:29,955.955 INFO:teuthology.orchestra.console:Power off for smithi003 completed 2021-03-15T02:09:29.962 INFO:teuthology.kill:2021-03-15 02:09:29,962.962 INFO:teuthology.orchestra.console:Power off for smithi068 completed 2021-03-15T02:09:30.101 INFO:teuthology.kill:2021-03-15 02:09:30,100.100 INFO:teuthology.lock.ops:unlocked smithi003.front.sepia.ceph.com 2021-03-15T02:09:30.110 INFO:teuthology.kill:2021-03-15 02:09:30,108.108 INFO:teuthology.lock.ops:unlocked smithi068.front.sepia.ceph.com 2021-03-15T02:11:30.450 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-03-15T02:11:30.551 INFO:teuthology.dispatcher.supervisor:Nuking machines... 2021-03-15T02:11:30.552 INFO:teuthology.nuke:Checking targets against current locks 2021-03-15T02:11:30.624 INFO:teuthology.nuke:Not nuking smithi003.front.sepia.ceph.com because description doesn't match 2021-03-15T02:11:30.625 INFO:teuthology.nuke:Not nuking smithi068.front.sepia.ceph.com because description doesn't match