2021-12-05T17:18:43.760 INFO:root:teuthology version: 1.1.0-844d1377 2021-12-05T17:18:43.882 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging 2021-12-05T17:18:43.883 INFO:teuthology.lock.ops:Updating [ubuntu@smithi090.front.sepia.ceph.com]: reset os type and version on server 2021-12-05T17:18:43.883 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2021-12-05T17:18:43.902 INFO:teuthology.lock.ops:Node 'ubuntu@smithi090.front.sepia.ceph.com' reimaging is complete 2021-12-05T17:18:43.903 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi104.front.sepia.ceph.com' reimaging 2021-12-05T17:18:43.903 INFO:teuthology.lock.ops:Updating [ubuntu@smithi104.front.sepia.ceph.com]: reset os type and version on server 2021-12-05T17:18:43.904 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2021-12-05T17:18:43.924 INFO:teuthology.lock.ops:Node 'ubuntu@smithi104.front.sepia.ceph.com' reimaging is complete 2021-12-05T17:18:44.102 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of centos 8.stream 2021-12-05T17:18:44.123 INFO:teuthology.provision.fog.smithi104:Scheduling deploy of centos 8.stream 2021-12-05T17:18:44.445 INFO:teuthology.orchestra.console:Power off smithi090 2021-12-05T17:18:44.459 INFO:teuthology.orchestra.console:Power off smithi104 2021-12-05T17:18:52.815 INFO:teuthology.orchestra.console:Power off for smithi090 completed 2021-12-05T17:18:52.825 INFO:teuthology.orchestra.console:Power off for smithi104 completed 2021-12-05T17:18:52.917 INFO:teuthology.orchestra.console:Power on smithi090 2021-12-05T17:18:52.927 INFO:teuthology.orchestra.console:Power on smithi104 2021-12-05T17:19:01.287 INFO:teuthology.orchestra.console:Power on for smithi090 completed 2021-12-05T17:19:01.293 INFO:teuthology.orchestra.console:Power on for smithi104 completed 2021-12-05T17:19:01.389 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish 2021-12-05T17:19:01.394 INFO:teuthology.provision.fog.smithi104:Waiting for deploy to finish 2021-12-05T17:23:51.004 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-05T17:23:51.274 INFO:teuthology.provision.fog.smithi104:Node is ready 2021-12-05T17:23:51.291 INFO:teuthology.orchestra.run.smithi104.stdout:smithi104.front.sepia.ceph.com 2021-12-05T17:23:51.348 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104 smithi104.front.sepia.ceph.com smithi104 2021-12-05T17:23:51.681 INFO:teuthology.provision.fog.smithi104:Deploy complete! 2021-12-05T17:24:18.340 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-05T17:24:18.669 INFO:teuthology.provision.fog.smithi090:Node is ready 2021-12-05T17:24:18.686 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090.front.sepia.ceph.com 2021-12-05T17:24:18.743 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90 smithi090.front.sepia.ceph.com smithi090 2021-12-05T17:24:19.100 INFO:teuthology.provision.fog.smithi090:Deploy complete! 2021-12-05T17:24:19.283 INFO:teuthology.lock.ops:Checking smithi090.front.sepia.ceph.com 2021-12-05T17:24:19.284 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-05T17:24:19.303 INFO:teuthology.lock.ops:Checking smithi104.front.sepia.ceph.com 2021-12-05T17:24:19.304 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-05T17:24:19.321 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: set os type and version on server 2021-12-05T17:24:19.322 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2021-12-05T17:24:19.509 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64 2021-12-05T17:24:19.565 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream" 2021-12-05T17:24:19.565 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8" 2021-12-05T17:24:19.565 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos" 2021-12-05T17:24:19.566 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora" 2021-12-05T17:24:19.566 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8" 2021-12-05T17:24:19.566 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8" 2021-12-05T17:24:19.566 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-05T17:24:19.566 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31" 2021-12-05T17:24:19.567 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-05T17:24:19.567 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/" 2021-12-05T17:24:19.567 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-05T17:24:19.567 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-05T17:24:19.568 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-05T17:24:19.569 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server 2021-12-05T17:24:19.587 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: set os type and version on server 2021-12-05T17:24:19.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2021-12-05T17:24:19.827 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2021-12-05T17:24:19.883 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2021-12-05T17:24:19.883 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="8" 2021-12-05T17:24:19.883 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2021-12-05T17:24:19.884 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2021-12-05T17:24:19.884 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="8" 2021-12-05T17:24:19.884 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el8" 2021-12-05T17:24:19.884 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 8" 2021-12-05T17:24:19.885 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2021-12-05T17:24:19.885 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-12-05T17:24:19.885 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2021-12-05T17:24:19.885 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-05T17:24:19.885 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-05T17:24:19.886 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2021-12-05T17:24:19.886 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2021-12-05T17:24:20.003 INFO:teuthology.dispatcher.supervisor:Running job 6546681 2021-12-05T17:24:20.003 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-12-05_15:06:27-rados-WIP_GBH_safe_fast_shutdown_1-distro-basic-smithi/6546681 --name benhanokh-2021-12-05_15:06:27-rados-WIP_GBH_safe_fast_shutdown_1-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_8.stream} thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} -- /home/teuthworker/archive/benhanokh-2021-12-05_15:06:27-rados-WIP_GBH_safe_fast_shutdown_1-distro-basic-smithi/6546681/orig.config.yaml 2021-12-05T17:24:20.007 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-12-05_15:06:27-rados-WIP_GBH_safe_fast_shutdown_1-distro-basic-smithi/6546681 2021-12-05T17:24:20.008 INFO:teuthology.dispatcher.supervisor:Job PID: 4847 2021-12-05T17:24:20.008 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-12-06T05:25:14.188 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-12-06T05:25:14.350 INFO:teuthology.kill:Killing Pids: {4847} 2021-12-06T05:25:15.554 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-12-06T05:25:15.555 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-12-06T05:25:15.555 INFO:teuthology.misc:Compressing logs... 2021-12-06T05:25:15.556 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com' 2021-12-06T05:25:15.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2021-12-06T05:25:16.049 INFO:teuthology.misc:Compressing logs... 2021-12-06T05:25:16.961 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-06T05:25:17.295 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-06T05:25:17.911 INFO:teuthology.misc:Compressing logs... 2021-12-06T05:25:24.292 INFO:teuthology.kill:No teuthology processes running 2021-12-06T05:25:25.458 INFO:teuthology.kill:Nuking machines: ['smithi090', 'smithi104'] 2021-12-06T05:25:26.459 INFO:teuthology.kill:2021-12-06 05:25:26,459.459 INFO:teuthology.nuke:targets: 2021-12-06T05:25:26.460 INFO:teuthology.kill: smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgL+Kqbrp5vNZTA1++L9xnQEP+/LtGUoti/lMXAUOtijoHqm6NbZQade7WtZBMODcYaATEzYVt6pT7U/zFCVMjqVDAKmID84C8nSDnsed12uKbsYnKHyOz++EUZ+UGDMGnn9Y2jffQPs1lR+whGfLahmqKcD8NgELNir6nU3Y/Cm3q+I5q4SvbDh18t7KO02RmCickTGCjHRaiU6Ld4EkIZg/gSm0lOKB138sGpw2BRJqvPnYfyT0nXb16gH+u6ApIL9MKPRLwxWALoGjNbHB1MSMBPZL3Ocv4zCVzsYBzH3YBmrYUhIkydtvuveU4ijt7T8yJ9+ff/dGmNy+2hxPTg3x3PeoHEJEOZgzSx/2N/A/pYfT0kWD8y51v+ppRcgcJB42NIwJSqOQT8K4OqYy0sNsVG4KDQ733Sqf68na+KDW6EG6W7kDV2osYGmjZE32Fik2knMbS3LES7ZLWyfpxqxB3n+3AO+eJLHpaRpEpwfvGCBXwFT/3kw9BHBuFBLE= 2021-12-06T05:25:26.460 INFO:teuthology.kill: smithi104.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDL1gcnNZV+Ii+V8Ji9jM/wp64uBtcJukZjKck+ezImrZsmNl/7lDVyItouC4VbTmz37GkFoSLHZWe99pADdHHTrSFKPFeIA8QCsqNWkIAgleio4wgPDT3xDrzZYCmtv+MqyGFW4yHp1kwTVRqW4CuO2enbjZXSsq1vZV6CO6lZKiIzKON/0D568JTQll8wHlfwfiID7JC8iYHZNmvhQ4iZVhLLv3JJ1ab6ea7vCVK2NttxN+gUJyFL9zqkX/NlSacvmdK9VLTSu/iHeRXS4E2CbpSGUgzgTg12/zJpExqPmIMm67E+DpAZ6pwtc8jFGlX43t9mCwD6VicenyhQYvZXDiSFYYbGQI6GZzhV3NRIuIbO0Z39MOtNsfjJNgkv6hMZOA639S+9ysZ55R9KtzAJVnm/UvvKsnAHS6zKcTkDUNOGuyDj9e00KJ0bntvozj+xXIkatJFYrRyDwFpKd2V7zUR2CSvs9hCK4F3O3amEb+lX4S1tHLZb3ZR4PpfjO1U= 2021-12-06T05:25:26.487 INFO:teuthology.kill:2021-12-06 05:25:26,486.486 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-06T05:25:26.491 INFO:teuthology.kill:2021-12-06 05:25:26,491.491 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-06T05:25:26.567 INFO:teuthology.kill:2021-12-06 05:25:26,567.567 INFO:teuthology.orchestra.console:Power off smithi090 2021-12-06T05:25:26.580 INFO:teuthology.kill:2021-12-06 05:25:26,580.580 INFO:teuthology.orchestra.console:Power off smithi104 2021-12-06T05:25:34.940 INFO:teuthology.kill:2021-12-06 05:25:34,939.939 INFO:teuthology.orchestra.console:Power off for smithi090 completed 2021-12-06T05:25:34.947 INFO:teuthology.kill:2021-12-06 05:25:34,947.947 INFO:teuthology.orchestra.console:Power off for smithi104 completed 2021-12-06T05:25:35.084 INFO:teuthology.kill:2021-12-06 05:25:35,084.084 INFO:teuthology.lock.ops:unlocked: smithi090.front.sepia.ceph.com 2021-12-06T05:25:35.090 INFO:teuthology.kill:2021-12-06 05:25:35,090.090 INFO:teuthology.lock.ops:unlocked: smithi104.front.sepia.ceph.com 2021-12-06T05:27:35.409 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-12-06T05:27:35.516 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi090 but it was locked by another job: /home/teuthworker/archive/vshankar-2021-12-06_05:23:54-orch:cephadm-wip-vshankar-testing-20211124-094330-distro-default-smithi/6547339 2021-12-06T05:27:35.517 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi104 but it was locked by another job: /home/teuthworker/archive/vshankar-2021-12-06_05:23:54-orch:cephadm-wip-vshankar-testing-20211124-094330-distro-default-smithi/6547339