2022-02-06T17:59:50.249 INFO:root:teuthology version: 1.1.0-3094160c 2022-02-06T17:59:50.371 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi067.front.sepia.ceph.com' reimaging 2022-02-06T17:59:50.372 INFO:teuthology.lock.ops:Updating [ubuntu@smithi067.front.sepia.ceph.com]: reset os type and version on server 2022-02-06T17:59:50.373 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2022-02-06T17:59:50.390 INFO:teuthology.lock.ops:Node 'ubuntu@smithi067.front.sepia.ceph.com' reimaging is complete 2022-02-06T17:59:50.390 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi103.front.sepia.ceph.com' reimaging 2022-02-06T17:59:50.390 INFO:teuthology.lock.ops:Updating [ubuntu@smithi103.front.sepia.ceph.com]: reset os type and version on server 2022-02-06T17:59:50.391 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2022-02-06T17:59:50.410 INFO:teuthology.lock.ops:Node 'ubuntu@smithi103.front.sepia.ceph.com' reimaging is complete 2022-02-06T17:59:50.598 INFO:teuthology.provision.fog.smithi067:Scheduling deploy of centos 8.stream 2022-02-06T17:59:50.612 INFO:teuthology.provision.fog.smithi103:Scheduling deploy of centos 8.stream 2022-02-06T17:59:50.966 INFO:teuthology.orchestra.console:Power off smithi067 2022-02-06T17:59:50.983 INFO:teuthology.orchestra.console:Power off smithi103 2022-02-06T17:59:51.112 INFO:teuthology.orchestra.console:Power off for smithi067 completed 2022-02-06T17:59:51.127 INFO:teuthology.orchestra.console:Power off for smithi103 completed 2022-02-06T17:59:51.214 INFO:teuthology.orchestra.console:Power on smithi067 2022-02-06T17:59:51.228 INFO:teuthology.orchestra.console:Power on smithi103 2022-02-06T17:59:59.584 INFO:teuthology.orchestra.console:Power on for smithi067 completed 2022-02-06T17:59:59.589 INFO:teuthology.orchestra.console:Power on for smithi103 completed 2022-02-06T17:59:59.686 INFO:teuthology.provision.fog.smithi067:Waiting for deploy to finish 2022-02-06T17:59:59.691 INFO:teuthology.provision.fog.smithi103:Waiting for deploy to finish 2022-02-06T18:04:43.937 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-06T18:04:44.199 INFO:teuthology.provision.fog.smithi067:Node is ready 2022-02-06T18:04:44.217 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067.front.sepia.ceph.com 2022-02-06T18:04:44.276 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67 smithi067.front.sepia.ceph.com smithi067 2022-02-06T18:04:44.630 INFO:teuthology.provision.fog.smithi067:Deploy complete! 2022-02-06T18:06:50.356 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-06T18:07:10.566 INFO:teuthology.provision.fog.smithi103:Node is ready 2022-02-06T18:07:10.591 INFO:teuthology.orchestra.run.smithi103.stdout:smithi103.front.sepia.ceph.com 2022-02-06T18:07:10.650 INFO:teuthology.orchestra.run.smithi103.stdout:172.21.15.103 smithi103.front.sepia.ceph.com smithi103 2022-02-06T18:07:11.017 INFO:teuthology.provision.fog.smithi103:Deploy complete! 2022-02-06T18:07:11.185 INFO:teuthology.lock.ops:Checking smithi067.front.sepia.ceph.com 2022-02-06T18:07:11.187 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-06T18:07:11.212 INFO:teuthology.lock.ops:Checking smithi103.front.sepia.ceph.com 2022-02-06T18:07:11.213 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-06T18:07:11.251 INFO:teuthology.lock.ops:Updating [smithi067.front.sepia.ceph.com]: set os type and version on server 2022-02-06T18:07:11.252 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2022-02-06T18:07:11.556 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2022-02-06T18:07:11.615 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2022-02-06T18:07:11.615 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="8" 2022-02-06T18:07:11.615 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2022-02-06T18:07:11.615 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2022-02-06T18:07:11.616 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="8" 2022-02-06T18:07:11.616 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el8" 2022-02-06T18:07:11.616 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 8" 2022-02-06T18:07:11.616 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2022-02-06T18:07:11.617 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-02-06T18:07:11.617 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2022-02-06T18:07:11.617 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-06T18:07:11.617 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-06T18:07:11.618 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-02-06T18:07:11.619 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2022-02-06T18:07:11.644 INFO:teuthology.lock.ops:Updating [smithi103.front.sepia.ceph.com]: set os type and version on server 2022-02-06T18:07:11.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2022-02-06T18:07:11.842 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2022-02-06T18:07:11.905 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2022-02-06T18:07:11.906 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="8" 2022-02-06T18:07:11.906 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2022-02-06T18:07:11.906 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2022-02-06T18:07:11.907 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="8" 2022-02-06T18:07:11.907 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el8" 2022-02-06T18:07:11.907 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 8" 2022-02-06T18:07:11.907 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2022-02-06T18:07:11.908 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-02-06T18:07:11.909 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2022-02-06T18:07:11.909 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-06T18:07:11.909 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-06T18:07:11.910 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-02-06T18:07:11.911 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2022-02-06T18:07:12.027 INFO:teuthology.dispatcher.supervisor:Running job 6665097 2022-02-06T18:07:12.027 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-02-06_17:57:58-rados-safe_shutdown_v2-distro-basic-smithi/6665097 --name benhanokh-2022-02-06_17:57:58-rados-safe_shutdown_v2-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/pggrow thrashosds-health workloads/cache} -- /home/teuthworker/archive/benhanokh-2022-02-06_17:57:58-rados-safe_shutdown_v2-distro-basic-smithi/6665097/orig.config.yaml 2022-02-06T18:07:12.035 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-02-06_17:57:58-rados-safe_shutdown_v2-distro-basic-smithi/6665097 2022-02-06T18:07:12.036 INFO:teuthology.dispatcher.supervisor:Job PID: 22353 2022-02-06T18:07:12.037 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-07T00:37:41.430 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-07T00:37:41.783 INFO:teuthology.kill:Killing Pids: {22353} 2022-02-07T00:37:42.948 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-02-07T00:37:42.949 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-02-07T00:37:42.949 INFO:teuthology.misc:Compressing logs... 2022-02-07T00:37:42.950 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2022-02-07T00:37:43.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2022-02-07T00:37:43.573 INFO:teuthology.misc:Compressing logs... 2022-02-07T00:37:44.140 INFO:teuthology.orchestra.run.smithi067.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-02-07T00:37:44.243 INFO:teuthology.orchestra.run.smithi103.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-02-07T00:37:44.627 INFO:teuthology.misc:Compressing logs... 2022-02-07T00:37:57.958 INFO:teuthology.kill:No teuthology processes running 2022-02-07T00:37:59.082 INFO:teuthology.kill:Nuking machines: ['smithi067', 'smithi103'] 2022-02-07T00:38:00.059 INFO:teuthology.kill:2022-02-07 00:38:00,059.059 INFO:teuthology.nuke:targets: 2022-02-07T00:38:00.060 INFO:teuthology.kill: smithi067.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDewq9cl3TG0bS+fcuTvtsV2jlnbrmDVU1IBIvP4bfU6wA3HISm7kCH+aCoqwvslms24dJZilhTxfjplZcuYSezu7ZKkaawtsazU44rbu2HgE4gESlZSbXduRilm45GazJZsqbaB741nUytUEsQqw3yPvFjl+EZGxW2L71iNSuDgs3dZB5Y9TVIxvncn+bX/mcYs6FG+TfySKsn6RAkwMcruzBQ8TZcU7NHkV/bAyV8kBkKLJoP+oB1FV/tTdKAAzvldm9ekHwfyTQ7A1V4/OEe85fw1fgi/8yely5trIYY5AdFsE0pRCndKTLMsaRWUf2Rm+wyK9tj7zS9WafgpfPGQai2YegG0IJdzvjIUENd8jw56PThtzrajgIeh7a/SC2FMSxEJKrhLyffKUz5NiIShhCwYPIcJ8s3j7MGo4tFCQOHG9D4BKjKEP1ASNX3de2rjRAORSaZixM3H1RoEkUmGkKyPVHIGvTTfSPHXsD+RTBVCPSd3JX2bdASiOpwSg8= 2022-02-07T00:38:00.060 INFO:teuthology.kill: smithi103.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCa5/7Ov+JtxepoxZTtisGv7CAl/VUYCO4bY7LyX+2iPGlIHHfRHDLBlifv8MiIMS8Vfqk2txD1NyYrvqFzx8Q3zurTxqzUNgSmcrjQGWIuESOoSO1+zh/i/8X7Y5izrC+GT7pt6lW6/9RzzpC8SsJ9wBxY908dHPNVZWVAUGeu9S+wfU0LOrTrz6bsMiLEmupKeKxjEPGjuO+mukV9IYAYODW+RFnbYL3RPgIh+ZGlvonp6JztNG3PX5UIcrZL/I3g47Vmsdj4EcBT7kJStM8RvSjLZUbQpkM58Nm+m/ZQJuRLKMzSlLInWabA1KyZLiQ4pdfwB4iXsuQfctHebzM9zR4+LEuOP1yL9oAdFYFsA0Yqg8CtrorVLyd43iVGicNacTjCvQ8VBeci2AMHMYkcjwnA8ShB2r+3pz5NhvRUV0FFhisG8hPIvhoa/JwY84afkk8FOgDS5egrYJOIpwqlhvRe1fv0uBJ0WChPSw+ea92SsQa3E67xXKhyqWvo/9k= 2022-02-07T00:38:00.086 INFO:teuthology.kill:2022-02-07 00:38:00,086.086 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-07T00:38:00.090 INFO:teuthology.kill:2022-02-07 00:38:00,090.090 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-07T00:38:00.167 INFO:teuthology.kill:2022-02-07 00:38:00,166.166 INFO:teuthology.orchestra.console:Power off smithi103 2022-02-07T00:38:00.205 INFO:teuthology.kill:2022-02-07 00:38:00,205.205 INFO:teuthology.orchestra.console:Power off smithi067 2022-02-07T00:38:08.567 INFO:teuthology.kill:2022-02-07 00:38:08,567.567 INFO:teuthology.orchestra.console:Power off for smithi067 completed 2022-02-07T00:38:08.712 INFO:teuthology.kill:2022-02-07 00:38:08,711.711 INFO:teuthology.lock.ops:unlocked: smithi067.front.sepia.ceph.com 2022-02-07T00:38:12.657 INFO:teuthology.kill:2022-02-07 00:38:12,656.656 INFO:teuthology.orchestra.console:Power off for smithi103 completed 2022-02-07T00:38:12.801 INFO:teuthology.kill:2022-02-07 00:38:12,801.801 INFO:teuthology.lock.ops:unlocked: smithi103.front.sepia.ceph.com 2022-02-07T00:40:13.113 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-02-07T00:40:13.216 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi067 but it was locked by another job: /home/teuthworker/archive/dis-2022-02-06_22:53:48-rbd-wip-rbd-quincy-batch-2-distro-default-smithi/6665475 2022-02-07T00:40:13.217 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi103 but it was locked by another job: /home/teuthworker/archive/dis-2022-02-06_22:53:48-rbd-wip-rbd-quincy-batch-2-distro-default-smithi/6665475