2022-03-07T00:28:45.128 INFO:root:teuthology version: 1.1.0-eeaeee38 2022-03-07T00:28:45.264 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi065.front.sepia.ceph.com' reimaging 2022-03-07T00:28:45.265 INFO:teuthology.lock.ops:Updating [ubuntu@smithi065.front.sepia.ceph.com]: reset os type and version on server 2022-03-07T00:28:45.266 INFO:teuthology.lock.ops:Updating smithi065.front.sepia.ceph.com on lock server 2022-03-07T00:28:45.288 INFO:teuthology.lock.ops:Node 'ubuntu@smithi065.front.sepia.ceph.com' reimaging is complete 2022-03-07T00:28:45.289 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi115.front.sepia.ceph.com' reimaging 2022-03-07T00:28:45.289 INFO:teuthology.lock.ops:Updating [ubuntu@smithi115.front.sepia.ceph.com]: reset os type and version on server 2022-03-07T00:28:45.290 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2022-03-07T00:28:45.314 INFO:teuthology.lock.ops:Node 'ubuntu@smithi115.front.sepia.ceph.com' reimaging is complete 2022-03-07T00:28:45.555 INFO:teuthology.provision.fog.smithi065:Scheduling deploy of centos 8.stream 2022-03-07T00:28:45.564 INFO:teuthology.provision.fog.smithi115:Scheduling deploy of centos 8.stream 2022-03-07T00:28:45.943 INFO:teuthology.orchestra.console:Power off smithi115 2022-03-07T00:28:45.984 INFO:teuthology.orchestra.console:Power off smithi065 2022-03-07T00:28:58.455 INFO:teuthology.orchestra.console:Power off for smithi115 completed 2022-03-07T00:28:58.463 INFO:teuthology.orchestra.console:Power off for smithi065 completed 2022-03-07T00:28:58.558 INFO:teuthology.orchestra.console:Power on smithi115 2022-03-07T00:28:58.571 INFO:teuthology.orchestra.console:Power on smithi065 2022-03-07T00:29:02.824 INFO:teuthology.orchestra.console:Power on for smithi115 completed 2022-03-07T00:29:02.926 INFO:teuthology.provision.fog.smithi115:Waiting for deploy to finish 2022-03-07T00:29:06.944 INFO:teuthology.orchestra.console:Power on for smithi065 completed 2022-03-07T00:29:07.045 INFO:teuthology.provision.fog.smithi065:Waiting for deploy to finish 2022-03-07T00:31:38.173 ERROR:teuthology.orchestra.connection:Error authenticating with smithi065.front.sepia.ceph.com: Authentication failed. 2022-03-07T00:33:49.874 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-07T00:33:50.105 INFO:teuthology.provision.fog.smithi115:Node is ready 2022-03-07T00:33:50.124 INFO:teuthology.orchestra.run.smithi115.stdout:smithi115.front.sepia.ceph.com 2022-03-07T00:33:50.183 INFO:teuthology.orchestra.run.smithi115.stdout:172.21.15.115 smithi115.front.sepia.ceph.com smithi115 2022-03-07T00:33:50.543 INFO:teuthology.provision.fog.smithi115:Deploy complete! 2022-03-07T00:33:51.529 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-03-07T00:33:51.780 INFO:teuthology.provision.fog.smithi065:Node is ready 2022-03-07T00:33:51.799 INFO:teuthology.orchestra.run.smithi065.stdout:smithi065.front.sepia.ceph.com 2022-03-07T00:33:51.861 INFO:teuthology.orchestra.run.smithi065.stdout:172.21.15.65 smithi065.front.sepia.ceph.com smithi065 2022-03-07T00:33:52.213 INFO:teuthology.provision.fog.smithi065:Deploy complete! 2022-03-07T00:33:52.363 INFO:teuthology.lock.ops:Checking smithi065.front.sepia.ceph.com 2022-03-07T00:33:52.363 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-07T00:33:52.384 INFO:teuthology.lock.ops:Checking smithi115.front.sepia.ceph.com 2022-03-07T00:33:52.385 INFO:teuthology.lock.ops:New key found. Updating... 2022-03-07T00:33:52.406 INFO:teuthology.lock.ops:Updating [smithi065.front.sepia.ceph.com]: set os type and version on server 2022-03-07T00:33:52.406 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi065.front.sepia.ceph.com' 2022-03-07T00:33:52.596 INFO:teuthology.orchestra.run.smithi065.stdout:x86_64 2022-03-07T00:33:52.656 INFO:teuthology.orchestra.run.smithi065.stdout:NAME="CentOS Stream" 2022-03-07T00:33:52.657 INFO:teuthology.orchestra.run.smithi065.stdout:VERSION="8" 2022-03-07T00:33:52.657 INFO:teuthology.orchestra.run.smithi065.stdout:ID="centos" 2022-03-07T00:33:52.657 INFO:teuthology.orchestra.run.smithi065.stdout:ID_LIKE="rhel fedora" 2022-03-07T00:33:52.657 INFO:teuthology.orchestra.run.smithi065.stdout:VERSION_ID="8" 2022-03-07T00:33:52.658 INFO:teuthology.orchestra.run.smithi065.stdout:PLATFORM_ID="platform:el8" 2022-03-07T00:33:52.658 INFO:teuthology.orchestra.run.smithi065.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-07T00:33:52.658 INFO:teuthology.orchestra.run.smithi065.stdout:ANSI_COLOR="0;31" 2022-03-07T00:33:52.658 INFO:teuthology.orchestra.run.smithi065.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-07T00:33:52.659 INFO:teuthology.orchestra.run.smithi065.stdout:HOME_URL="https://centos.org/" 2022-03-07T00:33:52.659 INFO:teuthology.orchestra.run.smithi065.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-07T00:33:52.659 INFO:teuthology.orchestra.run.smithi065.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-07T00:33:52.659 INFO:teuthology.orchestra.run.smithi065.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-07T00:33:52.660 INFO:teuthology.lock.ops:Updating smithi065.front.sepia.ceph.com on lock server 2022-03-07T00:33:52.709 INFO:teuthology.lock.ops:Updating [smithi115.front.sepia.ceph.com]: set os type and version on server 2022-03-07T00:33:52.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2022-03-07T00:33:52.964 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2022-03-07T00:33:53.023 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="CentOS Stream" 2022-03-07T00:33:53.023 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="8" 2022-03-07T00:33:53.024 INFO:teuthology.orchestra.run.smithi115.stdout:ID="centos" 2022-03-07T00:33:53.024 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="rhel fedora" 2022-03-07T00:33:53.024 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="8" 2022-03-07T00:33:53.024 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el8" 2022-03-07T00:33:53.024 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="CentOS Stream 8" 2022-03-07T00:33:53.025 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2022-03-07T00:33:53.025 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-03-07T00:33:53.025 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://centos.org/" 2022-03-07T00:33:53.025 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-03-07T00:33:53.026 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-03-07T00:33:53.026 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-03-07T00:33:53.026 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2022-03-07T00:33:53.134 INFO:teuthology.dispatcher.supervisor:Running job 6724341 2022-03-07T00:33:53.135 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_eeaeee38264fad8a01fd1ce912d8908676ed64bc/virtualenv/bin/teuthology -v --owner scheduled_rzarzynski@teuthology --archive /home/teuthworker/archive/rzarzynski-2022-03-07_00:23:56-rados-master-distro-basic-smithi/6724341 --name rzarzynski-2022-03-07_00:23:56-rados-master-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/redirect_promote_tests} -- /home/teuthworker/archive/rzarzynski-2022-03-07_00:23:56-rados-master-distro-basic-smithi/6724341/orig.config.yaml 2022-03-07T00:33:53.161 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rzarzynski-2022-03-07_00:23:56-rados-master-distro-basic-smithi/6724341 2022-03-07T00:33:53.162 INFO:teuthology.dispatcher.supervisor:Job PID: 9436 2022-03-07T00:33:53.163 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-03-07T07:04:04.314 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-03-07T07:04:04.495 INFO:teuthology.kill:Killing Pids: {9436} 2022-03-07T07:04:05.682 INFO:teuthology.task.internal:roles: ubuntu@smithi065.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-03-07T07:04:05.683 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-03-07T07:04:05.683 INFO:teuthology.misc:Compressing logs... 2022-03-07T07:04:05.684 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi065.front.sepia.ceph.com' 2022-03-07T07:04:05.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2022-03-07T07:04:06.205 INFO:teuthology.misc:Compressing logs... 2022-03-07T07:04:06.363 INFO:teuthology.misc:Compressing logs... 2022-03-07T07:04:39.329 INFO:teuthology.kill:No teuthology processes running 2022-03-07T07:04:40.496 INFO:teuthology.kill:Nuking machines: ['smithi065', 'smithi115'] 2022-03-07T07:04:41.462 INFO:teuthology.kill:2022-03-07 07:04:41,461.461 INFO:teuthology.nuke:targets: 2022-03-07T07:04:41.463 INFO:teuthology.kill: smithi065.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYf80+xcpWUnX0OR9WY2+MoKD2Yo1Z1/TlfVJo6YRp82DoUyahtvLBw/a56gp72gkAzho+043Pic3pVfUqAqNvYQWb3RWDklbXiwjmT2wtgcKTwFJ83uZNrtmNnwpAkTwmYW7ClvpsF2BWVzFmobWx1gjZDCDIlgYRy6rAcRrtbPY2wSBshg4PLYTAZE/uqMMOh0RFTio3r4DXHhdU/xXAUDXzAlqfloY6bD8D5MA/EzjGb8IjJqgwTb+qlFxTonI6a0wQw7mre/M3Ldy6PVsp9LOg9bKkljA60/rt2ZG4vYuzNVElw7yResapYvM0ZdgwhlMc4pv+biPWxtnRhnXVBaxsTPWMe6ragixuBo6MArM/RnigKE+qrvNqRK9o3QL0OnUW7AfAJ1NvAcrhWsmKtQmYVhKVptSP/f4r9C6JS+J2/+zRy2q9vs6p8rheBGTatVhKgSB1+hAhPT+H9WlNQXWoaoisFm65wkyioIx5bGi3KjpJgRqFmUjUzJrLKV0= 2022-03-07T07:04:41.463 INFO:teuthology.kill: smithi115.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+m3ebohKeC4DbMrfBEa2e45ONY/rKFyWWaVagfV0KugMxvrJ0MXm2w0MZ/fsfJPqCBvOM5IDQnmNLww7/7+8+rlQ6QPpzmXznYYmu2gYZLQW2IF2g21jOwoQRCT0xjM5RsnH/aggvhNWIcIO7A+G52grYuQXpmtMpQmKlWC2EXQfG/i0O2l6Qe9DWqQD8iphmtMulh+MjZnBBK3VhB1IfNvv24YtFxyX+AP0jg9XltVEb3DoSIErEfpuqM2raD61TvYxtGZRlHhKnuqXzGZatASaBtYU2FPN3plV/QbtX0p4iFuSOmfETsX+ooFaZUkooW6pv7omGV2ufvuAtwQoXdnWYrnwo5M5TQ3MVPHN8hec2v9QuUEaYigRZnmQGORtZfX/00L1KqUxD0AwqvBHwiQHt2sRtlyZJXVIlc+ca2FBWtQVbfWaQhMIVU6ZHJX6/iyBJTSJGbnPOPu+kQ2mSzDQcRtcaHr+LiMwBU/kiBX2Z1Uljky7K5IihRaCCnXE= 2022-03-07T07:04:41.488 INFO:teuthology.kill:2022-03-07 07:04:41,488.488 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-07T07:04:41.491 INFO:teuthology.kill:2022-03-07 07:04:41,491.491 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-03-07T07:04:41.566 INFO:teuthology.kill:2022-03-07 07:04:41,566.566 INFO:teuthology.orchestra.console:Power off smithi065 2022-03-07T07:04:41.580 INFO:teuthology.kill:2022-03-07 07:04:41,579.579 INFO:teuthology.orchestra.console:Power off smithi115 2022-03-07T07:04:49.951 INFO:teuthology.kill:2022-03-07 07:04:49,950.950 INFO:teuthology.orchestra.console:Power off for smithi115 completed 2022-03-07T07:04:50.095 INFO:teuthology.kill:2022-03-07 07:04:50,095.095 INFO:teuthology.lock.ops:unlocked: smithi115.front.sepia.ceph.com 2022-03-07T07:04:54.053 INFO:teuthology.kill:2022-03-07 07:04:54,053.053 INFO:teuthology.orchestra.console:Power off for smithi065 completed 2022-03-07T07:04:54.200 INFO:teuthology.kill:2022-03-07 07:04:54,199.199 INFO:teuthology.lock.ops:unlocked: smithi065.front.sepia.ceph.com 2022-03-07T07:06:54.428 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-03-07T07:06:54.525 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi115 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-03-07_06:07:03-rbd-pacific-distro-default-smithi/6724699