2023-03-29T23:12:16.226 INFO:root:teuthology version: 0.0.1.dev78+ge372916 2023-03-29T23:12:16.374 INFO:teuthology.lock.ops:Start node 'smithi053.front.sepia.ceph.com' reimaging 2023-03-29T23:12:16.375 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: reset os type and version on server 2023-03-29T23:12:16.376 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2023-03-29T23:12:16.410 INFO:teuthology.lock.ops:Node 'smithi053.front.sepia.ceph.com' reimaging is complete 2023-03-29T23:12:16.410 INFO:teuthology.lock.ops:Start node 'smithi171.front.sepia.ceph.com' reimaging 2023-03-29T23:12:16.411 INFO:teuthology.lock.ops:Updating [smithi171.front.sepia.ceph.com]: reset os type and version on server 2023-03-29T23:12:16.411 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2023-03-29T23:12:16.443 INFO:teuthology.lock.ops:Node 'smithi171.front.sepia.ceph.com' reimaging is complete 2023-03-29T23:12:16.585 INFO:teuthology.provision.fog.smithi053:Scheduling deploy of centos 8.stream 2023-03-29T23:12:16.627 INFO:teuthology.provision.fog.smithi171:Scheduling deploy of centos 8.stream 2023-03-29T23:12:16.897 INFO:teuthology.orchestra.console:Power off smithi171 2023-03-29T23:12:16.912 INFO:teuthology.orchestra.console:Power off smithi053 2023-03-29T23:12:25.273 INFO:teuthology.orchestra.console:Power off for smithi171 completed 2023-03-29T23:12:25.281 INFO:teuthology.orchestra.console:Power off for smithi053 completed 2023-03-29T23:12:25.375 INFO:teuthology.orchestra.console:Power on smithi171 2023-03-29T23:12:25.383 INFO:teuthology.orchestra.console:Power on smithi053 2023-03-29T23:12:33.750 INFO:teuthology.orchestra.console:Power on for smithi053 completed 2023-03-29T23:12:33.787 INFO:teuthology.orchestra.console:Power on for smithi171 completed 2023-03-29T23:12:33.852 INFO:teuthology.provision.fog.smithi053:Waiting for deploy to finish 2023-03-29T23:12:33.888 INFO:teuthology.provision.fog.smithi171:Waiting for deploy to finish 2023-03-29T23:15:07.881 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-03-29T23:15:16.969 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-03-29T23:15:22.949 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-03-29T23:15:26.021 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-03-29T23:15:32.009 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-03-29T23:15:35.081 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-03-29T23:15:41.093 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-03-29T23:15:44.165 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.53 2023-03-29T23:15:50.149 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-03-29T23:15:59.205 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-03-29T23:16:08.261 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-03-29T23:16:17.349 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.171 2023-03-29T23:16:21.969 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-29T23:16:42.186 INFO:teuthology.provision.fog.smithi053:Node is ready 2023-03-29T23:16:42.211 INFO:teuthology.orchestra.run.smithi053.stdout:smithi053.front.sepia.ceph.com 2023-03-29T23:16:42.273 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53 smithi053.front.sepia.ceph.com smithi053 2023-03-29T23:16:42.649 INFO:teuthology.provision.fog.smithi053:Deploy complete! 2023-03-29T23:17:23.389 WARNING:teuthology.provision.fog:timed out 2023-03-29T23:17:29.683 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-03-29T23:17:29.982 INFO:teuthology.provision.fog.smithi171:Node is ready 2023-03-29T23:17:30.000 INFO:teuthology.orchestra.run.smithi171.stdout:smithi171.front.sepia.ceph.com 2023-03-29T23:17:30.059 INFO:teuthology.orchestra.run.smithi171.stdout:172.21.15.171 smithi171.front.sepia.ceph.com smithi171 2023-03-29T23:17:30.413 INFO:teuthology.provision.fog.smithi171:Deploy complete! 2023-03-29T23:17:30.605 INFO:teuthology.lock.ops:Checking smithi053.front.sepia.ceph.com 2023-03-29T23:17:30.606 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-29T23:17:30.631 INFO:teuthology.lock.ops:Checking smithi171.front.sepia.ceph.com 2023-03-29T23:17:30.632 INFO:teuthology.lock.ops:New key found. Updating... 2023-03-29T23:17:30.648 INFO:teuthology.lock.ops:Updating [smithi053.front.sepia.ceph.com]: set os type and version on server 2023-03-29T23:17:30.649 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-03-29T23:17:30.912 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2023-03-29T23:17:30.972 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2023-03-29T23:17:30.972 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="8" 2023-03-29T23:17:30.972 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2023-03-29T23:17:30.972 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2023-03-29T23:17:30.973 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="8" 2023-03-29T23:17:30.973 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el8" 2023-03-29T23:17:30.973 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-29T23:17:30.973 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2023-03-29T23:17:30.973 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-29T23:17:30.973 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2023-03-29T23:17:30.973 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-29T23:17:30.974 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-29T23:17:30.974 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-29T23:17:30.975 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2023-03-29T23:17:30.998 INFO:teuthology.lock.ops:Updating [smithi171.front.sepia.ceph.com]: set os type and version on server 2023-03-29T23:17:30.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2023-03-29T23:17:31.186 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2023-03-29T23:17:31.244 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="CentOS Stream" 2023-03-29T23:17:31.244 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="8" 2023-03-29T23:17:31.244 INFO:teuthology.orchestra.run.smithi171.stdout:ID="centos" 2023-03-29T23:17:31.244 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="rhel fedora" 2023-03-29T23:17:31.244 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="8" 2023-03-29T23:17:31.245 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el8" 2023-03-29T23:17:31.245 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="CentOS Stream 8" 2023-03-29T23:17:31.245 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31" 2023-03-29T23:17:31.245 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-03-29T23:17:31.245 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://centos.org/" 2023-03-29T23:17:31.245 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-29T23:17:31.245 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-29T23:17:31.245 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-03-29T23:17:31.246 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2023-03-29T23:17:31.402 INFO:teuthology.dispatcher.supervisor:Running job 7224679 2023-03-29T23:17:31.403 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_e3729161eecf8e6d09a4069acbb883cdcd5e9f50/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2023-03-28_22:46:34-rados-WIP_GBH_snap_mapper_B-distro-default-smithi/7224679 --name benhanokh-2023-03-28_22:46:34-rados-WIP_GBH_snap_mapper_B-distro-default-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/small-objects-localized} -- /home/teuthworker/archive/benhanokh-2023-03-28_22:46:34-rados-WIP_GBH_snap_mapper_B-distro-default-smithi/7224679/orig.config.yaml 2023-03-29T23:17:31.408 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2023-03-28_22:46:34-rados-WIP_GBH_snap_mapper_B-distro-default-smithi/7224679 2023-03-29T23:17:31.410 INFO:teuthology.dispatcher.supervisor:Job PID: 4489 2023-03-29T23:17:31.410 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-03-30T11:18:29.443 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2023-03-30T11:18:29.596 INFO:teuthology.kill:Killing Pids: {4489} 2023-03-30T11:18:30.509 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-03-30T11:18:30.510 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-03-30T11:18:30.510 INFO:teuthology.misc:Compressing logs... 2023-03-30T11:18:30.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2023-03-30T11:18:30.654 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2023-03-30T11:18:31.025 INFO:teuthology.misc:Compressing logs... 2023-03-30T11:18:31.908 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2023-03-30T11:18:31.989 INFO:teuthology.orchestra.run.smithi053.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2023-03-30T11:18:58.869 INFO:teuthology.misc:Compressing logs... 2023-03-30T11:19:12.070 INFO:teuthology.kill:No teuthology processes running 2023-03-30T11:19:12.922 INFO:teuthology.kill:Nuking machines: ['smithi053', 'smithi171'] 2023-03-30T11:19:13.531 INFO:teuthology.kill:2023-03-30 11:19:13,530.530 INFO:teuthology.nuke:targets: 2023-03-30T11:19:13.532 INFO:teuthology.kill: smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA6SpPhoFSEshz7w99VQAd9OMyZ0qdkfLcbgRJRukJUPhFXGEsKYgQClh1hSr72HFn4KdGKQbrEI78SCIQx0vuQ= 2023-03-30T11:19:13.532 INFO:teuthology.kill: smithi171.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCTuCU91hRrRWQLGosYjBveu/TjjBJ71/GtIecbLKZFI0LyjyxwVXDInfhbsg+O2dnsnncT0m4DoXYFh/vY0zKmEvZJb2of8uYmDAU/+svlSkuMmPHQzbZY3epHOclK6O76mePUZpVYBk/ZftvvTSlSGwYqARo3N/C2bkYc0VHFxqmA+rhnY9TSpmB1Tz8W57ZD37hMDCu2VDJ+S4xE9bewN+xdbKFY0hK9d+NeikvVrjiPl6G7oSWJ5NfWOzHXuSe1LtHE2GDeNLueASIrCsDo4Wpn9bxvGqAh03TOCZegKdSfVACBAliJidcxc7AgNAYNHMXkY8LEsNDdArdFginj7P9Xy++tNtrD1OlqVzZCAWf/gSAhvt82zyp2/WvTSENzHqsDAmZNrMXRCmAfz5UuNIvQ5JrZdSE3ONuSnDCCVG63Yx6qZ6pl9kGWbLyBwPRWeET1UcT+oekc/sz3lKZJoy476hI1pKqhAtFhd/AgQLEzayTsJZRMygSorO9Pjic= 2023-03-30T11:19:13.532 INFO:teuthology.kill:2023-03-30 11:19:13,531.531 INFO:teuthology.nuke:Checking targets against current locks 2023-03-30T11:19:13.589 INFO:teuthology.kill:2023-03-30 11:19:13,589.589 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-30T11:19:13.608 INFO:teuthology.kill:2023-03-30 11:19:13,608.608 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-30T11:19:13.662 INFO:teuthology.kill:2023-03-30 11:19:13,661.661 INFO:teuthology.orchestra.console:Power off smithi053 2023-03-30T11:19:13.706 INFO:teuthology.kill:2023-03-30 11:19:13,706.706 INFO:teuthology.orchestra.console:Power off smithi171 2023-03-30T11:19:22.040 INFO:teuthology.kill:2023-03-30 11:19:22,040.040 INFO:teuthology.orchestra.console:Power off for smithi053 completed 2023-03-30T11:19:22.200 INFO:teuthology.kill:2023-03-30 11:19:22,199.199 INFO:teuthology.lock.ops:unlocked: smithi053.front.sepia.ceph.com 2023-03-30T11:19:26.187 INFO:teuthology.kill:2023-03-30 11:19:26,187.187 INFO:teuthology.orchestra.console:Power off for smithi171 completed 2023-03-30T11:19:26.362 INFO:teuthology.kill:2023-03-30 11:19:26,361.361 INFO:teuthology.lock.ops:unlocked: smithi171.front.sepia.ceph.com 2023-03-30T11:21:26.689 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2023-03-30T11:21:26.788 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi053 but it was locked by another job: /home/teuthworker/archive/gabrioux-2023-03-30_10:55:12-orch:cephadm-wip-guits-testing-3-2023-03-29-1515-distro-default-smithi/7226060