2022-01-22T12:44:24.072 INFO:root:teuthology version: 1.1.0-75f1bcd2
2022-01-22T12:44:24.223 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi136.front.sepia.ceph.com' reimaging
2022-01-22T12:44:24.224 INFO:teuthology.lock.ops:Updating [ubuntu@smithi136.front.sepia.ceph.com]: reset os type and version on server
2022-01-22T12:44:24.224 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server
2022-01-22T12:44:24.244 INFO:teuthology.lock.ops:Node 'ubuntu@smithi136.front.sepia.ceph.com' reimaging is complete
2022-01-22T12:44:24.245 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi175.front.sepia.ceph.com' reimaging
2022-01-22T12:44:24.245 INFO:teuthology.lock.ops:Updating [ubuntu@smithi175.front.sepia.ceph.com]: reset os type and version on server
2022-01-22T12:44:24.246 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server
2022-01-22T12:44:24.264 INFO:teuthology.lock.ops:Node 'ubuntu@smithi175.front.sepia.ceph.com' reimaging is complete
2022-01-22T12:44:24.459 INFO:teuthology.provision.fog.smithi136:Scheduling deploy of centos 8.stream
2022-01-22T12:44:24.487 INFO:teuthology.provision.fog.smithi175:Scheduling deploy of centos 8.stream
2022-01-22T12:44:24.856 INFO:teuthology.orchestra.console:Power off smithi136
2022-01-22T12:44:24.893 INFO:teuthology.orchestra.console:Power off smithi175
2022-01-22T12:44:33.225 INFO:teuthology.orchestra.console:Power off for smithi136 completed
2022-01-22T12:44:33.327 INFO:teuthology.orchestra.console:Power on smithi136
2022-01-22T12:44:37.365 INFO:teuthology.orchestra.console:Power off for smithi175 completed
2022-01-22T12:44:37.467 INFO:teuthology.orchestra.console:Power on smithi175
2022-01-22T12:44:41.693 INFO:teuthology.orchestra.console:Power on for smithi136 completed
2022-01-22T12:44:41.717 INFO:teuthology.orchestra.console:Power on for smithi175 completed
2022-01-22T12:44:41.795 INFO:teuthology.provision.fog.smithi136:Waiting for deploy to finish
2022-01-22T12:44:41.819 INFO:teuthology.provision.fog.smithi175:Waiting for deploy to finish
2022-01-22T12:49:13.679 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-22T12:49:13.972 INFO:teuthology.provision.fog.smithi136:Node is ready
2022-01-22T12:49:13.989 INFO:teuthology.orchestra.run.smithi136.stdout:smithi136.front.sepia.ceph.com
2022-01-22T12:49:14.046 INFO:teuthology.orchestra.run.smithi136.stdout:172.21.15.136 smithi136.front.sepia.ceph.com smithi136
2022-01-22T12:49:14.377 INFO:teuthology.provision.fog.smithi136:Deploy complete!
2022-01-22T12:49:22.675 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-01-22T12:49:22.967 INFO:teuthology.provision.fog.smithi175:Node is ready
2022-01-22T12:49:22.985 INFO:teuthology.orchestra.run.smithi175.stdout:smithi175.front.sepia.ceph.com
2022-01-22T12:49:23.043 INFO:teuthology.orchestra.run.smithi175.stdout:172.21.15.175 smithi175.front.sepia.ceph.com smithi175
2022-01-22T12:49:23.370 INFO:teuthology.provision.fog.smithi175:Deploy complete!
2022-01-22T12:49:23.516 INFO:teuthology.lock.ops:Checking smithi136.front.sepia.ceph.com
2022-01-22T12:49:23.517 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-22T12:49:23.535 INFO:teuthology.lock.ops:Checking smithi175.front.sepia.ceph.com
2022-01-22T12:49:23.536 INFO:teuthology.lock.ops:New key found. Updating...
2022-01-22T12:49:23.554 INFO:teuthology.lock.ops:Updating [smithi136.front.sepia.ceph.com]: set os type and version on server
2022-01-22T12:49:23.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com'
2022-01-22T12:49:23.751 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64
2022-01-22T12:49:23.809 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream"
2022-01-22T12:49:23.809 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="8"
2022-01-22T12:49:23.810 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos"
2022-01-22T12:49:23.810 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora"
2022-01-22T12:49:23.810 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="8"
2022-01-22T12:49:23.811 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el8"
2022-01-22T12:49:23.811 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 8"
2022-01-22T12:49:23.811 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31"
2022-01-22T12:49:23.811 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-01-22T12:49:23.811 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/"
2022-01-22T12:49:23.812 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-01-22T12:49:23.812 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-01-22T12:49:23.812 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-01-22T12:49:23.813 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server
2022-01-22T12:49:23.833 INFO:teuthology.lock.ops:Updating [smithi175.front.sepia.ceph.com]: set os type and version on server
2022-01-22T12:49:23.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com'
2022-01-22T12:49:24.019 INFO:teuthology.orchestra.run.smithi175.stdout:x86_64
2022-01-22T12:49:24.074 INFO:teuthology.orchestra.run.smithi175.stdout:NAME="CentOS Stream"
2022-01-22T12:49:24.074 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION="8"
2022-01-22T12:49:24.074 INFO:teuthology.orchestra.run.smithi175.stdout:ID="centos"
2022-01-22T12:49:24.075 INFO:teuthology.orchestra.run.smithi175.stdout:ID_LIKE="rhel fedora"
2022-01-22T12:49:24.075 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION_ID="8"
2022-01-22T12:49:24.075 INFO:teuthology.orchestra.run.smithi175.stdout:PLATFORM_ID="platform:el8"
2022-01-22T12:49:24.075 INFO:teuthology.orchestra.run.smithi175.stdout:PRETTY_NAME="CentOS Stream 8"
2022-01-22T12:49:24.076 INFO:teuthology.orchestra.run.smithi175.stdout:ANSI_COLOR="0;31"
2022-01-22T12:49:24.076 INFO:teuthology.orchestra.run.smithi175.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-01-22T12:49:24.076 INFO:teuthology.orchestra.run.smithi175.stdout:HOME_URL="https://centos.org/"
2022-01-22T12:49:24.076 INFO:teuthology.orchestra.run.smithi175.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-01-22T12:49:24.077 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-01-22T12:49:24.077 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-01-22T12:49:24.078 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server
2022-01-22T12:49:24.183 INFO:teuthology.dispatcher.supervisor:Running job 6633913
2022-01-22T12:49:24.183 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_75f1bcd237a8b6022305cd53ba87e106d45aa665/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-01-22_10:04:08-rados-WIP_GBH_NCB_new_alloc_map_A1-distro-basic-smithi/6633913 --name benhanokh-2022-01-22_10:04:08-rados-WIP_GBH_NCB_new_alloc_map_A1-distro-basic-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/benhanokh-2022-01-22_10:04:08-rados-WIP_GBH_NCB_new_alloc_map_A1-distro-basic-smithi/6633913/orig.config.yaml
2022-01-22T12:49:24.188 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-01-22_10:04:08-rados-WIP_GBH_NCB_new_alloc_map_A1-distro-basic-smithi/6633913
2022-01-22T12:49:24.188 INFO:teuthology.dispatcher.supervisor:Job PID: 7248
2022-01-22T12:49:24.189 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-01-22T19:19:53.013 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-01-22T19:19:54.319 INFO:teuthology.kill:Killing Pids: {7248}
2022-01-22T19:19:55.474 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2022-01-22T19:19:55.475 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2022-01-22T19:19:55.475 INFO:teuthology.misc:Compressing logs...
2022-01-22T19:19:55.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com'
2022-01-22T19:19:55.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com'
2022-01-22T19:19:55.954 INFO:teuthology.misc:Compressing logs...
2022-01-22T19:19:56.670 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-01-22T19:19:56.951 INFO:teuthology.misc:Compressing logs...
2022-01-22T19:20:01.280 INFO:teuthology.orchestra.run.smithi175.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-mon.smithi175.log: file size changed while zipping
2022-01-22T19:20:08.103 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi136.mmqwsw.log: file size changed while zipping
2022-01-22T19:20:13.254 INFO:teuthology.orchestra.run.smithi175.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-01-22T19:20:22.295 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-mon.smithi136.log: file size changed while zipping
2022-01-22T19:20:26.959 INFO:teuthology.orchestra.run.smithi175.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-01-22T19:20:33.959 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-01-22T19:20:37.180 INFO:teuthology.orchestra.run.smithi175.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-01-22T19:20:44.208 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-01-22T19:21:25.852 INFO:teuthology.orchestra.run.smithi175.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi175.niihjs.log: file size changed while zipping
2022-01-22T19:21:39.222 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi136.iabxtg.log: file size changed while zipping
2022-01-22T19:21:51.551 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-01-22T19:21:55.436 INFO:teuthology.orchestra.run.smithi175.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi175.ukyoaa.log: file size changed while zipping
2022-01-22T19:21:55.879 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/32154900-7b83-11ec-8c35-001a4aab830c/ceph-mgr.smithi136.dsxcqu.log: file size changed while zipping
2022-01-22T19:22:20.117 INFO:teuthology.orchestra.run.smithi136.stderr:tar: .: file changed as we read it
2022-01-22T19:22:43.649 INFO:teuthology.kill:No teuthology processes running
2022-01-22T19:22:44.717 INFO:teuthology.kill:Nuking machines: ['smithi136', 'smithi175']
2022-01-22T19:22:45.618 INFO:teuthology.kill:2022-01-22 19:22:45,618.618 INFO:teuthology.nuke:targets:
2022-01-22T19:22:45.619 INFO:teuthology.kill:  smithi136.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGUrD7QbFUXZHQg82/IRajJs08kslQkvFL2XP6pvJR2UOoFSUXjrFKjiJ8OUdCXpssktItdot/pGaeskEEnswGYFKu8t/Y7CqC3c+iCv3Nz5p7bv7MQemsP5pc2iP/u5dwBgbFXtD2CdLNCtlwIgvj0tl8MciwdU5Y51teoOKSnqfqHMB1ZKZyBKE8jOz5n1c3bw1wwoRgqx9ZRZ9EIU5ZXnHqFKIKbFBZYJl6ofGshdktax2xQqfKIBe6YLpArftC0k/tceRfbErP/siDaOT/5HYVPsbXveB1KDlg28JxyEwmdl+d4Ci4pzotEHDK+wKxb8dAHP5X/KX6Ekc/5gRl1GyyM71vCRoHInathsg6TsJHBgEh8ynxbhRtiMooToo59zhXbEFrrqT6dqgfD9jHp25zzP5A+ITgf+Upmrjn0d3j6B99nLKIbfEruUg1zZTywpicp2am9Xb/5M958woatq/Le6YeB7b2E2O4VaqQ5rvKGSxgD+6PhE3X39OecBM=
2022-01-22T19:22:45.619 INFO:teuthology.kill:  smithi175.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9FcVRFC5b2xvl3UHL5LmzXksuYOpvov/53F+7O4uncP8vOUJIMpJylkoftI8lifG0s2wW09051D2Q/PPgsRAzNdlRK0zp4rREYb7cqCl94w5P83yFG0Tx2sFNpd5QFXV6CApfaDIctEZmmOvMCteI+xcQ83CX3Y+QQL7CoIYzZqmL1Ubq3PjWa72u9lwcTIXaUjXR5ge+DR8JrYll5OSt/FO7BsZ+rbOyGbOyN7HT5TtAT9V1sgv7opifD4q/pIoZuvfCYs0OWsiIY6yIV+SZHxPYgraDG0R8+Ztctm+odSAot31+k2BOga0ELwgqjRoeOX/EmEO4vZTQY0Bd7xjJOc7WxhGtYTmAkb4+LdJucOM94Wa6tD3EiJatDx5Kh1GUejeTV/JstVgU6rWRj1SKTXjhoRTRs8nXSm0HqAFSYUkECKG/6MXsJFONogZobpwFrGqcWR1B70+P/YSgdFy+bJK8zSZ4HeygeAtSyUlqYPasryNlzziEXlc7kIWeQ+s=
2022-01-22T19:22:45.646 INFO:teuthology.kill:2022-01-22 19:22:45,645.645 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-22T19:22:45.648 INFO:teuthology.kill:2022-01-22 19:22:45,648.648 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-01-22T19:22:45.722 INFO:teuthology.kill:2022-01-22 19:22:45,722.722 INFO:teuthology.orchestra.console:Power off smithi136
2022-01-22T19:22:45.738 INFO:teuthology.kill:2022-01-22 19:22:45,737.737 INFO:teuthology.orchestra.console:Power off smithi175
2022-01-22T19:22:54.104 INFO:teuthology.kill:2022-01-22 19:22:54,104.104 INFO:teuthology.orchestra.console:Power off for smithi136 completed
2022-01-22T19:22:54.108 INFO:teuthology.kill:2022-01-22 19:22:54,108.108 INFO:teuthology.orchestra.console:Power off for smithi175 completed
2022-01-22T19:22:54.257 INFO:teuthology.kill:2022-01-22 19:22:54,256.256 INFO:teuthology.lock.ops:unlocked: smithi175.front.sepia.ceph.com
2022-01-22T19:22:54.261 INFO:teuthology.kill:2022-01-22 19:22:54,261.261 INFO:teuthology.lock.ops:unlocked: smithi136.front.sepia.ceph.com
2022-01-22T19:24:54.522 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-01-22T19:24:54.626 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi136 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-01-22_17:45:07-upgrade-wip-yuriw-quincy-1.22.22-distro-default-smithi/6634909
2022-01-22T19:24:54.627 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi175 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-01-22_17:45:07-upgrade-wip-yuriw-quincy-1.22.22-distro-default-smithi/6634909