2023-01-26T01:10:22.507 INFO:root:teuthology version: 0.0.1.dev22+g8e1bba1
2023-01-26T01:10:22.642 INFO:teuthology.lock.ops:Start node 'smithi090.front.sepia.ceph.com' reimaging
2023-01-26T01:10:22.643 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: reset os type and version on server
2023-01-26T01:10:22.643 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2023-01-26T01:10:22.661 INFO:teuthology.lock.ops:Node 'smithi090.front.sepia.ceph.com' reimaging is complete
2023-01-26T01:10:22.662 INFO:teuthology.lock.ops:Start node 'smithi178.front.sepia.ceph.com' reimaging
2023-01-26T01:10:22.662 INFO:teuthology.lock.ops:Updating [smithi178.front.sepia.ceph.com]: reset os type and version on server
2023-01-26T01:10:22.662 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server
2023-01-26T01:10:22.677 INFO:teuthology.lock.ops:Node 'smithi178.front.sepia.ceph.com' reimaging is complete
2023-01-26T01:10:22.821 INFO:teuthology.provision.fog.smithi178:Scheduling deploy of centos 8.stream
2023-01-26T01:10:22.824 INFO:teuthology.provision.fog.smithi090:Scheduling deploy of centos 8.stream
2023-01-26T01:10:23.054 INFO:teuthology.orchestra.console:Power off smithi178
2023-01-26T01:10:23.070 INFO:teuthology.orchestra.console:Power off smithi090
2023-01-26T01:10:23.199 INFO:teuthology.orchestra.console:Power off for smithi178 completed
2023-01-26T01:10:23.210 INFO:teuthology.orchestra.console:Power off for smithi090 completed
2023-01-26T01:10:23.300 INFO:teuthology.orchestra.console:Power on smithi178
2023-01-26T01:10:23.311 INFO:teuthology.orchestra.console:Power on smithi090
2023-01-26T01:10:27.560 INFO:teuthology.orchestra.console:Power on for smithi090 completed
2023-01-26T01:10:27.661 INFO:teuthology.provision.fog.smithi090:Waiting for deploy to finish
2023-01-26T01:10:31.665 INFO:teuthology.orchestra.console:Power on for smithi178 completed
2023-01-26T01:10:31.766 INFO:teuthology.provision.fog.smithi178:Waiting for deploy to finish
2023-01-26T01:13:17.822 ERROR:teuthology.orchestra.connection:Error authenticating with smithi178.front.sepia.ceph.com: Authentication failed.
2023-01-26T01:14:01.960 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-01-26T01:14:11.016 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-01-26T01:14:18.826 WARNING:teuthology.provision.fog:timed out
2023-01-26T01:14:20.072 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-01-26T01:14:29.128 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-01-26T01:14:38.184 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-01-26T01:14:47.240 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.90
2023-01-26T01:15:24.830 WARNING:teuthology.provision.fog:timed out
2023-01-26T01:15:31.088 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-26T01:15:31.361 INFO:teuthology.provision.fog.smithi178:Node is ready
2023-01-26T01:15:31.379 INFO:teuthology.orchestra.run.smithi178.stdout:smithi178.front.sepia.ceph.com
2023-01-26T01:15:31.437 INFO:teuthology.orchestra.run.smithi178.stdout:172.21.15.178 smithi178.front.sepia.ceph.com smithi178
2023-01-26T01:15:31.789 INFO:teuthology.provision.fog.smithi178:Deploy complete!
2023-01-26T01:15:53.243 WARNING:teuthology.provision.fog:timed out
2023-01-26T01:15:59.561 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-01-26T01:15:59.932 INFO:teuthology.provision.fog.smithi090:Node is ready
2023-01-26T01:15:59.952 INFO:teuthology.orchestra.run.smithi090.stdout:smithi090.front.sepia.ceph.com
2023-01-26T01:16:00.011 INFO:teuthology.orchestra.run.smithi090.stdout:172.21.15.90 smithi090.front.sepia.ceph.com smithi090
2023-01-26T01:16:00.369 INFO:teuthology.provision.fog.smithi090:Deploy complete!
2023-01-26T01:16:00.523 INFO:teuthology.lock.ops:Checking smithi090.front.sepia.ceph.com
2023-01-26T01:16:00.524 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-26T01:16:00.542 INFO:teuthology.lock.ops:Checking smithi178.front.sepia.ceph.com
2023-01-26T01:16:00.543 INFO:teuthology.lock.ops:New key found. Updating...
2023-01-26T01:16:00.558 INFO:teuthology.lock.ops:Updating [smithi090.front.sepia.ceph.com]: set os type and version on server
2023-01-26T01:16:00.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-01-26T01:16:00.792 INFO:teuthology.orchestra.run.smithi090.stdout:x86_64
2023-01-26T01:16:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:NAME="CentOS Stream"
2023-01-26T01:16:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION="8"
2023-01-26T01:16:00.849 INFO:teuthology.orchestra.run.smithi090.stdout:ID="centos"
2023-01-26T01:16:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:ID_LIKE="rhel fedora"
2023-01-26T01:16:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:VERSION_ID="8"
2023-01-26T01:16:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:PLATFORM_ID="platform:el8"
2023-01-26T01:16:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:PRETTY_NAME="CentOS Stream 8"
2023-01-26T01:16:00.850 INFO:teuthology.orchestra.run.smithi090.stdout:ANSI_COLOR="0;31"
2023-01-26T01:16:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-01-26T01:16:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:HOME_URL="https://centos.org/"
2023-01-26T01:16:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-01-26T01:16:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-01-26T01:16:00.851 INFO:teuthology.orchestra.run.smithi090.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-01-26T01:16:00.852 INFO:teuthology.lock.ops:Updating smithi090.front.sepia.ceph.com on lock server
2023-01-26T01:16:00.867 INFO:teuthology.lock.ops:Updating [smithi178.front.sepia.ceph.com]: set os type and version on server
2023-01-26T01:16:00.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com'
2023-01-26T01:16:01.123 INFO:teuthology.orchestra.run.smithi178.stdout:x86_64
2023-01-26T01:16:01.181 INFO:teuthology.orchestra.run.smithi178.stdout:NAME="CentOS Stream"
2023-01-26T01:16:01.181 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION="8"
2023-01-26T01:16:01.181 INFO:teuthology.orchestra.run.smithi178.stdout:ID="centos"
2023-01-26T01:16:01.181 INFO:teuthology.orchestra.run.smithi178.stdout:ID_LIKE="rhel fedora"
2023-01-26T01:16:01.182 INFO:teuthology.orchestra.run.smithi178.stdout:VERSION_ID="8"
2023-01-26T01:16:01.182 INFO:teuthology.orchestra.run.smithi178.stdout:PLATFORM_ID="platform:el8"
2023-01-26T01:16:01.182 INFO:teuthology.orchestra.run.smithi178.stdout:PRETTY_NAME="CentOS Stream 8"
2023-01-26T01:16:01.182 INFO:teuthology.orchestra.run.smithi178.stdout:ANSI_COLOR="0;31"
2023-01-26T01:16:01.182 INFO:teuthology.orchestra.run.smithi178.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-01-26T01:16:01.183 INFO:teuthology.orchestra.run.smithi178.stdout:HOME_URL="https://centos.org/"
2023-01-26T01:16:01.183 INFO:teuthology.orchestra.run.smithi178.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-01-26T01:16:01.183 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-01-26T01:16:01.183 INFO:teuthology.orchestra.run.smithi178.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-01-26T01:16:01.184 INFO:teuthology.lock.ops:Updating smithi178.front.sepia.ceph.com on lock server
2023-01-26T01:16:01.271 INFO:teuthology.dispatcher.supervisor:Running job 7138348
2023-01-26T01:16:01.271 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_8e1bba12e584b7ae912e92e94afe9c075ab51fa3/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138348 --name yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138348/orig.config.yaml
2023-01-26T01:16:01.274 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-01-25_15:33:20-fs-wip-yuri5-testing-2023-01-24-1425-quincy-distro-default-smithi/7138348
2023-01-26T01:16:01.275 INFO:teuthology.dispatcher.supervisor:Job PID: 11599
2023-01-26T01:16:01.275 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-01-26T13:16:21.566 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-01-26T13:16:21.769 INFO:teuthology.kill:Killing Pids: {11599}
2023-01-26T13:16:22.673 INFO:teuthology.task.internal:roles: ubuntu@smithi090.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2023-01-26T13:16:22.674 INFO:teuthology.task.internal:roles: ubuntu@smithi178.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2023-01-26T13:16:22.675 INFO:teuthology.misc:Compressing logs...
2023-01-26T13:16:22.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi090.front.sepia.ceph.com'
2023-01-26T13:16:22.826 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi178.front.sepia.ceph.com'
2023-01-26T13:16:23.121 INFO:teuthology.misc:Compressing logs...
2023-01-26T13:16:23.425 INFO:teuthology.misc:Compressing logs...
2023-01-26T13:16:30.883 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-mgr.smithi090.zlynra.log: file size changed while zipping
2023-01-26T13:16:45.295 INFO:teuthology.orchestra.run.smithi178.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2023-01-26T13:16:50.428 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2023-01-26T13:17:07.002 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2023-01-26T13:17:26.166 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-mon.smithi090.log: file size changed while zipping
2023-01-26T13:17:42.985 INFO:teuthology.orchestra.run.smithi178.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-mds.cephfs.smithi178.casnyk.log: file size changed while zipping
2023-01-26T13:17:50.331 INFO:teuthology.orchestra.run.smithi178.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-mon.smithi178.log: file size changed while zipping
2023-01-26T13:17:55.119 INFO:teuthology.orchestra.run.smithi178.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-mds.cephfs.smithi178.xafziz.log: file size changed while zipping
2023-01-26T13:18:03.219 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-mds.cephfs.smithi090.uqfirx.log: file size changed while zipping
2023-01-26T13:18:18.848 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2023-01-26T13:18:22.999 INFO:teuthology.orchestra.run.smithi178.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2023-01-26T13:18:25.068 INFO:teuthology.orchestra.run.smithi090.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-mds.cephfs.smithi090.cnsjej.log: file size changed while zipping
2023-01-26T13:18:37.554 INFO:teuthology.orchestra.run.smithi178.stderr:gzip: /var/log/ceph/13d70f38-9d19-11ed-9e56-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2023-01-26T13:19:29.977 INFO:teuthology.kill:No teuthology processes running
2023-01-26T13:19:30.666 INFO:teuthology.kill:Nuking machines: ['smithi090', 'smithi178']
2023-01-26T13:19:31.233 INFO:teuthology.kill:2023-01-26 13:19:31,232.232 INFO:teuthology.nuke:targets:
2023-01-26T13:19:31.233 INFO:teuthology.kill:  smithi090.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDU6Ajq0iyOG709tE05XReFeJ4KaHVW6dqhErfBbX8v5duenSV3NguRtSToBbyc9/mvT5/QhNfqk5y7R/Om2s4mSHbm7m0NFdbIqXNe/lWrWcnr+ZDGh0zMJRDxS86eWwqgh5KlHE0eZnuaBy/maTw0zLC+s0sIa3UUXpUvA0kJ0Sqb8rt7rr7gGMplHIslIfSwjd1am5ArUipYZ9rbEygYbp5UMgxUhogXMeuSQAeuiZtnc4dNrUvjVSCkzb1Hlq9B1HTpIsJIATYK/QG9G8U8hgXJy5ElxNzWI7uytjAz2+jg/ieWTh6W1zNOv7A8rg8XWhX4bMAgYmAj8rysMNl0iW4BhPRC2B647tKVL3Ls6/nB1fR5zEAuFvB5auJKn/Mxn5h5sTfzlOTC1wXoWKBQmdREEQYryYAStfXnsdiUd3+fZkCnZ5WOdSctWJmajZwfFR7XfSi6DRDZcF57uIbqc+Uizx0eMDRoS1kHndLuiL7UwMrfHLOiCO7q6TWHhrE=
2023-01-26T13:19:31.234 INFO:teuthology.kill:  smithi178.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO1YgD02u4nSCagr5bSeZ4coBvt2aTDgq4jWKmLP1AGhIck1WIvVE/syYQVG6XPZ2DG7QbMfamlQJ8o+2ZOV9zA=
2023-01-26T13:19:31.234 INFO:teuthology.kill:2023-01-26 13:19:31,232.232 INFO:teuthology.nuke:Checking targets against current locks
2023-01-26T13:19:31.282 INFO:teuthology.kill:2023-01-26 13:19:31,281.281 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-26T13:19:31.303 INFO:teuthology.kill:2023-01-26 13:19:31,303.303 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-01-26T13:19:31.353 INFO:teuthology.kill:2023-01-26 13:19:31,353.353 INFO:teuthology.orchestra.console:Power off smithi090
2023-01-26T13:19:31.387 INFO:teuthology.kill:2023-01-26 13:19:31,387.387 INFO:teuthology.orchestra.console:Power off smithi178
2023-01-26T13:19:39.718 INFO:teuthology.kill:2023-01-26 13:19:39,717.717 INFO:teuthology.orchestra.console:Power off for smithi090 completed
2023-01-26T13:19:39.744 INFO:teuthology.kill:2023-01-26 13:19:39,744.744 INFO:teuthology.orchestra.console:Power off for smithi178 completed
2023-01-26T13:19:39.858 INFO:teuthology.kill:2023-01-26 13:19:39,858.858 INFO:teuthology.lock.ops:unlocked: smithi090.front.sepia.ceph.com
2023-01-26T13:19:39.886 INFO:teuthology.kill:2023-01-26 13:19:39,886.886 INFO:teuthology.lock.ops:unlocked: smithi178.front.sepia.ceph.com
2023-01-26T13:21:40.076 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-01-26T13:21:40.180 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi090 but it was locked by another job: /home/teuthworker/archive/yuvalif-2023-01-25_08:55:27-rgw-wip-yuval-fix-58167-distro-default-smithi/7138218
2023-01-26T13:21:40.181 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi178 but it was locked by another job: /home/teuthworker/archive/yuvalif-2023-01-25_08:55:27-rgw-wip-yuval-fix-58167-distro-default-smithi/7138218