2023-02-16T15:52:34.963 INFO:root:teuthology version: 0.0.1.dev41+gfbbadb5
2023-02-16T15:52:35.114 INFO:teuthology.lock.ops:Start node 'smithi143.front.sepia.ceph.com' reimaging
2023-02-16T15:52:35.116 INFO:teuthology.lock.ops:Updating [smithi143.front.sepia.ceph.com]: reset os type and version on server
2023-02-16T15:52:35.116 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server
2023-02-16T15:52:35.174 INFO:teuthology.lock.ops:Node 'smithi143.front.sepia.ceph.com' reimaging is complete
2023-02-16T15:52:35.174 INFO:teuthology.lock.ops:Start node 'smithi153.front.sepia.ceph.com' reimaging
2023-02-16T15:52:35.175 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: reset os type and version on server
2023-02-16T15:52:35.175 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server
2023-02-16T15:52:35.215 INFO:teuthology.lock.ops:Node 'smithi153.front.sepia.ceph.com' reimaging is complete
2023-02-16T15:52:35.349 INFO:teuthology.provision.fog.smithi143:Scheduling deploy of centos 8.stream
2023-02-16T15:52:35.352 INFO:teuthology.provision.fog.smithi153:Scheduling deploy of centos 8.stream
2023-02-16T15:52:35.562 INFO:teuthology.orchestra.console:Power off smithi153
2023-02-16T15:52:35.603 INFO:teuthology.orchestra.console:Power off smithi143
2023-02-16T15:52:43.939 INFO:teuthology.orchestra.console:Power off for smithi153 completed
2023-02-16T15:52:44.040 INFO:teuthology.orchestra.console:Power on smithi153
2023-02-16T15:52:48.079 INFO:teuthology.orchestra.console:Power off for smithi143 completed
2023-02-16T15:52:48.181 INFO:teuthology.orchestra.console:Power on smithi143
2023-02-16T15:52:52.408 INFO:teuthology.orchestra.console:Power on for smithi153 completed
2023-02-16T15:52:52.509 INFO:teuthology.provision.fog.smithi153:Waiting for deploy to finish
2023-02-16T15:52:56.546 INFO:teuthology.orchestra.console:Power on for smithi143 completed
2023-02-16T15:52:56.647 INFO:teuthology.provision.fog.smithi143:Waiting for deploy to finish
2023-02-16T15:55:56.805 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.153
2023-02-16T15:56:00.901 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.143
2023-02-16T15:56:05.861 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.153
2023-02-16T15:56:09.957 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.143
2023-02-16T15:56:14.917 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.153
2023-02-16T15:56:19.013 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.143
2023-02-16T15:56:23.973 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.153
2023-02-16T15:56:28.073 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.143
2023-02-16T15:56:33.029 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.153
2023-02-16T15:56:37.125 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.143
2023-02-16T15:57:14.927 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-02-16T15:57:39.035 WARNING:teuthology.provision.fog:timed out
2023-02-16T15:57:40.446 INFO:teuthology.provision.fog.smithi143:Node is ready
2023-02-16T15:57:40.469 INFO:teuthology.orchestra.run.smithi143.stdout:smithi143.front.sepia.ceph.com
2023-02-16T15:57:40.529 INFO:teuthology.orchestra.run.smithi143.stdout:172.21.15.143 smithi143.front.sepia.ceph.com smithi143
2023-02-16T15:57:40.890 INFO:teuthology.provision.fog.smithi143:Deploy complete!
2023-02-16T15:57:45.317 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-02-16T15:57:45.548 INFO:teuthology.provision.fog.smithi153:Node is ready
2023-02-16T15:57:45.606 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153.front.sepia.ceph.com
2023-02-16T15:57:45.664 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153 smithi153.front.sepia.ceph.com smithi153
2023-02-16T15:57:46.021 INFO:teuthology.provision.fog.smithi153:Deploy complete!
2023-02-16T15:57:46.176 INFO:teuthology.lock.ops:Checking smithi143.front.sepia.ceph.com
2023-02-16T15:57:46.177 INFO:teuthology.lock.ops:New key found. Updating...
2023-02-16T15:57:46.226 INFO:teuthology.lock.ops:Checking smithi153.front.sepia.ceph.com
2023-02-16T15:57:46.227 INFO:teuthology.lock.ops:New key found. Updating...
2023-02-16T15:57:46.260 INFO:teuthology.lock.ops:Updating [smithi143.front.sepia.ceph.com]: set os type and version on server
2023-02-16T15:57:46.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com'
2023-02-16T15:57:46.446 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64
2023-02-16T15:57:46.503 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream"
2023-02-16T15:57:46.503 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="8"
2023-02-16T15:57:46.504 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos"
2023-02-16T15:57:46.504 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora"
2023-02-16T15:57:46.504 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="8"
2023-02-16T15:57:46.504 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el8"
2023-02-16T15:57:46.504 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 8"
2023-02-16T15:57:46.504 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31"
2023-02-16T15:57:46.504 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-02-16T15:57:46.505 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/"
2023-02-16T15:57:46.505 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-02-16T15:57:46.505 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-02-16T15:57:46.505 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-02-16T15:57:46.506 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server
2023-02-16T15:57:46.535 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: set os type and version on server
2023-02-16T15:57:46.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com'
2023-02-16T15:57:46.724 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64
2023-02-16T15:57:46.782 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream"
2023-02-16T15:57:46.783 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="8"
2023-02-16T15:57:46.783 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos"
2023-02-16T15:57:46.783 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora"
2023-02-16T15:57:46.783 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="8"
2023-02-16T15:57:46.783 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el8"
2023-02-16T15:57:46.783 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 8"
2023-02-16T15:57:46.783 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31"
2023-02-16T15:57:46.784 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2023-02-16T15:57:46.784 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/"
2023-02-16T15:57:46.784 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-02-16T15:57:46.784 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2023-02-16T15:57:46.784 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2023-02-16T15:57:46.785 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server
2023-02-16T15:57:46.932 INFO:teuthology.dispatcher.supervisor:Running job 7176317
2023-02-16T15:57:46.933 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/virtualenv/bin/teuthology -v --owner scheduled_gabrioux@teuthology --archive /home/teuthworker/archive/gabrioux-2023-02-16_15:51:01-orch:cephadm-wip-guits-testing-2-2023-02-16-0944-quincy-distro-default-smithi/7176317 --name gabrioux-2023-02-16_15:51:01-orch:cephadm-wip-guits-testing-2-2023-02-16-0944-quincy-distro-default-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/gabrioux-2023-02-16_15:51:01-orch:cephadm-wip-guits-testing-2-2023-02-16-0944-quincy-distro-default-smithi/7176317/orig.config.yaml
2023-02-16T15:57:46.936 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/gabrioux-2023-02-16_15:51:01-orch:cephadm-wip-guits-testing-2-2023-02-16-0944-quincy-distro-default-smithi/7176317
2023-02-16T15:57:46.937 INFO:teuthology.dispatcher.supervisor:Job PID: 23486
2023-02-16T15:57:46.937 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-02-17T03:58:45.505 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-02-17T03:58:48.767 INFO:teuthology.kill:Killing Pids: {23486}
2023-02-17T03:58:50.030 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2023-02-17T03:58:50.036 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2023-02-17T03:58:50.044 INFO:teuthology.misc:Compressing logs...
2023-02-17T03:58:50.065 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com'
2023-02-17T03:58:51.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com'
2023-02-17T03:58:52.058 INFO:teuthology.misc:Compressing logs...
2023-02-17T03:58:52.212 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2023-02-17T03:58:52.654 INFO:teuthology.misc:Compressing logs...
2023-02-17T03:58:53.057 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/ba059562-ae14-11ed-9ae8-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2023-02-17T03:58:53.107 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/ba059562-ae14-11ed-9ae8-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2023-02-17T03:58:53.435 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/ba059562-ae14-11ed-9ae8-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2023-02-17T03:58:53.605 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/ba059562-ae14-11ed-9ae8-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2023-02-17T03:58:53.741 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/ba059562-ae14-11ed-9ae8-001a4aab830c/ceph-mon.smithi143.log: file size changed while zipping
2023-02-17T03:58:54.258 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/ba059562-ae14-11ed-9ae8-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2023-02-17T03:58:54.388 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/ba059562-ae14-11ed-9ae8-001a4aab830c/ceph-mds.cephfs.smithi143.kqjaiv.log: file size changed while zipping
2023-02-17T03:58:55.138 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /var/log/ceph/ba059562-ae14-11ed-9ae8-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2023-02-17T03:59:47.594 INFO:teuthology.orchestra.run.smithi153.stderr:tar: .: file changed as we read it
2023-02-17T03:59:47.714 INFO:teuthology.kill:No teuthology processes running
2023-02-17T03:59:48.705 INFO:teuthology.kill:Nuking machines: ['smithi143', 'smithi153']
2023-02-17T03:59:49.378 INFO:teuthology.kill:2023-02-17 03:59:49,378.378 INFO:teuthology.nuke:targets:
2023-02-17T03:59:49.379 INFO:teuthology.kill:  smithi143.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCzXOSG/BBw+NjHPNH8li0vT/fCB5XHDcqUh3VOTPypy5ItKnEE1wRtN1ZzFY0BqN8NGtJle9IRGiqmLFxMuCUmobl+Zix9HPVWeMrdN+ZP1pDk+uVMl+AXe+c9SM0Zk0saKlSdwsBOM19CweAEkor3q4esA4S5f7Hu+R0R3ohGR3m+0dpNC60rLlsvd4UL23GM2QHS8um3yCHvUv9rwYH54p2d2fLmIk5bZLG67m0Dkd3PBwVBx+oPF7Tt2HsWXiDHB0EvtN6rRw1ejFvBZBavQo/K6+Loxqcnv8hRnIQiDmz6rR6E9GZb2NE03EA/UZ0rL7L5nUeWmPSXs4snEzptnbb9geE/5BfPW77lM9k6xwTuwwBVOabS2hhTvxvNSvMKucKqVr+K9p1KDcQrqEDTpAc7kQzdm2mbSsjO5pGw8yme/TAHm80uCPhsBBAR3pOYZWovjBVDpAZA8ApC8DtX4Pi5kTxJKYv9YXQWNcDnMSSgsRjQhAUNB/6EHEqG4Ns=
2023-02-17T03:59:49.380 INFO:teuthology.kill:  smithi153.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkkagd6DxQYWdxNpF3lAKwvzi5NnvtijyLmXOyMWbqv7+rCLA0AyeDa/AVMG+ObpAeewXrxp8m3YwpEay78VL5NEt4kIv91HH+LFw0hZ68F3WP2ntnA51Js01oixRasjub4AeOxBrtLV8tEF2WfHVdb3ao0NGb3FedmxCs4N0M+Fk7vWLl3J92773CMfq+5bJ/vZiZX9vKr/+p83iOatnEAcdHdYkSTiWRIytnS8CB04w6CAXF+oUZ0oUUU3uwgTIMr28whAbtV4bgJ13sojMOS29j2njKRIu4ZPTNAoe+iDG1OuRk142bFIgncrUcvq/YdE9nE6iL8n/e63boX3s+29nJU/uGfmarBaXZU5dNv7HYmdj1VjKiHo+pfq9gEK4r2mVt5MVcejsQ0LQPsSKrgwka4GNpk7kTPPAWMr5ObPyZpFocBhSfDaeS3l5TF8+ILA1nvCDVG+WVEtLOorna3rGr8gSnlP1wSb803cKCMOCIEhpV4Arzw7YG5hsZepE=
2023-02-17T03:59:49.380 INFO:teuthology.kill:2023-02-17 03:59:49,378.378 INFO:teuthology.nuke:Checking targets against current locks
2023-02-17T03:59:49.433 INFO:teuthology.kill:2023-02-17 03:59:49,433.433 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-02-17T03:59:49.460 INFO:teuthology.kill:2023-02-17 03:59:49,460.460 INFO:teuthology.task.internal.check_lock:Checking locks...
2023-02-17T03:59:49.512 INFO:teuthology.kill:2023-02-17 03:59:49,511.511 INFO:teuthology.orchestra.console:Power off smithi143
2023-02-17T03:59:49.549 INFO:teuthology.kill:2023-02-17 03:59:49,549.549 INFO:teuthology.orchestra.console:Power off smithi153
2023-02-17T03:59:57.888 INFO:teuthology.kill:2023-02-17 03:59:57,888.888 INFO:teuthology.orchestra.console:Power off for smithi143 completed
2023-02-17T03:59:57.915 INFO:teuthology.kill:2023-02-17 03:59:57,915.915 INFO:teuthology.orchestra.console:Power off for smithi153 completed
2023-02-17T03:59:58.061 INFO:teuthology.kill:2023-02-17 03:59:58,061.061 INFO:teuthology.lock.ops:unlocked: smithi143.front.sepia.ceph.com
2023-02-17T03:59:58.068 INFO:teuthology.kill:2023-02-17 03:59:58,068.068 INFO:teuthology.lock.ops:unlocked: smithi153.front.sepia.ceph.com
2023-02-17T04:01:58.441 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-02-17T04:01:58.560 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi143 but it was locked by another job: /home/teuthworker/archive/dis-2023-02-17_00:48:46-rbd-wip-dis-testing-distro-default-smithi/7177999
2023-02-17T04:01:58.561 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi153 but it was locked by another job: /home/teuthworker/archive/dis-2023-02-17_00:48:46-rbd-wip-dis-testing-distro-default-smithi/7177999