2022-03-30T11:41:36.098 INFO:root:teuthology version: 1.1.0-d877b023
2022-03-30T11:41:36.217 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi033.front.sepia.ceph.com' reimaging
2022-03-30T11:41:36.218 INFO:teuthology.lock.ops:Updating [ubuntu@smithi033.front.sepia.ceph.com]: reset os type and version on server
2022-03-30T11:41:36.219 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server
2022-03-30T11:41:36.239 INFO:teuthology.lock.ops:Node 'ubuntu@smithi033.front.sepia.ceph.com' reimaging is complete
2022-03-30T11:41:36.240 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi045.front.sepia.ceph.com' reimaging
2022-03-30T11:41:36.241 INFO:teuthology.lock.ops:Updating [ubuntu@smithi045.front.sepia.ceph.com]: reset os type and version on server
2022-03-30T11:41:36.242 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server
2022-03-30T11:41:36.261 INFO:teuthology.lock.ops:Node 'ubuntu@smithi045.front.sepia.ceph.com' reimaging is complete
2022-03-30T11:41:36.446 INFO:teuthology.provision.fog.smithi033:Scheduling deploy of centos 8.stream
2022-03-30T11:41:36.458 INFO:teuthology.provision.fog.smithi045:Scheduling deploy of centos 8.stream
2022-03-30T11:41:36.768 INFO:teuthology.orchestra.console:Power off smithi045
2022-03-30T11:41:36.784 INFO:teuthology.orchestra.console:Power off smithi033
2022-03-30T11:41:36.916 INFO:teuthology.orchestra.console:Power off for smithi045 completed
2022-03-30T11:41:36.928 INFO:teuthology.orchestra.console:Power off for smithi033 completed
2022-03-30T11:41:37.018 INFO:teuthology.orchestra.console:Power on smithi045
2022-03-30T11:41:37.029 INFO:teuthology.orchestra.console:Power on smithi033
2022-03-30T11:41:41.271 INFO:teuthology.orchestra.console:Power on for smithi045 completed
2022-03-30T11:41:41.373 INFO:teuthology.provision.fog.smithi045:Waiting for deploy to finish
2022-03-30T11:41:45.394 INFO:teuthology.orchestra.console:Power on for smithi033 completed
2022-03-30T11:41:45.496 INFO:teuthology.provision.fog.smithi033:Waiting for deploy to finish
2022-03-30T11:45:29.597 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-30T11:45:49.317 INFO:teuthology.provision.fog.smithi045:Node is ready
2022-03-30T11:45:49.342 INFO:teuthology.orchestra.run.smithi045.stdout:smithi045.front.sepia.ceph.com
2022-03-30T11:45:49.404 INFO:teuthology.orchestra.run.smithi045.stdout:172.21.15.45 smithi045.front.sepia.ceph.com smithi045
2022-03-30T11:45:49.795 INFO:teuthology.provision.fog.smithi045:Deploy complete!
2022-03-30T11:45:53.076 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-03-30T11:45:53.327 INFO:teuthology.provision.fog.smithi033:Node is ready
2022-03-30T11:45:53.346 INFO:teuthology.orchestra.run.smithi033.stdout:smithi033.front.sepia.ceph.com
2022-03-30T11:45:53.405 INFO:teuthology.orchestra.run.smithi033.stdout:172.21.15.33 smithi033.front.sepia.ceph.com smithi033
2022-03-30T11:45:53.767 INFO:teuthology.provision.fog.smithi033:Deploy complete!
2022-03-30T11:45:53.928 INFO:teuthology.lock.ops:Checking smithi033.front.sepia.ceph.com
2022-03-30T11:45:53.929 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-30T11:45:53.963 INFO:teuthology.lock.ops:Checking smithi045.front.sepia.ceph.com
2022-03-30T11:45:53.963 INFO:teuthology.lock.ops:New key found. Updating...
2022-03-30T11:45:53.982 INFO:teuthology.lock.ops:Updating [smithi033.front.sepia.ceph.com]: set os type and version on server
2022-03-30T11:45:53.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com'
2022-03-30T11:45:54.172 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64
2022-03-30T11:45:54.229 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream"
2022-03-30T11:45:54.231 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="8"
2022-03-30T11:45:54.231 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos"
2022-03-30T11:45:54.231 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora"
2022-03-30T11:45:54.231 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="8"
2022-03-30T11:45:54.232 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el8"
2022-03-30T11:45:54.232 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-30T11:45:54.232 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31"
2022-03-30T11:45:54.232 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-30T11:45:54.232 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/"
2022-03-30T11:45:54.233 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-30T11:45:54.233 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-30T11:45:54.233 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-30T11:45:54.234 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server
2022-03-30T11:45:54.253 INFO:teuthology.lock.ops:Updating [smithi045.front.sepia.ceph.com]: set os type and version on server
2022-03-30T11:45:54.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2022-03-30T11:45:54.452 INFO:teuthology.orchestra.run.smithi045.stdout:x86_64
2022-03-30T11:45:54.512 INFO:teuthology.orchestra.run.smithi045.stdout:NAME="CentOS Stream"
2022-03-30T11:45:54.512 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION="8"
2022-03-30T11:45:54.512 INFO:teuthology.orchestra.run.smithi045.stdout:ID="centos"
2022-03-30T11:45:54.513 INFO:teuthology.orchestra.run.smithi045.stdout:ID_LIKE="rhel fedora"
2022-03-30T11:45:54.513 INFO:teuthology.orchestra.run.smithi045.stdout:VERSION_ID="8"
2022-03-30T11:45:54.513 INFO:teuthology.orchestra.run.smithi045.stdout:PLATFORM_ID="platform:el8"
2022-03-30T11:45:54.513 INFO:teuthology.orchestra.run.smithi045.stdout:PRETTY_NAME="CentOS Stream 8"
2022-03-30T11:45:54.514 INFO:teuthology.orchestra.run.smithi045.stdout:ANSI_COLOR="0;31"
2022-03-30T11:45:54.514 INFO:teuthology.orchestra.run.smithi045.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-03-30T11:45:54.514 INFO:teuthology.orchestra.run.smithi045.stdout:HOME_URL="https://centos.org/"
2022-03-30T11:45:54.514 INFO:teuthology.orchestra.run.smithi045.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-03-30T11:45:54.514 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-03-30T11:45:54.515 INFO:teuthology.orchestra.run.smithi045.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-03-30T11:45:54.515 INFO:teuthology.lock.ops:Updating smithi045.front.sepia.ceph.com on lock server
2022-03-30T11:45:54.617 INFO:teuthology.dispatcher.supervisor:Running job 6768772
2022-03-30T11:45:54.617 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_d877b023166308f282187444de8e8e725ffb27d2/virtualenv/bin/teuthology -v --owner scheduled_vshankar@teuthology --archive /home/teuthworker/archive/vshankar-2022-03-30_07:58:23-fs-wip-vshankar-testing1-20220330-104256-testing-default-smithi/6768772 --name vshankar-2022-03-30_07:58:23-fs-wip-vshankar-testing1-20220330-104256-testing-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/vshankar-2022-03-30_07:58:23-fs-wip-vshankar-testing1-20220330-104256-testing-default-smithi/6768772/orig.config.yaml
2022-03-30T11:45:54.621 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/vshankar-2022-03-30_07:58:23-fs-wip-vshankar-testing1-20220330-104256-testing-default-smithi/6768772
2022-03-30T11:45:54.622 INFO:teuthology.dispatcher.supervisor:Job PID: 13914
2022-03-30T11:45:54.622 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-03-30T18:16:24.096 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-03-30T18:16:24.283 INFO:teuthology.kill:Killing Pids: {13914}
2022-03-30T18:16:25.484 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2']
2022-03-30T18:16:25.485 INFO:teuthology.task.internal:roles: ubuntu@smithi045.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5']
2022-03-30T18:16:25.485 INFO:teuthology.misc:Compressing logs...
2022-03-30T18:16:25.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com'
2022-03-30T18:16:25.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi045.front.sepia.ceph.com'
2022-03-30T18:16:26.191 INFO:teuthology.misc:Compressing logs...
2022-03-30T18:16:26.656 INFO:teuthology.misc:Compressing logs...
2022-03-30T18:16:26.885 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-mgr.smithi045.vneosk.log: file size changed while zipping
2022-03-30T18:16:35.659 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-03-30T18:16:43.342 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-03-30T18:16:52.481 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi045.thrwrp.log: file size changed while zipping
2022-03-30T18:16:55.019 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-mon.smithi033.log: file size changed while zipping
2022-03-30T18:16:57.161 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi045.nhweyv.log: file size changed while zipping
2022-03-30T18:16:57.701 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi033.sejbqg.log: file size changed while zipping
2022-03-30T18:17:00.004 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-mgr.smithi033.pzfmav.log: file size changed while zipping
2022-03-30T18:17:10.127 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-03-30T18:17:13.856 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-mon.smithi045.log: file size changed while zipping
2022-03-30T18:17:19.834 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi033.datzaz.log: file size changed while zipping
2022-03-30T18:17:26.219 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-03-30T18:17:29.244 INFO:teuthology.orchestra.run.smithi033.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-03-30T18:17:34.484 INFO:teuthology.orchestra.run.smithi045.stderr:gzip: /var/log/ceph/2b49595e-b021-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-03-30T18:18:01.063 INFO:teuthology.kill:No teuthology processes running
2022-03-30T18:18:02.205 INFO:teuthology.kill:Nuking machines: ['smithi033', 'smithi045']
2022-03-30T18:18:03.189 INFO:teuthology.kill:2022-03-30 18:18:03,189.189 INFO:teuthology.nuke:targets:
2022-03-30T18:18:03.190 INFO:teuthology.kill:  smithi033.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAkvtO4yaH5dad131XIKpHG+DGvyEsfSgJPnvBFTKUbcAa2uSJdDWq4qzsWVvQcn9glqrHlqQuBeK4SxH9ynbBRzy+O0FfPL/P6AduMU0xGtjyUCMi1VD4pzNXYBQgVnEw9dXH15lSQ+J0OomMDXm0MUer4ylDxzKCaeig2j9ixUss565UfMCfvQnOhT5/MRSb61rNrR4YqXLa3UXjiZDsOadOiMef5IvZOlE9NsVZb3IjbGzxkIF3yPmwI3ZaKDHJ6Kj9GRHlGS9aQ8MvlAuFkvpmiu2U6iC2jJ2yfvID0+V5ELXwDk91h/rvhdvC8ip5Kjmdq/r+9salgelXVtvYA99Pbp5torQ20kDR0StkquzEB0EvzUV88WodOxpzHaQl36KPjyukLZSX/yjb/regD08F7pHcLJAjgwLq3wtKhbPPTLM7LqvAAEjPGCiAbo/aNmN9rtnsaq+TVO4kW/bMuM98hBDqP5/7oIBHah15dfLreJfNxnuphzKQAP3XCpU=
2022-03-30T18:18:03.191 INFO:teuthology.kill:  smithi045.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOdW5H1mhYmq3TdYRq68YwRU1HAbWRmZrIqvprBXU6K69OtvDypq4B9X5IyDBNXKAthbyIsWH65RRJMB9x5ssKW7ni+3iZXc3NtjgOwL5wZcxkujSxL8xbKxTzF4UcEXVCmLXqb4S7NHwUyYugbOr1C5CmHUSTQ1SBD6IrGz2AjBKOUEfVRyZ0cmYUHaqgLvYLQogBTOKl7wgqvqKxp1Mtjmn59idP7RsM35ZGtRcaWdsb6gcMQwW3u1+FTu1LSiVtRAHf8JQtBChDGwrcrxnaTv80PdGHgdX67MEzRbDg4VPUzvzwhym1okqpzZvJMZW7DGBKBkadYppuFGIv+GDkdwoBMTkDcJWq+NJtGnxczbAK1Ifpr/eE5OchsAfLiDMdmwzZSob5ZfBPtJa0dY45yR3w9uN6qUkA50WbSL2KZ2ZczkrcMBmii13pnFHNzpvNTdtkGXQIYFfZEPHNKEknUdp5wQjspbSirxQ7qNie+1qRgoWRcqMZBPOlEJyEPZc=
2022-03-30T18:18:03.216 INFO:teuthology.kill:2022-03-30 18:18:03,216.216 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-30T18:18:03.219 INFO:teuthology.kill:2022-03-30 18:18:03,219.219 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-03-30T18:18:03.293 INFO:teuthology.kill:2022-03-30 18:18:03,293.293 INFO:teuthology.orchestra.console:Power off smithi033
2022-03-30T18:18:03.309 INFO:teuthology.kill:2022-03-30 18:18:03,309.309 INFO:teuthology.orchestra.console:Power off smithi045
2022-03-30T18:18:11.673 INFO:teuthology.kill:2022-03-30 18:18:11,672.672 INFO:teuthology.orchestra.console:Power off for smithi033 completed
2022-03-30T18:18:11.678 INFO:teuthology.kill:2022-03-30 18:18:11,678.678 INFO:teuthology.orchestra.console:Power off for smithi045 completed
2022-03-30T18:18:11.822 INFO:teuthology.kill:2022-03-30 18:18:11,821.821 INFO:teuthology.lock.ops:unlocked: smithi045.front.sepia.ceph.com
2022-03-30T18:18:11.825 INFO:teuthology.kill:2022-03-30 18:18:11,825.825 INFO:teuthology.lock.ops:unlocked: smithi033.front.sepia.ceph.com
2022-03-30T18:20:12.035 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-03-30T18:20:12.146 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi033 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-03-30_18:13:29-upgrade:octopus-x-quincy-distro-default-smithi/6769194
2022-03-30T18:20:12.147 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi045 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-03-30_18:13:29-upgrade:octopus-x-quincy-distro-default-smithi/6769194