2022-05-08T17:42:07.285 INFO:root:teuthology version: 1.1.1.dev331+gfd5c3166 2022-05-08T17:42:07.412 INFO:teuthology.lock.ops:Start node 'smithi163.front.sepia.ceph.com' reimaging 2022-05-08T17:42:07.414 INFO:teuthology.lock.ops:Updating [smithi163.front.sepia.ceph.com]: reset os type and version on server 2022-05-08T17:42:07.415 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2022-05-08T17:42:07.436 INFO:teuthology.lock.ops:Node 'smithi163.front.sepia.ceph.com' reimaging is complete 2022-05-08T17:42:07.436 INFO:teuthology.lock.ops:Start node 'smithi176.front.sepia.ceph.com' reimaging 2022-05-08T17:42:07.437 INFO:teuthology.lock.ops:Updating [smithi176.front.sepia.ceph.com]: reset os type and version on server 2022-05-08T17:42:07.437 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2022-05-08T17:42:07.460 INFO:teuthology.lock.ops:Node 'smithi176.front.sepia.ceph.com' reimaging is complete 2022-05-08T17:42:07.679 INFO:teuthology.provision.fog.smithi163:Scheduling deploy of centos 8.stream 2022-05-08T17:42:07.727 INFO:teuthology.provision.fog.smithi176:Scheduling deploy of centos 8.stream 2022-05-08T17:42:08.095 INFO:teuthology.orchestra.console:Power off smithi163 2022-05-08T17:42:08.150 INFO:teuthology.orchestra.console:Power off smithi176 2022-05-08T17:42:16.463 INFO:teuthology.orchestra.console:Power off for smithi163 completed 2022-05-08T17:42:16.511 INFO:teuthology.orchestra.console:Power off for smithi176 completed 2022-05-08T17:42:16.564 INFO:teuthology.orchestra.console:Power on smithi163 2022-05-08T17:42:16.612 INFO:teuthology.orchestra.console:Power on smithi176 2022-05-08T17:42:24.926 INFO:teuthology.orchestra.console:Power on for smithi163 completed 2022-05-08T17:42:24.990 INFO:teuthology.orchestra.console:Power on for smithi176 completed 2022-05-08T17:42:25.028 INFO:teuthology.provision.fog.smithi163:Waiting for deploy to finish 2022-05-08T17:42:25.091 INFO:teuthology.provision.fog.smithi176:Waiting for deploy to finish 2022-05-08T17:46:56.730 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-05-08T17:46:56.989 INFO:teuthology.provision.fog.smithi176:Node is ready 2022-05-08T17:46:57.006 INFO:teuthology.orchestra.run.smithi176.stdout:smithi176.front.sepia.ceph.com 2022-05-08T17:46:57.064 INFO:teuthology.orchestra.run.smithi176.stdout:172.21.15.176 smithi176.front.sepia.ceph.com smithi176 2022-05-08T17:46:57.402 INFO:teuthology.provision.fog.smithi176:Deploy complete! 2022-05-08T17:47:45.076 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-05-08T17:47:45.355 INFO:teuthology.provision.fog.smithi163:Node is ready 2022-05-08T17:47:45.374 INFO:teuthology.orchestra.run.smithi163.stdout:smithi163.front.sepia.ceph.com 2022-05-08T17:47:45.434 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.15.163 smithi163.front.sepia.ceph.com smithi163 2022-05-08T17:47:45.783 INFO:teuthology.provision.fog.smithi163:Deploy complete! 2022-05-08T17:47:46.007 INFO:teuthology.lock.ops:Checking smithi163.front.sepia.ceph.com 2022-05-08T17:47:46.009 INFO:teuthology.lock.ops:New key found. Updating... 2022-05-08T17:47:46.032 INFO:teuthology.lock.ops:Checking smithi176.front.sepia.ceph.com 2022-05-08T17:47:46.033 INFO:teuthology.lock.ops:New key found. Updating... 2022-05-08T17:47:46.054 INFO:teuthology.lock.ops:Updating [smithi163.front.sepia.ceph.com]: set os type and version on server 2022-05-08T17:47:46.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2022-05-08T17:47:46.258 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2022-05-08T17:47:46.316 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2022-05-08T17:47:46.316 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="8" 2022-05-08T17:47:46.317 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2022-05-08T17:47:46.317 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2022-05-08T17:47:46.317 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="8" 2022-05-08T17:47:46.317 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el8" 2022-05-08T17:47:46.318 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 8" 2022-05-08T17:47:46.318 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2022-05-08T17:47:46.318 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-05-08T17:47:46.318 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2022-05-08T17:47:46.319 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-05-08T17:47:46.319 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-05-08T17:47:46.319 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-05-08T17:47:46.321 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2022-05-08T17:47:46.348 INFO:teuthology.lock.ops:Updating [smithi176.front.sepia.ceph.com]: set os type and version on server 2022-05-08T17:47:46.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2022-05-08T17:47:46.626 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2022-05-08T17:47:46.683 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2022-05-08T17:47:46.683 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="8" 2022-05-08T17:47:46.683 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2022-05-08T17:47:46.684 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2022-05-08T17:47:46.684 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="8" 2022-05-08T17:47:46.684 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el8" 2022-05-08T17:47:46.684 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 8" 2022-05-08T17:47:46.685 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2022-05-08T17:47:46.685 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-05-08T17:47:46.685 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2022-05-08T17:47:46.686 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-05-08T17:47:46.687 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-05-08T17:47:46.687 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-05-08T17:47:46.688 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2022-05-08T17:47:46.862 INFO:teuthology.dispatcher.supervisor:Running job 6819151 2022-05-08T17:47:46.863 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-05-03_03:15:03-fs-quincy-distro-default-smithi/6819151 --name teuthology-2022-05-03_03:15:03-fs-quincy-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/teuthology-2022-05-03_03:15:03-fs-quincy-distro-default-smithi/6819151/orig.config.yaml 2022-05-08T17:47:46.868 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-05-03_03:15:03-fs-quincy-distro-default-smithi/6819151 2022-05-08T17:47:46.868 INFO:teuthology.dispatcher.supervisor:Job PID: 5120 2022-05-08T17:47:46.869 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-05-09T00:18:15.915 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-05-09T00:18:16.133 INFO:teuthology.kill:Killing Pids: {5120} 2022-05-09T00:18:17.539 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-05-09T00:18:17.540 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-05-09T00:18:17.540 INFO:teuthology.misc:Compressing logs... 2022-05-09T00:18:17.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2022-05-09T00:18:17.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2022-05-09T00:18:18.067 INFO:teuthology.misc:Compressing logs... 2022-05-09T00:18:18.309 INFO:teuthology.misc:Compressing logs... 2022-05-09T00:18:25.756 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-05-09T00:18:26.108 INFO:teuthology.orchestra.run.smithi176.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-05-09T00:18:29.125 INFO:teuthology.orchestra.run.smithi176.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-mds.cephfs.smithi176.dbcvvr.log: file size changed while zipping 2022-05-09T00:18:39.603 INFO:teuthology.orchestra.run.smithi176.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-05-09T00:18:45.197 INFO:teuthology.orchestra.run.smithi176.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-mon.smithi176.log: file size changed while zipping 2022-05-09T00:18:49.339 INFO:teuthology.orchestra.run.smithi176.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-mds.cephfs.smithi176.sygfbg.log: file size changed while zipping 2022-05-09T00:18:52.850 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-mds.cephfs.smithi163.oqqpnz.log: file size changed while zipping 2022-05-09T00:18:59.914 INFO:teuthology.orchestra.run.smithi176.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-05-09T00:19:17.070 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-mds.cephfs.smithi163.lrlrmb.log: file size changed while zipping 2022-05-09T00:19:25.641 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-05-09T00:19:27.856 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-mgr.smithi163.lulsrd.log: file size changed while zipping 2022-05-09T00:19:39.671 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-05-09T00:19:51.029 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/d0415fb6-cef7-11ec-8c39-001a4aab830c/ceph-mon.smithi163.log: file size changed while zipping 2022-05-09T00:20:18.351 INFO:teuthology.kill:No teuthology processes running 2022-05-09T00:20:19.709 INFO:teuthology.kill:Nuking machines: ['smithi163', 'smithi176'] 2022-05-09T00:20:20.857 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-05-09T00:20:20.858 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-05-09T00:20:20.858 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_fd5c31669e51177d3c57bc0420279c5d1e944497/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated 2022-05-09T00:20:20.858 INFO:teuthology.kill: "class": algorithms.Blowfish, 2022-05-09T00:20:20.858 INFO:teuthology.kill:2022-05-09 00:20:20,856.856 INFO:teuthology.nuke:targets: 2022-05-09T00:20:20.859 INFO:teuthology.kill: smithi163.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCxmcZL1B+OfKEVwpXa+oYiPN2VlKPulj+qZNwoOfWDG6UXFAVupOa/xl3M7iN2l/g+jaDxybp9VibRN5fT1URCN8T52C6o6QDNIKylkCVBVMbjkx4367PMW0ziEY4+cDkwiAoMgCSeL6T0p2GtL+XjU4W8wT4+qnLI0dftiH7tDPnqHSD3lyY57FBEkdCckYPiHRawWW3KmkkizBRnmuc+vdl3P1NMx2834yrLDmZnnCplhVP1FFCLFp3nFJ92ta09hXai1WljoB2UVLNER2ZMjzLWj+R2eKb7Xj7HXwDTAEIJDwiDph0GbSjX8R07F4mwjfmv/M3oUFLQ7OKS+DEPaJ+zZ3eJlFzP12R8+SY0ln1cO2n6oS75smRUDE2jVrarJgo6F11lmXAEJ+HHqMghZcF+jVE6jKDM38663cbKXN9zZQGz4GNt3MayRReejvM8mtzzhGqEf00Jkl3RjW3w/hIDVAN6zTHJCSYAUSl5AMuPDCgnbLH05cHdb+I5pqM= 2022-05-09T00:20:20.859 INFO:teuthology.kill: smithi176.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpVT20LR9spLwWDMsEI242Owg3N9nRm0aemyynI31LJhL8gUYkj8kLR8cEt14h+Ys7ezM5YS011WQ6HkeL3ROwM0xOcpmXbAFnuOdn0/SxNgoW9DuOfFRwwT4k2g6dGtyBqznu6nmnZg4p9liCCtZXoM6I+dLjyS2rzMt2WepZK1fNHc63MrprxPMRoLAtdjN8dLzia4nSCt3ETxRIgYHAd8fN1IkPBTyzF2Curhzuv0YsAqnfU8JyiR3mJeD9awsbSg0p1ISoRh5JilsY958cOKcLU8rR0RPlxo0MRoF5mycwL1Y2Y2Dn1PlqUYLfVI9BJKd5Au2ZV6gZc7ApPkUGGmI8+RelB6TnGiOyuOqFkeB52wz6qmqQ6R91E+GvhfibJXKolTs6ebNu6DePfMkDpQG8X98OwZiBaqxCPN8InZeL42v2m9F01QxPORZBxxr9NiIfjYZDyNn65R7rfg9FLiB0EyDTQgJrud6iqH60O8/Wn+PpMhSiFEsrtPrH8B8= 2022-05-09T00:20:20.886 INFO:teuthology.kill:2022-05-09 00:20:20,886.886 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-09T00:20:20.891 INFO:teuthology.kill:2022-05-09 00:20:20,891.891 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-05-09T00:20:20.963 INFO:teuthology.kill:2022-05-09 00:20:20,962.962 INFO:teuthology.orchestra.console:Power off smithi163 2022-05-09T00:20:20.975 INFO:teuthology.kill:2022-05-09 00:20:20,975.975 INFO:teuthology.orchestra.console:Power off smithi176 2022-05-09T00:20:29.335 INFO:teuthology.kill:2022-05-09 00:20:29,334.334 INFO:teuthology.orchestra.console:Power off for smithi176 completed 2022-05-09T00:20:29.482 INFO:teuthology.kill:2022-05-09 00:20:29,481.481 INFO:teuthology.lock.ops:unlocked: smithi176.front.sepia.ceph.com 2022-05-09T00:20:33.441 INFO:teuthology.kill:2022-05-09 00:20:33,441.441 INFO:teuthology.orchestra.console:Power off for smithi163 completed 2022-05-09T00:20:33.588 INFO:teuthology.kill:2022-05-09 00:20:33,587.587 INFO:teuthology.lock.ops:unlocked: smithi163.front.sepia.ceph.com 2022-05-09T00:22:33.929 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-05-09T00:22:34.046 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi163 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-05-06_03:20:03-krbd-quincy-testing-default-smithi/6824138 2022-05-09T00:22:34.047 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi176 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-05-06_03:20:03-krbd-quincy-testing-default-smithi/6824138