2022-10-21T05:58:31.833 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-21T05:58:31.988 INFO:teuthology.lock.ops:Start node 'smithi132.front.sepia.ceph.com' reimaging 2022-10-21T05:58:31.990 INFO:teuthology.lock.ops:Updating [smithi132.front.sepia.ceph.com]: reset os type and version on server 2022-10-21T05:58:31.990 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2022-10-21T05:58:32.010 INFO:teuthology.lock.ops:Node 'smithi132.front.sepia.ceph.com' reimaging is complete 2022-10-21T05:58:32.011 INFO:teuthology.lock.ops:Start node 'smithi138.front.sepia.ceph.com' reimaging 2022-10-21T05:58:32.011 INFO:teuthology.lock.ops:Updating [smithi138.front.sepia.ceph.com]: reset os type and version on server 2022-10-21T05:58:32.012 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2022-10-21T05:58:32.031 INFO:teuthology.lock.ops:Node 'smithi138.front.sepia.ceph.com' reimaging is complete 2022-10-21T05:58:32.031 INFO:teuthology.lock.ops:Start node 'smithi201.front.sepia.ceph.com' reimaging 2022-10-21T05:58:32.031 INFO:teuthology.lock.ops:Updating [smithi201.front.sepia.ceph.com]: reset os type and version on server 2022-10-21T05:58:32.032 INFO:teuthology.lock.ops:Updating smithi201.front.sepia.ceph.com on lock server 2022-10-21T05:58:32.050 INFO:teuthology.lock.ops:Node 'smithi201.front.sepia.ceph.com' reimaging is complete 2022-10-21T05:58:32.223 INFO:teuthology.provision.fog.smithi132:Scheduling deploy of centos 8.stream 2022-10-21T05:58:32.261 INFO:teuthology.provision.fog.smithi138:Scheduling deploy of centos 8.stream 2022-10-21T05:58:32.284 INFO:teuthology.provision.fog.smithi201:Scheduling deploy of centos 8.stream 2022-10-21T05:58:32.514 INFO:teuthology.orchestra.console:Power off smithi132 2022-10-21T05:58:32.568 INFO:teuthology.orchestra.console:Power off smithi138 2022-10-21T05:58:32.582 INFO:teuthology.orchestra.console:Power off smithi201 2022-10-21T05:58:32.652 INFO:teuthology.orchestra.console:Power off for smithi132 completed 2022-10-21T05:58:32.707 INFO:teuthology.orchestra.console:Power off for smithi138 completed 2022-10-21T05:58:32.719 INFO:teuthology.orchestra.console:Power off for smithi201 completed 2022-10-21T05:58:32.753 INFO:teuthology.orchestra.console:Power on smithi132 2022-10-21T05:58:32.809 INFO:teuthology.orchestra.console:Power on smithi138 2022-10-21T05:58:32.820 INFO:teuthology.orchestra.console:Power on smithi201 2022-10-21T05:58:41.116 INFO:teuthology.orchestra.console:Power on for smithi132 completed 2022-10-21T05:58:41.170 INFO:teuthology.orchestra.console:Power on for smithi201 completed 2022-10-21T05:58:41.203 INFO:teuthology.orchestra.console:Power on for smithi138 completed 2022-10-21T05:58:41.217 INFO:teuthology.provision.fog.smithi132:Waiting for deploy to finish 2022-10-21T05:58:41.271 INFO:teuthology.provision.fog.smithi201:Waiting for deploy to finish 2022-10-21T05:58:41.304 INFO:teuthology.provision.fog.smithi138:Waiting for deploy to finish 2022-10-21T06:03:10.129 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-10-21T06:03:10.398 INFO:teuthology.provision.fog.smithi201:Node is ready 2022-10-21T06:03:10.416 INFO:teuthology.orchestra.run.smithi201.stdout:smithi201.front.sepia.ceph.com 2022-10-21T06:03:10.476 INFO:teuthology.orchestra.run.smithi201.stdout:172.21.15.201 smithi201.front.sepia.ceph.com smithi201 2022-10-21T06:03:10.822 INFO:teuthology.provision.fog.smithi201:Deploy complete! 2022-10-21T06:03:19.193 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-10-21T06:03:19.199 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-10-21T06:03:19.461 INFO:teuthology.provision.fog.smithi138:Node is ready 2022-10-21T06:03:19.473 INFO:teuthology.provision.fog.smithi132:Node is ready 2022-10-21T06:03:19.480 INFO:teuthology.orchestra.run.smithi138.stdout:smithi138.front.sepia.ceph.com 2022-10-21T06:03:19.491 INFO:teuthology.orchestra.run.smithi132.stdout:smithi132.front.sepia.ceph.com 2022-10-21T06:03:19.539 INFO:teuthology.orchestra.run.smithi138.stdout:172.21.15.138 smithi138.front.sepia.ceph.com smithi138 2022-10-21T06:03:19.549 INFO:teuthology.orchestra.run.smithi132.stdout:172.21.15.132 smithi132.front.sepia.ceph.com smithi132 2022-10-21T06:03:19.884 INFO:teuthology.provision.fog.smithi138:Deploy complete! 2022-10-21T06:03:19.890 INFO:teuthology.provision.fog.smithi132:Deploy complete! 2022-10-21T06:03:20.083 INFO:teuthology.lock.ops:Checking smithi132.front.sepia.ceph.com 2022-10-21T06:03:20.083 INFO:teuthology.lock.ops:New key found. Updating... 2022-10-21T06:03:20.102 INFO:teuthology.lock.ops:Checking smithi138.front.sepia.ceph.com 2022-10-21T06:03:20.102 INFO:teuthology.lock.ops:New key found. Updating... 2022-10-21T06:03:20.122 INFO:teuthology.lock.ops:Checking smithi201.front.sepia.ceph.com 2022-10-21T06:03:20.122 INFO:teuthology.lock.ops:New key found. Updating... 2022-10-21T06:03:20.143 INFO:teuthology.lock.ops:Updating [smithi132.front.sepia.ceph.com]: set os type and version on server 2022-10-21T06:03:20.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2022-10-21T06:03:20.341 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2022-10-21T06:03:20.398 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2022-10-21T06:03:20.398 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="8" 2022-10-21T06:03:20.398 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2022-10-21T06:03:20.399 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2022-10-21T06:03:20.399 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="8" 2022-10-21T06:03:20.399 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el8" 2022-10-21T06:03:20.399 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 8" 2022-10-21T06:03:20.400 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2022-10-21T06:03:20.400 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-10-21T06:03:20.400 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2022-10-21T06:03:20.400 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-10-21T06:03:20.400 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-10-21T06:03:20.401 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-10-21T06:03:20.402 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2022-10-21T06:03:20.420 INFO:teuthology.lock.ops:Updating [smithi138.front.sepia.ceph.com]: set os type and version on server 2022-10-21T06:03:20.421 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2022-10-21T06:03:20.573 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2022-10-21T06:03:20.631 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Stream" 2022-10-21T06:03:20.631 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="8" 2022-10-21T06:03:20.631 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos" 2022-10-21T06:03:20.632 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora" 2022-10-21T06:03:20.632 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="8" 2022-10-21T06:03:20.632 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el8" 2022-10-21T06:03:20.632 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Stream 8" 2022-10-21T06:03:20.632 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2022-10-21T06:03:20.633 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-10-21T06:03:20.633 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://centos.org/" 2022-10-21T06:03:20.633 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-10-21T06:03:20.633 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-10-21T06:03:20.633 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-10-21T06:03:20.634 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2022-10-21T06:03:20.652 INFO:teuthology.lock.ops:Updating [smithi201.front.sepia.ceph.com]: set os type and version on server 2022-10-21T06:03:20.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi201.front.sepia.ceph.com' 2022-10-21T06:03:20.854 INFO:teuthology.orchestra.run.smithi201.stdout:x86_64 2022-10-21T06:03:20.913 INFO:teuthology.orchestra.run.smithi201.stdout:NAME="CentOS Stream" 2022-10-21T06:03:20.914 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION="8" 2022-10-21T06:03:20.914 INFO:teuthology.orchestra.run.smithi201.stdout:ID="centos" 2022-10-21T06:03:20.914 INFO:teuthology.orchestra.run.smithi201.stdout:ID_LIKE="rhel fedora" 2022-10-21T06:03:20.914 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION_ID="8" 2022-10-21T06:03:20.915 INFO:teuthology.orchestra.run.smithi201.stdout:PLATFORM_ID="platform:el8" 2022-10-21T06:03:20.915 INFO:teuthology.orchestra.run.smithi201.stdout:PRETTY_NAME="CentOS Stream 8" 2022-10-21T06:03:20.915 INFO:teuthology.orchestra.run.smithi201.stdout:ANSI_COLOR="0;31" 2022-10-21T06:03:20.915 INFO:teuthology.orchestra.run.smithi201.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-10-21T06:03:20.916 INFO:teuthology.orchestra.run.smithi201.stdout:HOME_URL="https://centos.org/" 2022-10-21T06:03:20.916 INFO:teuthology.orchestra.run.smithi201.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-10-21T06:03:20.916 INFO:teuthology.orchestra.run.smithi201.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-10-21T06:03:20.916 INFO:teuthology.orchestra.run.smithi201.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-10-21T06:03:20.917 INFO:teuthology.lock.ops:Updating smithi201.front.sepia.ceph.com on lock server 2022-10-21T06:03:21.014 INFO:teuthology.dispatcher.supervisor:Running job 7075738 2022-10-21T06:03:21.014 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075738 --name yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi --description fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} -- /home/teuthworker/archive/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075738/orig.config.yaml 2022-10-21T06:03:21.018 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075738 2022-10-21T06:03:21.018 INFO:teuthology.dispatcher.supervisor:Job PID: 27570 2022-10-21T06:03:21.019 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-10-21T18:04:07.019 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-10-21T18:04:07.994 INFO:teuthology.kill:Killing Pids: {27570} 2022-10-21T18:04:09.298 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2022-10-21T18:04:09.299 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2022-10-21T18:04:09.299 INFO:teuthology.task.internal:roles: ubuntu@smithi201.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2022-10-21T18:04:09.300 INFO:teuthology.misc:Compressing logs... 2022-10-21T18:04:09.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2022-10-21T18:04:09.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2022-10-21T18:04:09.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi201.front.sepia.ceph.com' 2022-10-21T18:04:10.280 INFO:teuthology.kill:No teuthology processes running 2022-10-21T18:04:11.448 INFO:teuthology.kill:Nuking machines: ['smithi132', 'smithi138', 'smithi201'] 2022-10-21T18:04:12.444 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-10-21T18:04:12.445 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-10-21T18:04:12.445 INFO:teuthology.kill:2022-10-21 18:04:12,443.443 INFO:teuthology.nuke:targets: 2022-10-21T18:04:12.445 INFO:teuthology.kill: smithi132.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQClEYcmLdrydbXA5DxqpATmrNjyLzw7ZIQaMOp5w1jBau/500ZN08FK99/7BlOPLFiIDDA3Dt+yLozzsFVRR2f7xOXwTeF3vIbbMzbWjms95GceFOa3XUIqitiOFjIftYEnaosC4RZCAhFFGaSabcFvT4CRYOnS8sR7o/5n4Vc2Pt4HYRqTxUKtLwRJqS9ZMOY+6v5NGnL0i2JfdDGV+RdcbGCt5r1fPhCVvjJI4Vvo2ZQctHZKK6RdE3SB1vHB9SA6U3m+cr9pGq0AcsXik/HXaE609aeG/XxvOsV/yAziG0cxUjPr2/KOa/v248GvUoJF4uXv26JCx8WjzFImrBgvw1doFUdSePtparNOyuU9OOTZ8Ae5i19krmxx3pSgyuQRV1DhCcJMIhtWEsDFxN2tDJQFL25vnIwObEERqXmY9nyH2ygmJpheXdWu97/Ah9tXJRyr+UMHXp9XbCzv3OK2GP0dKpyQzXVIfCLl6zjn097pzy9j8cf3JXRZrI7e8BE= 2022-10-21T18:04:12.446 INFO:teuthology.kill: smithi138.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHybDoDimp7tp5JlTnrmMeC7y1WnDa2iUWw9oXsSQHHyv3Kikv+2IYL0eo9/8ye+5LS6uPsEGI9ps6q7uytstQKrOFc9AyTJo5REPNELH/0QYQG/DOt7ModusCMHfHuvHYkJNeMhCkM40gqjoXW9nVOnJr5kJGBf4Ft6ZiTRoxP3mRNX/Or+sCOXAb6HqtvCWylVUn/8TSzKI9Ppav31RsZFlzmktvk2qrwo0GFqBRschp3oeV3JGpKiW/OrAtb8Yx4zgoRQMxlfuNS2qU1jsKG6WmDnosAbtiIOtI/tR8qD3P5vMM+CmXk4heabw6BQj7j1QdFZz+2D1p6mL9UGM6koNDsFuhaRsnl+3dIyZn5kfNWFUaQ0o2QyT26g260muSFlQtFihUO9M+h8ncyh+7ly6tjNKVN58TuBsWBdZsFe6xkH3ArXAk6YFQ7ZJOc0qVwHZmTFVlCgsl5s9RPF37fLGowtQ+3s8E7kZun9pGwJEKeDfWq2XL/gJwMaMH1r0= 2022-10-21T18:04:12.446 INFO:teuthology.kill: smithi201.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBD/X1c1yh7adkIkIQ1cseIMH1Rb65jn+tU2bQ/+IMc8HkR+77PesKVLlK+/OhBiNj4m7uasQ4ygap7EO6iNahso= 2022-10-21T18:04:12.446 INFO:teuthology.kill:2022-10-21 18:04:12,443.443 INFO:teuthology.nuke:Checking targets against current locks 2022-10-21T18:04:12.496 INFO:teuthology.kill:2022-10-21 18:04:12,496.496 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-10-21T18:04:12.518 INFO:teuthology.kill:2022-10-21 18:04:12,518.518 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-10-21T18:04:12.536 INFO:teuthology.kill:2022-10-21 18:04:12,536.536 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-10-21T18:04:12.570 INFO:teuthology.kill:2022-10-21 18:04:12,570.570 INFO:teuthology.orchestra.console:Power off smithi132 2022-10-21T18:04:12.608 INFO:teuthology.kill:2022-10-21 18:04:12,608.608 INFO:teuthology.orchestra.console:Power off smithi201 2022-10-21T18:04:12.619 INFO:teuthology.kill:2022-10-21 18:04:12,619.619 INFO:teuthology.orchestra.console:Power off smithi138 2022-10-21T18:04:20.963 INFO:teuthology.kill:2022-10-21 18:04:20,963.963 INFO:teuthology.orchestra.console:Power off for smithi201 completed 2022-10-21T18:04:21.110 INFO:teuthology.kill:2022-10-21 18:04:21,109.109 INFO:teuthology.lock.ops:unlocked: smithi201.front.sepia.ceph.com 2022-10-21T18:04:25.065 INFO:teuthology.kill:2022-10-21 18:04:25,064.064 INFO:teuthology.orchestra.console:Power off for smithi132 completed 2022-10-21T18:04:25.080 INFO:teuthology.kill:2022-10-21 18:04:25,080.080 INFO:teuthology.orchestra.console:Power off for smithi138 completed 2022-10-21T18:04:25.215 INFO:teuthology.kill:2022-10-21 18:04:25,214.214 INFO:teuthology.lock.ops:unlocked: smithi132.front.sepia.ceph.com 2022-10-21T18:04:25.225 INFO:teuthology.kill:2022-10-21 18:04:25,224.224 INFO:teuthology.lock.ops:unlocked: smithi138.front.sepia.ceph.com 2022-10-21T18:06:25.468 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-10-21T18:06:25.599 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi132 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075789 2022-10-21T18:06:25.600 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi138 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-10-20_14:33:23-fs-wip-yuri2-testing-2022-10-19-1252-pacific-distro-default-smithi/7075789 2022-10-21T18:06:25.600 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi201 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-10-20_14:31:28-rados-wip-yuri11-testing-2022-10-19-1648-distro-default-smithi/7075790