2022-10-23T22:44:30.470 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-23T22:44:30.623 INFO:teuthology.lock.ops:Start node 'smithi044.front.sepia.ceph.com' reimaging 2022-10-23T22:44:30.624 INFO:teuthology.lock.ops:Updating [smithi044.front.sepia.ceph.com]: reset os type and version on server 2022-10-23T22:44:30.625 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2022-10-23T22:44:30.648 INFO:teuthology.lock.ops:Node 'smithi044.front.sepia.ceph.com' reimaging is complete 2022-10-23T22:44:30.648 INFO:teuthology.lock.ops:Start node 'smithi094.front.sepia.ceph.com' reimaging 2022-10-23T22:44:30.649 INFO:teuthology.lock.ops:Updating [smithi094.front.sepia.ceph.com]: reset os type and version on server 2022-10-23T22:44:30.649 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2022-10-23T22:44:30.672 INFO:teuthology.lock.ops:Node 'smithi094.front.sepia.ceph.com' reimaging is complete 2022-10-23T22:44:30.868 INFO:teuthology.provision.fog.smithi044:Scheduling deploy of centos 8.stream 2022-10-23T22:44:30.882 INFO:teuthology.provision.fog.smithi094:Scheduling deploy of centos 8.stream 2022-10-23T22:44:31.209 INFO:teuthology.orchestra.console:Power off smithi044 2022-10-23T22:44:31.228 INFO:teuthology.orchestra.console:Power off smithi094 2022-10-23T22:44:39.608 INFO:teuthology.orchestra.console:Power off for smithi094 completed 2022-10-23T22:44:39.709 INFO:teuthology.orchestra.console:Power on smithi094 2022-10-23T22:44:43.713 INFO:teuthology.orchestra.console:Power off for smithi044 completed 2022-10-23T22:44:43.814 INFO:teuthology.orchestra.console:Power on smithi044 2022-10-23T22:44:48.084 INFO:teuthology.orchestra.console:Power on for smithi094 completed 2022-10-23T22:44:48.185 INFO:teuthology.provision.fog.smithi094:Waiting for deploy to finish 2022-10-23T22:44:52.305 INFO:teuthology.orchestra.console:Power on for smithi044 completed 2022-10-23T22:44:52.406 INFO:teuthology.provision.fog.smithi044:Waiting for deploy to finish 2022-10-23T22:48:57.625 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-10-23T22:49:12.248 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-10-23T22:49:12.519 INFO:teuthology.provision.fog.smithi044:Node is ready 2022-10-23T22:49:12.539 INFO:teuthology.orchestra.run.smithi044.stdout:smithi044.front.sepia.ceph.com 2022-10-23T22:49:12.599 INFO:teuthology.orchestra.run.smithi044.stdout:172.21.15.44 smithi044.front.sepia.ceph.com smithi044 2022-10-23T22:49:12.957 INFO:teuthology.provision.fog.smithi044:Deploy complete! 2022-10-23T22:49:22.805 INFO:teuthology.provision.fog.smithi094:Node is ready 2022-10-23T22:49:22.828 INFO:teuthology.orchestra.run.smithi094.stdout:smithi094.front.sepia.ceph.com 2022-10-23T22:49:22.889 INFO:teuthology.orchestra.run.smithi094.stdout:172.21.15.94 smithi094.front.sepia.ceph.com smithi094 2022-10-23T22:49:23.254 INFO:teuthology.provision.fog.smithi094:Deploy complete! 2022-10-23T22:49:23.484 INFO:teuthology.lock.ops:Checking smithi044.front.sepia.ceph.com 2022-10-23T22:49:23.485 INFO:teuthology.lock.ops:New key found. Updating... 2022-10-23T22:49:23.506 INFO:teuthology.lock.ops:Checking smithi094.front.sepia.ceph.com 2022-10-23T22:49:23.506 INFO:teuthology.lock.ops:New key found. Updating... 2022-10-23T22:49:23.530 INFO:teuthology.lock.ops:Updating [smithi044.front.sepia.ceph.com]: set os type and version on server 2022-10-23T22:49:23.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2022-10-23T22:49:23.816 INFO:teuthology.orchestra.run.smithi044.stdout:x86_64 2022-10-23T22:49:23.876 INFO:teuthology.orchestra.run.smithi044.stdout:NAME="CentOS Stream" 2022-10-23T22:49:23.877 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION="8" 2022-10-23T22:49:23.877 INFO:teuthology.orchestra.run.smithi044.stdout:ID="centos" 2022-10-23T22:49:23.878 INFO:teuthology.orchestra.run.smithi044.stdout:ID_LIKE="rhel fedora" 2022-10-23T22:49:23.878 INFO:teuthology.orchestra.run.smithi044.stdout:VERSION_ID="8" 2022-10-23T22:49:23.879 INFO:teuthology.orchestra.run.smithi044.stdout:PLATFORM_ID="platform:el8" 2022-10-23T22:49:23.879 INFO:teuthology.orchestra.run.smithi044.stdout:PRETTY_NAME="CentOS Stream 8" 2022-10-23T22:49:23.880 INFO:teuthology.orchestra.run.smithi044.stdout:ANSI_COLOR="0;31" 2022-10-23T22:49:23.880 INFO:teuthology.orchestra.run.smithi044.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-10-23T22:49:23.880 INFO:teuthology.orchestra.run.smithi044.stdout:HOME_URL="https://centos.org/" 2022-10-23T22:49:23.881 INFO:teuthology.orchestra.run.smithi044.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-10-23T22:49:23.881 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-10-23T22:49:23.882 INFO:teuthology.orchestra.run.smithi044.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-10-23T22:49:23.884 INFO:teuthology.lock.ops:Updating smithi044.front.sepia.ceph.com on lock server 2022-10-23T22:49:23.903 INFO:teuthology.lock.ops:Updating [smithi094.front.sepia.ceph.com]: set os type and version on server 2022-10-23T22:49:23.904 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2022-10-23T22:49:24.111 INFO:teuthology.orchestra.run.smithi094.stdout:x86_64 2022-10-23T22:49:24.168 INFO:teuthology.orchestra.run.smithi094.stdout:NAME="CentOS Stream" 2022-10-23T22:49:24.169 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION="8" 2022-10-23T22:49:24.169 INFO:teuthology.orchestra.run.smithi094.stdout:ID="centos" 2022-10-23T22:49:24.169 INFO:teuthology.orchestra.run.smithi094.stdout:ID_LIKE="rhel fedora" 2022-10-23T22:49:24.170 INFO:teuthology.orchestra.run.smithi094.stdout:VERSION_ID="8" 2022-10-23T22:49:24.170 INFO:teuthology.orchestra.run.smithi094.stdout:PLATFORM_ID="platform:el8" 2022-10-23T22:49:24.170 INFO:teuthology.orchestra.run.smithi094.stdout:PRETTY_NAME="CentOS Stream 8" 2022-10-23T22:49:24.170 INFO:teuthology.orchestra.run.smithi094.stdout:ANSI_COLOR="0;31" 2022-10-23T22:49:24.171 INFO:teuthology.orchestra.run.smithi094.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-10-23T22:49:24.171 INFO:teuthology.orchestra.run.smithi094.stdout:HOME_URL="https://centos.org/" 2022-10-23T22:49:24.171 INFO:teuthology.orchestra.run.smithi094.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-10-23T22:49:24.171 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-10-23T22:49:24.171 INFO:teuthology.orchestra.run.smithi094.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-10-23T22:49:24.172 INFO:teuthology.lock.ops:Updating smithi094.front.sepia.ceph.com on lock server 2022-10-23T22:49:24.276 INFO:teuthology.dispatcher.supervisor:Running job 7071587 2022-10-23T22:49:24.277 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-10-18_03:15:03-fs-quincy-distro-default-smithi/7071587 --name teuthology-2022-10-18_03:15:03-fs-quincy-distro-default-smithi --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/teuthology-2022-10-18_03:15:03-fs-quincy-distro-default-smithi/7071587/orig.config.yaml 2022-10-23T22:49:24.281 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-10-18_03:15:03-fs-quincy-distro-default-smithi/7071587 2022-10-23T22:49:24.282 INFO:teuthology.dispatcher.supervisor:Job PID: 2568 2022-10-23T22:49:24.282 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-10-24T10:50:19.398 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-10-24T10:50:19.587 INFO:teuthology.kill:Killing Pids: {2568} 2022-10-24T10:50:20.941 INFO:teuthology.task.internal:roles: ubuntu@smithi044.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-10-24T10:50:20.942 INFO:teuthology.task.internal:roles: ubuntu@smithi094.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-10-24T10:50:20.942 INFO:teuthology.misc:Compressing logs... 2022-10-24T10:50:20.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi044.front.sepia.ceph.com' 2022-10-24T10:50:21.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi094.front.sepia.ceph.com' 2022-10-24T10:50:21.450 INFO:teuthology.misc:Compressing logs... 2022-10-24T10:50:21.768 INFO:teuthology.misc:Compressing logs... 2022-10-24T10:50:34.602 INFO:teuthology.orchestra.run.smithi094.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-10-24T10:50:35.404 INFO:teuthology.orchestra.run.smithi044.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-mon.smithi044.log: file size changed while zipping 2022-10-24T10:50:41.569 INFO:teuthology.orchestra.run.smithi094.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-10-24T10:50:42.545 INFO:teuthology.orchestra.run.smithi094.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-mds.cephfs.smithi094.yotkpa.log: file size changed while zipping 2022-10-24T10:50:43.945 INFO:teuthology.orchestra.run.smithi044.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-10-24T10:50:47.192 INFO:teuthology.orchestra.run.smithi094.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-mon.smithi094.log: file size changed while zipping 2022-10-24T10:50:48.037 INFO:teuthology.orchestra.run.smithi094.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-mds.cephfs.smithi094.ddhbbd.log: file size changed while zipping 2022-10-24T10:50:51.991 INFO:teuthology.orchestra.run.smithi044.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-10-24T10:50:52.670 INFO:teuthology.orchestra.run.smithi044.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-mds.cephfs.smithi044.lkpscy.log: file size changed while zipping 2022-10-24T10:51:00.486 INFO:teuthology.orchestra.run.smithi094.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-10-24T10:51:01.294 INFO:teuthology.orchestra.run.smithi044.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-10-24T10:51:06.900 INFO:teuthology.orchestra.run.smithi044.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-mgr.smithi044.ogimwv.log: file size changed while zipping 2022-10-24T10:51:07.845 INFO:teuthology.orchestra.run.smithi044.stderr:gzip: /var/log/ceph/46a5bc3a-5326-11ed-8438-001a4aab830c/ceph-mds.cephfs.smithi044.isbaku.log: file size changed while zipping 2022-10-24T10:52:02.638 INFO:teuthology.kill:No teuthology processes running 2022-10-24T10:52:03.934 INFO:teuthology.kill:Nuking machines: ['smithi044', 'smithi094'] 2022-10-24T10:52:05.085 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-10-24T10:52:05.086 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-10-24T10:52:05.086 INFO:teuthology.kill:2022-10-24 10:52:05,085.085 INFO:teuthology.nuke:targets: 2022-10-24T10:52:05.087 INFO:teuthology.kill: smithi044.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZaoKwHwUcTqAV53jJMUljtqy6tpAS3HqNjAszaUC9zjKlQxNz8Bq7jveIaDBwZmpgc8NC0PTjYZXFKzSeUL4TpZc+NYwr/4APG3j7RbqsLioYjOpeFlhcuRJemYdy5cX1H+3jJW1FL5PCXHF4NtdIYPfcIpSA6WFBGDzdAmhYktqhtaQ72/QQR3sazRC3W75tL6Tf2twQhHbhsQoZhe8ktM9FQcsuQm0ZXsjZLHRvn/cnerPexhfXrbz+NEj0gb8Iwist3VmOMP6OD/NRAgqTVgxZI76iU11BcKMLLW/yKzrKrkUOAZD0wRjgd/K1mtMz/PMPEJtprTiZww39+jqtApSPaD4KUlfuP0GzGkFRfOSvOPtrB8z/gSp6GDJHe9MvJwGTVeN9d+BjMR1kHQ2mHbwQXpkq5z6qdziKIdcoA2hq7qwxEqWlA8WJf4ciYbei51wJmyEX4Mw7IHZxDsJWBx+kZeZqVDSzKzmiUTXGIk0rbgmaoNNa6NPGMdpcdEs= 2022-10-24T10:52:05.087 INFO:teuthology.kill: smithi094.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG/gzBRTtqT0WmjaWkctGVcppB8w0FifLxRPoDRBSSoqBzgp4D8zgqJcgc4On/dv2LwttmYvgEoGGB6ENcsR284= 2022-10-24T10:52:05.087 INFO:teuthology.kill:2022-10-24 10:52:05,085.085 INFO:teuthology.nuke:Checking targets against current locks 2022-10-24T10:52:05.138 INFO:teuthology.kill:2022-10-24 10:52:05,138.138 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-10-24T10:52:05.164 INFO:teuthology.kill:2022-10-24 10:52:05,163.163 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-10-24T10:52:05.224 INFO:teuthology.kill:2022-10-24 10:52:05,224.224 INFO:teuthology.orchestra.console:Power off smithi044 2022-10-24T10:52:05.262 INFO:teuthology.kill:2022-10-24 10:52:05,262.262 INFO:teuthology.orchestra.console:Power off smithi094 2022-10-24T10:52:13.596 INFO:teuthology.kill:2022-10-24 10:52:13,596.596 INFO:teuthology.orchestra.console:Power off for smithi044 completed 2022-10-24T10:52:13.631 INFO:teuthology.kill:2022-10-24 10:52:13,630.630 INFO:teuthology.orchestra.console:Power off for smithi094 completed 2022-10-24T10:52:13.749 INFO:teuthology.kill:2022-10-24 10:52:13,748.748 INFO:teuthology.lock.ops:unlocked: smithi044.front.sepia.ceph.com 2022-10-24T10:52:13.779 INFO:teuthology.kill:2022-10-24 10:52:13,779.779 INFO:teuthology.lock.ops:unlocked: smithi094.front.sepia.ceph.com 2022-10-24T10:54:14.113 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-10-24T10:54:14.234 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi044 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-10-24_06:07:02-rbd-pacific-distro-default-smithi/7079096 2022-10-24T10:54:14.235 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi094 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-10-24_06:07:02-rbd-pacific-distro-default-smithi/7079096