2022-06-26T05:57:26.022 INFO:root:teuthology version: 1.1.1.dev376+g082ae7ef 2022-06-26T05:57:26.189 INFO:teuthology.lock.ops:Start node 'smithi018.front.sepia.ceph.com' reimaging 2022-06-26T05:57:26.190 INFO:teuthology.lock.ops:Updating [smithi018.front.sepia.ceph.com]: reset os type and version on server 2022-06-26T05:57:26.191 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2022-06-26T05:57:26.232 INFO:teuthology.lock.ops:Node 'smithi018.front.sepia.ceph.com' reimaging is complete 2022-06-26T05:57:26.233 INFO:teuthology.lock.ops:Start node 'smithi111.front.sepia.ceph.com' reimaging 2022-06-26T05:57:26.234 INFO:teuthology.lock.ops:Updating [smithi111.front.sepia.ceph.com]: reset os type and version on server 2022-06-26T05:57:26.234 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2022-06-26T05:57:26.278 INFO:teuthology.lock.ops:Node 'smithi111.front.sepia.ceph.com' reimaging is complete 2022-06-26T05:57:26.517 INFO:teuthology.provision.fog.smithi018:Scheduling deploy of centos 8.stream 2022-06-26T05:57:26.555 INFO:teuthology.provision.fog.smithi111:Scheduling deploy of centos 8.stream 2022-06-26T05:57:27.386 INFO:teuthology.orchestra.console:Power off smithi111 2022-06-26T05:57:27.471 INFO:teuthology.orchestra.console:Power off smithi018 2022-06-26T05:57:35.887 INFO:teuthology.orchestra.console:Power off for smithi111 completed 2022-06-26T05:57:35.989 INFO:teuthology.orchestra.console:Power on smithi111 2022-06-26T05:57:40.065 INFO:teuthology.orchestra.console:Power off for smithi018 completed 2022-06-26T05:57:40.169 INFO:teuthology.orchestra.console:Power on smithi018 2022-06-26T05:57:44.552 INFO:teuthology.orchestra.console:Power on for smithi111 completed 2022-06-26T05:57:44.657 INFO:teuthology.provision.fog.smithi111:Waiting for deploy to finish 2022-06-26T05:57:48.628 INFO:teuthology.orchestra.console:Power on for smithi018 completed 2022-06-26T05:57:48.729 INFO:teuthology.provision.fog.smithi018:Waiting for deploy to finish 2022-06-26T06:01:56.742 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-06-26T06:01:57.084 INFO:teuthology.provision.fog.smithi018:Node is ready 2022-06-26T06:01:57.106 INFO:teuthology.orchestra.run.smithi018.stdout:smithi018.front.sepia.ceph.com 2022-06-26T06:01:57.175 INFO:teuthology.orchestra.run.smithi018.stdout:172.21.15.18 smithi018.front.sepia.ceph.com smithi018 2022-06-26T06:01:57.594 INFO:teuthology.provision.fog.smithi018:Deploy complete! 2022-06-26T06:02:50.282 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-06-26T06:02:50.585 INFO:teuthology.provision.fog.smithi111:Node is ready 2022-06-26T06:02:50.608 INFO:teuthology.orchestra.run.smithi111.stdout:smithi111.front.sepia.ceph.com 2022-06-26T06:02:50.669 INFO:teuthology.orchestra.run.smithi111.stdout:172.21.15.111 smithi111.front.sepia.ceph.com smithi111 2022-06-26T06:02:51.123 INFO:teuthology.provision.fog.smithi111:Deploy complete! 2022-06-26T06:02:51.446 INFO:teuthology.lock.ops:Checking smithi018.front.sepia.ceph.com 2022-06-26T06:02:51.447 INFO:teuthology.lock.ops:New key found. Updating... 2022-06-26T06:02:51.476 INFO:teuthology.lock.ops:Checking smithi111.front.sepia.ceph.com 2022-06-26T06:02:51.477 INFO:teuthology.lock.ops:New key found. Updating... 2022-06-26T06:02:51.497 INFO:teuthology.lock.ops:Updating [smithi018.front.sepia.ceph.com]: set os type and version on server 2022-06-26T06:02:51.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-06-26T06:02:51.783 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2022-06-26T06:02:51.842 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="CentOS Stream" 2022-06-26T06:02:51.842 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="8" 2022-06-26T06:02:51.843 INFO:teuthology.orchestra.run.smithi018.stdout:ID="centos" 2022-06-26T06:02:51.843 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE="rhel fedora" 2022-06-26T06:02:51.843 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="8" 2022-06-26T06:02:51.844 INFO:teuthology.orchestra.run.smithi018.stdout:PLATFORM_ID="platform:el8" 2022-06-26T06:02:51.844 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="CentOS Stream 8" 2022-06-26T06:02:51.844 INFO:teuthology.orchestra.run.smithi018.stdout:ANSI_COLOR="0;31" 2022-06-26T06:02:51.844 INFO:teuthology.orchestra.run.smithi018.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-06-26T06:02:51.845 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://centos.org/" 2022-06-26T06:02:51.845 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-06-26T06:02:51.845 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-06-26T06:02:51.845 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-06-26T06:02:51.847 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2022-06-26T06:02:51.869 INFO:teuthology.lock.ops:Updating [smithi111.front.sepia.ceph.com]: set os type and version on server 2022-06-26T06:02:51.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-06-26T06:02:52.054 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2022-06-26T06:02:52.116 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2022-06-26T06:02:52.116 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="8" 2022-06-26T06:02:52.117 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2022-06-26T06:02:52.117 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2022-06-26T06:02:52.117 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="8" 2022-06-26T06:02:52.118 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el8" 2022-06-26T06:02:52.118 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 8" 2022-06-26T06:02:52.119 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2022-06-26T06:02:52.119 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-06-26T06:02:52.120 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2022-06-26T06:02:52.120 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-06-26T06:02:52.121 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-06-26T06:02:52.121 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-06-26T06:02:52.122 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2022-06-26T06:02:52.249 INFO:teuthology.dispatcher.supervisor:Running job 6898815 2022-06-26T06:02:52.249 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-06-26_03:31:02-rados-pacific-distro-default-smithi/6898815 --name teuthology-2022-06-26_03:31:02-rados-pacific-distro-default-smithi --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/teuthology-2022-06-26_03:31:02-rados-pacific-distro-default-smithi/6898815/orig.config.yaml 2022-06-26T06:02:52.254 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-06-26_03:31:02-rados-pacific-distro-default-smithi/6898815 2022-06-26T06:02:52.255 INFO:teuthology.dispatcher.supervisor:Job PID: 7783 2022-06-26T06:02:52.256 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-06-26T18:03:46.865 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-06-26T18:03:47.652 INFO:teuthology.kill:Killing Pids: {7783} 2022-06-26T18:03:49.143 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-06-26T18:03:49.144 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-06-26T18:03:49.144 INFO:teuthology.misc:Compressing logs... 2022-06-26T18:03:49.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2022-06-26T18:03:49.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2022-06-26T18:03:49.686 INFO:teuthology.misc:Compressing logs... 2022-06-26T18:03:50.173 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-06-26T18:03:50.394 INFO:teuthology.misc:Compressing logs... 2022-06-26T18:03:57.347 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/8fd98396-f516-11ec-842c-001a4aab830c/ceph-mgr.smithi018.mmtcdm.log: file size changed while zipping 2022-06-26T18:04:06.446 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/8fd98396-f516-11ec-842c-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-06-26T18:04:13.363 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/8fd98396-f516-11ec-842c-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-06-26T18:04:30.986 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/8fd98396-f516-11ec-842c-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-06-26T18:04:56.767 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/8fd98396-f516-11ec-842c-001a4aab830c/ceph-mon.smithi018.log: file size changed while zipping 2022-06-26T18:05:10.740 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/8fd98396-f516-11ec-842c-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-06-26T18:05:18.327 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/8fd98396-f516-11ec-842c-001a4aab830c/ceph-mon.smithi111.log: file size changed while zipping 2022-06-26T18:05:31.996 INFO:teuthology.orchestra.run.smithi111.stderr:gzip: /var/log/ceph/8fd98396-f516-11ec-842c-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-06-26T18:06:06.838 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/8fd98396-f516-11ec-842c-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-06-26T18:06:10.346 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/8fd98396-f516-11ec-842c-001a4aab830c/ceph-mds.cephfs.smithi018.bsgfdk.log: file size changed while zipping 2022-06-26T18:06:58.476 INFO:teuthology.kill:No teuthology processes running 2022-06-26T18:06:59.755 INFO:teuthology.kill:Nuking machines: ['smithi018', 'smithi111'] 2022-06-26T18:07:00.811 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-06-26T18:07:00.812 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-06-26T18:07:00.813 INFO:teuthology.kill:2022-06-26 18:07:00,811.811 INFO:teuthology.nuke:targets: 2022-06-26T18:07:00.813 INFO:teuthology.kill: smithi018.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMw9e7XHkqyMJbrg5sa9LMcalcnZ28OOqbKJ4xBL+VNkPvADKZlLRrE8Uw6RnREw1HmiLbn1Mo4T6Pp91x4KgWEdGh8UEnNO9gvIvibVrciH9X9xKWnFNSz/pwpzWDv3K7ExRILuwlTcWpUykoh3RF/Gi9P3D8qBSXRayBT/JJ2zP2P+PXtlHtrGx+DNb2tqUYlUDz3GqvYT1MSjKibbqkjfCpV4EmIFGbZlSJDLKpV8G9ab+jXlnSE/WDdume+f3QYTl6xg4P2vctAilq9KDVY9TF+WNNvWa3jTKI9M5GcO//dh1d68+Q7FsIuwJyl8y8BhiqXq7ZN4PT1xtlTvMFKBiQzmhrAhant8RnuW53gC7+g87ldy8hHSBcTWdT8WKM9lcv5JdBfBGbUoUYvZctKT3pjnTpC44hBLEIcZQ7IZ1kfJSC8+9oQkcrkKVBp9o/q/h5iE1nomlkSqlqclggMvy1dYKmwr2AozZEAX6UJGjoxnk9wYa106/AFJTaVm0= 2022-06-26T18:07:00.813 INFO:teuthology.kill: smithi111.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwSVxQfR7jBYpF5wZLf8Oj7s/gi9iaWrjDMXantkG+zJqDS4FVfW4+Wr4lk/vcff+6tdLF4+1yvGgERRBZq/NTaOohOI7uI8VEollxP7KExgYumgf+DAlae1XsMkCyLCAWg0UaI3ZGKbvKhEr/xGIvgBndkNtbfTbF8Epcgi6xgzEMEGbhe5LSr3TG1HASgoxLv6Y+SJhR+BadbKT3DZ69KrMM7seLvbIeD0h0Ck8Xak0Z+OBnLVjXDBj6Mop7bjMy+VlOxmrm0EgNPEEsci1ktKXuDdyt6+zNczmVPbM+sWpjFaPur14vshxBABm6gkyR0kaP5nnI0s0PoD9xQKhyooliSGPIQ38ZntxQcmcJldJGKxlgK6ge3imosIZD2bQc3ahindcJypVzQRAAllVgrLy5R6mt/R4NrKwtRLyKrtm9IHc4uNKAYRehPPQyYBlHvt27aqTnMJpWUE+Wz/08oPt/rdBvMtIQI3UlVzPPxKRIpjN2ex84fwt+oxyePk8= 2022-06-26T18:07:00.814 INFO:teuthology.kill:2022-06-26 18:07:00,811.811 INFO:teuthology.nuke:Checking targets against current locks 2022-06-26T18:07:00.866 INFO:teuthology.kill:2022-06-26 18:07:00,866.866 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-06-26T18:07:00.888 INFO:teuthology.kill:2022-06-26 18:07:00,888.888 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-06-26T18:07:00.947 INFO:teuthology.kill:2022-06-26 18:07:00,946.946 INFO:teuthology.orchestra.console:Power off smithi018 2022-06-26T18:07:00.992 INFO:teuthology.kill:2022-06-26 18:07:00,992.992 INFO:teuthology.orchestra.console:Power off smithi111 2022-06-26T18:07:09.327 INFO:teuthology.kill:2022-06-26 18:07:09,326.326 INFO:teuthology.orchestra.console:Power off for smithi018 completed 2022-06-26T18:07:09.485 INFO:teuthology.kill:2022-06-26 18:07:09,485.485 INFO:teuthology.lock.ops:unlocked: smithi018.front.sepia.ceph.com 2022-06-26T18:07:13.470 INFO:teuthology.kill:2022-06-26 18:07:13,469.469 INFO:teuthology.orchestra.console:Power off for smithi111 completed 2022-06-26T18:07:13.620 INFO:teuthology.kill:2022-06-26 18:07:13,620.620 INFO:teuthology.lock.ops:unlocked: smithi111.front.sepia.ceph.com 2022-06-26T18:09:13.974 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15