2022-01-10T14:57:52.304 INFO:root:teuthology version: 1.1.0-00e09f05 2022-01-10T14:57:52.428 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi118.front.sepia.ceph.com' reimaging 2022-01-10T14:57:52.429 INFO:teuthology.lock.ops:Updating [ubuntu@smithi118.front.sepia.ceph.com]: reset os type and version on server 2022-01-10T14:57:52.429 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2022-01-10T14:57:52.448 INFO:teuthology.lock.ops:Node 'ubuntu@smithi118.front.sepia.ceph.com' reimaging is complete 2022-01-10T14:57:52.449 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi139.front.sepia.ceph.com' reimaging 2022-01-10T14:57:52.449 INFO:teuthology.lock.ops:Updating [ubuntu@smithi139.front.sepia.ceph.com]: reset os type and version on server 2022-01-10T14:57:52.450 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2022-01-10T14:57:52.470 INFO:teuthology.lock.ops:Node 'ubuntu@smithi139.front.sepia.ceph.com' reimaging is complete 2022-01-10T14:57:52.658 INFO:teuthology.provision.fog.smithi118:Scheduling deploy of centos 8.stream 2022-01-10T14:57:52.671 INFO:teuthology.provision.fog.smithi139:Scheduling deploy of centos 8.stream 2022-01-10T14:57:53.042 INFO:teuthology.orchestra.console:Power off smithi139 2022-01-10T14:57:53.059 INFO:teuthology.orchestra.console:Power off smithi118 2022-01-10T14:58:01.422 INFO:teuthology.orchestra.console:Power off for smithi118 completed 2022-01-10T14:58:01.523 INFO:teuthology.orchestra.console:Power on smithi118 2022-01-10T14:58:05.528 INFO:teuthology.orchestra.console:Power off for smithi139 completed 2022-01-10T14:58:05.630 INFO:teuthology.orchestra.console:Power on smithi139 2022-01-10T14:58:09.886 INFO:teuthology.orchestra.console:Power on for smithi139 completed 2022-01-10T14:58:09.891 INFO:teuthology.orchestra.console:Power on for smithi118 completed 2022-01-10T14:58:09.988 INFO:teuthology.provision.fog.smithi139:Waiting for deploy to finish 2022-01-10T14:58:09.993 INFO:teuthology.provision.fog.smithi118:Waiting for deploy to finish 2022-01-10T15:03:14.805 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-10T15:03:15.080 INFO:teuthology.provision.fog.smithi139:Node is ready 2022-01-10T15:03:15.097 INFO:teuthology.orchestra.run.smithi139.stdout:smithi139.front.sepia.ceph.com 2022-01-10T15:03:15.155 INFO:teuthology.orchestra.run.smithi139.stdout:172.21.15.139 smithi139.front.sepia.ceph.com smithi139 2022-01-10T15:03:15.501 INFO:teuthology.provision.fog.smithi139:Deploy complete! 2022-01-10T15:03:29.706 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-10T15:03:29.994 INFO:teuthology.provision.fog.smithi118:Node is ready 2022-01-10T15:03:30.011 INFO:teuthology.orchestra.run.smithi118.stdout:smithi118.front.sepia.ceph.com 2022-01-10T15:03:30.069 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118 smithi118.front.sepia.ceph.com smithi118 2022-01-10T15:03:30.406 INFO:teuthology.provision.fog.smithi118:Deploy complete! 2022-01-10T15:03:30.551 INFO:teuthology.lock.ops:Checking smithi118.front.sepia.ceph.com 2022-01-10T15:03:30.552 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-10T15:03:30.571 INFO:teuthology.lock.ops:Checking smithi139.front.sepia.ceph.com 2022-01-10T15:03:30.571 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-10T15:03:30.588 INFO:teuthology.lock.ops:Updating [smithi118.front.sepia.ceph.com]: set os type and version on server 2022-01-10T15:03:30.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2022-01-10T15:03:30.776 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2022-01-10T15:03:30.831 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2022-01-10T15:03:30.831 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="8" 2022-01-10T15:03:30.832 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2022-01-10T15:03:30.832 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2022-01-10T15:03:30.832 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="8" 2022-01-10T15:03:30.833 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el8" 2022-01-10T15:03:30.833 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 8" 2022-01-10T15:03:30.833 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2022-01-10T15:03:30.834 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-10T15:03:30.834 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2022-01-10T15:03:30.834 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-01-10T15:03:30.834 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-01-10T15:03:30.834 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-01-10T15:03:30.836 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2022-01-10T15:03:30.871 INFO:teuthology.lock.ops:Updating [smithi139.front.sepia.ceph.com]: set os type and version on server 2022-01-10T15:03:30.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2022-01-10T15:03:31.115 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2022-01-10T15:03:31.172 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Stream" 2022-01-10T15:03:31.172 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="8" 2022-01-10T15:03:31.172 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2022-01-10T15:03:31.172 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2022-01-10T15:03:31.173 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="8" 2022-01-10T15:03:31.173 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el8" 2022-01-10T15:03:31.173 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Stream 8" 2022-01-10T15:03:31.173 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2022-01-10T15:03:31.174 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-10T15:03:31.174 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://centos.org/" 2022-01-10T15:03:31.174 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-01-10T15:03:31.174 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-01-10T15:03:31.175 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-01-10T15:03:31.175 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2022-01-10T15:03:31.272 INFO:teuthology.dispatcher.supervisor:Running job 6605980 2022-01-10T15:03:31.272 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_00e09f05e9fb2ab7c7f497f5e446613531743ae2/virtualenv/bin/teuthology -v --owner scheduled_gabrioux@teuthology --archive /home/teuthworker/archive/gabrioux-2022-01-10_14:38:45-rados:cephadm-wip-guits-testing-2022-01-07-0853-bis-pacific-distro-default-smithi/6605980 --name gabrioux-2022-01-10_14:38:45-rados:cephadm-wip-guits-testing-2022-01-07-0853-bis-pacific-distro-default-smithi --description rados:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/gabrioux-2022-01-10_14:38:45-rados:cephadm-wip-guits-testing-2022-01-07-0853-bis-pacific-distro-default-smithi/6605980/orig.config.yaml 2022-01-10T15:03:31.276 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/gabrioux-2022-01-10_14:38:45-rados:cephadm-wip-guits-testing-2022-01-07-0853-bis-pacific-distro-default-smithi/6605980 2022-01-10T15:03:31.277 INFO:teuthology.dispatcher.supervisor:Job PID: 27261 2022-01-10T15:03:31.277 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-01-11T03:04:25.411 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-01-11T03:04:25.628 INFO:teuthology.kill:Killing Pids: {27261} 2022-01-11T03:04:26.806 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-01-11T03:04:26.806 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-01-11T03:04:26.807 INFO:teuthology.misc:Compressing logs... 2022-01-11T03:04:26.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2022-01-11T03:04:27.053 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2022-01-11T03:04:27.360 INFO:teuthology.misc:Compressing logs... 2022-01-11T03:04:27.768 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-01-11T03:04:29.476 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-01-11T03:04:30.226 INFO:teuthology.misc:Compressing logs... 2022-01-11T03:04:30.809 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/cephadm.log: file size changed while zipping 2022-01-11T03:04:38.074 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-mon.smithi139.log: file size changed while zipping 2022-01-11T03:04:41.561 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-mgr.smithi118.vxtyrw.log: file size changed while zipping 2022-01-11T03:04:45.943 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-mds.cephfs.smithi118.xuqaiz.log: file size changed while zipping 2022-01-11T03:04:55.405 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-01-11T03:05:10.088 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-mon.smithi118.log: file size changed while zipping 2022-01-11T03:05:27.165 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-01-11T03:05:57.933 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-mds.cephfs.smithi139.vgfyja.log: file size changed while zipping 2022-01-11T03:06:19.002 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-01-11T03:06:34.293 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-01-11T03:06:41.699 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-mds.cephfs.smithi139.fudmac.log: file size changed while zipping 2022-01-11T03:09:03.985 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-mds.cephfs.smithi118.yzkacf.log: file size changed while zipping 2022-01-11T03:09:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-01-11T03:09:39.296 INFO:teuthology.orchestra.run.smithi118.stderr:gzip: /var/log/ceph/ff2c32d8-7227-11ec-8c32-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-01-11T03:10:42.026 INFO:teuthology.orchestra.run.smithi118.stderr:tar: ./ff2c32d8-7227-11ec-8c32-001a4aab830c: file changed as we read it 2022-01-11T03:10:42.026 INFO:teuthology.orchestra.run.smithi118.stderr:tar: .: file changed as we read it 2022-01-11T03:11:08.856 INFO:teuthology.kill:No teuthology processes running 2022-01-11T03:11:09.939 INFO:teuthology.kill:Nuking machines: ['smithi118', 'smithi139'] 2022-01-11T03:11:10.890 INFO:teuthology.kill:2022-01-11 03:11:10,890.890 INFO:teuthology.nuke:targets: 2022-01-11T03:11:10.891 INFO:teuthology.kill: smithi118.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4VbUyOwfYrgtL7knSsM4ni+jm0IVaB8mqx0AaOZSeUfzgPVTEbNVK2/Ee9g+oMuH0J0YSNoRjOWvEyPhVWqGpQmjLVW9lJN0TPBusdP8J6PfKP5XqZkpTO1/pEVhrIEawZqXcEWtkE2lsqWkY/BsvAjd2bGUpmcgG3n8BrC7BRKAc2uh1ktBgPMbIGR5KYxu0Dvr2SaxnxiK+hpsjDgxPK1Xak5IEVsP4pHwU5gcbcxQent53bCf+VSwHhlXKaiv65lymE5KdQ+XhXp/rV4poQwOeqXL5dtseYyByw5R8UseAHEwjE80Nf7WNM1Co2Q5UTSDvj4y5xYHjJ/vF1zaqrlSCoqJ7BAAA8FxQco4XUqwfLeLw7vMO/M0ok1L+IA1QY40C9FzvXB63Rp+wqt82uHCJ9ZdYaS+8HJxWAzLGv1oqW54LezwvtB/REqRh+WwV5ToR4HUIMjgHloNYHYHtp6miTYLJK65M2R6Ukctjc5ZuGD+6IHa7HGzhtkUNfe0= 2022-01-11T03:11:10.892 INFO:teuthology.kill: smithi139.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMQdTBRzpa6QX8VbK+mt0aYWYCuWyDKFBUoNiN3uaNOSShOjM42ubvK65DJqZ9pZ2fvdxOVhRh37s5r/1C64SgGlyJRqjF2K3GykURilUkzT5e9XHaV36fA718ELkpwFyUyqnmtqdcWfuMAKLOAqJy1v9R32dRM/vRcNX33YY6nE8VJepXWiwy9L7rQZzn34vbS3idqSEn32cz2q6V4M55D0qvvYjlx4qVh2yYN2RUCmARuEweDzJWG14T9yJCDeV3jClrRfhKFYka25NQyBeB1G/LiZo1hvnrweFJ/wGv29Auz36vrbb2lrF5qGG1OennstL4fDIG8oVyBWGZcz2MZiXgAgsOMA8/RavZsVxYL8INaljXJPtY3jhLUSB/SE9GsnL5V3Za2HPnzCcYZdSpJVi5TRYuIEgi2z4jOJoAhQgZR8qTs6dy9MjWpzxomeV9uUzBXCMuEy2Y5rdPG9Wf5r6wfsjIXMfGjF0yfxAIRlfhU/NWo+veU6BrRCK2A5M= 2022-01-11T03:11:10.917 INFO:teuthology.kill:2022-01-11 03:11:10,917.917 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-11T03:11:10.920 INFO:teuthology.kill:2022-01-11 03:11:10,920.920 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-11T03:11:10.995 INFO:teuthology.kill:2022-01-11 03:11:10,995.995 INFO:teuthology.orchestra.console:Power off smithi118 2022-01-11T03:11:11.007 INFO:teuthology.kill:2022-01-11 03:11:11,006.006 INFO:teuthology.orchestra.console:Power off smithi139 2022-01-11T03:11:19.365 INFO:teuthology.kill:2022-01-11 03:11:19,364.364 INFO:teuthology.orchestra.console:Power off for smithi118 completed 2022-01-11T03:11:19.511 INFO:teuthology.kill:2022-01-11 03:11:19,509.509 INFO:teuthology.lock.ops:unlocked: smithi118.front.sepia.ceph.com 2022-01-11T03:11:23.481 INFO:teuthology.kill:2022-01-11 03:11:23,481.481 INFO:teuthology.orchestra.console:Power off for smithi139 completed 2022-01-11T03:11:23.627 INFO:teuthology.kill:2022-01-11 03:11:23,627.627 INFO:teuthology.lock.ops:unlocked: smithi139.front.sepia.ceph.com 2022-01-11T03:13:23.919 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-01-11T03:13:24.031 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi118 but it was locked by another job: /home/teuthworker/archive/gsalomon-2022-01-10_21:29:21-rgw:verify-wip-s3select-parquet-object-processing-2-distro-basic-smithi/6606478