2022-01-03T17:06:28.073 INFO:root:teuthology version: 1.1.0-95a7d479 2022-01-03T17:06:28.204 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi135.front.sepia.ceph.com' reimaging 2022-01-03T17:06:28.205 INFO:teuthology.lock.ops:Updating [ubuntu@smithi135.front.sepia.ceph.com]: reset os type and version on server 2022-01-03T17:06:28.205 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2022-01-03T17:06:28.224 INFO:teuthology.lock.ops:Node 'ubuntu@smithi135.front.sepia.ceph.com' reimaging is complete 2022-01-03T17:06:28.225 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi195.front.sepia.ceph.com' reimaging 2022-01-03T17:06:28.225 INFO:teuthology.lock.ops:Updating [ubuntu@smithi195.front.sepia.ceph.com]: reset os type and version on server 2022-01-03T17:06:28.225 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2022-01-03T17:06:28.244 INFO:teuthology.lock.ops:Node 'ubuntu@smithi195.front.sepia.ceph.com' reimaging is complete 2022-01-03T17:06:28.448 INFO:teuthology.provision.fog.smithi135:Scheduling deploy of centos 8.stream 2022-01-03T17:06:28.472 INFO:teuthology.provision.fog.smithi195:Scheduling deploy of centos 8.stream 2022-01-03T17:06:28.921 INFO:teuthology.orchestra.console:Power off smithi135 2022-01-03T17:06:28.983 INFO:teuthology.orchestra.console:Power off smithi195 2022-01-03T17:06:29.070 INFO:teuthology.orchestra.console:Power off for smithi135 completed 2022-01-03T17:06:29.138 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2022-01-03T17:06:29.172 INFO:teuthology.orchestra.console:Power on smithi135 2022-01-03T17:06:29.239 INFO:teuthology.orchestra.console:Power on smithi195 2022-01-03T17:06:33.494 INFO:teuthology.orchestra.console:Power on for smithi195 completed 2022-01-03T17:06:33.596 INFO:teuthology.provision.fog.smithi195:Waiting for deploy to finish 2022-01-03T17:06:37.545 INFO:teuthology.orchestra.console:Power on for smithi135 completed 2022-01-03T17:06:37.647 INFO:teuthology.provision.fog.smithi135:Waiting for deploy to finish 2022-01-03T17:11:25.261 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-03T17:11:25.553 INFO:teuthology.provision.fog.smithi195:Node is ready 2022-01-03T17:11:25.571 INFO:teuthology.orchestra.run.smithi195.stdout:smithi195.front.sepia.ceph.com 2022-01-03T17:11:25.627 INFO:teuthology.orchestra.run.smithi195.stdout:172.21.15.195 smithi195.front.sepia.ceph.com smithi195 2022-01-03T17:11:25.972 INFO:teuthology.provision.fog.smithi195:Deploy complete! 2022-01-03T17:11:29.996 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-03T17:11:30.288 INFO:teuthology.provision.fog.smithi135:Node is ready 2022-01-03T17:11:30.305 INFO:teuthology.orchestra.run.smithi135.stdout:smithi135.front.sepia.ceph.com 2022-01-03T17:11:30.362 INFO:teuthology.orchestra.run.smithi135.stdout:172.21.15.135 smithi135.front.sepia.ceph.com smithi135 2022-01-03T17:11:30.689 INFO:teuthology.provision.fog.smithi135:Deploy complete! 2022-01-03T17:11:30.841 INFO:teuthology.lock.ops:Checking smithi135.front.sepia.ceph.com 2022-01-03T17:11:30.842 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-03T17:11:30.862 INFO:teuthology.lock.ops:Checking smithi195.front.sepia.ceph.com 2022-01-03T17:11:30.862 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-03T17:11:30.880 INFO:teuthology.lock.ops:Updating [smithi135.front.sepia.ceph.com]: set os type and version on server 2022-01-03T17:11:30.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2022-01-03T17:11:31.066 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2022-01-03T17:11:31.121 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2022-01-03T17:11:31.121 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="8" 2022-01-03T17:11:31.121 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2022-01-03T17:11:31.121 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2022-01-03T17:11:31.122 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="8" 2022-01-03T17:11:31.122 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el8" 2022-01-03T17:11:31.122 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 8" 2022-01-03T17:11:31.122 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2022-01-03T17:11:31.123 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-03T17:11:31.123 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2022-01-03T17:11:31.123 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-01-03T17:11:31.124 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-01-03T17:11:31.124 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-01-03T17:11:31.125 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2022-01-03T17:11:31.148 INFO:teuthology.lock.ops:Updating [smithi195.front.sepia.ceph.com]: set os type and version on server 2022-01-03T17:11:31.148 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2022-01-03T17:11:31.343 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2022-01-03T17:11:31.401 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2022-01-03T17:11:31.401 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="8" 2022-01-03T17:11:31.401 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2022-01-03T17:11:31.402 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2022-01-03T17:11:31.402 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="8" 2022-01-03T17:11:31.402 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el8" 2022-01-03T17:11:31.402 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 8" 2022-01-03T17:11:31.403 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2022-01-03T17:11:31.403 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-01-03T17:11:31.403 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2022-01-03T17:11:31.404 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-01-03T17:11:31.404 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-01-03T17:11:31.404 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-01-03T17:11:31.405 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2022-01-03T17:11:31.509 INFO:teuthology.dispatcher.supervisor:Running job 6592331 2022-01-03T17:11:31.510 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_swagner@teuthology --archive /home/teuthworker/archive/swagner-2022-01-03_16:19:45-orch:cephadm-wip-pereman2-testing-2021-11-01-2310-distro-default-smithi/6592331 --name swagner-2022-01-03_16:19:45-orch:cephadm-wip-pereman2-testing-2021-11-01-2310-distro-default-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/swagner-2022-01-03_16:19:45-orch:cephadm-wip-pereman2-testing-2021-11-01-2310-distro-default-smithi/6592331/orig.config.yaml 2022-01-03T17:11:31.513 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/swagner-2022-01-03_16:19:45-orch:cephadm-wip-pereman2-testing-2021-11-01-2310-distro-default-smithi/6592331 2022-01-03T17:11:31.514 INFO:teuthology.dispatcher.supervisor:Job PID: 31802 2022-01-03T17:11:31.514 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-01-04T05:12:22.903 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-01-04T05:12:23.050 INFO:teuthology.kill:Killing Pids: {31802} 2022-01-04T05:12:24.317 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-01-04T05:12:24.318 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-01-04T05:12:24.319 INFO:teuthology.misc:Compressing logs... 2022-01-04T05:12:24.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2022-01-04T05:12:24.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2022-01-04T05:12:24.800 INFO:teuthology.misc:Compressing logs... 2022-01-04T05:12:26.052 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-01-04T05:12:26.483 INFO:teuthology.misc:Compressing logs... 2022-01-04T05:12:28.330 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/4657887a-6cb9-11ec-8c31-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-01-04T05:12:28.784 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/4657887a-6cb9-11ec-8c31-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-01-04T05:12:30.573 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/4657887a-6cb9-11ec-8c31-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-01-04T05:12:30.779 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/4657887a-6cb9-11ec-8c31-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-01-04T05:12:32.296 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/4657887a-6cb9-11ec-8c31-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-01-04T05:12:33.377 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/4657887a-6cb9-11ec-8c31-001a4aab830c/ceph-mon.smithi195.log: file size changed while zipping 2022-01-04T05:12:33.533 INFO:teuthology.orchestra.run.smithi195.stderr:gzip: /var/log/ceph/4657887a-6cb9-11ec-8c31-001a4aab830c/ceph-mds.cephfs.smithi195.zyvsep.log: file size changed while zipping 2022-01-04T05:12:35.026 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/4657887a-6cb9-11ec-8c31-001a4aab830c/ceph-mon.smithi135.log: file size changed while zipping 2022-01-04T05:12:37.354 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/4657887a-6cb9-11ec-8c31-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-01-04T05:12:38.490 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/4657887a-6cb9-11ec-8c31-001a4aab830c/ceph-mgr.smithi135.fvpxmd.log: file size changed while zipping 2022-01-04T05:13:22.666 INFO:teuthology.orchestra.run.smithi195.stderr:tar: .: file changed as we read it 2022-01-04T05:13:22.747 INFO:teuthology.kill:No teuthology processes running 2022-01-04T05:13:23.909 INFO:teuthology.kill:Nuking machines: ['smithi135', 'smithi195'] 2022-01-04T05:13:24.952 INFO:teuthology.kill:2022-01-04 05:13:24,951.951 INFO:teuthology.nuke:targets: 2022-01-04T05:13:24.953 INFO:teuthology.kill: smithi135.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvlMzTU8WX7FPZhC8Qw3ZTghnlYf0DVI81v0ZLPUSbhv1X9qXDcEWGlGHPl7s82nFky/Tmc16jRJJP5L6U1q/49nWHs2vFlLoq0X1OuLau3yHn66XiNfIfmM+vQgQ64bHEE7Fc4cdHu6pr7bMEbt/b3a4NgelxRTx9ml3yciHEbAhw5P+RT2pH08vaod67HB73sM45hJRSIlXpfFzgRVZlK5Sg8WZh1hZI+OxFX2/dXzF9aIR22o7P2h5joVQHgqgcoC+OYqnuirXZN9RF37OEfwj10TwwsV3wtZzLNAmbm6ulO/v3jxSYcuHBLJpLU+NK+4r8V8h5PjkqCC8G4k752EEImMYpNKGM9fhImg6tSJAcT9gn2JrqU65cKSzZIbObZ/9uMPYb6LdT/zUcsqISKheI8Z/xLHUcMlh/nGoZa7JVevvBFKZcmaCeP9vGY9YlzCY5BKOdRZK6MefovuXVY8YllpVcLCLq1jr4eGOunYLNFcXGFRatsLaZ3DuthSk= 2022-01-04T05:13:24.953 INFO:teuthology.kill: smithi195.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTmdR85/UmjCaG8zB1lhKNE8Ap9zrFGWvSM7Wy3F3DP4k6gZ2Wm9vmnuFF+bY7Aix+zIbGzqQe6/K+Pm4UXTBW8UwOqRyfIAXxxRR0sqk1DoRSmYXIPx8u92XPXqYpoxnvCEOu1P7LkKEk1fbC5JZ3cbpFR3R0ZEUV1bBfGfLFQDIAaPpNZBSRb9AuHzzUujdADtT4xqSVu4hX7Jr9TscNEc2IfiTWELLG92VnpjD3AWmR4wKk2FMzCdIgHHEfZFto+04xweWNm7Hl0DuZrPxM/4YZCTj9zzvUjwP3uxLD8uWaoe542w6U08smQRRVwbRIQNU2KJdoI7qHhsROgulleCqPS47/FtnNnzVVkzfBfZuhEzIBJusROIxVsAFFZpAYeet78NdWIth319PP2kVBVNEErDCQPv6nMPBsyzzmcZ1f17Njr3mdlitcP7wih84hmQjk9M4oWf7SI6nPH4cf8ahH0mbGw0nat+QxLfNP8Eoy3jU6WK2K+NNAazwGoZc= 2022-01-04T05:13:24.979 INFO:teuthology.kill:2022-01-04 05:13:24,979.979 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-04T05:13:24.983 INFO:teuthology.kill:2022-01-04 05:13:24,983.983 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-04T05:13:25.059 INFO:teuthology.kill:2022-01-04 05:13:25,059.059 INFO:teuthology.orchestra.console:Power off smithi135 2022-01-04T05:13:25.077 INFO:teuthology.kill:2022-01-04 05:13:25,077.077 INFO:teuthology.orchestra.console:Power off smithi195 2022-01-04T05:13:37.549 INFO:teuthology.kill:2022-01-04 05:13:37,549.549 INFO:teuthology.orchestra.console:Power off for smithi135 completed 2022-01-04T05:13:37.557 INFO:teuthology.kill:2022-01-04 05:13:37,557.557 INFO:teuthology.orchestra.console:Power off for smithi195 completed 2022-01-04T05:13:37.699 INFO:teuthology.kill:2022-01-04 05:13:37,699.699 INFO:teuthology.lock.ops:unlocked: smithi195.front.sepia.ceph.com 2022-01-04T05:13:37.703 INFO:teuthology.kill:2022-01-04 05:13:37,702.702 INFO:teuthology.lock.ops:unlocked: smithi135.front.sepia.ceph.com 2022-01-04T05:15:38.031 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15