2022-02-04T17:57:27.560 INFO:root:teuthology version: 1.1.0-3094160c 2022-02-04T17:57:27.684 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi135.front.sepia.ceph.com' reimaging 2022-02-04T17:57:27.685 INFO:teuthology.lock.ops:Updating [ubuntu@smithi135.front.sepia.ceph.com]: reset os type and version on server 2022-02-04T17:57:27.685 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2022-02-04T17:57:27.704 INFO:teuthology.lock.ops:Node 'ubuntu@smithi135.front.sepia.ceph.com' reimaging is complete 2022-02-04T17:57:27.705 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi152.front.sepia.ceph.com' reimaging 2022-02-04T17:57:27.705 INFO:teuthology.lock.ops:Updating [ubuntu@smithi152.front.sepia.ceph.com]: reset os type and version on server 2022-02-04T17:57:27.705 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2022-02-04T17:57:27.725 INFO:teuthology.lock.ops:Node 'ubuntu@smithi152.front.sepia.ceph.com' reimaging is complete 2022-02-04T17:57:27.918 INFO:teuthology.provision.fog.smithi135:Scheduling deploy of centos 8.stream 2022-02-04T17:57:27.941 INFO:teuthology.provision.fog.smithi152:Scheduling deploy of centos 8.stream 2022-02-04T17:57:28.298 INFO:teuthology.orchestra.console:Power off smithi135 2022-02-04T17:57:28.315 INFO:teuthology.orchestra.console:Power off smithi152 2022-02-04T17:57:36.674 INFO:teuthology.orchestra.console:Power off for smithi135 completed 2022-02-04T17:57:36.678 INFO:teuthology.orchestra.console:Power off for smithi152 completed 2022-02-04T17:57:36.776 INFO:teuthology.orchestra.console:Power on smithi135 2022-02-04T17:57:36.784 INFO:teuthology.orchestra.console:Power on smithi152 2022-02-04T17:57:45.148 INFO:teuthology.orchestra.console:Power on for smithi152 completed 2022-02-04T17:57:45.248 INFO:teuthology.orchestra.console:Power on for smithi135 completed 2022-02-04T17:57:45.250 INFO:teuthology.provision.fog.smithi152:Waiting for deploy to finish 2022-02-04T17:57:45.350 INFO:teuthology.provision.fog.smithi135:Waiting for deploy to finish 2022-02-04T18:02:32.152 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-04T18:02:32.405 INFO:teuthology.provision.fog.smithi135:Node is ready 2022-02-04T18:02:32.464 INFO:teuthology.orchestra.run.smithi135.stdout:smithi135.front.sepia.ceph.com 2022-02-04T18:02:32.523 INFO:teuthology.orchestra.run.smithi135.stdout:172.21.15.135 smithi135.front.sepia.ceph.com smithi135 2022-02-04T18:02:32.874 INFO:teuthology.provision.fog.smithi135:Deploy complete! 2022-02-04T18:02:59.370 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-04T18:02:59.626 INFO:teuthology.provision.fog.smithi152:Node is ready 2022-02-04T18:02:59.695 INFO:teuthology.orchestra.run.smithi152.stdout:smithi152.front.sepia.ceph.com 2022-02-04T18:02:59.762 INFO:teuthology.orchestra.run.smithi152.stdout:172.21.15.152 smithi152.front.sepia.ceph.com smithi152 2022-02-04T18:03:00.138 INFO:teuthology.provision.fog.smithi152:Deploy complete! 2022-02-04T18:03:00.296 INFO:teuthology.lock.ops:Checking smithi135.front.sepia.ceph.com 2022-02-04T18:03:00.297 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-04T18:03:00.316 INFO:teuthology.lock.ops:Checking smithi152.front.sepia.ceph.com 2022-02-04T18:03:00.317 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-04T18:03:00.336 INFO:teuthology.lock.ops:Updating [smithi135.front.sepia.ceph.com]: set os type and version on server 2022-02-04T18:03:00.336 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2022-02-04T18:03:00.587 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2022-02-04T18:03:00.645 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2022-02-04T18:03:00.646 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="8" 2022-02-04T18:03:00.646 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2022-02-04T18:03:00.646 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2022-02-04T18:03:00.647 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="8" 2022-02-04T18:03:00.647 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el8" 2022-02-04T18:03:00.647 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 8" 2022-02-04T18:03:00.647 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2022-02-04T18:03:00.648 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-02-04T18:03:00.648 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2022-02-04T18:03:00.648 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-04T18:03:00.648 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-04T18:03:00.649 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-02-04T18:03:00.650 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2022-02-04T18:03:00.669 INFO:teuthology.lock.ops:Updating [smithi152.front.sepia.ceph.com]: set os type and version on server 2022-02-04T18:03:00.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2022-02-04T18:03:00.867 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2022-02-04T18:03:00.926 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2022-02-04T18:03:00.926 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="8" 2022-02-04T18:03:00.926 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2022-02-04T18:03:00.927 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2022-02-04T18:03:00.927 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="8" 2022-02-04T18:03:00.927 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el8" 2022-02-04T18:03:00.927 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 8" 2022-02-04T18:03:00.927 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2022-02-04T18:03:00.928 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-02-04T18:03:00.928 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2022-02-04T18:03:00.928 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-04T18:03:00.928 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-04T18:03:00.929 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-02-04T18:03:00.929 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2022-02-04T18:03:01.065 INFO:teuthology.dispatcher.supervisor:Running job 6661682 2022-02-04T18:03:01.065 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_3094160cc590b786a43d55faaf7c99d5de71ce56/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2022-02-04_17:39:45-orch:cephadm-wip-adk-testing-2022-02-03-0853-distro-basic-smithi/6661682 --name nojha-2022-02-04_17:39:45-orch:cephadm-wip-adk-testing-2022-02-03-0853-distro-basic-smithi --description orch:cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/nojha-2022-02-04_17:39:45-orch:cephadm-wip-adk-testing-2022-02-03-0853-distro-basic-smithi/6661682/orig.config.yaml 2022-02-04T18:03:01.071 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2022-02-04_17:39:45-orch:cephadm-wip-adk-testing-2022-02-03-0853-distro-basic-smithi/6661682 2022-02-04T18:03:01.071 INFO:teuthology.dispatcher.supervisor:Job PID: 10063 2022-02-04T18:03:01.072 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-05T00:33:30.256 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-05T00:33:30.849 INFO:teuthology.kill:Killing Pids: {10063} 2022-02-05T00:33:32.125 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-02-05T00:33:32.126 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-02-05T00:33:32.126 INFO:teuthology.misc:Compressing logs... 2022-02-05T00:33:32.127 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2022-02-05T00:33:32.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2022-02-05T00:33:32.614 INFO:teuthology.misc:Compressing logs... 2022-02-05T00:33:33.274 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-02-05T00:33:33.528 INFO:teuthology.misc:Compressing logs... 2022-02-05T00:33:36.293 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-mgr.smithi135.shafmp.log: file size changed while zipping 2022-02-05T00:33:48.133 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-02-05T00:33:58.152 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-02-05T00:34:05.008 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi152.fkjlza.log: file size changed while zipping 2022-02-05T00:34:09.122 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-mon.smithi152.log: file size changed while zipping 2022-02-05T00:34:09.686 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-mon.smithi135.log: file size changed while zipping 2022-02-05T00:34:21.293 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-02-05T00:34:52.862 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi135.anpszi.log: file size changed while zipping 2022-02-05T00:35:11.341 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi135.yipsug.log: file size changed while zipping 2022-02-05T00:35:12.699 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-mds.cephfs.smithi152.rhkfui.log: file size changed while zipping 2022-02-05T00:35:24.222 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-02-05T00:35:27.295 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-02-05T00:35:37.861 INFO:teuthology.orchestra.run.smithi152.stderr:gzip: /var/log/ceph/8456e70c-85e5-11ec-8c35-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-02-05T00:36:26.327 INFO:teuthology.kill:No teuthology processes running 2022-02-05T00:36:27.719 INFO:teuthology.kill:Nuking machines: ['smithi135', 'smithi152'] 2022-02-05T00:36:28.718 INFO:teuthology.kill:2022-02-05 00:36:28,717.717 INFO:teuthology.nuke:targets: 2022-02-05T00:36:28.719 INFO:teuthology.kill: smithi135.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFR1hEGheBmX3YZGingHIYoSJU2h2PN9ZxAvtMnrvxp2pN1n6W4WZlKjlCT0d7yHWW27TMKLxmje/cKBfrkJrU6K3N28ZA9K5VmIVQ+2c4QfFBKsQGk1Kqyj8EGRvHq9smkpUsa8nCxFn8LHK40P57jGabNxPKar8Iv9Q5RS79t5hrCEyw41CcyHC4Qla2GjA+4k/ApmwPspY5RcUXjR7FoVXsTdeBHnD1krKMdWn3rtwgCy3kAQtWyaPX/syphs8nZ7W5hqLSxwg82pcW+zwGyPO+yb7R/Qtu7s8YpJ5BYI3hRix1K+90SO1liAbZnai+hypOLzQ4CvpSgRAlFO5pT4ZgBEp3YpAxdDQtggnHFS/PzE2u4ECKEGP3SjUPuaRF+1rnr0E6mC0NOQmJxAcbmUj/1KYlYDHwHB4ZcwM+xShjBlhqrbcKxX6upPSBdqdPtwJKv5+BFFib+iOHf14LuZE97WYqUqFKH6xkARHFaXGczW506urcG3x6EYHIm4s= 2022-02-05T00:36:28.719 INFO:teuthology.kill: smithi152.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC9Osffpbx5z+8vj2/ujHcN+RuJMHDzlMKi/MnqcUsrM2jGqfyCsJnaDd/eEQbIUmzyTBgjAN+UJDFHNDvHPnkBcNcvEnLR2KuroGtiYP1GeOY8T2vXxof57xsgjhkZyn2+FoK3ULaiiv34zE7fXND3xmrk3G8V0mbAYVrt46IjApgUpUtVoR+gWYnnIz5uFcoc9LE2AYBAiXPgm8wkvOVKawd6upwHX9RaA6RvVqIgPIL61oXjav+HSgtGfytTBJjt9aKFns0Jsjxu4J3ULaKr2oQOZVZHCqusDqY3i/V+xLIhUR/xr8q5n+bm2vWSzk7Rn9fNwqMmHYdd+o3iMQ4Hy+HilES8a17FPuby6jOzJNzl7oYWsfTrxfTDCwGJY7OaxbeECu3TaES0gcvgcx0IR6KezpjC8FguWZjFpVIDGOzfX4WY64xsX+5UdFsBurCqt4UvWihIw/ZQlhl8e9Vd+nlvBNKbT4zs7l4B4WJQGaArHC9GBqFGqqxoKNQjFg0= 2022-02-05T00:36:28.747 INFO:teuthology.kill:2022-02-05 00:36:28,747.747 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-05T00:36:28.754 INFO:teuthology.kill:2022-02-05 00:36:28,754.754 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-05T00:36:28.836 INFO:teuthology.kill:2022-02-05 00:36:28,835.835 INFO:teuthology.orchestra.console:Power off smithi152 2022-02-05T00:36:28.851 INFO:teuthology.kill:2022-02-05 00:36:28,851.851 INFO:teuthology.orchestra.console:Power off smithi135 2022-02-05T00:36:37.209 INFO:teuthology.kill:2022-02-05 00:36:37,208.208 INFO:teuthology.orchestra.console:Power off for smithi152 completed 2022-02-05T00:36:37.220 INFO:teuthology.kill:2022-02-05 00:36:37,220.220 INFO:teuthology.orchestra.console:Power off for smithi135 completed 2022-02-05T00:36:37.356 INFO:teuthology.kill:2022-02-05 00:36:37,355.355 INFO:teuthology.lock.ops:unlocked: smithi152.front.sepia.ceph.com 2022-02-05T00:36:37.365 INFO:teuthology.kill:2022-02-05 00:36:37,364.364 INFO:teuthology.lock.ops:unlocked: smithi135.front.sepia.ceph.com 2022-02-05T00:38:37.713 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-02-05T00:38:37.827 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi135 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-04_22:20:21-rados-wip-yuri8-testing-2022-02-04-1220-quincy-distro-default-smithi/6662026 2022-02-05T00:38:37.828 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi152 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-02-04_22:20:21-rados-wip-yuri8-testing-2022-02-04-1220-quincy-distro-default-smithi/6662026