2022-07-04T17:41:40.488 INFO:root:teuthology version: 1.1.1.dev379+gf58709e3 2022-07-04T17:41:40.640 INFO:teuthology.lock.ops:Start node 'smithi135.front.sepia.ceph.com' reimaging 2022-07-04T17:41:40.642 INFO:teuthology.lock.ops:Updating [smithi135.front.sepia.ceph.com]: reset os type and version on server 2022-07-04T17:41:40.642 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2022-07-04T17:41:40.672 INFO:teuthology.lock.ops:Node 'smithi135.front.sepia.ceph.com' reimaging is complete 2022-07-04T17:41:40.673 INFO:teuthology.lock.ops:Start node 'smithi150.front.sepia.ceph.com' reimaging 2022-07-04T17:41:40.673 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: reset os type and version on server 2022-07-04T17:41:40.673 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2022-07-04T17:41:40.694 INFO:teuthology.lock.ops:Node 'smithi150.front.sepia.ceph.com' reimaging is complete 2022-07-04T17:41:40.900 INFO:teuthology.provision.fog.smithi135:Scheduling deploy of centos 8.stream 2022-07-04T17:41:40.922 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of centos 8.stream 2022-07-04T17:41:41.319 INFO:teuthology.orchestra.console:Power off smithi135 2022-07-04T17:41:41.337 INFO:teuthology.orchestra.console:Power off smithi150 2022-07-04T17:41:49.692 INFO:teuthology.orchestra.console:Power off for smithi135 completed 2022-07-04T17:41:49.703 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2022-07-04T17:41:49.795 INFO:teuthology.orchestra.console:Power on smithi135 2022-07-04T17:41:49.804 INFO:teuthology.orchestra.console:Power on smithi150 2022-07-04T17:41:54.080 INFO:teuthology.orchestra.console:Power on for smithi135 completed 2022-07-04T17:41:54.182 INFO:teuthology.provision.fog.smithi135:Waiting for deploy to finish 2022-07-04T17:41:58.204 INFO:teuthology.orchestra.console:Power on for smithi150 completed 2022-07-04T17:41:58.305 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish 2022-07-04T17:46:00.600 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-04T17:46:26.132 INFO:teuthology.provision.fog.smithi135:Node is ready 2022-07-04T17:46:26.154 INFO:teuthology.orchestra.run.smithi135.stdout:smithi135.front.sepia.ceph.com 2022-07-04T17:46:26.213 INFO:teuthology.orchestra.run.smithi135.stdout:172.21.15.135 smithi135.front.sepia.ceph.com smithi135 2022-07-04T17:46:26.581 INFO:teuthology.provision.fog.smithi135:Deploy complete! 2022-07-04T17:46:39.367 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-07-04T17:46:39.649 INFO:teuthology.provision.fog.smithi150:Node is ready 2022-07-04T17:46:39.668 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150.front.sepia.ceph.com 2022-07-04T17:46:39.727 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150 smithi150.front.sepia.ceph.com smithi150 2022-07-04T17:46:40.077 INFO:teuthology.provision.fog.smithi150:Deploy complete! 2022-07-04T17:46:40.280 INFO:teuthology.lock.ops:Checking smithi135.front.sepia.ceph.com 2022-07-04T17:46:40.281 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-04T17:46:40.306 INFO:teuthology.lock.ops:Checking smithi150.front.sepia.ceph.com 2022-07-04T17:46:40.307 INFO:teuthology.lock.ops:New key found. Updating... 2022-07-04T17:46:40.328 INFO:teuthology.lock.ops:Updating [smithi135.front.sepia.ceph.com]: set os type and version on server 2022-07-04T17:46:40.329 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2022-07-04T17:46:40.600 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2022-07-04T17:46:40.659 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2022-07-04T17:46:40.659 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="8" 2022-07-04T17:46:40.659 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2022-07-04T17:46:40.660 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2022-07-04T17:46:40.660 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="8" 2022-07-04T17:46:40.660 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el8" 2022-07-04T17:46:40.660 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 8" 2022-07-04T17:46:40.661 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2022-07-04T17:46:40.661 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-07-04T17:46:40.661 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2022-07-04T17:46:40.661 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-04T17:46:40.661 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-04T17:46:40.662 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-07-04T17:46:40.663 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2022-07-04T17:46:40.687 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: set os type and version on server 2022-07-04T17:46:40.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2022-07-04T17:46:40.889 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2022-07-04T17:46:40.946 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="CentOS Stream" 2022-07-04T17:46:40.946 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8" 2022-07-04T17:46:40.947 INFO:teuthology.orchestra.run.smithi150.stdout:ID="centos" 2022-07-04T17:46:40.947 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="rhel fedora" 2022-07-04T17:46:40.948 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8" 2022-07-04T17:46:40.948 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8" 2022-07-04T17:46:40.949 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="CentOS Stream 8" 2022-07-04T17:46:40.949 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2022-07-04T17:46:40.950 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-07-04T17:46:40.950 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://centos.org/" 2022-07-04T17:46:40.951 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-07-04T17:46:40.951 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-07-04T17:46:40.952 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-07-04T17:46:40.953 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2022-07-04T17:46:41.096 INFO:teuthology.dispatcher.supervisor:Running job 6912657 2022-07-04T17:46:41.096 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/bin/teuthology -v --owner scheduled_dparmar@teuthology --archive /home/teuthworker/archive/dparmar-2022-07-04_17:09:23-fs:upgrade-wip-dparmar-cephadm-55715-testing-distro-default-smithi/6912657 --name dparmar-2022-07-04_17:09:23-fs:upgrade-wip-dparmar-cephadm-55715-testing-distro-default-smithi --description fs:upgrade/upgrade_without_reducing_max_mds/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/dparmar-2022-07-04_17:09:23-fs:upgrade-wip-dparmar-cephadm-55715-testing-distro-default-smithi/6912657/orig.config.yaml 2022-07-04T17:46:41.101 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/dparmar-2022-07-04_17:09:23-fs:upgrade-wip-dparmar-cephadm-55715-testing-distro-default-smithi/6912657 2022-07-04T17:46:41.101 INFO:teuthology.dispatcher.supervisor:Job PID: 8009 2022-07-04T17:46:41.102 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-07-05T05:47:39.211 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-07-05T05:47:39.464 INFO:teuthology.kill:Killing Pids: {8009} 2022-07-05T05:47:41.034 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2022-07-05T05:47:41.035 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2022-07-05T05:47:41.035 INFO:teuthology.misc:Compressing logs... 2022-07-05T05:47:41.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2022-07-05T05:47:41.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2022-07-05T05:47:41.721 INFO:teuthology.misc:Compressing logs... 2022-07-05T05:47:42.025 INFO:teuthology.misc:Compressing logs... 2022-07-05T05:47:43.523 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/2be7afcc-fbc2-11ec-842c-001a4aab830c/ceph-mon.smithi150.log: file size changed while zipping 2022-07-05T05:47:44.601 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/2be7afcc-fbc2-11ec-842c-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-07-05T05:47:45.706 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/2be7afcc-fbc2-11ec-842c-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-07-05T05:47:46.859 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/2be7afcc-fbc2-11ec-842c-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-07-05T05:47:47.784 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/2be7afcc-fbc2-11ec-842c-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-07-05T05:47:47.806 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/2be7afcc-fbc2-11ec-842c-001a4aab830c/ceph-mgr.smithi150.rmankt.log: file size changed while zipping 2022-07-05T05:47:50.614 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /var/log/ceph/2be7afcc-fbc2-11ec-842c-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-07-05T05:47:51.213 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/2be7afcc-fbc2-11ec-842c-001a4aab830c/ceph-mon.smithi135.log: file size changed while zipping 2022-07-05T05:47:53.856 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/2be7afcc-fbc2-11ec-842c-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-07-05T05:47:54.886 INFO:teuthology.orchestra.run.smithi135.stderr:gzip: /var/log/ceph/2be7afcc-fbc2-11ec-842c-001a4aab830c/ceph-mgr.smithi135.wexmvj.log: file size changed while zipping 2022-07-05T05:48:46.711 INFO:teuthology.kill:No teuthology processes running 2022-07-05T05:48:48.597 INFO:teuthology.kill:Nuking machines: ['smithi135', 'smithi150'] 2022-07-05T05:48:50.276 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-07-05T05:48:50.277 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-07-05T05:48:50.277 INFO:teuthology.kill:2022-07-05 05:48:50,276.276 INFO:teuthology.nuke:targets: 2022-07-05T05:48:50.278 INFO:teuthology.kill: smithi135.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2Chj06ZtUO4nQ7SuS+IXbk5I1il/zJ+boNmJfUMZuG3/XKxpIeLbzgWmRieJR/JS7Thtl+jqtzraXSJzFq/Kk4aiZJF3L/yP97SYDAskspQxnwCVNKC7A+8voWyTlqsgri6+DndZzOqfMTtiZSJjJgq23QgTx4XGmjKyv2E9aqg+El5P63jSwaNaK1+J2/v3wkXqSRx5yRNONFDpRWYaVYifEYoxpJ8kR16cAH9OfHcQpvxcLmJVzwS5+GWNS5tCLkoaaR0tp4hZxQcl6Lbaadi/Yaik1LMVPeosYeyOZ+gvwmvE+Rzf0sKlsZmGFCl29/vcbamKJAHAMOH6h3MFUsJlrsHQpwiXIyB7pw8lzmDzvCboM0PAUYaD357+6ulKVClqMiz48DqEGQvyf2H/By8EPWM2MsDZzAutl56T74BKsG79fpNDv5W6TC8BLfNDTLriL+YIcOtqyiJcMVCB64pRArkviq+uW6THjlwdIoE7fAXPR+MsXsLj1/4bijCU= 2022-07-05T05:48:50.278 INFO:teuthology.kill: smithi150.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsYO7npPuH6UfmSVIDkeTp0aj/zoK530DogLdVBGlaRngzvm5XT4TAPL+ddIvkQjNsLKeozsZRgJcGXllXxnceDd4Wj1yu/lc3VDhyyoqK0XChjqO/bytQy3GSbxqOmtXIm2jROZgD65/sYLvhilVnlTmsl+VUqY7/DPMV9iKRZHWranXK7+fh2I4mPjR/T3fDP6RdLwKmA8V1IkLuU9XWhOMJfTnU7hUe/y/ERiZvbZBGBqP9LDkwsj/oP+XpiNkhi8rVHWnHJIcGEEmJVcGkFYj3XvhfK/L4BlN9mM6n7+628Pgo+NelYmN8OXrx2OJcr42Myf7enpqQF5M20YzNTxY80Amk9H9vxX0ziCa6mfYRdP8+mrEsQeAdyLOO388atHWblpP4GSaHDWowPjpKto0UM+2fI/w5BPIl5oVIOoDhYYSR6FtWFpBFr7zcKFVjd3yeaTwiGIoO7H9ztbFsP1goqK1EzbVMlwMtWDHKATBZ7ABC7wEHUpUtoJyhe8U= 2022-07-05T05:48:50.278 INFO:teuthology.kill:2022-07-05 05:48:50,278.278 INFO:teuthology.nuke:Checking targets against current locks 2022-07-05T05:48:50.340 INFO:teuthology.kill:2022-07-05 05:48:50,339.339 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-05T05:48:50.371 INFO:teuthology.kill:2022-07-05 05:48:50,370.370 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-05T05:48:50.436 INFO:teuthology.kill:2022-07-05 05:48:50,435.435 INFO:teuthology.orchestra.console:Power off smithi135 2022-07-05T05:48:50.489 INFO:teuthology.kill:2022-07-05 05:48:50,489.489 INFO:teuthology.orchestra.console:Power off smithi150 2022-07-05T05:49:02.954 INFO:teuthology.kill:2022-07-05 05:49:02,954.954 INFO:teuthology.orchestra.console:Power off for smithi135 completed 2022-07-05T05:49:02.993 INFO:teuthology.kill:2022-07-05 05:49:02,993.993 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2022-07-05T05:49:03.131 INFO:teuthology.kill:2022-07-05 05:49:03,130.130 INFO:teuthology.lock.ops:unlocked: smithi135.front.sepia.ceph.com 2022-07-05T05:49:03.161 INFO:teuthology.kill:2022-07-05 05:49:03,161.161 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com 2022-07-05T05:51:03.594 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-07-05T05:51:03.785 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi135 but it was locked by another job: /home/teuthworker/archive/soumyakoduri-2022-07-05_05:48:04-rgw-wip-skoduri-dbstore-lc-distro-default-smithi/6913787 2022-07-05T05:51:03.786 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi150 but it was locked by another job: /home/teuthworker/archive/soumyakoduri-2022-07-05_05:48:04-rgw-wip-skoduri-dbstore-lc-distro-default-smithi/6913786