2022-09-12T21:40:54.666 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-12T21:40:55.128 INFO:teuthology.lock.ops:Start node 'smithi055.front.sepia.ceph.com' reimaging 2022-09-12T21:40:55.130 INFO:teuthology.lock.ops:Updating [smithi055.front.sepia.ceph.com]: reset os type and version on server 2022-09-12T21:40:55.131 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2022-09-12T21:40:55.168 INFO:teuthology.lock.ops:Node 'smithi055.front.sepia.ceph.com' reimaging is complete 2022-09-12T21:40:55.169 INFO:teuthology.lock.ops:Start node 'smithi171.front.sepia.ceph.com' reimaging 2022-09-12T21:40:55.169 INFO:teuthology.lock.ops:Updating [smithi171.front.sepia.ceph.com]: reset os type and version on server 2022-09-12T21:40:55.170 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2022-09-12T21:40:55.236 INFO:teuthology.lock.ops:Node 'smithi171.front.sepia.ceph.com' reimaging is complete 2022-09-12T21:40:55.501 INFO:teuthology.provision.fog.smithi055:Scheduling deploy of rhel 8.6 2022-09-12T21:40:55.598 INFO:teuthology.provision.fog.smithi171:Scheduling deploy of rhel 8.6 2022-09-12T21:40:56.019 INFO:teuthology.orchestra.console:Power off smithi055 2022-09-12T21:40:56.225 INFO:teuthology.orchestra.console:Power off smithi171 2022-09-12T21:41:08.712 INFO:teuthology.orchestra.console:Power off for smithi055 completed 2022-09-12T21:41:08.759 INFO:teuthology.orchestra.console:Power off for smithi171 completed 2022-09-12T21:41:08.814 INFO:teuthology.orchestra.console:Power on smithi055 2022-09-12T21:41:08.860 INFO:teuthology.orchestra.console:Power on smithi171 2022-09-12T21:41:13.103 INFO:teuthology.orchestra.console:Power on for smithi055 completed 2022-09-12T21:41:13.208 INFO:teuthology.provision.fog.smithi055:Waiting for deploy to finish 2022-09-12T21:41:17.247 INFO:teuthology.orchestra.console:Power on for smithi171 completed 2022-09-12T21:41:17.349 INFO:teuthology.provision.fog.smithi171:Waiting for deploy to finish 2022-09-12T21:45:33.307 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-12T21:45:33.581 INFO:teuthology.provision.fog.smithi055:Node is ready 2022-09-12T21:45:33.657 INFO:teuthology.orchestra.run.smithi055.stdout:smithi055.front.sepia.ceph.com 2022-09-12T21:45:33.748 INFO:teuthology.orchestra.run.smithi055.stdout:172.21.15.55 smithi055.front.sepia.ceph.com smithi055 2022-09-12T21:45:34.164 INFO:teuthology.provision.fog.smithi055:Deploy complete! 2022-09-12T21:46:19.662 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-12T21:46:19.946 INFO:teuthology.provision.fog.smithi171:Node is ready 2022-09-12T21:46:19.966 INFO:teuthology.orchestra.run.smithi171.stdout:smithi171.front.sepia.ceph.com 2022-09-12T21:46:20.028 INFO:teuthology.orchestra.run.smithi171.stdout:172.21.15.171 smithi171.front.sepia.ceph.com smithi171 2022-09-12T21:46:20.382 INFO:teuthology.provision.fog.smithi171:Deploy complete! 2022-09-12T21:46:20.606 INFO:teuthology.lock.ops:Checking smithi055.front.sepia.ceph.com 2022-09-12T21:46:20.608 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-12T21:46:20.634 INFO:teuthology.lock.ops:Checking smithi171.front.sepia.ceph.com 2022-09-12T21:46:20.635 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-12T21:46:20.660 INFO:teuthology.lock.ops:Updating [smithi055.front.sepia.ceph.com]: set os type and version on server 2022-09-12T21:46:20.661 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2022-09-12T21:46:20.981 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2022-09-12T21:46:21.041 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="Red Hat Enterprise Linux" 2022-09-12T21:46:21.042 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="8.6 (Ootpa)" 2022-09-12T21:46:21.042 INFO:teuthology.orchestra.run.smithi055.stdout:ID="rhel" 2022-09-12T21:46:21.042 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE="fedora" 2022-09-12T21:46:21.042 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="8.6" 2022-09-12T21:46:21.043 INFO:teuthology.orchestra.run.smithi055.stdout:PLATFORM_ID="platform:el8" 2022-09-12T21:46:21.043 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-12T21:46:21.043 INFO:teuthology.orchestra.run.smithi055.stdout:ANSI_COLOR="0;31" 2022-09-12T21:46:21.043 INFO:teuthology.orchestra.run.smithi055.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-12T21:46:21.044 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://www.redhat.com/" 2022-09-12T21:46:21.044 INFO:teuthology.orchestra.run.smithi055.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-12T21:46:21.044 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-12T21:46:21.044 INFO:teuthology.orchestra.run.smithi055.stdout: 2022-09-12T21:46:21.045 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-12T21:46:21.045 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-12T21:46:21.045 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-12T21:46:21.046 INFO:teuthology.orchestra.run.smithi055.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-12T21:46:21.048 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2022-09-12T21:46:21.074 INFO:teuthology.lock.ops:Updating [smithi171.front.sepia.ceph.com]: set os type and version on server 2022-09-12T21:46:21.074 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2022-09-12T21:46:21.311 INFO:teuthology.orchestra.run.smithi171.stdout:x86_64 2022-09-12T21:46:21.370 INFO:teuthology.orchestra.run.smithi171.stdout:NAME="Red Hat Enterprise Linux" 2022-09-12T21:46:21.371 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION="8.6 (Ootpa)" 2022-09-12T21:46:21.371 INFO:teuthology.orchestra.run.smithi171.stdout:ID="rhel" 2022-09-12T21:46:21.371 INFO:teuthology.orchestra.run.smithi171.stdout:ID_LIKE="fedora" 2022-09-12T21:46:21.372 INFO:teuthology.orchestra.run.smithi171.stdout:VERSION_ID="8.6" 2022-09-12T21:46:21.372 INFO:teuthology.orchestra.run.smithi171.stdout:PLATFORM_ID="platform:el8" 2022-09-12T21:46:21.372 INFO:teuthology.orchestra.run.smithi171.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-12T21:46:21.372 INFO:teuthology.orchestra.run.smithi171.stdout:ANSI_COLOR="0;31" 2022-09-12T21:46:21.372 INFO:teuthology.orchestra.run.smithi171.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-12T21:46:21.373 INFO:teuthology.orchestra.run.smithi171.stdout:HOME_URL="https://www.redhat.com/" 2022-09-12T21:46:21.373 INFO:teuthology.orchestra.run.smithi171.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-12T21:46:21.373 INFO:teuthology.orchestra.run.smithi171.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-12T21:46:21.373 INFO:teuthology.orchestra.run.smithi171.stdout: 2022-09-12T21:46:21.373 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-12T21:46:21.374 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-12T21:46:21.374 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-12T21:46:21.374 INFO:teuthology.orchestra.run.smithi171.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-12T21:46:21.375 INFO:teuthology.lock.ops:Updating smithi171.front.sepia.ceph.com on lock server 2022-09-12T21:46:21.488 INFO:teuthology.dispatcher.supervisor:Running job 7029427 2022-09-12T21:46:21.489 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-09-12_21:02:31-rados-wip-yuri4-testing-2022-09-09-1105-distro-default-smithi/7029427 --name yuriw-2022-09-12_21:02:31-rados-wip-yuri4-testing-2022-09-09-1105-distro-default-smithi --description rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} -- /home/teuthworker/archive/yuriw-2022-09-12_21:02:31-rados-wip-yuri4-testing-2022-09-09-1105-distro-default-smithi/7029427/orig.config.yaml 2022-09-12T21:46:21.497 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-09-12_21:02:31-rados-wip-yuri4-testing-2022-09-09-1105-distro-default-smithi/7029427 2022-09-12T21:46:21.498 INFO:teuthology.dispatcher.supervisor:Job PID: 31966 2022-09-12T21:46:21.498 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-09-13T09:47:23.812 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-09-13T09:47:24.036 INFO:teuthology.kill:Killing Pids: {31966} 2022-09-13T09:47:25.450 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2022-09-13T09:47:25.451 INFO:teuthology.task.internal:roles: ubuntu@smithi171.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2022-09-13T09:47:25.452 INFO:teuthology.misc:Compressing logs... 2022-09-13T09:47:25.452 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2022-09-13T09:47:25.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi171.front.sepia.ceph.com' 2022-09-13T09:47:26.047 INFO:teuthology.misc:Compressing logs... 2022-09-13T09:47:26.371 INFO:teuthology.misc:Compressing logs... 2022-09-13T09:47:34.211 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-osd.2.log: file size changed while zipping 2022-09-13T09:47:35.512 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-osd.5.log: file size changed while zipping 2022-09-13T09:47:43.575 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-osd.6.log: file size changed while zipping 2022-09-13T09:47:44.423 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-osd.0.log: file size changed while zipping 2022-09-13T09:47:51.055 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-osd.4.log: file size changed while zipping 2022-09-13T09:47:53.792 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-osd.3.log: file size changed while zipping 2022-09-13T09:48:00.865 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-osd.7.log: file size changed while zipping 2022-09-13T09:48:03.455 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-osd.1.log: file size changed while zipping 2022-09-13T09:48:04.541 INFO:teuthology.orchestra.run.smithi171.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-mon.b.log: file size changed while zipping 2022-09-13T09:48:08.374 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-mon.c.log: file size changed while zipping 2022-09-13T09:48:22.008 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-mon.a.log: file size changed while zipping 2022-09-13T09:48:24.725 INFO:teuthology.orchestra.run.smithi055.stderr:gzip: /var/log/ceph/32337180-32e6-11ed-8431-001a4aab830c/ceph-mgr.y.log: file size changed while zipping 2022-09-13T09:49:23.649 INFO:teuthology.kill:No teuthology processes running 2022-09-13T09:49:25.383 INFO:teuthology.kill:Nuking machines: ['smithi055', 'smithi171'] 2022-09-13T09:49:26.619 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-09-13T09:49:26.620 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-09-13T09:49:26.620 INFO:teuthology.kill:2022-09-13 09:49:26,619.619 INFO:teuthology.nuke:targets: 2022-09-13T09:49:26.620 INFO:teuthology.kill: smithi055.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCqvPAheQY8fdrhqoUsg6npRDsupmGhEFBxKXJLoEnJwJ9IrWJJGPedQRQh+byLOdtMhjqFJxUUsO8pXa1V0KcUXdHgQgWGMwG4vvS8WWisZURxZr2+QtDd0NSG+vFsk0fZhhdc1hurAksVVt1WJfUvUrkVhtPGNvtcJBFOsBKN6bR+2d8a5EY3c24vkCX11rKsaUeeXJsmbwwKa+AaqOfGQSVoO6Sey+0AkgW/rtHvuYlVH8C9nyS/xznOE8SfCRE1SptP424b7gZDYQw+Hs5KWAF4XsZ+UZcf6+/AN/cNkjibGck374cPVbCeEHcbLwPA9d08Es+M09iN/+T8uATkr8vOiNUrPlKN6GExtLzX2zCbNX2QHbygY7fK3Pqwn2I0JEGafBin3U5GzfqJTFKFqtTnh1udJmH6ffY9LvZnyU5DMze40hsDxNZhK1qLI1Bu/1wfw0U5Vc1qUtb32MKr52xR4MIVGuXHtix2AKvnnj5Wa1/8WD6QmRBjvYCfNs= 2022-09-13T09:49:26.621 INFO:teuthology.kill: smithi171.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBImGxh/sVxH/odISiQ0vvYdRkbYnfvIBsLMbivM5CqfTKGulsu9xlGBRJsoADjrXZtceepf3VltGr81v1uK2wnY= 2022-09-13T09:49:26.621 INFO:teuthology.kill:2022-09-13 09:49:26,620.620 INFO:teuthology.nuke:Checking targets against current locks 2022-09-13T09:49:26.680 INFO:teuthology.kill:2022-09-13 09:49:26,679.679 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-13T09:49:26.707 INFO:teuthology.kill:2022-09-13 09:49:26,706.706 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-13T09:49:26.774 INFO:teuthology.kill:2022-09-13 09:49:26,774.774 INFO:teuthology.orchestra.console:Power off smithi055 2022-09-13T09:49:26.807 INFO:teuthology.kill:2022-09-13 09:49:26,807.807 INFO:teuthology.orchestra.console:Power off smithi171 2022-09-13T09:49:35.177 INFO:teuthology.kill:2022-09-13 09:49:35,177.177 INFO:teuthology.orchestra.console:Power off for smithi055 completed 2022-09-13T09:49:35.178 INFO:teuthology.kill:2022-09-13 09:49:35,177.177 INFO:teuthology.orchestra.console:Power off for smithi171 completed 2022-09-13T09:49:35.333 INFO:teuthology.kill:2022-09-13 09:49:35,332.332 INFO:teuthology.lock.ops:unlocked: smithi055.front.sepia.ceph.com 2022-09-13T09:49:35.338 INFO:teuthology.kill:2022-09-13 09:49:35,337.337 INFO:teuthology.lock.ops:unlocked: smithi171.front.sepia.ceph.com 2022-09-13T09:51:35.705 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-09-13T09:51:35.843 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi055 but it was locked by another job: /home/teuthworker/archive/vshankar-2022-09-13_09:03:58-fs-wip-vshankar-testing-20220909-120801-testing-default-smithi/7030456 2022-09-13T09:51:35.843 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi171 but it was locked by another job: /home/teuthworker/archive/vshankar-2022-09-13_09:03:58-fs-wip-vshankar-testing-20220909-120801-testing-default-smithi/7030456