2022-09-03T10:56:52.734 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-03T10:56:52.889 INFO:teuthology.lock.ops:Start node 'smithi081.front.sepia.ceph.com' reimaging 2022-09-03T10:56:52.890 INFO:teuthology.lock.ops:Updating [smithi081.front.sepia.ceph.com]: reset os type and version on server 2022-09-03T10:56:52.891 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2022-09-03T10:56:52.912 INFO:teuthology.lock.ops:Node 'smithi081.front.sepia.ceph.com' reimaging is complete 2022-09-03T10:56:52.913 INFO:teuthology.lock.ops:Start node 'smithi175.front.sepia.ceph.com' reimaging 2022-09-03T10:56:52.913 INFO:teuthology.lock.ops:Updating [smithi175.front.sepia.ceph.com]: reset os type and version on server 2022-09-03T10:56:52.913 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server 2022-09-03T10:56:52.939 INFO:teuthology.lock.ops:Node 'smithi175.front.sepia.ceph.com' reimaging is complete 2022-09-03T10:56:53.145 INFO:teuthology.provision.fog.smithi081:Scheduling deploy of rhel 8.6 2022-09-03T10:56:53.176 INFO:teuthology.provision.fog.smithi175:Scheduling deploy of rhel 8.6 2022-09-03T10:56:53.496 INFO:teuthology.orchestra.console:Power off smithi081 2022-09-03T10:56:53.541 INFO:teuthology.orchestra.console:Power off smithi175 2022-09-03T10:57:01.882 INFO:teuthology.orchestra.console:Power off for smithi081 completed 2022-09-03T10:57:01.909 INFO:teuthology.orchestra.console:Power off for smithi175 completed 2022-09-03T10:57:01.984 INFO:teuthology.orchestra.console:Power on smithi081 2022-09-03T10:57:02.018 INFO:teuthology.orchestra.console:Power on smithi175 2022-09-03T10:57:10.356 INFO:teuthology.orchestra.console:Power on for smithi081 completed 2022-09-03T10:57:10.384 INFO:teuthology.orchestra.console:Power on for smithi175 completed 2022-09-03T10:57:10.458 INFO:teuthology.provision.fog.smithi081:Waiting for deploy to finish 2022-09-03T10:57:10.485 INFO:teuthology.provision.fog.smithi175:Waiting for deploy to finish 2022-09-03T11:01:42.347 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-03T11:01:42.590 INFO:teuthology.provision.fog.smithi175:Node is ready 2022-09-03T11:01:42.651 INFO:teuthology.orchestra.run.smithi175.stdout:smithi175.front.sepia.ceph.com 2022-09-03T11:01:42.712 INFO:teuthology.orchestra.run.smithi175.stdout:172.21.15.175 smithi175.front.sepia.ceph.com smithi175 2022-09-03T11:01:43.060 INFO:teuthology.provision.fog.smithi175:Deploy complete! 2022-09-03T11:01:51.324 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-03T11:01:51.595 INFO:teuthology.provision.fog.smithi081:Node is ready 2022-09-03T11:01:51.654 INFO:teuthology.orchestra.run.smithi081.stdout:smithi081 2022-09-03T11:01:51.712 INFO:teuthology.orchestra.run.smithi081.stdout:172.21.15.81 smithi081.front.sepia.ceph.com smithi081 2022-09-03T11:01:52.051 INFO:teuthology.provision.fog.smithi081:Deploy complete! 2022-09-03T11:01:52.250 INFO:teuthology.lock.ops:Checking smithi081.front.sepia.ceph.com 2022-09-03T11:01:52.251 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-03T11:01:52.272 INFO:teuthology.lock.ops:Checking smithi175.front.sepia.ceph.com 2022-09-03T11:01:52.272 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-03T11:01:52.292 INFO:teuthology.lock.ops:Updating [smithi081.front.sepia.ceph.com]: set os type and version on server 2022-09-03T11:01:52.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-09-03T11:01:52.492 INFO:teuthology.orchestra.run.smithi081.stdout:x86_64 2022-09-03T11:01:52.551 INFO:teuthology.orchestra.run.smithi081.stdout:NAME="Red Hat Enterprise Linux" 2022-09-03T11:01:52.551 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION="8.6 (Ootpa)" 2022-09-03T11:01:52.552 INFO:teuthology.orchestra.run.smithi081.stdout:ID="rhel" 2022-09-03T11:01:52.553 INFO:teuthology.orchestra.run.smithi081.stdout:ID_LIKE="fedora" 2022-09-03T11:01:52.553 INFO:teuthology.orchestra.run.smithi081.stdout:VERSION_ID="8.6" 2022-09-03T11:01:52.554 INFO:teuthology.orchestra.run.smithi081.stdout:PLATFORM_ID="platform:el8" 2022-09-03T11:01:52.558 INFO:teuthology.orchestra.run.smithi081.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-03T11:01:52.559 INFO:teuthology.orchestra.run.smithi081.stdout:ANSI_COLOR="0;31" 2022-09-03T11:01:52.559 INFO:teuthology.orchestra.run.smithi081.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-03T11:01:52.559 INFO:teuthology.orchestra.run.smithi081.stdout:HOME_URL="https://www.redhat.com/" 2022-09-03T11:01:52.559 INFO:teuthology.orchestra.run.smithi081.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-03T11:01:52.560 INFO:teuthology.orchestra.run.smithi081.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-03T11:01:52.560 INFO:teuthology.orchestra.run.smithi081.stdout: 2022-09-03T11:01:52.560 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-03T11:01:52.560 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-03T11:01:52.561 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-03T11:01:52.561 INFO:teuthology.orchestra.run.smithi081.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-03T11:01:52.563 INFO:teuthology.lock.ops:Updating smithi081.front.sepia.ceph.com on lock server 2022-09-03T11:01:52.584 INFO:teuthology.lock.ops:Updating [smithi175.front.sepia.ceph.com]: set os type and version on server 2022-09-03T11:01:52.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2022-09-03T11:01:52.806 INFO:teuthology.orchestra.run.smithi175.stdout:x86_64 2022-09-03T11:01:52.867 INFO:teuthology.orchestra.run.smithi175.stdout:NAME="Red Hat Enterprise Linux" 2022-09-03T11:01:52.867 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION="8.6 (Ootpa)" 2022-09-03T11:01:52.868 INFO:teuthology.orchestra.run.smithi175.stdout:ID="rhel" 2022-09-03T11:01:52.868 INFO:teuthology.orchestra.run.smithi175.stdout:ID_LIKE="fedora" 2022-09-03T11:01:52.868 INFO:teuthology.orchestra.run.smithi175.stdout:VERSION_ID="8.6" 2022-09-03T11:01:52.869 INFO:teuthology.orchestra.run.smithi175.stdout:PLATFORM_ID="platform:el8" 2022-09-03T11:01:52.869 INFO:teuthology.orchestra.run.smithi175.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-03T11:01:52.869 INFO:teuthology.orchestra.run.smithi175.stdout:ANSI_COLOR="0;31" 2022-09-03T11:01:52.869 INFO:teuthology.orchestra.run.smithi175.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-03T11:01:52.870 INFO:teuthology.orchestra.run.smithi175.stdout:HOME_URL="https://www.redhat.com/" 2022-09-03T11:01:52.870 INFO:teuthology.orchestra.run.smithi175.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-03T11:01:52.870 INFO:teuthology.orchestra.run.smithi175.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-03T11:01:52.871 INFO:teuthology.orchestra.run.smithi175.stdout: 2022-09-03T11:01:52.871 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-03T11:01:52.871 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-03T11:01:52.871 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-03T11:01:52.872 INFO:teuthology.orchestra.run.smithi175.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-03T11:01:52.872 INFO:teuthology.lock.ops:Updating smithi175.front.sepia.ceph.com on lock server 2022-09-03T11:01:52.977 INFO:teuthology.dispatcher.supervisor:Running job 7004207 2022-09-03T11:01:52.977 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-09-01_10:32:47-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi/7004207 --name benhanokh-2022-09-01_10:32:47-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi --description rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-stupid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} -- /home/teuthworker/archive/benhanokh-2022-09-01_10:32:47-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi/7004207/orig.config.yaml 2022-09-03T11:01:52.982 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-09-01_10:32:47-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi/7004207 2022-09-03T11:01:52.983 INFO:teuthology.dispatcher.supervisor:Job PID: 22828 2022-09-03T11:01:52.983 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-09-03T23:02:50.665 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-09-03T23:02:50.835 INFO:teuthology.kill:Killing Pids: {22828} 2022-09-03T23:02:52.313 INFO:teuthology.task.internal:roles: ubuntu@smithi081.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-09-03T23:02:52.314 INFO:teuthology.task.internal:roles: ubuntu@smithi175.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-09-03T23:02:52.315 INFO:teuthology.misc:Compressing logs... 2022-09-03T23:02:52.315 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi081.front.sepia.ceph.com' 2022-09-03T23:02:52.518 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi175.front.sepia.ceph.com' 2022-09-03T23:02:52.962 INFO:teuthology.misc:Compressing logs... 2022-09-03T23:02:54.053 INFO:teuthology.orchestra.run.smithi081.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-03T23:02:54.069 INFO:teuthology.orchestra.run.smithi175.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-03T23:05:51.304 INFO:teuthology.misc:Compressing logs... 2022-09-03T23:05:53.851 INFO:teuthology.kill:No teuthology processes running 2022-09-03T23:05:55.243 INFO:teuthology.kill:Nuking machines: ['smithi081', 'smithi175'] 2022-09-03T23:05:56.370 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-09-03T23:05:56.371 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-09-03T23:05:56.372 INFO:teuthology.kill:2022-09-03 23:05:56,370.370 INFO:teuthology.nuke:targets: 2022-09-03T23:05:56.372 INFO:teuthology.kill: smithi081.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOthNY8GJLHuSMrfILbkVme7GCWOmX4/UBw+mCM/wc7isD9rzWEJ/1mZ7jlEKdxtbtV4rRao1aGqTPvdj1/YjSVdFqqyOrL2oqAj7O3eVuZ2dfCV8e1uU1r0bi8wn8HzZLNHsaBn71dYo1qqL8wQvpAOmRnNDNtmA9HXWyrVPmFjOZyuVy+JHh5D0rczChRU8GtvsWKPY74eIfeUUk1TaYT+nIDIxdQsCiwFJCVLrqTqJpUV7nHz9815snoB7oRTi+U3AYCtZc2UC9wF+kg/OFjZnR2zBbbpB1prnaGExXSrnNRYHJG/3po1naRRiad/lAwOYMh1A9GDX+xkAULUiKeR+pSYprnorXiWcMY4ccxm2PozNYqaENh1R0A/u/pYBtseZtkQLPVfsLrgIjxN8oYCVq7VQg4ngk1wwCySUmWApDWlUT5PyLm4JcxA6LV/oLS1+b4VQEfdFeq005ifRCu04CbpRSK8GP7HO133KleczhX+L2cJIbCKTQigK+YtM= 2022-09-03T23:05:56.372 INFO:teuthology.kill: smithi175.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUA+JkIfvO6O8yT8SG/OvNejk94wP0yHjo+PZQATPGjwHdnruyvNF1Mt/C35DObIIWrzPCEBqdbtsLXE3aPUevoZcbjlzqakYbX6NSJ2KTwLY137TMZy9AjkRfWnA3em2ZAIvtitP3iz1zNATtMsXH4KFW+kpvu6XSnk9xLsUYr1CZXC+UZKMYOaBkiwq9OudOzfnreTBxTQQPTxasp/xVj6X9JolnVkGeRe1Yq/cX/jobCXnTC++98j1VPNbjMxd8gK1pCsyklcZE7vbBljnYdPswgr4jNz3uN2SdujqOsILQl1DfZfYxQjW1nmPeIjRMZ18Fr0Gu7PuVZMj7aKZXKYSE4V3gPOy7Wa/mwQfmbV5YtpiOJZ7VNqsLtJBNN3saF7bXDnZPEwaGeCqvk8iUNsS5zBJiS+sunHSFJ1xv8kqpMsuuJ1u592HZU1QZZ00CwvYiDC4ak/SzoejMgWJyA2eyisaZO5gN2+u5OyZriLMTdFA/+S3bY8DbQrkqTi0= 2022-09-03T23:05:56.373 INFO:teuthology.kill:2022-09-03 23:05:56,370.370 INFO:teuthology.nuke:Checking targets against current locks 2022-09-03T23:05:56.422 INFO:teuthology.kill:2022-09-03 23:05:56,422.422 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-03T23:05:56.448 INFO:teuthology.kill:2022-09-03 23:05:56,448.448 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-03T23:05:56.510 INFO:teuthology.kill:2022-09-03 23:05:56,509.509 INFO:teuthology.orchestra.console:Power off smithi081 2022-09-03T23:05:56.550 INFO:teuthology.kill:2022-09-03 23:05:56,550.550 INFO:teuthology.orchestra.console:Power off smithi175 2022-09-03T23:06:04.886 INFO:teuthology.kill:2022-09-03 23:06:04,885.885 INFO:teuthology.orchestra.console:Power off for smithi081 completed 2022-09-03T23:06:04.915 INFO:teuthology.kill:2022-09-03 23:06:04,915.915 INFO:teuthology.orchestra.console:Power off for smithi175 completed 2022-09-03T23:06:05.040 INFO:teuthology.kill:2022-09-03 23:06:05,040.040 INFO:teuthology.lock.ops:unlocked: smithi081.front.sepia.ceph.com 2022-09-03T23:06:05.065 INFO:teuthology.kill:2022-09-03 23:06:05,065.065 INFO:teuthology.lock.ops:unlocked: smithi175.front.sepia.ceph.com 2022-09-03T23:08:05.434 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-09-03T23:08:05.548 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi081 but it was locked by another job: /home/teuthworker/archive/benhanokh-2022-09-01_10:54:32-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi/7004418 2022-09-03T23:08:05.548 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi175 but it was locked by another job: /home/teuthworker/archive/benhanokh-2022-09-01_10:54:32-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi/7004417