2022-09-05T10:36:47.004 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-05T10:36:47.142 INFO:teuthology.lock.ops:Start node 'smithi062.front.sepia.ceph.com' reimaging 2022-09-05T10:36:47.143 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: reset os type and version on server 2022-09-05T10:36:47.144 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2022-09-05T10:36:47.168 INFO:teuthology.lock.ops:Node 'smithi062.front.sepia.ceph.com' reimaging is complete 2022-09-05T10:36:47.169 INFO:teuthology.lock.ops:Start node 'smithi150.front.sepia.ceph.com' reimaging 2022-09-05T10:36:47.170 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: reset os type and version on server 2022-09-05T10:36:47.170 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2022-09-05T10:36:47.194 INFO:teuthology.lock.ops:Node 'smithi150.front.sepia.ceph.com' reimaging is complete 2022-09-05T10:36:47.392 INFO:teuthology.provision.fog.smithi062:Scheduling deploy of rhel 8.6 2022-09-05T10:36:47.402 INFO:teuthology.provision.fog.smithi150:Scheduling deploy of rhel 8.6 2022-09-05T10:36:47.757 INFO:teuthology.orchestra.console:Power off smithi150 2022-09-05T10:36:47.774 INFO:teuthology.orchestra.console:Power off smithi062 2022-09-05T10:36:56.141 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2022-09-05T10:36:56.242 INFO:teuthology.orchestra.console:Power on smithi062 2022-09-05T10:37:00.247 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2022-09-05T10:37:00.349 INFO:teuthology.orchestra.console:Power on smithi150 2022-09-05T10:37:04.609 INFO:teuthology.orchestra.console:Power on for smithi062 completed 2022-09-05T10:37:04.710 INFO:teuthology.provision.fog.smithi062:Waiting for deploy to finish 2022-09-05T10:37:08.726 INFO:teuthology.orchestra.console:Power on for smithi150 completed 2022-09-05T10:37:08.828 INFO:teuthology.provision.fog.smithi150:Waiting for deploy to finish 2022-09-05T10:41:36.444 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-05T10:41:36.688 INFO:teuthology.provision.fog.smithi062:Node is ready 2022-09-05T10:41:36.707 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062.front.sepia.ceph.com 2022-09-05T10:41:36.767 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62 smithi062.front.sepia.ceph.com smithi062 2022-09-05T10:41:37.108 INFO:teuthology.provision.fog.smithi062:Deploy complete! 2022-09-05T10:41:55.541 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-05T10:41:55.799 INFO:teuthology.provision.fog.smithi150:Node is ready 2022-09-05T10:41:55.818 INFO:teuthology.orchestra.run.smithi150.stdout:smithi150 2022-09-05T10:41:55.883 INFO:teuthology.orchestra.run.smithi150.stdout:172.21.15.150 smithi150.front.sepia.ceph.com smithi150 2022-09-05T10:41:56.228 INFO:teuthology.provision.fog.smithi150:Deploy complete! 2022-09-05T10:41:56.433 INFO:teuthology.lock.ops:Checking smithi062.front.sepia.ceph.com 2022-09-05T10:41:56.434 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-05T10:41:56.464 INFO:teuthology.lock.ops:Checking smithi150.front.sepia.ceph.com 2022-09-05T10:41:56.465 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-05T10:41:56.487 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: set os type and version on server 2022-09-05T10:41:56.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2022-09-05T10:41:56.791 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2022-09-05T10:41:56.852 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="Red Hat Enterprise Linux" 2022-09-05T10:41:56.852 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="8.6 (Ootpa)" 2022-09-05T10:41:56.853 INFO:teuthology.orchestra.run.smithi062.stdout:ID="rhel" 2022-09-05T10:41:56.853 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="fedora" 2022-09-05T10:41:56.854 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="8.6" 2022-09-05T10:41:56.855 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el8" 2022-09-05T10:41:56.855 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-05T10:41:56.856 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2022-09-05T10:41:56.856 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-05T10:41:56.857 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.redhat.com/" 2022-09-05T10:41:56.857 INFO:teuthology.orchestra.run.smithi062.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-05T10:41:56.858 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-05T10:41:56.859 INFO:teuthology.orchestra.run.smithi062.stdout: 2022-09-05T10:41:56.859 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-05T10:41:56.860 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-05T10:41:56.860 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-05T10:41:56.861 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-05T10:41:56.863 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2022-09-05T10:41:56.885 INFO:teuthology.lock.ops:Updating [smithi150.front.sepia.ceph.com]: set os type and version on server 2022-09-05T10:41:56.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2022-09-05T10:41:57.106 INFO:teuthology.orchestra.run.smithi150.stdout:x86_64 2022-09-05T10:41:57.160 INFO:teuthology.orchestra.run.smithi150.stdout:NAME="Red Hat Enterprise Linux" 2022-09-05T10:41:57.160 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION="8.6 (Ootpa)" 2022-09-05T10:41:57.161 INFO:teuthology.orchestra.run.smithi150.stdout:ID="rhel" 2022-09-05T10:41:57.161 INFO:teuthology.orchestra.run.smithi150.stdout:ID_LIKE="fedora" 2022-09-05T10:41:57.161 INFO:teuthology.orchestra.run.smithi150.stdout:VERSION_ID="8.6" 2022-09-05T10:41:57.162 INFO:teuthology.orchestra.run.smithi150.stdout:PLATFORM_ID="platform:el8" 2022-09-05T10:41:57.162 INFO:teuthology.orchestra.run.smithi150.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-05T10:41:57.162 INFO:teuthology.orchestra.run.smithi150.stdout:ANSI_COLOR="0;31" 2022-09-05T10:41:57.162 INFO:teuthology.orchestra.run.smithi150.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-05T10:41:57.163 INFO:teuthology.orchestra.run.smithi150.stdout:HOME_URL="https://www.redhat.com/" 2022-09-05T10:41:57.163 INFO:teuthology.orchestra.run.smithi150.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-05T10:41:57.163 INFO:teuthology.orchestra.run.smithi150.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-05T10:41:57.163 INFO:teuthology.orchestra.run.smithi150.stdout: 2022-09-05T10:41:57.163 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-05T10:41:57.164 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-05T10:41:57.164 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-05T10:41:57.164 INFO:teuthology.orchestra.run.smithi150.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-05T10:41:57.165 INFO:teuthology.lock.ops:Updating smithi150.front.sepia.ceph.com on lock server 2022-09-05T10:41:57.296 INFO:teuthology.dispatcher.supervisor:Running job 7011001 2022-09-05T10:41:57.297 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-09-04_20:55:20-rados-WIP_GBH_snap_mapper_one_wayX1-distro-default-smithi/7011001 --name benhanokh-2022-09-04_20:55:20-rados-WIP_GBH_snap_mapper_one_wayX1-distro-default-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache} -- /home/teuthworker/archive/benhanokh-2022-09-04_20:55:20-rados-WIP_GBH_snap_mapper_one_wayX1-distro-default-smithi/7011001/orig.config.yaml 2022-09-05T10:41:57.303 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-09-04_20:55:20-rados-WIP_GBH_snap_mapper_one_wayX1-distro-default-smithi/7011001 2022-09-05T10:41:57.304 INFO:teuthology.dispatcher.supervisor:Job PID: 7536 2022-09-05T10:41:57.304 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-09-05T22:42:54.672 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-09-05T22:42:54.871 INFO:teuthology.kill:Killing Pids: {7536} 2022-09-05T22:42:56.878 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-09-05T22:42:56.880 INFO:teuthology.task.internal:roles: ubuntu@smithi150.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-09-05T22:42:56.881 INFO:teuthology.misc:Compressing logs... 2022-09-05T22:42:56.882 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2022-09-05T22:42:57.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi150.front.sepia.ceph.com' 2022-09-05T22:42:57.526 INFO:teuthology.misc:Compressing logs... 2022-09-05T22:42:58.476 INFO:teuthology.orchestra.run.smithi150.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-05T22:42:58.843 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-05T22:43:19.233 INFO:teuthology.misc:Compressing logs... 2022-09-05T22:43:34.242 INFO:teuthology.kill:No teuthology processes running 2022-09-05T22:43:35.686 INFO:teuthology.kill:Nuking machines: ['smithi062', 'smithi150'] 2022-09-05T22:43:36.817 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-09-05T22:43:36.818 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-09-05T22:43:36.818 INFO:teuthology.kill:2022-09-05 22:43:36,816.816 INFO:teuthology.nuke:targets: 2022-09-05T22:43:36.818 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgAVoOE2IUPmgnqRvNxKuNnjxLc1KMlG46PE/i0G57Hx6Wis3a0031V0zhf282xBP5YaX6Z/1fW5pAzJXxS42HaiisCXyJES/Yl+h2cb8wSTXPiGavROZIvZrznNqwH/LB03l0feSr7M7dhcJptTFIsqar2wEiSe/nl+VC4xaHzsVOzrIdduObDEImmJMT2gC+krVJBiyr0A4lXb3BcLz7580eCd42YdM4fqI1VsPcX1jPXkkj1IoFQKed/pCsnaHn398sWuzrqOG8l4IkCkji+NaoWyOYhaBiXQtHy4SUhQxavAMy4uwvEzGPhEvnX0Vo2F4IKZoeSbXGI3PvBdsFjeXTQD7qoTW4v+T8k2+/YDgffaRHzEAS14w8Pz70u44xNoIOg8kBz2jpBUxK/J4z6xbtfS4IvJA1mIX7MbcFSEjcwPyaOP4d1U86lPKGKmstSsEFLVbFDWp3quitH1BTautmBIKKIIX9MfUGyQc72NMWCtzsFK1Fhtj0B+Vwdf8= 2022-09-05T22:43:36.819 INFO:teuthology.kill: smithi150.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxqwnCmxA6FEJSWxxVQh4ayKbmLZdRaoE9DjWRB8ZJ9fYcPo0+LSLYt4LfDkU9+sfRPjWoWhbmMEibLCMZuzBqE73UbvFSgef/nCQ3SdyNA9jklQ54L7iKV6xURdBtxWeyWdR0YQti8DrzB3hjef0fklm/RClzbFN0qWLyZZ5HVFUSvx08LjJa0EBjYSOtzoTjHL9J6cQoZP4Lq9Yac5rWSJG3mDPt2b0UVWI9QC5YNXbnEBR1macO4BhK67cqRCeUSyIQLdd2ONhu0JjRguU/G8SURr5OnFE55ZJmAn7fOoWveyKz3YUd2upyUVWGc+thWgXIiovikFm2oMnKbD4VA01GwNURXUF9QnekHKBJSyAV8m0ZLKDSTCA0h1GnTgPrKCTSEF15jgACBm5f86yZZBwErLMomO+g1nWQDW2YwSV0HWEvYdL0umIABdRK+Kf+aHxe1FxvsBx+SYdYBm/hvIEYy4OkenIKEZGytLjRV03mzpzvekzoNQ6EpPWKP9M= 2022-09-05T22:43:36.819 INFO:teuthology.kill:2022-09-05 22:43:36,817.817 INFO:teuthology.nuke:Checking targets against current locks 2022-09-05T22:43:36.875 INFO:teuthology.kill:2022-09-05 22:43:36,875.875 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-05T22:43:36.902 INFO:teuthology.kill:2022-09-05 22:43:36,902.902 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-05T22:43:36.963 INFO:teuthology.kill:2022-09-05 22:43:36,962.962 INFO:teuthology.orchestra.console:Power off smithi062 2022-09-05T22:43:37.007 INFO:teuthology.kill:2022-09-05 22:43:37,007.007 INFO:teuthology.orchestra.console:Power off smithi150 2022-09-05T22:43:45.344 INFO:teuthology.kill:2022-09-05 22:43:45,343.343 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2022-09-05T22:43:45.372 INFO:teuthology.kill:2022-09-05 22:43:45,372.372 INFO:teuthology.orchestra.console:Power off for smithi150 completed 2022-09-05T22:43:45.506 INFO:teuthology.kill:2022-09-05 22:43:45,505.505 INFO:teuthology.lock.ops:unlocked: smithi062.front.sepia.ceph.com 2022-09-05T22:43:45.529 INFO:teuthology.kill:2022-09-05 22:43:45,529.529 INFO:teuthology.lock.ops:unlocked: smithi150.front.sepia.ceph.com 2022-09-05T22:45:45.913 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-09-05T22:45:46.056 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi062 but it was locked by another job: /home/teuthworker/archive/adking-2022-09-05_21:29:03-orch:cephadm-wip-adk-testing-2022-09-05-1506-distro-default-smithi/7012661 2022-09-05T22:45:46.057 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi150 but it was locked by another job: /home/teuthworker/archive/adking-2022-09-05_21:29:03-orch:cephadm-wip-adk-testing-2022-09-05-1506-distro-default-smithi/7012658