2022-09-03T12:44:47.785 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-03T12:44:47.958 INFO:teuthology.lock.ops:Start node 'smithi027.front.sepia.ceph.com' reimaging 2022-09-03T12:44:47.961 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: reset os type and version on server 2022-09-03T12:44:47.961 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2022-09-03T12:44:47.982 INFO:teuthology.lock.ops:Node 'smithi027.front.sepia.ceph.com' reimaging is complete 2022-09-03T12:44:47.983 INFO:teuthology.lock.ops:Start node 'smithi093.front.sepia.ceph.com' reimaging 2022-09-03T12:44:47.983 INFO:teuthology.lock.ops:Updating [smithi093.front.sepia.ceph.com]: reset os type and version on server 2022-09-03T12:44:47.983 INFO:teuthology.lock.ops:Updating smithi093.front.sepia.ceph.com on lock server 2022-09-03T12:44:48.008 INFO:teuthology.lock.ops:Node 'smithi093.front.sepia.ceph.com' reimaging is complete 2022-09-03T12:44:48.203 INFO:teuthology.provision.fog.smithi027:Scheduling deploy of rhel 8.6 2022-09-03T12:44:48.232 INFO:teuthology.provision.fog.smithi093:Scheduling deploy of rhel 8.6 2022-09-03T12:44:48.528 INFO:teuthology.orchestra.console:Power off smithi027 2022-09-03T12:44:48.573 INFO:teuthology.orchestra.console:Power off smithi093 2022-09-03T12:44:56.911 INFO:teuthology.orchestra.console:Power off for smithi027 completed 2022-09-03T12:44:56.942 INFO:teuthology.orchestra.console:Power off for smithi093 completed 2022-09-03T12:44:57.014 INFO:teuthology.orchestra.console:Power on smithi027 2022-09-03T12:44:57.046 INFO:teuthology.orchestra.console:Power on smithi093 2022-09-03T12:45:05.390 INFO:teuthology.orchestra.console:Power on for smithi027 completed 2022-09-03T12:45:05.413 INFO:teuthology.orchestra.console:Power on for smithi093 completed 2022-09-03T12:45:05.492 INFO:teuthology.provision.fog.smithi027:Waiting for deploy to finish 2022-09-03T12:45:05.514 INFO:teuthology.provision.fog.smithi093:Waiting for deploy to finish 2022-09-03T12:49:11.792 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-03T12:49:36.947 INFO:teuthology.provision.fog.smithi093:Node is ready 2022-09-03T12:49:36.966 INFO:teuthology.orchestra.run.smithi093.stdout:smithi093.front.sepia.ceph.com 2022-09-03T12:49:37.023 INFO:teuthology.orchestra.run.smithi093.stdout:172.21.15.93 smithi093.front.sepia.ceph.com smithi093 2022-09-03T12:49:37.428 INFO:teuthology.provision.fog.smithi093:Deploy complete! 2022-09-03T12:50:25.542 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-03T12:50:25.789 INFO:teuthology.provision.fog.smithi027:Node is ready 2022-09-03T12:50:25.847 INFO:teuthology.orchestra.run.smithi027.stdout:smithi027.front.sepia.ceph.com 2022-09-03T12:50:25.907 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.15.27 smithi027.front.sepia.ceph.com smithi027 2022-09-03T12:50:26.254 INFO:teuthology.provision.fog.smithi027:Deploy complete! 2022-09-03T12:50:26.515 INFO:teuthology.lock.ops:Checking smithi027.front.sepia.ceph.com 2022-09-03T12:50:26.516 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-03T12:50:26.540 INFO:teuthology.lock.ops:Checking smithi093.front.sepia.ceph.com 2022-09-03T12:50:26.540 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-03T12:50:26.562 INFO:teuthology.lock.ops:Updating [smithi027.front.sepia.ceph.com]: set os type and version on server 2022-09-03T12:50:26.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2022-09-03T12:50:26.724 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2022-09-03T12:50:26.781 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="Red Hat Enterprise Linux" 2022-09-03T12:50:26.782 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8.6 (Ootpa)" 2022-09-03T12:50:26.782 INFO:teuthology.orchestra.run.smithi027.stdout:ID="rhel" 2022-09-03T12:50:26.782 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="fedora" 2022-09-03T12:50:26.783 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8.6" 2022-09-03T12:50:26.783 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8" 2022-09-03T12:50:26.783 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-03T12:50:26.783 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2022-09-03T12:50:26.784 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-03T12:50:26.784 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://www.redhat.com/" 2022-09-03T12:50:26.784 INFO:teuthology.orchestra.run.smithi027.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-03T12:50:26.784 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-03T12:50:26.785 INFO:teuthology.orchestra.run.smithi027.stdout: 2022-09-03T12:50:26.785 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-03T12:50:26.785 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-03T12:50:26.785 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-03T12:50:26.786 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-03T12:50:26.787 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2022-09-03T12:50:26.809 INFO:teuthology.lock.ops:Updating [smithi093.front.sepia.ceph.com]: set os type and version on server 2022-09-03T12:50:26.809 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2022-09-03T12:50:27.086 INFO:teuthology.orchestra.run.smithi093.stdout:x86_64 2022-09-03T12:50:27.147 INFO:teuthology.orchestra.run.smithi093.stdout:NAME="Red Hat Enterprise Linux" 2022-09-03T12:50:27.148 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION="8.6 (Ootpa)" 2022-09-03T12:50:27.148 INFO:teuthology.orchestra.run.smithi093.stdout:ID="rhel" 2022-09-03T12:50:27.148 INFO:teuthology.orchestra.run.smithi093.stdout:ID_LIKE="fedora" 2022-09-03T12:50:27.149 INFO:teuthology.orchestra.run.smithi093.stdout:VERSION_ID="8.6" 2022-09-03T12:50:27.149 INFO:teuthology.orchestra.run.smithi093.stdout:PLATFORM_ID="platform:el8" 2022-09-03T12:50:27.149 INFO:teuthology.orchestra.run.smithi093.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-03T12:50:27.149 INFO:teuthology.orchestra.run.smithi093.stdout:ANSI_COLOR="0;31" 2022-09-03T12:50:27.150 INFO:teuthology.orchestra.run.smithi093.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-03T12:50:27.150 INFO:teuthology.orchestra.run.smithi093.stdout:HOME_URL="https://www.redhat.com/" 2022-09-03T12:50:27.150 INFO:teuthology.orchestra.run.smithi093.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-03T12:50:27.151 INFO:teuthology.orchestra.run.smithi093.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-03T12:50:27.151 INFO:teuthology.orchestra.run.smithi093.stdout: 2022-09-03T12:50:27.151 INFO:teuthology.orchestra.run.smithi093.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-03T12:50:27.151 INFO:teuthology.orchestra.run.smithi093.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-03T12:50:27.152 INFO:teuthology.orchestra.run.smithi093.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-03T12:50:27.152 INFO:teuthology.orchestra.run.smithi093.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-03T12:50:27.152 INFO:teuthology.lock.ops:Updating smithi093.front.sepia.ceph.com on lock server 2022-09-03T12:50:30.133 INFO:teuthology.dispatcher.supervisor:Running job 7004347 2022-09-03T12:50:30.133 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-09-01_10:32:47-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi/7004347 --name benhanokh-2022-09-01_10:32:47-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-pool-snaps-readproxy} -- /home/teuthworker/archive/benhanokh-2022-09-01_10:32:47-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi/7004347/orig.config.yaml 2022-09-03T12:50:30.138 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-09-01_10:32:47-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi/7004347 2022-09-03T12:50:30.139 INFO:teuthology.dispatcher.supervisor:Job PID: 15183 2022-09-03T12:50:30.139 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-09-04T00:50:52.872 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-09-04T00:50:53.055 INFO:teuthology.kill:Killing Pids: {15183} 2022-09-04T00:50:54.550 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-09-04T00:50:54.551 INFO:teuthology.task.internal:roles: ubuntu@smithi093.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-09-04T00:50:54.551 INFO:teuthology.misc:Compressing logs... 2022-09-04T00:50:54.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2022-09-04T00:50:54.772 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi093.front.sepia.ceph.com' 2022-09-04T00:50:55.151 INFO:teuthology.misc:Compressing logs... 2022-09-04T00:50:56.068 INFO:teuthology.orchestra.run.smithi093.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-04T00:50:56.446 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-04T00:52:23.266 INFO:teuthology.misc:Compressing logs... 2022-09-04T00:52:29.697 INFO:teuthology.kill:No teuthology processes running 2022-09-04T00:52:31.287 INFO:teuthology.kill:Nuking machines: ['smithi027', 'smithi093'] 2022-09-04T00:52:32.476 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-09-04T00:52:32.477 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-09-04T00:52:32.478 INFO:teuthology.kill:2022-09-04 00:52:32,476.476 INFO:teuthology.nuke:targets: 2022-09-04T00:52:32.478 INFO:teuthology.kill: smithi027.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChKPbcabntPj8i3eRnRpZWEHiXx49iGySevuTXqNKZJmESQyKsuJVJWUYepl1YSuh3O4wD3uSAV9oQnzjCnv+Yz04ueuTMOayBtr/P628U/jfzPFlxTlvDRCVf7yvRTwahKStS6kiFq+cvvwgxm44MUziLaN342erW0FP79n4RiB3HbUjRfbx4dVHE5sqFDkET6XG+GU/B61YRT86BhXkw7COeJQSXyAcw1RHxip43p+uZo0BEDZSQP4TWXwL+8nX6isGjRWw2HqhOZo7TIFy/fIyOsqJA9Wy/kLAqab8WQPzLUn9fcc5eEmtNL0LZF8YlOsaGPWwc8ty6xs+CSSFQBYSOBmVRlWFTSYA6Wp7Oxn0XvtHb0krEZwBV/f605Cd1PF4dyI0I1FGtB1BrT3nlqs7r+YSK3cBVwX8gUDhMuBSdWqPI08xlznxcoxcjQJU55R3/pGGLXfXRIYxa49S/e9slhVrTFYa5J/36LBxOEJ1ZXoi6TTdoFBpkQgpSSD8= 2022-09-04T00:52:32.478 INFO:teuthology.kill: smithi093.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDjvHnXSbGHpwGc0gSws/2JQzEHg4LhnFvJOmfMuqN7x2OoOdgfWszMCSUJ9yuvxLEzD4t85HmGcZBaGpM0pglDETcwJnFotpgh6HxJ916SnVY7hN2rcAsvMEPS+tYwLOebdTe0ZMpb4MKJOezNVhNpGnA9Luaj90COWWZ1TGk/uZCCfVcKc4G2060o2bJQNxnKp8JUGEjBUWdsQJ0PKkd/UYAGF+IJKP54UWEWZNfcaTmAQmU5aPe1U6cffXCkHq7IlzHstXhxK/0QbA+8HasbfP0uuvTOpnOonbQVsVSdc0yrrXk26fRRqnSFMXm1ltP+u9XPKpq9t1PNw2FxxW2uR7jK8e+Kpg0RFE8jHHagPoBDE+/dx7K9isYBhy/2gNRKMaECRulyXO32ZrL67/+RS+S/sowoKHYgcBAljWoWHcLBpRpfRjsWIzUmFgSF3pY84UdAWvDkm+oDL8mYq/8mwvvj5eSVnJsk5v4SdJIpFdoRbk7taQMVMJvR1V4mHCk= 2022-09-04T00:52:32.479 INFO:teuthology.kill:2022-09-04 00:52:32,477.477 INFO:teuthology.nuke:Checking targets against current locks 2022-09-04T00:52:32.552 INFO:teuthology.kill:2022-09-04 00:52:32,551.551 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-04T00:52:32.577 INFO:teuthology.kill:2022-09-04 00:52:32,577.577 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-04T00:52:32.641 INFO:teuthology.kill:2022-09-04 00:52:32,640.640 INFO:teuthology.orchestra.console:Power off smithi027 2022-09-04T00:52:32.681 INFO:teuthology.kill:2022-09-04 00:52:32,680.680 INFO:teuthology.orchestra.console:Power off smithi093 2022-09-04T00:52:41.014 INFO:teuthology.kill:2022-09-04 00:52:41,014.014 INFO:teuthology.orchestra.console:Power off for smithi027 completed 2022-09-04T00:52:41.053 INFO:teuthology.kill:2022-09-04 00:52:41,052.052 INFO:teuthology.orchestra.console:Power off for smithi093 completed 2022-09-04T00:52:41.172 INFO:teuthology.kill:2022-09-04 00:52:41,169.169 INFO:teuthology.lock.ops:unlocked: smithi027.front.sepia.ceph.com 2022-09-04T00:52:41.204 INFO:teuthology.kill:2022-09-04 00:52:41,204.204 INFO:teuthology.lock.ops:unlocked: smithi093.front.sepia.ceph.com 2022-09-04T00:54:41.593 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-09-04T00:54:41.743 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi027 but it was locked by another job: /home/teuthworker/archive/adking-2022-09-03_23:13:39-orch:cephadm-wip-adk3-testing-2022-09-03-1327-quincy-distro-default-smithi/7009746 2022-09-04T00:54:41.744 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi093 but it was locked by another job: /home/teuthworker/archive/adking-2022-09-03_23:13:39-orch:cephadm-wip-adk3-testing-2022-09-03-1327-quincy-distro-default-smithi/7009746