2022-09-05T10:54:30.153 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-05T10:54:30.304 INFO:teuthology.lock.ops:Start node 'smithi119.front.sepia.ceph.com' reimaging 2022-09-05T10:54:30.305 INFO:teuthology.lock.ops:Updating [smithi119.front.sepia.ceph.com]: reset os type and version on server 2022-09-05T10:54:30.306 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2022-09-05T10:54:30.327 INFO:teuthology.lock.ops:Node 'smithi119.front.sepia.ceph.com' reimaging is complete 2022-09-05T10:54:30.328 INFO:teuthology.lock.ops:Start node 'smithi165.front.sepia.ceph.com' reimaging 2022-09-05T10:54:30.329 INFO:teuthology.lock.ops:Updating [smithi165.front.sepia.ceph.com]: reset os type and version on server 2022-09-05T10:54:30.329 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2022-09-05T10:54:30.354 INFO:teuthology.lock.ops:Node 'smithi165.front.sepia.ceph.com' reimaging is complete 2022-09-05T10:54:30.592 INFO:teuthology.provision.fog.smithi119:Scheduling deploy of rhel 8.6 2022-09-05T10:54:30.609 INFO:teuthology.provision.fog.smithi165:Scheduling deploy of rhel 8.6 2022-09-05T10:54:30.983 INFO:teuthology.orchestra.console:Power off smithi119 2022-09-05T10:54:31.034 INFO:teuthology.orchestra.console:Power off smithi165 2022-09-05T10:54:31.144 INFO:teuthology.orchestra.console:Power off for smithi119 completed 2022-09-05T10:54:31.245 INFO:teuthology.orchestra.console:Power on smithi119 2022-09-05T10:54:39.609 INFO:teuthology.orchestra.console:Power on for smithi119 completed 2022-09-05T10:54:39.711 INFO:teuthology.provision.fog.smithi119:Waiting for deploy to finish 2022-09-05T10:54:43.518 INFO:teuthology.orchestra.console:Power off for smithi165 completed 2022-09-05T10:54:43.620 INFO:teuthology.orchestra.console:Power on smithi165 2022-09-05T10:54:47.874 INFO:teuthology.orchestra.console:Power on for smithi165 completed 2022-09-05T10:54:47.975 INFO:teuthology.provision.fog.smithi165:Waiting for deploy to finish 2022-09-05T10:58:46.191 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-05T10:59:11.322 INFO:teuthology.provision.fog.smithi119:Node is ready 2022-09-05T10:59:11.343 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119.front.sepia.ceph.com 2022-09-05T10:59:11.400 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119 smithi119.front.sepia.ceph.com smithi119 2022-09-05T10:59:11.769 INFO:teuthology.provision.fog.smithi119:Deploy complete! 2022-09-05T10:59:18.417 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-05T10:59:43.712 INFO:teuthology.provision.fog.smithi165:Node is ready 2022-09-05T10:59:43.731 INFO:teuthology.orchestra.run.smithi165.stdout:smithi165.front.sepia.ceph.com 2022-09-05T10:59:43.788 INFO:teuthology.orchestra.run.smithi165.stdout:172.21.15.165 smithi165.front.sepia.ceph.com smithi165 2022-09-05T10:59:44.142 INFO:teuthology.provision.fog.smithi165:Deploy complete! 2022-09-05T10:59:44.347 INFO:teuthology.lock.ops:Checking smithi119.front.sepia.ceph.com 2022-09-05T10:59:44.348 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-05T10:59:44.369 INFO:teuthology.lock.ops:Checking smithi165.front.sepia.ceph.com 2022-09-05T10:59:44.370 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-05T10:59:44.390 INFO:teuthology.lock.ops:Updating [smithi119.front.sepia.ceph.com]: set os type and version on server 2022-09-05T10:59:44.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2022-09-05T10:59:44.622 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2022-09-05T10:59:44.681 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="Red Hat Enterprise Linux" 2022-09-05T10:59:44.682 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="8.6 (Ootpa)" 2022-09-05T10:59:44.686 INFO:teuthology.orchestra.run.smithi119.stdout:ID="rhel" 2022-09-05T10:59:44.686 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="fedora" 2022-09-05T10:59:44.686 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="8.6" 2022-09-05T10:59:44.687 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el8" 2022-09-05T10:59:44.687 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-05T10:59:44.688 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2022-09-05T10:59:44.688 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-05T10:59:44.688 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://www.redhat.com/" 2022-09-05T10:59:44.689 INFO:teuthology.orchestra.run.smithi119.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-05T10:59:44.689 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-05T10:59:44.690 INFO:teuthology.orchestra.run.smithi119.stdout: 2022-09-05T10:59:44.690 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-05T10:59:44.691 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-05T10:59:44.691 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-05T10:59:44.691 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-05T10:59:44.693 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2022-09-05T10:59:44.715 INFO:teuthology.lock.ops:Updating [smithi165.front.sepia.ceph.com]: set os type and version on server 2022-09-05T10:59:44.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2022-09-05T10:59:44.932 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2022-09-05T10:59:44.989 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="Red Hat Enterprise Linux" 2022-09-05T10:59:44.990 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="8.6 (Ootpa)" 2022-09-05T10:59:44.991 INFO:teuthology.orchestra.run.smithi165.stdout:ID="rhel" 2022-09-05T10:59:44.991 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="fedora" 2022-09-05T10:59:44.992 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="8.6" 2022-09-05T10:59:44.994 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el8" 2022-09-05T10:59:44.995 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-05T10:59:44.996 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2022-09-05T10:59:44.996 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-05T10:59:44.996 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://www.redhat.com/" 2022-09-05T10:59:44.996 INFO:teuthology.orchestra.run.smithi165.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-05T10:59:44.998 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-05T10:59:44.998 INFO:teuthology.orchestra.run.smithi165.stdout: 2022-09-05T10:59:44.999 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-05T10:59:44.999 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-05T10:59:44.999 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-05T10:59:45.000 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-05T10:59:45.001 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2022-09-05T10:59:45.134 INFO:teuthology.dispatcher.supervisor:Running job 7011029 2022-09-05T10:59:45.135 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-09-04_20:55:20-rados-WIP_GBH_snap_mapper_one_wayX1-distro-default-smithi/7011029 --name benhanokh-2022-09-04_20:55:20-rados-WIP_GBH_snap_mapper_one_wayX1-distro-default-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/pool-snaps-few-objects} -- /home/teuthworker/archive/benhanokh-2022-09-04_20:55:20-rados-WIP_GBH_snap_mapper_one_wayX1-distro-default-smithi/7011029/orig.config.yaml 2022-09-05T10:59:45.141 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-09-04_20:55:20-rados-WIP_GBH_snap_mapper_one_wayX1-distro-default-smithi/7011029 2022-09-05T10:59:45.142 INFO:teuthology.dispatcher.supervisor:Job PID: 30559 2022-09-05T10:59:45.142 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-09-05T23:00:42.500 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-09-05T23:00:42.690 INFO:teuthology.kill:Killing Pids: {30559} 2022-09-05T23:00:44.317 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-09-05T23:00:44.318 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-09-05T23:00:44.318 INFO:teuthology.misc:Compressing logs... 2022-09-05T23:00:44.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2022-09-05T23:00:44.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2022-09-05T23:00:44.994 INFO:teuthology.misc:Compressing logs... 2022-09-05T23:00:45.966 INFO:teuthology.orchestra.run.smithi165.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-05T23:00:46.232 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-05T23:02:02.185 INFO:teuthology.misc:Compressing logs... 2022-09-05T23:02:05.423 INFO:teuthology.kill:No teuthology processes running 2022-09-05T23:02:07.128 INFO:teuthology.kill:Nuking machines: ['smithi119', 'smithi165'] 2022-09-05T23:02:08.458 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-09-05T23:02:08.459 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-09-05T23:02:08.460 INFO:teuthology.kill:2022-09-05 23:02:08,458.458 INFO:teuthology.nuke:targets: 2022-09-05T23:02:08.460 INFO:teuthology.kill: smithi119.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAaFc1kkvJ2GrqPNy4NeRBuB1TERkzOC38Gzg8jUpk8Yguucong24jGu/GcmMYkXTKphpr9zaic3ocU0EaHyMFYJnYcAnMCwvjlwWvjarPfaJQWCwxLlYZskXDMqNHTdLAXK9ctbF2xbWqo5zHO+b/3uWkf8SgE0qQnwkMsgM7j4LQBwROb/vH5m6SFniM9ImMWa7DK3VbakjbJR+rxt2brFDIZwXt/hdYfWyDxSLDYg2C/PG95XshbL3VbtGr2qy6gAY03TQIEjsXj85k0Xf7c/58K4KVHTz8TKGJpl0to5zVMLfd5Yce7FAAE1P3eJDaRXozoFd7o485d4viqgp4Fnvb/DoBN7CawhNwO8afzTL0H+iJDKSfidG9LBOXg2zfj4o0ndkCfIe7Z7NrM72V2kH16cTbecgsoMyqznRB0nkAn5ZFdY6v2+WPq0gP2JZsv25GHAwFV9NoccsypoMdDMekORaGZjYhabFXa8P1Yezb3Z8TItPy8pcLachkltM= 2022-09-05T23:02:08.460 INFO:teuthology.kill: smithi165.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDiuL00PGd59av8Rd9120baKbYJvWFmWWWmejTS0v5QZkXzC+7uBM8LaegknV54KyBqpWy0bCCUAKYqoWe7F123RP4MRItJGGcngPsWBEqMviV2lg1YWAqKC66kyKL4Bikplbn4GjXn27fyJa7khIjEV5xSLbYnKAxMIR9d0M8CCNpaLAMdFRSyqbSNTs5qA4XDdjMAiBVU1IhT+UKX5Qv2It7MuppFPWfQWPJl/rzpLnAqa+5ibtYQfs6TOttjSSk0PFnjmTHh+jrFEwq/ufkMDUqcGpYwhvHvz1c7/z66zj8SCnxlL18Ppnu1/blm/eUuRWCg8HWfEnvr0nzSu3KvCDM48BfW+beJLp8/4OvvSX7B1yTsLlrhCWbzZFWVjfV6/Rw34RBy0ydXPSWooQstHPgfr5KVbSqBr1EEvw6nG+EXTcYazwaAvQiJtzsqjN6+VGi30cwmH+voF5GrsVBVR4OM5U/sZIzA1qS68RoJ/OZuU9oToZ4y6ZObKxfLp6c= 2022-09-05T23:02:08.461 INFO:teuthology.kill:2022-09-05 23:02:08,461.461 INFO:teuthology.nuke:Checking targets against current locks 2022-09-05T23:02:08.528 INFO:teuthology.kill:2022-09-05 23:02:08,528.528 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-05T23:02:08.554 INFO:teuthology.kill:2022-09-05 23:02:08,553.553 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-05T23:02:08.634 INFO:teuthology.kill:2022-09-05 23:02:08,633.633 INFO:teuthology.orchestra.console:Power off smithi119 2022-09-05T23:02:08.656 INFO:teuthology.kill:2022-09-05 23:02:08,655.655 INFO:teuthology.orchestra.console:Power off smithi165 2022-09-05T23:02:21.138 INFO:teuthology.kill:2022-09-05 23:02:21,137.137 INFO:teuthology.orchestra.console:Power off for smithi119 completed 2022-09-05T23:02:21.154 INFO:teuthology.kill:2022-09-05 23:02:21,154.154 INFO:teuthology.orchestra.console:Power off for smithi165 completed 2022-09-05T23:02:21.318 INFO:teuthology.kill:2022-09-05 23:02:21,316.316 INFO:teuthology.lock.ops:unlocked: smithi165.front.sepia.ceph.com 2022-09-05T23:02:21.322 INFO:teuthology.kill:2022-09-05 23:02:21,321.321 INFO:teuthology.lock.ops:unlocked: smithi119.front.sepia.ceph.com 2022-09-05T23:04:21.747 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-09-05T23:04:21.866 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi119 but it was locked by another job: /home/teuthworker/archive/dis-2022-09-05_22:46:00-rbd-wip-dis-testing-distro-default-smithi/7012723 2022-09-05T23:04:21.867 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi165 but it was locked by another job: /home/teuthworker/archive/dis-2022-09-05_22:46:00-rbd-wip-dis-testing-distro-default-smithi/7012723