2022-08-26T18:19:56.094 INFO:root:teuthology version: 1.1.1.dev412+gb1d387f1 2022-08-26T18:19:56.322 INFO:teuthology.lock.ops:Start node 'smithi074.front.sepia.ceph.com' reimaging 2022-08-26T18:19:56.324 INFO:teuthology.lock.ops:Updating [smithi074.front.sepia.ceph.com]: reset os type and version on server 2022-08-26T18:19:56.324 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server 2022-08-26T18:19:56.350 INFO:teuthology.lock.ops:Node 'smithi074.front.sepia.ceph.com' reimaging is complete 2022-08-26T18:19:56.350 INFO:teuthology.lock.ops:Start node 'smithi099.front.sepia.ceph.com' reimaging 2022-08-26T18:19:56.351 INFO:teuthology.lock.ops:Updating [smithi099.front.sepia.ceph.com]: reset os type and version on server 2022-08-26T18:19:56.352 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2022-08-26T18:19:56.377 INFO:teuthology.lock.ops:Node 'smithi099.front.sepia.ceph.com' reimaging is complete 2022-08-26T18:19:56.377 INFO:teuthology.lock.ops:Start node 'smithi186.front.sepia.ceph.com' reimaging 2022-08-26T18:19:56.378 INFO:teuthology.lock.ops:Updating [smithi186.front.sepia.ceph.com]: reset os type and version on server 2022-08-26T18:19:56.378 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2022-08-26T18:19:56.402 INFO:teuthology.lock.ops:Node 'smithi186.front.sepia.ceph.com' reimaging is complete 2022-08-26T18:19:56.591 INFO:teuthology.provision.fog.smithi074:Scheduling deploy of ubuntu 20.04 2022-08-26T18:19:56.600 INFO:teuthology.provision.fog.smithi099:Scheduling deploy of ubuntu 20.04 2022-08-26T18:19:56.641 INFO:teuthology.provision.fog.smithi186:Scheduling deploy of ubuntu 20.04 2022-08-26T18:19:56.937 INFO:teuthology.orchestra.console:Power off smithi074 2022-08-26T18:19:56.976 INFO:teuthology.orchestra.console:Power off smithi099 2022-08-26T18:19:57.023 INFO:teuthology.orchestra.console:Power off smithi186 2022-08-26T18:19:57.094 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2022-08-26T18:19:57.133 INFO:teuthology.orchestra.console:Power off for smithi099 completed 2022-08-26T18:19:57.167 INFO:teuthology.orchestra.console:Power off for smithi186 completed 2022-08-26T18:19:57.195 INFO:teuthology.orchestra.console:Power on smithi074 2022-08-26T18:19:57.239 INFO:teuthology.orchestra.console:Power on smithi099 2022-08-26T18:19:57.274 INFO:teuthology.orchestra.console:Power on smithi186 2022-08-26T18:20:01.474 INFO:teuthology.orchestra.console:Power on for smithi074 completed 2022-08-26T18:20:01.578 INFO:teuthology.provision.fog.smithi074:Waiting for deploy to finish 2022-08-26T18:20:05.620 INFO:teuthology.orchestra.console:Power on for smithi099 completed 2022-08-26T18:20:05.648 INFO:teuthology.orchestra.console:Power on for smithi186 completed 2022-08-26T18:20:05.722 INFO:teuthology.provision.fog.smithi099:Waiting for deploy to finish 2022-08-26T18:20:05.749 INFO:teuthology.provision.fog.smithi186:Waiting for deploy to finish 2022-08-26T18:25:08.101 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-08-26T18:25:08.571 INFO:teuthology.provision.fog.smithi074:Node is ready 2022-08-26T18:25:08.621 INFO:teuthology.orchestra.run.smithi074.stdout:smithi074.front.sepia.ceph.com 2022-08-26T18:25:08.628 INFO:teuthology.orchestra.run.smithi074.stdout:172.21.15.74 smithi074.front.sepia.ceph.com smithi074 2022-08-26T18:25:08.871 INFO:teuthology.provision.fog.smithi074:Deploy complete! 2022-08-26T18:25:12.207 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-08-26T18:25:12.736 INFO:teuthology.provision.fog.smithi099:Node is ready 2022-08-26T18:25:12.745 INFO:teuthology.orchestra.run.smithi099.stdout:smithi099.front.sepia.ceph.com 2022-08-26T18:25:12.794 INFO:teuthology.orchestra.run.smithi099.stdout:172.21.15.99 smithi099.front.sepia.ceph.com smithi099 2022-08-26T18:25:12.986 INFO:teuthology.provision.fog.smithi099:Deploy complete! 2022-08-26T18:25:16.651 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-08-26T18:25:17.187 INFO:teuthology.provision.fog.smithi186:Node is ready 2022-08-26T18:25:17.262 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186.front.sepia.ceph.com 2022-08-26T18:25:17.294 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.186 smithi186.front.sepia.ceph.com smithi186 2022-08-26T18:25:17.675 INFO:teuthology.provision.fog.smithi186:Deploy complete! 2022-08-26T18:25:18.147 INFO:teuthology.lock.ops:Checking smithi074.front.sepia.ceph.com 2022-08-26T18:25:18.148 INFO:teuthology.lock.ops:New key found. Updating... 2022-08-26T18:25:18.205 INFO:teuthology.lock.ops:Checking smithi099.front.sepia.ceph.com 2022-08-26T18:25:18.206 INFO:teuthology.lock.ops:New key found. Updating... 2022-08-26T18:25:18.235 INFO:teuthology.lock.ops:Checking smithi186.front.sepia.ceph.com 2022-08-26T18:25:18.236 INFO:teuthology.lock.ops:New key found. Updating... 2022-08-26T18:25:18.307 INFO:teuthology.lock.ops:Updating [smithi074.front.sepia.ceph.com]: set os type and version on server 2022-08-26T18:25:18.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi074.front.sepia.ceph.com' 2022-08-26T18:25:18.637 INFO:teuthology.orchestra.run.smithi074.stdout:x86_64 2022-08-26T18:25:18.694 INFO:teuthology.orchestra.run.smithi074.stdout:NAME="Ubuntu" 2022-08-26T18:25:18.695 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-08-26T18:25:18.696 INFO:teuthology.orchestra.run.smithi074.stdout:ID=ubuntu 2022-08-26T18:25:18.697 INFO:teuthology.orchestra.run.smithi074.stdout:ID_LIKE=debian 2022-08-26T18:25:18.697 INFO:teuthology.orchestra.run.smithi074.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-08-26T18:25:18.698 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION_ID="20.04" 2022-08-26T18:25:18.699 INFO:teuthology.orchestra.run.smithi074.stdout:HOME_URL="https://www.ubuntu.com/" 2022-08-26T18:25:18.699 INFO:teuthology.orchestra.run.smithi074.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-08-26T18:25:18.700 INFO:teuthology.orchestra.run.smithi074.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-08-26T18:25:18.700 INFO:teuthology.orchestra.run.smithi074.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-08-26T18:25:18.701 INFO:teuthology.orchestra.run.smithi074.stdout:VERSION_CODENAME=focal 2022-08-26T18:25:18.702 INFO:teuthology.orchestra.run.smithi074.stdout:UBUNTU_CODENAME=focal 2022-08-26T18:25:18.705 INFO:teuthology.lock.ops:Updating smithi074.front.sepia.ceph.com on lock server 2022-08-26T18:25:18.733 INFO:teuthology.lock.ops:Updating [smithi099.front.sepia.ceph.com]: set os type and version on server 2022-08-26T18:25:18.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2022-08-26T18:25:18.988 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2022-08-26T18:25:19.002 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="Ubuntu" 2022-08-26T18:25:19.003 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-08-26T18:25:19.003 INFO:teuthology.orchestra.run.smithi099.stdout:ID=ubuntu 2022-08-26T18:25:19.003 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE=debian 2022-08-26T18:25:19.004 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-08-26T18:25:19.004 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="20.04" 2022-08-26T18:25:19.004 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://www.ubuntu.com/" 2022-08-26T18:25:19.004 INFO:teuthology.orchestra.run.smithi099.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-08-26T18:25:19.005 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-08-26T18:25:19.005 INFO:teuthology.orchestra.run.smithi099.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-08-26T18:25:19.005 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_CODENAME=focal 2022-08-26T18:25:19.006 INFO:teuthology.orchestra.run.smithi099.stdout:UBUNTU_CODENAME=focal 2022-08-26T18:25:19.006 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2022-08-26T18:25:19.028 INFO:teuthology.lock.ops:Updating [smithi186.front.sepia.ceph.com]: set os type and version on server 2022-08-26T18:25:19.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-08-26T18:25:19.259 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2022-08-26T18:25:19.268 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="Ubuntu" 2022-08-26T18:25:19.269 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-08-26T18:25:19.269 INFO:teuthology.orchestra.run.smithi186.stdout:ID=ubuntu 2022-08-26T18:25:19.269 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE=debian 2022-08-26T18:25:19.270 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-08-26T18:25:19.271 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="20.04" 2022-08-26T18:25:19.271 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.ubuntu.com/" 2022-08-26T18:25:19.272 INFO:teuthology.orchestra.run.smithi186.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-08-26T18:25:19.272 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-08-26T18:25:19.273 INFO:teuthology.orchestra.run.smithi186.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-08-26T18:25:19.273 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_CODENAME=focal 2022-08-26T18:25:19.273 INFO:teuthology.orchestra.run.smithi186.stdout:UBUNTU_CODENAME=focal 2022-08-26T18:25:19.274 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2022-08-26T18:25:19.413 INFO:teuthology.dispatcher.supervisor:Running job 6993139 2022-08-26T18:25:19.414 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b1d387f12b117399cb87c86aaa341398fa0c0919/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-08-25_18:36:20-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi/6993139 --name benhanokh-2022-08-25_18:36:20-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/fastclose objectstore/bluestore-low-osd-mem-target rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=lrc-k=4-m=2-l=3} -- /home/teuthworker/archive/benhanokh-2022-08-25_18:36:20-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi/6993139/orig.config.yaml 2022-08-26T18:25:19.422 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-08-25_18:36:20-rados-WIP_GBH_snap_mapper_one_way-distro-default-smithi/6993139 2022-08-26T18:25:19.423 INFO:teuthology.dispatcher.supervisor:Job PID: 32221 2022-08-26T18:25:19.423 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-08-27T06:26:20.295 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-08-27T06:26:20.490 INFO:teuthology.kill:Killing Pids: {32221} 2022-08-27T06:26:21.997 INFO:teuthology.task.internal:roles: ubuntu@smithi074.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'] 2022-08-27T06:26:21.998 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'] 2022-08-27T06:26:21.999 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c'] 2022-08-27T06:26:21.999 INFO:teuthology.misc:Compressing logs... 2022-08-27T06:26:22.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi074.front.sepia.ceph.com' 2022-08-27T06:26:22.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2022-08-27T06:26:22.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-08-27T06:26:23.160 INFO:teuthology.misc:Compressing logs... 2022-08-27T06:30:24.133 INFO:teuthology.misc:Compressing logs... 2022-08-27T06:30:28.203 INFO:teuthology.kill:No teuthology processes running 2022-08-27T06:30:29.671 INFO:teuthology.kill:Nuking machines: ['smithi074', 'smithi099', 'smithi186'] 2022-08-27T06:30:30.974 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b1d387f12b117399cb87c86aaa341398fa0c0919/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-08-27T06:30:30.975 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-08-27T06:30:30.975 INFO:teuthology.kill:2022-08-27 06:30:30,973.973 INFO:teuthology.nuke:targets: 2022-08-27T06:30:30.975 INFO:teuthology.kill: smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXJwe7hQdWMU/+eYaiPsoDsKUcs1d0BcNGgXH+xAnwn2c/wE7eK0BnevSowzNCDFufZJizxU0St+aLhiX25E4Zu79cnUBgrS11hBe1KF+6iCaJYTVUtQnCPZhIIDqiZ1Y6PEvvkq5AykVC6h8h6zXBftdoYg7QcjWr9DrmcFkfUolTJcoBuMI31X371gKuPDHwBUh+9c1o6gwMTi51Lg0u/5m52pbCfs5NJqx3pPO98wAR2QGtpJTDGQgfVE/M3mvxZQSZ31ewBwe39/74dFzD58dg7U5O1NvVYGGQOeWJrgAXmiDPrrvvETqoDUX6wMYoYfTMnCj+kw3BnhgSX6DDaIXnxI4mA9xv2gU3STbzar9amsGUQx88K/uyi9PXonSGR1AckhvMNCFXhD/zIHPV7cRbSPxNwK/vOUPRhNGVcBtUQfBRekR/suOPTCm1QkWcZkFVUuXw/EyNNTFLvtxDThsU2J7D1hrOxxVfDYdUT9zHMmrUfWdUtJkUyEkWZ3E= 2022-08-27T06:30:30.976 INFO:teuthology.kill: smithi099.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDP8jjvEQ5X0oWhFwmuNqk5NsB/FwFNnfaPHPz1G9NbTOdAAUxCXQCXIosIDMk+KZqVBxASRqDzu3SCWbcFUlQ2IQqjVE7t4HXLvqcKJmGvtGRXeGr8ZxPgg6tzYFnudswMkJJqoQsF7esw7n1o65HvhIHL7Dmg3JrJwkn9efM+sNkmMdFiNmqFXXHrjfDCPp1IN32EXU3T/Ueh0wajtL8aJ1KHGhU5yAQ8SO6g7Y0g1vcwWt1oMmodPQLOcXDFcKjp6Jv7uu8HptxtY3Ezxap7XNDUAIe9dXtHD4bJa3HumvRojQ1dArKtvJV5F7Oe3Od4bMkwHykUi/eJHGVhLnNinGLLlV6onEGX7Dha4BldQjgkApj3fYF1PMKw5XMt9ck2oBAeucFezvGDcGcGFSFdbWOoedQmGwb0xs+iaRNjX8a7wIb6awN4zbJRDvBVlmT81VVIOBkm9TTTP3iIhT3jIvqdxnsBvQGeVCOvmm1XT12CQPbjdXOG/8HVNERe78= 2022-08-27T06:30:30.976 INFO:teuthology.kill: smithi186.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD91RV87tme5Cr+Ovx4QsNSUPo1DtgnL9sEBRh1BZxP2kladrrZwq5wdPu4hXKUYdWVyOkBXz2XYG/uPU8o4omaZx97LLIi/qMXCRNTXTlx2d/6LVCyViaW3dL1OR3p2cMejw6vikWZfapYcGjSwQJKWSK9FSHiTMNeipJrWzQ2IyTSXfc0Fg5MpGwoCGlVM/ChPWJsF+54uoYesPFXEwq7tpQhz1Q0vTH7uajZs1OAmAZW0AptePBRQS/cLAQfvTBFENQZpvUpNFEGgaFHBBCZBlSFu+l4nPFvhU1OgQE3hAgWFI1qvOCRkAhCUCKHjMP+YhSrs9WfCkz0mf7QSy9CUJSNBFxYeive4QmSf9Ew4dyb0M76E2+/mBGP3yyRHduqzULyJwMqiFDOjfRTgAPv2GCcPJj6DwE1nHvBzLuN2CdCD6sE0i/2I9nAHIq1Enlei/93e439q6PvWKxLEux+oPvpE113MK/cIkFScpyEajcZxW+WpDesTPSkVvupQhs= 2022-08-27T06:30:30.976 INFO:teuthology.kill:2022-08-27 06:30:30,975.975 INFO:teuthology.nuke:Checking targets against current locks 2022-08-27T06:30:31.049 INFO:teuthology.kill:2022-08-27 06:30:31,049.049 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-08-27T06:30:31.093 INFO:teuthology.kill:2022-08-27 06:30:31,092.092 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-08-27T06:30:31.126 INFO:teuthology.kill:2022-08-27 06:30:31,126.126 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-08-27T06:30:31.169 INFO:teuthology.kill:2022-08-27 06:30:31,169.169 INFO:teuthology.orchestra.console:Power off smithi074 2022-08-27T06:30:31.217 INFO:teuthology.kill:2022-08-27 06:30:31,217.217 INFO:teuthology.orchestra.console:Power off smithi099 2022-08-27T06:30:31.267 INFO:teuthology.kill:2022-08-27 06:30:31,266.266 INFO:teuthology.orchestra.console:Power off smithi186 2022-08-27T06:30:39.565 INFO:teuthology.kill:2022-08-27 06:30:39,564.564 INFO:teuthology.orchestra.console:Power off for smithi074 completed 2022-08-27T06:30:39.716 INFO:teuthology.kill:2022-08-27 06:30:39,715.715 INFO:teuthology.lock.ops:unlocked: smithi074.front.sepia.ceph.com 2022-08-27T06:30:43.722 INFO:teuthology.kill:2022-08-27 06:30:43,722.722 INFO:teuthology.orchestra.console:Power off for smithi099 completed 2022-08-27T06:30:43.751 INFO:teuthology.kill:2022-08-27 06:30:43,750.750 INFO:teuthology.orchestra.console:Power off for smithi186 completed 2022-08-27T06:30:43.897 INFO:teuthology.kill:2022-08-27 06:30:43,897.897 INFO:teuthology.lock.ops:unlocked: smithi099.front.sepia.ceph.com 2022-08-27T06:30:43.925 INFO:teuthology.kill:2022-08-27 06:30:43,925.925 INFO:teuthology.lock.ops:unlocked: smithi186.front.sepia.ceph.com 2022-08-27T06:32:44.324 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-08-27T06:32:44.464 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi074 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-08-23_03:15:03-fs-quincy-distro-default-smithi/6986783 2022-08-27T06:32:44.464 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi186 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-08-23_03:15:03-fs-quincy-distro-default-smithi/6986786