2021-11-04T06:26:54.577 INFO:root:teuthology version: 1.1.0-c56135d1 2021-11-04T06:26:54.740 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi077.front.sepia.ceph.com' reimaging 2021-11-04T06:26:54.742 INFO:teuthology.lock.ops:Updating [ubuntu@smithi077.front.sepia.ceph.com]: reset os type and version on server 2021-11-04T06:26:54.742 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2021-11-04T06:26:54.761 INFO:teuthology.lock.ops:Node 'ubuntu@smithi077.front.sepia.ceph.com' reimaging is complete 2021-11-04T06:26:54.762 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi121.front.sepia.ceph.com' reimaging 2021-11-04T06:26:54.762 INFO:teuthology.lock.ops:Updating [ubuntu@smithi121.front.sepia.ceph.com]: reset os type and version on server 2021-11-04T06:26:54.762 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2021-11-04T06:26:54.785 INFO:teuthology.lock.ops:Node 'ubuntu@smithi121.front.sepia.ceph.com' reimaging is complete 2021-11-04T06:26:54.786 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi192.front.sepia.ceph.com' reimaging 2021-11-04T06:26:54.787 INFO:teuthology.lock.ops:Updating [ubuntu@smithi192.front.sepia.ceph.com]: reset os type and version on server 2021-11-04T06:26:54.787 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2021-11-04T06:26:54.808 INFO:teuthology.lock.ops:Node 'ubuntu@smithi192.front.sepia.ceph.com' reimaging is complete 2021-11-04T06:26:55.006 INFO:teuthology.provision.fog.smithi077:Scheduling deploy of ubuntu 20.04 2021-11-04T06:26:55.010 INFO:teuthology.provision.fog.smithi121:Scheduling deploy of ubuntu 20.04 2021-11-04T06:26:55.030 INFO:teuthology.provision.fog.smithi192:Scheduling deploy of ubuntu 20.04 2021-11-04T06:26:55.319 INFO:teuthology.orchestra.console:Power off smithi077 2021-11-04T06:26:55.342 INFO:teuthology.orchestra.console:Power off smithi192 2021-11-04T06:26:55.357 INFO:teuthology.orchestra.console:Power off smithi121 2021-11-04T06:27:03.717 INFO:teuthology.orchestra.console:Power off for smithi077 completed 2021-11-04T06:27:03.725 INFO:teuthology.orchestra.console:Power off for smithi192 completed 2021-11-04T06:27:03.819 INFO:teuthology.orchestra.console:Power on smithi077 2021-11-04T06:27:03.828 INFO:teuthology.orchestra.console:Power on smithi192 2021-11-04T06:27:07.844 INFO:teuthology.orchestra.console:Power off for smithi121 completed 2021-11-04T06:27:07.946 INFO:teuthology.orchestra.console:Power on smithi121 2021-11-04T06:27:12.190 INFO:teuthology.orchestra.console:Power on for smithi077 completed 2021-11-04T06:27:12.196 INFO:teuthology.orchestra.console:Power on for smithi192 completed 2021-11-04T06:27:12.201 INFO:teuthology.orchestra.console:Power on for smithi121 completed 2021-11-04T06:27:12.292 INFO:teuthology.provision.fog.smithi077:Waiting for deploy to finish 2021-11-04T06:27:12.297 INFO:teuthology.provision.fog.smithi192:Waiting for deploy to finish 2021-11-04T06:27:12.303 INFO:teuthology.provision.fog.smithi121:Waiting for deploy to finish 2021-11-04T06:32:47.247 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-04T06:32:47.335 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-04T06:32:47.811 INFO:teuthology.provision.fog.smithi192:Node is ready 2021-11-04T06:32:47.818 INFO:teuthology.orchestra.run.smithi192.stdout:smithi192.front.sepia.ceph.com 2021-11-04T06:32:47.867 INFO:teuthology.orchestra.run.smithi192.stdout:172.21.15.192 smithi192.front.sepia.ceph.com smithi192 2021-11-04T06:32:47.900 INFO:teuthology.provision.fog.smithi121:Node is ready 2021-11-04T06:32:47.906 INFO:teuthology.orchestra.run.smithi121.stdout:smithi121.front.sepia.ceph.com 2021-11-04T06:32:47.955 INFO:teuthology.orchestra.run.smithi121.stdout:172.21.15.121 smithi121.front.sepia.ceph.com smithi121 2021-11-04T06:32:48.109 INFO:teuthology.provision.fog.smithi192:Deploy complete! 2021-11-04T06:32:48.141 INFO:teuthology.provision.fog.smithi121:Deploy complete! 2021-11-04T06:33:00.538 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-11-04T06:33:01.084 INFO:teuthology.provision.fog.smithi077:Node is ready 2021-11-04T06:33:01.091 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077.front.sepia.ceph.com 2021-11-04T06:33:01.099 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.15.77 smithi077.front.sepia.ceph.com smithi077 2021-11-04T06:33:01.253 INFO:teuthology.provision.fog.smithi077:Deploy complete! 2021-11-04T06:33:01.486 INFO:teuthology.lock.ops:Checking smithi077.front.sepia.ceph.com 2021-11-04T06:33:01.487 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-04T06:33:01.508 INFO:teuthology.lock.ops:Checking smithi121.front.sepia.ceph.com 2021-11-04T06:33:01.509 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-04T06:33:01.525 INFO:teuthology.lock.ops:Checking smithi192.front.sepia.ceph.com 2021-11-04T06:33:01.525 INFO:teuthology.lock.ops:New key found. Updating... 2021-11-04T06:33:01.544 INFO:teuthology.lock.ops:Updating [smithi077.front.sepia.ceph.com]: set os type and version on server 2021-11-04T06:33:01.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2021-11-04T06:33:01.723 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2021-11-04T06:33:01.730 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Ubuntu" 2021-11-04T06:33:01.730 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2021-11-04T06:33:01.730 INFO:teuthology.orchestra.run.smithi077.stdout:ID=ubuntu 2021-11-04T06:33:01.730 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE=debian 2021-11-04T06:33:01.731 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2021-11-04T06:33:01.731 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="20.04" 2021-11-04T06:33:01.731 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.ubuntu.com/" 2021-11-04T06:33:01.731 INFO:teuthology.orchestra.run.smithi077.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-11-04T06:33:01.732 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-11-04T06:33:01.732 INFO:teuthology.orchestra.run.smithi077.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-11-04T06:33:01.732 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_CODENAME=focal 2021-11-04T06:33:01.732 INFO:teuthology.orchestra.run.smithi077.stdout:UBUNTU_CODENAME=focal 2021-11-04T06:33:01.734 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2021-11-04T06:33:01.755 INFO:teuthology.lock.ops:Updating [smithi121.front.sepia.ceph.com]: set os type and version on server 2021-11-04T06:33:01.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2021-11-04T06:33:02.022 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2021-11-04T06:33:02.029 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Ubuntu" 2021-11-04T06:33:02.030 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2021-11-04T06:33:02.030 INFO:teuthology.orchestra.run.smithi121.stdout:ID=ubuntu 2021-11-04T06:33:02.030 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE=debian 2021-11-04T06:33:02.031 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2021-11-04T06:33:02.031 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="20.04" 2021-11-04T06:33:02.034 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.ubuntu.com/" 2021-11-04T06:33:02.034 INFO:teuthology.orchestra.run.smithi121.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-11-04T06:33:02.035 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-11-04T06:33:02.035 INFO:teuthology.orchestra.run.smithi121.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-11-04T06:33:02.035 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_CODENAME=focal 2021-11-04T06:33:02.035 INFO:teuthology.orchestra.run.smithi121.stdout:UBUNTU_CODENAME=focal 2021-11-04T06:33:02.036 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2021-11-04T06:33:02.057 INFO:teuthology.lock.ops:Updating [smithi192.front.sepia.ceph.com]: set os type and version on server 2021-11-04T06:33:02.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2021-11-04T06:33:02.314 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2021-11-04T06:33:02.321 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="Ubuntu" 2021-11-04T06:33:02.321 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2021-11-04T06:33:02.321 INFO:teuthology.orchestra.run.smithi192.stdout:ID=ubuntu 2021-11-04T06:33:02.322 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE=debian 2021-11-04T06:33:02.322 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2021-11-04T06:33:02.322 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="20.04" 2021-11-04T06:33:02.323 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://www.ubuntu.com/" 2021-11-04T06:33:02.323 INFO:teuthology.orchestra.run.smithi192.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-11-04T06:33:02.323 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-11-04T06:33:02.323 INFO:teuthology.orchestra.run.smithi192.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-11-04T06:33:02.324 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_CODENAME=focal 2021-11-04T06:33:02.324 INFO:teuthology.orchestra.run.smithi192.stdout:UBUNTU_CODENAME=focal 2021-11-04T06:33:02.324 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2021-11-04T06:33:02.425 INFO:teuthology.dispatcher.supervisor:Running job 6482866 2021-11-04T06:33:02.425 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2021-11-03_18:53:10-rados-wip-yuri5-testing-2021-11-02-0745-distro-basic-smithi/6482866 --name yuriw-2021-11-03_18:53:10-rados-wip-yuri5-testing-2021-11-02-0745-distro-basic-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/yuriw-2021-11-03_18:53:10-rados-wip-yuri5-testing-2021-11-02-0745-distro-basic-smithi/6482866/orig.config.yaml 2021-11-04T06:33:02.429 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2021-11-03_18:53:10-rados-wip-yuri5-testing-2021-11-02-0745-distro-basic-smithi/6482866 2021-11-04T06:33:02.429 INFO:teuthology.dispatcher.supervisor:Job PID: 29002 2021-11-04T06:33:02.430 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-11-04T18:33:57.075 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-11-04T18:33:57.306 INFO:teuthology.kill:Killing Pids: {29002} 2021-11-04T18:33:58.465 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'] 2021-11-04T18:33:58.465 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'] 2021-11-04T18:33:58.466 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c'] 2021-11-04T18:33:58.466 INFO:teuthology.misc:Compressing logs... 2021-11-04T18:33:58.467 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2021-11-04T18:33:58.638 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2021-11-04T18:33:58.873 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2021-11-04T18:33:59.129 INFO:teuthology.misc:Compressing logs... 2021-11-04T18:34:00.293 INFO:teuthology.misc:Compressing logs... 2021-11-04T18:37:42.318 INFO:teuthology.kill:No teuthology processes running 2021-11-04T18:37:43.440 INFO:teuthology.kill:Nuking machines: ['smithi077', 'smithi121', 'smithi192'] 2021-11-04T18:37:44.384 INFO:teuthology.kill:2021-11-04 18:37:44,384.384 INFO:teuthology.nuke:targets: 2021-11-04T18:37:44.385 INFO:teuthology.kill: smithi077.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmec0P39utmwZ9C2NpjY/6PIuA/LLr89yB/Ej17VLM0CfNobwCWJ4v55+ZFxINa/k4ltGKeqFXZMDNuXEOfFvvS/hfWraHQ+lvY731TzlpZ8hot8CDUX4RZvMmG84dy8yoNHXjAu9tBD4SPvFuA58pPpPvhSS7yNvmIe5tS1rpal7aAqRyyQ2pPAQEe9sreP1pr9g3Dp1PidKvernuWbX2jbBNahkUczhsXalMCEwJH+iWw3SKvqfp38Ac+iPqT722GtfrIcg89OpA6kPJUd/ba9km2JPpCPxnT1F27iwzMQqVmbffbUXWzkujQm1kBHm8dl0md9GScmyyaWJ4leSLouZXUeIkYd1Xjp8ReEph4K/VWoLiDHl0xnOh/Dv+1t39G0Txm3T+qj6Kt9NMkiWHkwfY3GJHYCkzrZY0eMg9Hl4C2DcMqxryRrrfFMZiJlGk6J/UC40fnTKtaq/sHKKi4aB6fYE0Daj0lQClFF9lu9wRLAZ4ifYTHb0HJgUoWOk= 2021-11-04T18:37:44.385 INFO:teuthology.kill: smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDxZW964j+ZsMCs4zr50viZORtKhJz0SMTYpLWtGpsp/GmOtdCv6yMI4waoKtf/JZnx1zIvwbRm9Lmfww1kqvpROnwy+hTU5e+1r1vMWMaxD5efqgJDP+EX5bJrWVytOaBTB+8Ajm3jrH+WfslzB5DuXViCtCOEDSAkI544O6zAPMDM4ya5ezsVnN205KRpnXNfmW4znSLmAXTes+ffYiWT2rkQPUl/TH9jWW0m002oG/6W361gxXwQfeMtYKXu4qQpCFOwG41KGZq44iK/8jfN+/jSZzJebDgoE5f1jfVlioiCMOLuPNelvOeW5UYGbQ+gwHIgFbtOBMMoUR/p5L3HeEyNYqH3nOfIVfhMiQ6AvXYRlMhiQm0rDvJGp8QBdkP+myPOk9fudl3f8C+dLRp7VCv3fxXqy/w8vo1s/sMiTNil6N3piX6YSLbndjUSZtsOgHZlknyr418t3sB0IiuQtYCkKaewsYqrAP6XfqXzl5/LWBr7/9VfVyaLPgxFVQ8= 2021-11-04T18:37:44.386 INFO:teuthology.kill: smithi192.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrLfIrXiSnlWEXeC5NfjaNchseSHkjbZ5tr+3seBcVv64Y6UldAD663c/WGIkFA+hP1xr2GtBLWlxuIZZUamrMSCwhMwx0oW5oCyKxmdsg4IeK2BtqMk1lb7xRgETL63Rd5xLBeVoRAkX51Js0aOxrA11UOvlnqq/c4XsaLkQe9Wq7McJpgZD0NBn1PMeFEJ2NP2/oA7PMQDiNk0cSIP0heu81A0VnpchGTV51Fm8PtSDsdFuVOzQso8GIVsOPCVK3NQAtrvaEzMfmfINDomo1JXI/ILlcSM+nRUCL6OEqMm9E0mIjPEL4iAZLVn8/M4xU2SUpKMG54d2qyQT5goz59S2V/qp7Rl7FmjqoZ/Izu/f0CjKzKNWTSFrZ1VbVVnPbSDsckZOKCCPvO4EIDVjzxwazPU+SIlJQOHj2XEHeVmjfInw+eS2eOyFyn+c3ejnkngNu/SJv5Ck1WT0u9oFPQ9RHYPXNyeJHbLGlwikq/EmxrDMc3yEwixdT2JK0iGU= 2021-11-04T18:37:44.415 INFO:teuthology.kill:2021-11-04 18:37:44,414.414 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-04T18:37:44.418 INFO:teuthology.kill:2021-11-04 18:37:44,418.418 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-04T18:37:44.420 INFO:teuthology.kill:2021-11-04 18:37:44,419.419 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-11-04T18:37:44.504 INFO:teuthology.kill:2021-11-04 18:37:44,504.504 INFO:teuthology.orchestra.console:Power off smithi077 2021-11-04T18:37:44.520 INFO:teuthology.kill:2021-11-04 18:37:44,519.519 INFO:teuthology.orchestra.console:Power off smithi192 2021-11-04T18:37:44.530 INFO:teuthology.kill:2021-11-04 18:37:44,529.529 INFO:teuthology.orchestra.console:Power off smithi121 2021-11-04T18:37:52.890 INFO:teuthology.kill:2021-11-04 18:37:52,890.890 INFO:teuthology.orchestra.console:Power off for smithi192 completed 2021-11-04T18:37:53.039 INFO:teuthology.kill:2021-11-04 18:37:53,038.038 INFO:teuthology.lock.ops:unlocked: smithi192.front.sepia.ceph.com 2021-11-04T18:37:56.999 INFO:teuthology.kill:2021-11-04 18:37:56,999.999 INFO:teuthology.orchestra.console:Power off for smithi077 completed 2021-11-04T18:37:57.006 INFO:teuthology.kill:2021-11-04 18:37:57,006.006 INFO:teuthology.orchestra.console:Power off for smithi121 completed 2021-11-04T18:37:57.152 INFO:teuthology.kill:2021-11-04 18:37:57,152.152 INFO:teuthology.lock.ops:unlocked: smithi077.front.sepia.ceph.com 2021-11-04T18:37:57.162 INFO:teuthology.kill:2021-11-04 18:37:57,161.161 INFO:teuthology.lock.ops:unlocked: smithi121.front.sepia.ceph.com 2021-11-04T18:39:57.507 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-11-04T18:39:57.631 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi077 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-02_19:49:55-rados-wip-yuri8-testing-2021-11-02-1009-pacific-distro-basic-smithi/6479349 2021-11-04T18:39:57.631 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi121 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-02_19:49:55-rados-wip-yuri8-testing-2021-11-02-1009-pacific-distro-basic-smithi/6479348 2021-11-04T18:39:57.632 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi192 but it was locked by another job: /home/teuthworker/archive/yuriw-2021-11-02_19:49:55-rados-wip-yuri8-testing-2021-11-02-1009-pacific-distro-basic-smithi/6479347