2022-01-09T02:10:12.295 INFO:root:teuthology version: 1.1.0-95a7d479 2022-01-09T02:10:12.463 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi089.front.sepia.ceph.com' reimaging 2022-01-09T02:10:12.465 INFO:teuthology.lock.ops:Updating [ubuntu@smithi089.front.sepia.ceph.com]: reset os type and version on server 2022-01-09T02:10:12.465 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2022-01-09T02:10:12.488 INFO:teuthology.lock.ops:Node 'ubuntu@smithi089.front.sepia.ceph.com' reimaging is complete 2022-01-09T02:10:12.489 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi100.front.sepia.ceph.com' reimaging 2022-01-09T02:10:12.489 INFO:teuthology.lock.ops:Updating [ubuntu@smithi100.front.sepia.ceph.com]: reset os type and version on server 2022-01-09T02:10:12.489 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2022-01-09T02:10:12.515 INFO:teuthology.lock.ops:Node 'ubuntu@smithi100.front.sepia.ceph.com' reimaging is complete 2022-01-09T02:10:12.516 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi161.front.sepia.ceph.com' reimaging 2022-01-09T02:10:12.517 INFO:teuthology.lock.ops:Updating [ubuntu@smithi161.front.sepia.ceph.com]: reset os type and version on server 2022-01-09T02:10:12.517 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2022-01-09T02:10:12.537 INFO:teuthology.lock.ops:Node 'ubuntu@smithi161.front.sepia.ceph.com' reimaging is complete 2022-01-09T02:10:12.711 INFO:teuthology.provision.fog.smithi089:Scheduling deploy of ubuntu 20.04 2022-01-09T02:10:12.747 INFO:teuthology.provision.fog.smithi161:Scheduling deploy of ubuntu 20.04 2022-01-09T02:10:12.752 INFO:teuthology.provision.fog.smithi100:Scheduling deploy of ubuntu 20.04 2022-01-09T02:10:13.018 INFO:teuthology.orchestra.console:Power off smithi089 2022-01-09T02:10:13.066 INFO:teuthology.orchestra.console:Power off smithi100 2022-01-09T02:10:13.081 INFO:teuthology.orchestra.console:Power off smithi161 2022-01-09T02:10:21.432 INFO:teuthology.orchestra.console:Power off for smithi100 completed 2022-01-09T02:10:21.534 INFO:teuthology.orchestra.console:Power on smithi100 2022-01-09T02:10:25.527 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2022-01-09T02:10:25.556 INFO:teuthology.orchestra.console:Power off for smithi161 completed 2022-01-09T02:10:25.629 INFO:teuthology.orchestra.console:Power on smithi089 2022-01-09T02:10:25.660 INFO:teuthology.orchestra.console:Power on smithi161 2022-01-09T02:10:25.791 INFO:teuthology.orchestra.console:Power on for smithi100 completed 2022-01-09T02:10:25.892 INFO:teuthology.provision.fog.smithi100:Waiting for deploy to finish 2022-01-09T02:10:29.888 INFO:teuthology.orchestra.console:Power on for smithi089 completed 2022-01-09T02:10:29.910 INFO:teuthology.orchestra.console:Power on for smithi161 completed 2022-01-09T02:10:29.990 INFO:teuthology.provision.fog.smithi089:Waiting for deploy to finish 2022-01-09T02:10:30.011 INFO:teuthology.provision.fog.smithi161:Waiting for deploy to finish 2022-01-09T02:16:01.110 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-09T02:16:01.626 INFO:teuthology.provision.fog.smithi100:Node is ready 2022-01-09T02:16:01.636 INFO:teuthology.orchestra.run.smithi100.stdout:smithi100.front.sepia.ceph.com 2022-01-09T02:16:01.689 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.15.100 smithi100.front.sepia.ceph.com smithi100 2022-01-09T02:16:01.840 INFO:teuthology.provision.fog.smithi100:Deploy complete! 2022-01-09T02:16:04.891 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-09T02:16:05.470 INFO:teuthology.provision.fog.smithi089:Node is ready 2022-01-09T02:16:05.517 INFO:teuthology.orchestra.run.smithi089.stdout:smithi089.front.sepia.ceph.com 2022-01-09T02:16:05.525 INFO:teuthology.orchestra.run.smithi089.stdout:172.21.15.89 smithi089.front.sepia.ceph.com smithi089 2022-01-09T02:16:05.736 INFO:teuthology.provision.fog.smithi089:Deploy complete! 2022-01-09T02:16:33.156 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-01-09T02:16:33.715 INFO:teuthology.provision.fog.smithi161:Node is ready 2022-01-09T02:16:33.721 INFO:teuthology.orchestra.run.smithi161.stdout:smithi161.front.sepia.ceph.com 2022-01-09T02:16:33.728 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.15.161 smithi161.front.sepia.ceph.com smithi161 2022-01-09T02:16:33.955 INFO:teuthology.provision.fog.smithi161:Deploy complete! 2022-01-09T02:16:34.179 INFO:teuthology.lock.ops:Checking smithi089.front.sepia.ceph.com 2022-01-09T02:16:34.180 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-09T02:16:34.204 INFO:teuthology.lock.ops:Checking smithi100.front.sepia.ceph.com 2022-01-09T02:16:34.204 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-09T02:16:34.222 INFO:teuthology.lock.ops:Checking smithi161.front.sepia.ceph.com 2022-01-09T02:16:34.222 INFO:teuthology.lock.ops:New key found. Updating... 2022-01-09T02:16:34.243 INFO:teuthology.lock.ops:Updating [smithi089.front.sepia.ceph.com]: set os type and version on server 2022-01-09T02:16:34.243 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2022-01-09T02:16:34.510 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2022-01-09T02:16:34.516 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="Ubuntu" 2022-01-09T02:16:34.516 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-01-09T02:16:34.516 INFO:teuthology.orchestra.run.smithi089.stdout:ID=ubuntu 2022-01-09T02:16:34.517 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE=debian 2022-01-09T02:16:34.517 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-01-09T02:16:34.517 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="20.04" 2022-01-09T02:16:34.517 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://www.ubuntu.com/" 2022-01-09T02:16:34.517 INFO:teuthology.orchestra.run.smithi089.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-01-09T02:16:34.518 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-01-09T02:16:34.518 INFO:teuthology.orchestra.run.smithi089.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-01-09T02:16:34.518 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_CODENAME=focal 2022-01-09T02:16:34.518 INFO:teuthology.orchestra.run.smithi089.stdout:UBUNTU_CODENAME=focal 2022-01-09T02:16:34.519 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2022-01-09T02:16:34.540 INFO:teuthology.lock.ops:Updating [smithi100.front.sepia.ceph.com]: set os type and version on server 2022-01-09T02:16:34.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2022-01-09T02:16:34.794 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2022-01-09T02:16:34.800 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="Ubuntu" 2022-01-09T02:16:34.801 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-01-09T02:16:34.801 INFO:teuthology.orchestra.run.smithi100.stdout:ID=ubuntu 2022-01-09T02:16:34.801 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE=debian 2022-01-09T02:16:34.801 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-01-09T02:16:34.802 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="20.04" 2022-01-09T02:16:34.802 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://www.ubuntu.com/" 2022-01-09T02:16:34.803 INFO:teuthology.orchestra.run.smithi100.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-01-09T02:16:34.803 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-01-09T02:16:34.803 INFO:teuthology.orchestra.run.smithi100.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-01-09T02:16:34.803 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_CODENAME=focal 2022-01-09T02:16:34.804 INFO:teuthology.orchestra.run.smithi100.stdout:UBUNTU_CODENAME=focal 2022-01-09T02:16:34.804 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2022-01-09T02:16:34.822 INFO:teuthology.lock.ops:Updating [smithi161.front.sepia.ceph.com]: set os type and version on server 2022-01-09T02:16:34.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2022-01-09T02:16:35.002 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2022-01-09T02:16:35.009 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="Ubuntu" 2022-01-09T02:16:35.009 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="20.04.3 LTS (Focal Fossa)" 2022-01-09T02:16:35.009 INFO:teuthology.orchestra.run.smithi161.stdout:ID=ubuntu 2022-01-09T02:16:35.010 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE=debian 2022-01-09T02:16:35.010 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS" 2022-01-09T02:16:35.010 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="20.04" 2022-01-09T02:16:35.010 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://www.ubuntu.com/" 2022-01-09T02:16:35.011 INFO:teuthology.orchestra.run.smithi161.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-01-09T02:16:35.011 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-01-09T02:16:35.011 INFO:teuthology.orchestra.run.smithi161.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-01-09T02:16:35.011 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_CODENAME=focal 2022-01-09T02:16:35.011 INFO:teuthology.orchestra.run.smithi161.stdout:UBUNTU_CODENAME=focal 2022-01-09T02:16:35.012 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2022-01-09T02:16:35.121 INFO:teuthology.dispatcher.supervisor:Running job 6602958 2022-01-09T02:16:35.121 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-01-08_15:53:33-rados-wip-yuriw-master-1.7.22-distro-default-smithi/6602958 --name yuriw-2022-01-08_15:53:33-rados-wip-yuriw-master-1.7.22-distro-default-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/yuriw-2022-01-08_15:53:33-rados-wip-yuriw-master-1.7.22-distro-default-smithi/6602958/orig.config.yaml 2022-01-09T02:16:35.124 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-01-08_15:53:33-rados-wip-yuriw-master-1.7.22-distro-default-smithi/6602958 2022-01-09T02:16:35.125 INFO:teuthology.dispatcher.supervisor:Job PID: 742 2022-01-09T02:16:35.125 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-01-09T14:17:28.797 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-01-09T14:17:29.006 INFO:teuthology.kill:Killing Pids: {742} 2022-01-09T14:17:30.994 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a'] 2022-01-09T14:17:30.996 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x'] 2022-01-09T14:17:30.997 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c'] 2022-01-09T14:17:30.997 INFO:teuthology.misc:Compressing logs... 2022-01-09T14:17:30.998 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2022-01-09T14:17:31.280 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2022-01-09T14:17:31.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2022-01-09T14:17:31.761 INFO:teuthology.misc:Compressing logs... 2022-01-09T14:17:32.935 INFO:teuthology.misc:Compressing logs... 2022-01-09T14:19:45.037 INFO:teuthology.kill:No teuthology processes running 2022-01-09T14:19:46.260 INFO:teuthology.kill:Nuking machines: ['smithi089', 'smithi100', 'smithi161'] 2022-01-09T14:19:47.311 INFO:teuthology.kill:2022-01-09 14:19:47,304.304 INFO:teuthology.nuke:targets: 2022-01-09T14:19:47.312 INFO:teuthology.kill: smithi089.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEw5hehGE2z3yci+BlhwNpB5zESzznCzbLdZdbfYOqasA5jLnoPfI4+ADzrgz2fqrV5u1ot+ds53jzmUIVBPmWXYMseEqzkZ/EcxEXTTBRisDsW/T4NdfEbm3x5weWBDLpDm5sK4wrB+mAw2WdOfap5SvfMMFZO0lFaEBsmrSv0TThTIobPLIeDiNF4pq2HRYApxSm+NgckxsKe+/0aWMmdsgnOl9k+vybDzZ9zPhmd1beQ5F9Wk+MwTWJiw/el1OUs82qord79F9PZyOK74I283mdLbeP9d3oGMsitodnG3ktg7jB6ErCyZKOX+A9yfqhS3QVGh24dG7Cj16vXPmEV8dHkDXfQKjMR9k2uHdQOJ5uIIHcbn+eIo+fWqxDNoz56usVPAti3igeTf7kpoAAj8ry3cRRGAIYQimTAiZd0Mc1VEJfb9Uh3ccqDcEJNsxkXmArd/kpu5Ogem40m5f++5jQy3pbL/ylTbmFq0m9Ih3tzlFk7ZEmAGHx4YbTEpk= 2022-01-09T14:19:47.312 INFO:teuthology.kill: smithi100.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbT6aW+JB88+/xi7d1deI2gHkqZ7mxl62KSuIhPqrIPhDIuea/q7PQWsQGp7Po58g1UN+psFkcA+6EzSOoBtOs6dRZfvGOrP6sq9qfkUTnHa2iKr9qcIR9I990i9+8w9kOFn9iWn82KeZQYYlViUxo1+IrCctTf3NlLHzOFT+CRnNqecMhYDaPjdFN5Z4Mws5M0bME2DnYQJEjLUSi0EZxKUlalAyCPqNAe3P0Ch2Kd+l/oFQOc/XtBe0hbvyJ3+bvucU+X6SfJZvy3c1Ree7phTOyf9cG7eO69VLLAQv8JusOTF+Z+s3989+zXvnkrCY5miWDwINYlRqm/4FiBthSWvH1zckjpB8kTN3J16YVTOyx6Wq8gKDJrAvBCCPbznJ9i7kFiOoiyYyjOyGV2TeR16p3VWgPfKnti+rLVKfUp7KHN9Ce9tBlNJWYQRyjqOUQFz+KCLePeeYPVgiRcFybNW9y2R5kxv2Y2qUQ/98gBzFUU8w6aCspQodPGDXksVs= 2022-01-09T14:19:47.312 INFO:teuthology.kill: smithi161.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu44Nv+c/bCucDlWM0FbmMzJFouE7yJW6ca/l0oahV13S/vLKEXWmLZFIz2s6C0Ydw5UWj1Sv2wEfWOohhQ721CgTvQqhvTI3+j4LZjEdfIXJ5+ELTpdQTFfLV92pufOt17irhck12pmvAOxnjA+vjxoIC446D0IXORlusOzZzkHvQ2E9ki/maDdnP/rF/gBKxpTGSpk1niGW5J3HdB/U5Ql3eoDZpDLzT7J1fmKDrTEF7lNFe2OmniizQP+Ohp5wNzXmWmgEBkMW/fA/4Hk92/UMi3qWBjY/lFfh/ZjYbx3qs6imQPPYhUfg7cQiPGeDyCvdbI0u1p/aiDO1Ev92gkBPLe8OGCyDN17Nw4mqABx9D1pUBUEnpnpxrye0PPiVyRqs7/iItb+Fq94/1QoZPPBdYCBPzwZhsQzrP7rVU9Y5RSyTnhY9waQeF5YBUDXxrwh0Xf9hugdEo3fs1ArMx+WOAhc4XZ5EKhB+c94aXk0S5bOrJNCkTZu94nSqrTtU= 2022-01-09T14:19:47.347 INFO:teuthology.kill:2022-01-09 14:19:47,347.347 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-09T14:19:47.362 INFO:teuthology.kill:2022-01-09 14:19:47,361.361 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-09T14:19:47.387 INFO:teuthology.kill:2022-01-09 14:19:47,386.386 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-01-09T14:19:47.442 INFO:teuthology.kill:2022-01-09 14:19:47,442.442 INFO:teuthology.orchestra.console:Power off smithi100 2022-01-09T14:19:47.479 INFO:teuthology.kill:2022-01-09 14:19:47,479.479 INFO:teuthology.orchestra.console:Power off smithi089 2022-01-09T14:19:47.494 INFO:teuthology.kill:2022-01-09 14:19:47,494.494 INFO:teuthology.orchestra.console:Power off smithi161 2022-01-09T14:19:55.854 INFO:teuthology.kill:2022-01-09 14:19:55,853.853 INFO:teuthology.orchestra.console:Power off for smithi161 completed 2022-01-09T14:19:56.010 INFO:teuthology.kill:2022-01-09 14:19:56,010.010 INFO:teuthology.lock.ops:unlocked: smithi161.front.sepia.ceph.com 2022-01-09T14:19:59.947 INFO:teuthology.kill:2022-01-09 14:19:59,946.946 INFO:teuthology.orchestra.console:Power off for smithi100 completed 2022-01-09T14:19:59.954 INFO:teuthology.kill:2022-01-09 14:19:59,954.954 INFO:teuthology.orchestra.console:Power off for smithi089 completed 2022-01-09T14:20:00.100 INFO:teuthology.kill:2022-01-09 14:20:00,099.099 INFO:teuthology.lock.ops:unlocked: smithi100.front.sepia.ceph.com 2022-01-09T14:20:00.114 INFO:teuthology.kill:2022-01-09 14:20:00,114.114 INFO:teuthology.lock.ops:unlocked: smithi089.front.sepia.ceph.com 2022-01-09T14:22:00.481 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-01-09T14:22:00.686 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi089 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-01-09_03:31:03-rados-pacific-distro-default-smithi/6603649 2022-01-09T14:22:00.686 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi100 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-01-09_03:31:03-rados-pacific-distro-default-smithi/6603650 2022-01-09T14:22:00.687 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi161 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-01-09_03:31:03-rados-pacific-distro-default-smithi/6603650