2022-02-17T06:38:08.862 INFO:root:teuthology version: 1.1.0-c6442aa9
2022-02-17T06:38:09.029 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi037.front.sepia.ceph.com' reimaging
2022-02-17T06:38:09.031 INFO:teuthology.lock.ops:Updating [ubuntu@smithi037.front.sepia.ceph.com]: reset os type and version on server
2022-02-17T06:38:09.031 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2022-02-17T06:38:09.075 INFO:teuthology.lock.ops:Node 'ubuntu@smithi037.front.sepia.ceph.com' reimaging is complete
2022-02-17T06:38:09.075 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi050.front.sepia.ceph.com' reimaging
2022-02-17T06:38:09.076 INFO:teuthology.lock.ops:Updating [ubuntu@smithi050.front.sepia.ceph.com]: reset os type and version on server
2022-02-17T06:38:09.076 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server
2022-02-17T06:38:09.094 INFO:teuthology.lock.ops:Node 'ubuntu@smithi050.front.sepia.ceph.com' reimaging is complete
2022-02-17T06:38:09.094 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi141.front.sepia.ceph.com' reimaging
2022-02-17T06:38:09.095 INFO:teuthology.lock.ops:Updating [ubuntu@smithi141.front.sepia.ceph.com]: reset os type and version on server
2022-02-17T06:38:09.095 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server
2022-02-17T06:38:09.133 INFO:teuthology.lock.ops:Node 'ubuntu@smithi141.front.sepia.ceph.com' reimaging is complete
2022-02-17T06:38:09.287 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of ubuntu 20.04
2022-02-17T06:38:09.307 INFO:teuthology.provision.fog.smithi050:Scheduling deploy of ubuntu 20.04
2022-02-17T06:38:09.352 INFO:teuthology.provision.fog.smithi141:Scheduling deploy of ubuntu 20.04
2022-02-17T06:38:09.585 INFO:teuthology.orchestra.console:Power off smithi037
2022-02-17T06:38:09.627 INFO:teuthology.orchestra.console:Power off smithi050
2022-02-17T06:38:09.670 INFO:teuthology.orchestra.console:Power off smithi141
2022-02-17T06:38:09.744 INFO:teuthology.orchestra.console:Power off for smithi037 completed
2022-02-17T06:38:09.782 INFO:teuthology.orchestra.console:Power off for smithi050 completed
2022-02-17T06:38:09.815 INFO:teuthology.orchestra.console:Power off for smithi141 completed
2022-02-17T06:38:09.847 INFO:teuthology.orchestra.console:Power on smithi037
2022-02-17T06:38:09.884 INFO:teuthology.orchestra.console:Power on smithi050
2022-02-17T06:38:09.916 INFO:teuthology.orchestra.console:Power on smithi141
2022-02-17T06:38:14.141 INFO:teuthology.orchestra.console:Power on for smithi050 completed
2022-02-17T06:38:14.172 INFO:teuthology.orchestra.console:Power on for smithi141 completed
2022-02-17T06:38:14.243 INFO:teuthology.provision.fog.smithi050:Waiting for deploy to finish
2022-02-17T06:38:14.274 INFO:teuthology.provision.fog.smithi141:Waiting for deploy to finish
2022-02-17T06:38:18.230 INFO:teuthology.orchestra.console:Power on for smithi037 completed
2022-02-17T06:38:18.331 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish
2022-02-17T06:42:51.814 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-17T06:42:52.380 INFO:teuthology.provision.fog.smithi050:Node is ready
2022-02-17T06:42:52.387 INFO:teuthology.orchestra.run.smithi050.stdout:smithi050.front.sepia.ceph.com
2022-02-17T06:42:52.395 INFO:teuthology.orchestra.run.smithi050.stdout:172.21.15.50 smithi050.front.sepia.ceph.com smithi050
2022-02-17T06:42:52.641 INFO:teuthology.provision.fog.smithi050:Deploy complete!
2022-02-17T06:42:55.804 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-17T06:42:56.359 INFO:teuthology.provision.fog.smithi037:Node is ready
2022-02-17T06:42:56.366 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com
2022-02-17T06:42:56.372 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037
2022-02-17T06:42:56.635 INFO:teuthology.provision.fog.smithi037:Deploy complete!
2022-02-17T06:43:14.519 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-17T06:43:14.950 INFO:teuthology.provision.fog.smithi141:Node is ready
2022-02-17T06:43:14.957 INFO:teuthology.orchestra.run.smithi141.stdout:smithi141.front.sepia.ceph.com
2022-02-17T06:43:14.964 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141 smithi141.front.sepia.ceph.com smithi141
2022-02-17T06:43:15.145 INFO:teuthology.provision.fog.smithi141:Deploy complete!
2022-02-17T06:43:15.390 INFO:teuthology.lock.ops:Checking smithi037.front.sepia.ceph.com
2022-02-17T06:43:15.392 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-17T06:43:15.421 INFO:teuthology.lock.ops:Checking smithi050.front.sepia.ceph.com
2022-02-17T06:43:15.422 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-17T06:43:15.442 INFO:teuthology.lock.ops:Checking smithi141.front.sepia.ceph.com
2022-02-17T06:43:15.442 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-17T06:43:15.461 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: set os type and version on server
2022-02-17T06:43:15.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2022-02-17T06:43:15.726 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64
2022-02-17T06:43:15.732 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Ubuntu"
2022-02-17T06:43:15.732 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-17T06:43:15.733 INFO:teuthology.orchestra.run.smithi037.stdout:ID=ubuntu
2022-02-17T06:43:15.733 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE=debian
2022-02-17T06:43:15.733 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-17T06:43:15.734 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="20.04"
2022-02-17T06:43:15.734 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-17T06:43:15.734 INFO:teuthology.orchestra.run.smithi037.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-17T06:43:15.734 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-17T06:43:15.735 INFO:teuthology.orchestra.run.smithi037.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-17T06:43:15.735 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_CODENAME=focal
2022-02-17T06:43:15.735 INFO:teuthology.orchestra.run.smithi037.stdout:UBUNTU_CODENAME=focal
2022-02-17T06:43:15.737 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server
2022-02-17T06:43:15.757 INFO:teuthology.lock.ops:Updating [smithi050.front.sepia.ceph.com]: set os type and version on server
2022-02-17T06:43:15.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com'
2022-02-17T06:43:16.027 INFO:teuthology.orchestra.run.smithi050.stdout:x86_64
2022-02-17T06:43:16.075 INFO:teuthology.orchestra.run.smithi050.stdout:NAME="Ubuntu"
2022-02-17T06:43:16.075 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-17T06:43:16.076 INFO:teuthology.orchestra.run.smithi050.stdout:ID=ubuntu
2022-02-17T06:43:16.076 INFO:teuthology.orchestra.run.smithi050.stdout:ID_LIKE=debian
2022-02-17T06:43:16.076 INFO:teuthology.orchestra.run.smithi050.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-17T06:43:16.076 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_ID="20.04"
2022-02-17T06:43:16.077 INFO:teuthology.orchestra.run.smithi050.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-17T06:43:16.077 INFO:teuthology.orchestra.run.smithi050.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-17T06:43:16.077 INFO:teuthology.orchestra.run.smithi050.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-17T06:43:16.077 INFO:teuthology.orchestra.run.smithi050.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-17T06:43:16.078 INFO:teuthology.orchestra.run.smithi050.stdout:VERSION_CODENAME=focal
2022-02-17T06:43:16.078 INFO:teuthology.orchestra.run.smithi050.stdout:UBUNTU_CODENAME=focal
2022-02-17T06:43:16.078 INFO:teuthology.lock.ops:Updating smithi050.front.sepia.ceph.com on lock server
2022-02-17T06:43:16.098 INFO:teuthology.lock.ops:Updating [smithi141.front.sepia.ceph.com]: set os type and version on server
2022-02-17T06:43:16.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2022-02-17T06:43:16.269 INFO:teuthology.orchestra.run.smithi141.stdout:x86_64
2022-02-17T06:43:16.316 INFO:teuthology.orchestra.run.smithi141.stdout:NAME="Ubuntu"
2022-02-17T06:43:16.316 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-17T06:43:16.317 INFO:teuthology.orchestra.run.smithi141.stdout:ID=ubuntu
2022-02-17T06:43:16.317 INFO:teuthology.orchestra.run.smithi141.stdout:ID_LIKE=debian
2022-02-17T06:43:16.317 INFO:teuthology.orchestra.run.smithi141.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-17T06:43:16.317 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_ID="20.04"
2022-02-17T06:43:16.318 INFO:teuthology.orchestra.run.smithi141.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-17T06:43:16.318 INFO:teuthology.orchestra.run.smithi141.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-17T06:43:16.318 INFO:teuthology.orchestra.run.smithi141.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-17T06:43:16.318 INFO:teuthology.orchestra.run.smithi141.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-17T06:43:16.319 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_CODENAME=focal
2022-02-17T06:43:16.319 INFO:teuthology.orchestra.run.smithi141.stdout:UBUNTU_CODENAME=focal
2022-02-17T06:43:16.319 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server
2022-02-17T06:43:16.412 INFO:teuthology.dispatcher.supervisor:Running job 6690438
2022-02-17T06:43:16.412 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c6442aa9e509aace1201390ea4c2d6a80e3d0800/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi/6690438 --name benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi/6690438/orig.config.yaml
2022-02-17T06:43:16.416 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi/6690438
2022-02-17T06:43:16.417 INFO:teuthology.dispatcher.supervisor:Job PID: 20782
2022-02-17T06:43:16.417 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-02-17T13:13:46.118 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-02-17T13:13:46.342 INFO:teuthology.kill:Killing Pids: {20782}
2022-02-17T13:13:47.525 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a']
2022-02-17T13:13:47.526 INFO:teuthology.task.internal:roles: ubuntu@smithi050.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x']
2022-02-17T13:13:47.526 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c']
2022-02-17T13:13:47.527 INFO:teuthology.misc:Compressing logs...
2022-02-17T13:13:47.528 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com'
2022-02-17T13:13:47.715 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi050.front.sepia.ceph.com'
2022-02-17T13:13:47.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com'
2022-02-17T13:13:48.162 INFO:teuthology.misc:Compressing logs...
2022-02-17T13:13:52.471 INFO:teuthology.misc:Compressing logs...
2022-02-17T13:14:23.641 INFO:teuthology.kill:No teuthology processes running
2022-02-17T13:14:24.845 INFO:teuthology.kill:Nuking machines: ['smithi037', 'smithi050', 'smithi141']
2022-02-17T13:14:25.839 INFO:teuthology.kill:2022-02-17 13:14:25,839.839 INFO:teuthology.nuke:targets:
2022-02-17T13:14:25.841 INFO:teuthology.kill:  smithi037.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDeKQH8Uw4mRwtwq4gxeDUimrC8+To91nNhJyT2vWmiksug0QV6hYu9GIqJJu1YD0voLUXUPANzDCbXm6J6suXRqyEhr05mD4P1DhXg8bOnwteCxlesmh5YXyafOkaXq8/fDZfBoeSqLK8kgjZZ90BgaBwnkcLw954M4oBzCj5RSsOP5bGWUJAQeQwGzDdl4W4SmI/QkjXiFoJixX3PMSACM+pK9nZvI7Y2FdDPNjQpO7oVhlYjEgu7pPZLZB1ry5ix/VbOQ2qu+h9+IVzgbYnSMcWKC81HCuqs2gZ+cG+Sv5NyHPzbL/x6Add4Lnllem9fz8PIZ8IxwXiOyq/fECoDWyhNw6zIeCzgMP8k0DbskQi8Mjm+XCvuE8y21vEnMaFZXzBt8EXdA58sZ6Nqw1a+R/zSE5cQ+98Kl/3jyfU/VvBkR/w6AZR2dwxtZrM+WIzWXycFEvg34R5acIIgukYQY1HiGqb0xcvK4UDhLZVnoNwp69Ls1gr4+VmldV7DNlM=
2022-02-17T13:14:25.841 INFO:teuthology.kill:  smithi050.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6/wMUnwW3hqLu9KqgweyZEDHLb+0UpL4ExZ84EvXAwW5LRlANJbsblWE7NDMHzxdnLinwiW1SFu7MX/8g9HISdRy/XY3Lqov+pOGw7UMjwaHpzyqaRDdfkMMWdmRsTR7p73D8m7GB7teZXuYfZpZpM613NhOZ8qegDHCWsJvicnVxN/M4Aiw00oNQBX9czUdrKm6KNYB9Px0s9LegeKPo9EtipoDQGA1khGaQ2esIl+P3atbgHkZABCmbV67nwruJvuEEN62tdzCMtKWXYIHG9zfN4BM2TVbJFPpi35L7y15fY9hxxIrQGkDx8bZEFmp7PK1wfTazmbHJvDX64BPrxZmn7pUTsEopVQJs6mHX2o7z4F0loeWo7aMFmNIsJRMw+J+OUgWPlJ9ShSTg4gPdgw+tqELd2YWrvB2vG1uiqRqya8Qy2nGyfQNyD2gI4VZsuc0gaF3buEcy8YCNrzkYIffvTuzzVn2vx2Rm/rd/GiVN06ZapV6VvCtpmSx5v3U=
2022-02-17T13:14:25.842 INFO:teuthology.kill:  smithi141.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7hOTWs365cQNzWJCip6kpNJxzly5ZneB8USaHE1n5r8k0yfkjIkwvA3GQafNQzuBPobxQMsmzCWaXFBV6i2Trdg3Fp/XjKM+LpNTkO3U1Z+aJu5ZeH9Se3d0Hfrs8RVOFJJi7F5XOdKAFeYa2WFjsYk9CZvAscd+Kf+zWpAv+E3mBsAZ40coEOcw3QgES0pBb/a5aEztzhdSlhP1/am7Cw3XlwKTbkxKNcH6l6eJxh65H8XN9yA9sylszBL1aD0iXNQtaEQQGLTvGZcMvEPj6wizLvVV3zw8mMT4eFopVgMC3OkjXhyOOcHqUi5sShSMJNxKWM5QSGWwyp73UzDh2XZ4yoYBidE7+NFc1iBFDivBr7NFgO1OCND1HIJHbR/VGHYcnzd2HuWfQvPykx/I8WA9j0n7RgKcd5tFX4PIs4agUOn8zlXNV+V6EiuET8UkKZaCvppou3QK5U/8clvPmwoJS1XfS7etgvrdkglVye4uoXkmX0m/+CWGnpSX6N3E=
2022-02-17T13:14:25.868 INFO:teuthology.kill:2022-02-17 13:14:25,867.867 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-17T13:14:25.872 INFO:teuthology.kill:2022-02-17 13:14:25,872.872 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-17T13:14:25.873 INFO:teuthology.kill:2022-02-17 13:14:25,873.873 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-17T13:14:25.958 INFO:teuthology.kill:2022-02-17 13:14:25,958.958 INFO:teuthology.orchestra.console:Power off smithi050
2022-02-17T13:14:25.973 INFO:teuthology.kill:2022-02-17 13:14:25,973.973 INFO:teuthology.orchestra.console:Power off smithi037
2022-02-17T13:14:25.986 INFO:teuthology.kill:2022-02-17 13:14:25,985.985 INFO:teuthology.orchestra.console:Power off smithi141
2022-02-17T13:14:34.352 INFO:teuthology.kill:2022-02-17 13:14:34,351.351 INFO:teuthology.orchestra.console:Power off for smithi141 completed
2022-02-17T13:14:34.498 INFO:teuthology.kill:2022-02-17 13:14:34,498.498 INFO:teuthology.lock.ops:unlocked: smithi141.front.sepia.ceph.com
2022-02-17T13:14:38.454 INFO:teuthology.kill:2022-02-17 13:14:38,454.454 INFO:teuthology.orchestra.console:Power off for smithi050 completed
2022-02-17T13:14:38.460 INFO:teuthology.kill:2022-02-17 13:14:38,460.460 INFO:teuthology.orchestra.console:Power off for smithi037 completed
2022-02-17T13:14:38.601 INFO:teuthology.kill:2022-02-17 13:14:38,601.601 INFO:teuthology.lock.ops:unlocked: smithi050.front.sepia.ceph.com
2022-02-17T13:14:38.608 INFO:teuthology.kill:2022-02-17 13:14:38,608.608 INFO:teuthology.lock.ops:unlocked: smithi037.front.sepia.ceph.com
2022-02-17T13:16:38.897 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-02-17T13:16:39.028 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi037 but it was locked by another job: /home/teuthworker/archive/rfriedma-2022-02-17_12:48:58-rados:thrash-wip-rf-scrubBEv1_2-distro-basic-smithi/6691000
2022-02-17T13:16:39.028 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi050 but it was locked by another job: /home/teuthworker/archive/rfriedma-2022-02-17_12:48:58-rados:thrash-wip-rf-scrubBEv1_2-distro-basic-smithi/6691000
2022-02-17T13:16:39.029 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi141 but it was locked by another job: /home/teuthworker/archive/rfriedma-2022-02-17_12:48:58-rados:thrash-wip-rf-scrubBEv1_2-distro-basic-smithi/6690999