2022-02-17T06:40:39.991 INFO:root:teuthology version: 1.1.0-c6442aa9
2022-02-17T06:40:40.148 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi029.front.sepia.ceph.com' reimaging
2022-02-17T06:40:40.149 INFO:teuthology.lock.ops:Updating [ubuntu@smithi029.front.sepia.ceph.com]: reset os type and version on server
2022-02-17T06:40:40.150 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2022-02-17T06:40:40.171 INFO:teuthology.lock.ops:Node 'ubuntu@smithi029.front.sepia.ceph.com' reimaging is complete
2022-02-17T06:40:40.171 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi164.front.sepia.ceph.com' reimaging
2022-02-17T06:40:40.171 INFO:teuthology.lock.ops:Updating [ubuntu@smithi164.front.sepia.ceph.com]: reset os type and version on server
2022-02-17T06:40:40.172 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server
2022-02-17T06:40:40.192 INFO:teuthology.lock.ops:Node 'ubuntu@smithi164.front.sepia.ceph.com' reimaging is complete
2022-02-17T06:40:40.192 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi184.front.sepia.ceph.com' reimaging
2022-02-17T06:40:40.193 INFO:teuthology.lock.ops:Updating [ubuntu@smithi184.front.sepia.ceph.com]: reset os type and version on server
2022-02-17T06:40:40.193 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server
2022-02-17T06:40:40.214 INFO:teuthology.lock.ops:Node 'ubuntu@smithi184.front.sepia.ceph.com' reimaging is complete
2022-02-17T06:40:40.381 INFO:teuthology.provision.fog.smithi029:Scheduling deploy of ubuntu 20.04
2022-02-17T06:40:40.428 INFO:teuthology.provision.fog.smithi164:Scheduling deploy of ubuntu 20.04
2022-02-17T06:40:40.444 INFO:teuthology.provision.fog.smithi184:Scheduling deploy of ubuntu 20.04
2022-02-17T06:40:40.754 INFO:teuthology.orchestra.console:Power off smithi029
2022-02-17T06:40:40.820 INFO:teuthology.orchestra.console:Power off smithi164
2022-02-17T06:40:40.835 INFO:teuthology.orchestra.console:Power off smithi184
2022-02-17T06:40:40.898 INFO:teuthology.orchestra.console:Power off for smithi029 completed
2022-02-17T06:40:40.964 INFO:teuthology.orchestra.console:Power off for smithi164 completed
2022-02-17T06:40:40.976 INFO:teuthology.orchestra.console:Power off for smithi184 completed
2022-02-17T06:40:41.000 INFO:teuthology.orchestra.console:Power on smithi029
2022-02-17T06:40:41.066 INFO:teuthology.orchestra.console:Power on smithi164
2022-02-17T06:40:41.077 INFO:teuthology.orchestra.console:Power on smithi184
2022-02-17T06:40:45.318 INFO:teuthology.orchestra.console:Power on for smithi164 completed
2022-02-17T06:40:45.420 INFO:teuthology.provision.fog.smithi164:Waiting for deploy to finish
2022-02-17T06:40:49.371 INFO:teuthology.orchestra.console:Power on for smithi029 completed
2022-02-17T06:40:49.436 INFO:teuthology.orchestra.console:Power on for smithi184 completed
2022-02-17T06:40:49.473 INFO:teuthology.provision.fog.smithi029:Waiting for deploy to finish
2022-02-17T06:40:49.538 INFO:teuthology.provision.fog.smithi184:Waiting for deploy to finish
2022-02-17T06:45:46.944 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-17T06:45:47.419 INFO:teuthology.provision.fog.smithi184:Node is ready
2022-02-17T06:45:47.426 INFO:teuthology.orchestra.run.smithi184.stdout:smithi184.front.sepia.ceph.com
2022-02-17T06:45:47.433 INFO:teuthology.orchestra.run.smithi184.stdout:172.21.15.184 smithi184.front.sepia.ceph.com smithi184
2022-02-17T06:45:47.611 INFO:teuthology.provision.fog.smithi184:Deploy complete!
2022-02-17T06:45:56.335 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-17T06:45:56.809 INFO:teuthology.provision.fog.smithi164:Node is ready
2022-02-17T06:45:56.816 INFO:teuthology.orchestra.run.smithi164.stdout:smithi164.front.sepia.ceph.com
2022-02-17T06:45:56.824 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.15.164 smithi164.front.sepia.ceph.com smithi164
2022-02-17T06:45:56.950 INFO:teuthology.provision.fog.smithi164:Deploy complete!
2022-02-17T06:46:27.597 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-17T06:46:28.247 INFO:teuthology.provision.fog.smithi029:Node is ready
2022-02-17T06:46:28.255 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029.front.sepia.ceph.com
2022-02-17T06:46:28.264 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29 smithi029.front.sepia.ceph.com smithi029
2022-02-17T06:46:28.523 INFO:teuthology.provision.fog.smithi029:Deploy complete!
2022-02-17T06:46:28.801 INFO:teuthology.lock.ops:Checking smithi029.front.sepia.ceph.com
2022-02-17T06:46:28.801 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-17T06:46:28.820 INFO:teuthology.lock.ops:Checking smithi164.front.sepia.ceph.com
2022-02-17T06:46:28.821 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-17T06:46:28.838 INFO:teuthology.lock.ops:Checking smithi184.front.sepia.ceph.com
2022-02-17T06:46:28.838 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-17T06:46:28.857 INFO:teuthology.lock.ops:Updating [smithi029.front.sepia.ceph.com]: set os type and version on server
2022-02-17T06:46:28.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com'
2022-02-17T06:46:29.047 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64
2022-02-17T06:46:29.054 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="Ubuntu"
2022-02-17T06:46:29.054 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-17T06:46:29.055 INFO:teuthology.orchestra.run.smithi029.stdout:ID=ubuntu
2022-02-17T06:46:29.055 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE=debian
2022-02-17T06:46:29.055 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-17T06:46:29.055 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="20.04"
2022-02-17T06:46:29.055 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-17T06:46:29.056 INFO:teuthology.orchestra.run.smithi029.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-17T06:46:29.056 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-17T06:46:29.056 INFO:teuthology.orchestra.run.smithi029.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-17T06:46:29.056 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_CODENAME=focal
2022-02-17T06:46:29.057 INFO:teuthology.orchestra.run.smithi029.stdout:UBUNTU_CODENAME=focal
2022-02-17T06:46:29.058 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server
2022-02-17T06:46:29.077 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: set os type and version on server
2022-02-17T06:46:29.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com'
2022-02-17T06:46:29.377 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64
2022-02-17T06:46:29.384 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Ubuntu"
2022-02-17T06:46:29.385 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-17T06:46:29.385 INFO:teuthology.orchestra.run.smithi164.stdout:ID=ubuntu
2022-02-17T06:46:29.385 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE=debian
2022-02-17T06:46:29.385 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-17T06:46:29.386 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="20.04"
2022-02-17T06:46:29.386 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-17T06:46:29.386 INFO:teuthology.orchestra.run.smithi164.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-17T06:46:29.387 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-17T06:46:29.387 INFO:teuthology.orchestra.run.smithi164.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-17T06:46:29.387 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_CODENAME=focal
2022-02-17T06:46:29.387 INFO:teuthology.orchestra.run.smithi164.stdout:UBUNTU_CODENAME=focal
2022-02-17T06:46:29.388 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server
2022-02-17T06:46:29.436 INFO:teuthology.lock.ops:Updating [smithi184.front.sepia.ceph.com]: set os type and version on server
2022-02-17T06:46:29.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com'
2022-02-17T06:46:29.627 INFO:teuthology.orchestra.run.smithi184.stdout:x86_64
2022-02-17T06:46:29.633 INFO:teuthology.orchestra.run.smithi184.stdout:NAME="Ubuntu"
2022-02-17T06:46:29.634 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-17T06:46:29.634 INFO:teuthology.orchestra.run.smithi184.stdout:ID=ubuntu
2022-02-17T06:46:29.634 INFO:teuthology.orchestra.run.smithi184.stdout:ID_LIKE=debian
2022-02-17T06:46:29.634 INFO:teuthology.orchestra.run.smithi184.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-17T06:46:29.635 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_ID="20.04"
2022-02-17T06:46:29.635 INFO:teuthology.orchestra.run.smithi184.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-17T06:46:29.635 INFO:teuthology.orchestra.run.smithi184.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-17T06:46:29.635 INFO:teuthology.orchestra.run.smithi184.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-17T06:46:29.636 INFO:teuthology.orchestra.run.smithi184.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-17T06:46:29.636 INFO:teuthology.orchestra.run.smithi184.stdout:VERSION_CODENAME=focal
2022-02-17T06:46:29.636 INFO:teuthology.orchestra.run.smithi184.stdout:UBUNTU_CODENAME=focal
2022-02-17T06:46:29.637 INFO:teuthology.lock.ops:Updating smithi184.front.sepia.ceph.com on lock server
2022-02-17T06:46:29.732 INFO:teuthology.dispatcher.supervisor:Running job 6690440
2022-02-17T06:46:29.732 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c6442aa9e509aace1201390ea4c2d6a80e3d0800/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi/6690440 --name benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi/6690440/orig.config.yaml
2022-02-17T06:46:29.736 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi/6690440
2022-02-17T06:46:29.736 INFO:teuthology.dispatcher.supervisor:Job PID: 26780
2022-02-17T06:46:29.737 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-02-17T13:16:59.199 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-02-17T13:16:59.385 INFO:teuthology.kill:Killing Pids: {26780}
2022-02-17T13:17:00.620 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a']
2022-02-17T13:17:00.621 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x']
2022-02-17T13:17:00.621 INFO:teuthology.task.internal:roles: ubuntu@smithi184.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c']
2022-02-17T13:17:00.621 INFO:teuthology.misc:Compressing logs...
2022-02-17T13:17:00.622 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com'
2022-02-17T13:17:00.806 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com'
2022-02-17T13:17:00.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi184.front.sepia.ceph.com'
2022-02-17T13:17:01.254 INFO:teuthology.misc:Compressing logs...
2022-02-17T13:17:01.651 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-02-17T13:17:01.674 INFO:teuthology.orchestra.run.smithi184.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-02-17T13:17:02.010 INFO:teuthology.misc:Compressing logs...
2022-02-17T13:17:59.958 INFO:teuthology.kill:No teuthology processes running
2022-02-17T13:18:01.147 INFO:teuthology.kill:Nuking machines: ['smithi029', 'smithi164', 'smithi184']
2022-02-17T13:18:02.133 INFO:teuthology.kill:2022-02-17 13:18:02,133.133 INFO:teuthology.nuke:targets:
2022-02-17T13:18:02.134 INFO:teuthology.kill:  smithi029.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDG3UpGNAfdgnYQRPK9u8prKH9FzP00zY7zqSmPJKZ9g5ePKkF1FXtFMyEfKbiy62fxjg+ohACc07yZWRx2uFGb78wbqXRAjxkfhi6SQzCHtMspLSXMi+tAiqqZCKzdsSPamhh8cB4sSpMQDB2A8+cB+/RL0f7RzNdx0WqJmFeEUeWEnYWn2NkTD3sBjfxlt3WxrHnVwgb/LSwi7ASuSp+1SXdJGIlIOQifXEBU1/BaEmSgtPUs1WwbydBopMCXtGUqBTj53Lm6gJ+Cnl1CiZ1Mu3Q/inY5n0oWJIeSwhZZTqz8pfd088iR5keGvQdEPdD73UcsG7ag3keVttvwhZuPhgf2+H/XtN0Zv/cTKQHUtKoIH+csjiVG6UtwCNPNa1S9LTRVeh/q3VMX+tuR99CvwmBsUM1eTDPKjAPH6H0oXpk0gSolW2rkQXZAfaJyhCxM2c3Z9tW4NFXl6F244qqaoVUytw+BXTg/E70exMhvFuzyl4Dm96M5oesHwbQRT5M=
2022-02-17T13:18:02.135 INFO:teuthology.kill:  smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDES2lSP3W8pSHjg75/DqrgSlfr4/w5aJDHcglHbPoNMhvGz3VPkuKCX8v47FtX5jdnebWWsbQg92E51FQuSDqQxrLRNgxx6MY1lpfCXAXwGw3/kjzlpiRXvTGhZ1a3l3S+HgOLsPx2osiLOqrImmJ01zQ15XI+gv5m8MFy8TJocKJE08odx6FMLfWjnHGEkBZDY/nS6ajMkTCR/xU2tEj89hu0M348iShClesWSLg2C/LvsCzzr0FCzETSIVqL+ZJOyhsjkuEwygl95g2htfnAuWmk972HK/k7S3C1q85kdTF0ROlalyNRbpNR14RexAdXqmfLemDxJMUL3qH3RodLf7Xgjt4ZzBw/g02h6cY/8I6MrNsGjZEbDJThvcMP3v2LVu9Fa0w++gworPn8sYHx/6fsmKNdtszC8DBIpXYX0Dt0GYCDz77JyQbfmgi5kTBGNyv7L8r9xSOsD223Xcvs0OaycIEVGFWAlYNwNKisH4dmJPUgaH5mlyoz8EKiRnE=
2022-02-17T13:18:02.135 INFO:teuthology.kill:  smithi184.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDM9tODuyXKuj81Rm65EFfoEIeATsgRp1xkYSvCwDhgRGNC1GxD6+c88quC8dbXfvfm0PnC2za3OH3IEhDmTmVArP8GoGyDO+HfSl516x30g5e4AZ9/QwWJoDq2/qUSz0QYsDkIHbRZ9dS/Z1lSvV7vZZVYJFZdiTFNaaSZ7aloKnRsY6FcYAtFKPO9nfL5naaQ62l6qYKsyd6NbQ3KVyVXL2UFG8uVfA4HWK9mZoK1Gmgn2yECY/ti5gttXKUsX7zBGobu+4zGwPh3QKOtqfNWhl7B8mT81V3XSwcbYte983rsR69jG8EE+FKl7hTIWOqKo2ErDPJQmQUDQJ3ZyYmGs24Ku28xet5f/pLRmRU42Q3QuSSK5vyrYrkqAGb5SXOn76YjOn0d4KLFcVc0hy1UD3CIMhWOyrrc65aKZxH4Jd8z2R9T0uYhJPDiH4I/zdfKGuKIdh7bRpy8uXy6ap/KS/1/O50b8iDMXM3aFND5TvCvIe6ShPoDyfPy+TI4o+s=
2022-02-17T13:18:02.163 INFO:teuthology.kill:2022-02-17 13:18:02,162.162 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-17T13:18:02.167 INFO:teuthology.kill:2022-02-17 13:18:02,166.166 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-17T13:18:02.168 INFO:teuthology.kill:2022-02-17 13:18:02,168.168 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-17T13:18:02.245 INFO:teuthology.kill:2022-02-17 13:18:02,245.245 INFO:teuthology.orchestra.console:Power off smithi029
2022-02-17T13:18:02.260 INFO:teuthology.kill:2022-02-17 13:18:02,260.260 INFO:teuthology.orchestra.console:Power off smithi184
2022-02-17T13:18:02.271 INFO:teuthology.kill:2022-02-17 13:18:02,271.271 INFO:teuthology.orchestra.console:Power off smithi164
2022-02-17T13:18:14.740 INFO:teuthology.kill:2022-02-17 13:18:14,740.740 INFO:teuthology.orchestra.console:Power off for smithi029 completed
2022-02-17T13:18:14.744 INFO:teuthology.kill:2022-02-17 13:18:14,744.744 INFO:teuthology.orchestra.console:Power off for smithi184 completed
2022-02-17T13:18:14.750 INFO:teuthology.kill:2022-02-17 13:18:14,750.750 INFO:teuthology.orchestra.console:Power off for smithi164 completed
2022-02-17T13:18:14.897 INFO:teuthology.kill:2022-02-17 13:18:14,896.896 INFO:teuthology.lock.ops:unlocked: smithi184.front.sepia.ceph.com
2022-02-17T13:18:14.905 INFO:teuthology.kill:2022-02-17 13:18:14,905.905 INFO:teuthology.lock.ops:unlocked: smithi164.front.sepia.ceph.com
2022-02-17T13:18:14.906 INFO:teuthology.kill:2022-02-17 13:18:14,906.906 INFO:teuthology.lock.ops:unlocked: smithi029.front.sepia.ceph.com
2022-02-17T13:20:15.208 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-02-17T13:20:15.333 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi029 but it was locked by another job: /home/teuthworker/archive/rfriedma-2022-02-17_12:48:58-rados:thrash-wip-rf-scrubBEv1_2-distro-basic-smithi/6691002
2022-02-17T13:20:15.334 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi164 but it was locked by another job: /home/teuthworker/archive/rfriedma-2022-02-17_12:48:58-rados:thrash-wip-rf-scrubBEv1_2-distro-basic-smithi/6691002
2022-02-17T13:20:15.334 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi184 but it was locked by another job: /home/teuthworker/archive/rfriedma-2022-02-17_12:48:58-rados:thrash-wip-rf-scrubBEv1_2-distro-basic-smithi/6691003