2022-02-17T08:40:52.134 INFO:root:teuthology version: 1.1.0-c6442aa9
2022-02-17T08:40:52.304 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi057.front.sepia.ceph.com' reimaging
2022-02-17T08:40:52.305 INFO:teuthology.lock.ops:Updating [ubuntu@smithi057.front.sepia.ceph.com]: reset os type and version on server
2022-02-17T08:40:52.305 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server
2022-02-17T08:40:52.442 INFO:teuthology.lock.ops:Node 'ubuntu@smithi057.front.sepia.ceph.com' reimaging is complete
2022-02-17T08:40:52.442 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi198.front.sepia.ceph.com' reimaging
2022-02-17T08:40:52.442 INFO:teuthology.lock.ops:Updating [ubuntu@smithi198.front.sepia.ceph.com]: reset os type and version on server
2022-02-17T08:40:52.443 INFO:teuthology.lock.ops:Updating smithi198.front.sepia.ceph.com on lock server
2022-02-17T08:40:52.467 INFO:teuthology.lock.ops:Node 'ubuntu@smithi198.front.sepia.ceph.com' reimaging is complete
2022-02-17T08:40:52.467 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi201.front.sepia.ceph.com' reimaging
2022-02-17T08:40:52.467 INFO:teuthology.lock.ops:Updating [ubuntu@smithi201.front.sepia.ceph.com]: reset os type and version on server
2022-02-17T08:40:52.468 INFO:teuthology.lock.ops:Updating smithi201.front.sepia.ceph.com on lock server
2022-02-17T08:40:52.624 INFO:teuthology.lock.ops:Node 'ubuntu@smithi201.front.sepia.ceph.com' reimaging is complete
2022-02-17T08:40:52.669 INFO:teuthology.provision.fog.smithi057:Scheduling deploy of ubuntu 20.04
2022-02-17T08:40:52.673 INFO:teuthology.provision.fog.smithi198:Scheduling deploy of ubuntu 20.04
2022-02-17T08:40:52.854 INFO:teuthology.provision.fog.smithi201:Scheduling deploy of ubuntu 20.04
2022-02-17T08:40:52.964 INFO:teuthology.orchestra.console:Power off smithi057
2022-02-17T08:40:52.986 INFO:teuthology.orchestra.console:Power off smithi198
2022-02-17T08:40:53.117 INFO:teuthology.orchestra.console:Power off for smithi057 completed
2022-02-17T08:40:53.126 INFO:teuthology.orchestra.console:Power off for smithi198 completed
2022-02-17T08:40:53.172 INFO:teuthology.orchestra.console:Power off smithi201
2022-02-17T08:40:53.218 INFO:teuthology.orchestra.console:Power on smithi057
2022-02-17T08:40:53.227 INFO:teuthology.orchestra.console:Power on smithi198
2022-02-17T08:40:53.313 INFO:teuthology.orchestra.console:Power off for smithi201 completed
2022-02-17T08:40:53.414 INFO:teuthology.orchestra.console:Power on smithi201
2022-02-17T08:40:57.476 INFO:teuthology.orchestra.console:Power on for smithi057 completed
2022-02-17T08:40:57.482 INFO:teuthology.orchestra.console:Power on for smithi198 completed
2022-02-17T08:40:57.577 INFO:teuthology.provision.fog.smithi057:Waiting for deploy to finish
2022-02-17T08:40:57.584 INFO:teuthology.provision.fog.smithi198:Waiting for deploy to finish
2022-02-17T08:40:57.668 INFO:teuthology.orchestra.console:Power on for smithi201 completed
2022-02-17T08:40:57.769 INFO:teuthology.provision.fog.smithi201:Waiting for deploy to finish
2022-02-17T08:43:43.746 ERROR:teuthology.orchestra.connection:Error authenticating with smithi057.front.sepia.ceph.com: Authentication failed.
2022-02-17T08:45:54.713 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-17T08:45:54.761 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-17T08:45:55.220 INFO:teuthology.provision.fog.smithi198:Node is ready
2022-02-17T08:45:55.227 INFO:teuthology.orchestra.run.smithi198.stdout:smithi198.front.sepia.ceph.com
2022-02-17T08:45:55.234 INFO:teuthology.provision.fog.smithi201:Node is ready
2022-02-17T08:45:55.240 INFO:teuthology.orchestra.run.smithi201.stdout:smithi201.front.sepia.ceph.com
2022-02-17T08:45:55.246 INFO:teuthology.orchestra.run.smithi201.stdout:172.21.15.201 smithi201.front.sepia.ceph.com smithi201
2022-02-17T08:45:55.276 INFO:teuthology.orchestra.run.smithi198.stdout:172.21.15.198 smithi198.front.sepia.ceph.com smithi198
2022-02-17T08:45:55.376 INFO:teuthology.provision.fog.smithi201:Deploy complete!
2022-02-17T08:45:55.440 INFO:teuthology.provision.fog.smithi198:Deploy complete!
2022-02-17T08:45:56.971 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-17T08:45:57.437 INFO:teuthology.provision.fog.smithi057:Node is ready
2022-02-17T08:45:57.443 INFO:teuthology.orchestra.run.smithi057.stdout:smithi057.front.sepia.ceph.com
2022-02-17T08:45:57.451 INFO:teuthology.orchestra.run.smithi057.stdout:172.21.15.57 smithi057.front.sepia.ceph.com smithi057
2022-02-17T08:45:57.598 INFO:teuthology.provision.fog.smithi057:Deploy complete!
2022-02-17T08:45:57.825 INFO:teuthology.lock.ops:Checking smithi057.front.sepia.ceph.com
2022-02-17T08:45:57.826 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-17T08:45:57.849 INFO:teuthology.lock.ops:Checking smithi198.front.sepia.ceph.com
2022-02-17T08:45:57.850 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-17T08:45:57.867 INFO:teuthology.lock.ops:Checking smithi201.front.sepia.ceph.com
2022-02-17T08:45:57.867 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-17T08:45:57.884 INFO:teuthology.lock.ops:Updating [smithi057.front.sepia.ceph.com]: set os type and version on server
2022-02-17T08:45:57.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2022-02-17T08:45:58.067 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64
2022-02-17T08:45:58.073 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="Ubuntu"
2022-02-17T08:45:58.074 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-17T08:45:58.074 INFO:teuthology.orchestra.run.smithi057.stdout:ID=ubuntu
2022-02-17T08:45:58.074 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE=debian
2022-02-17T08:45:58.075 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-17T08:45:58.075 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="20.04"
2022-02-17T08:45:58.075 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-17T08:45:58.075 INFO:teuthology.orchestra.run.smithi057.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-17T08:45:58.076 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-17T08:45:58.076 INFO:teuthology.orchestra.run.smithi057.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-17T08:45:58.076 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_CODENAME=focal
2022-02-17T08:45:58.076 INFO:teuthology.orchestra.run.smithi057.stdout:UBUNTU_CODENAME=focal
2022-02-17T08:45:58.077 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server
2022-02-17T08:45:58.096 INFO:teuthology.lock.ops:Updating [smithi198.front.sepia.ceph.com]: set os type and version on server
2022-02-17T08:45:58.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi198.front.sepia.ceph.com'
2022-02-17T08:45:58.280 INFO:teuthology.orchestra.run.smithi198.stdout:x86_64
2022-02-17T08:45:58.285 INFO:teuthology.orchestra.run.smithi198.stdout:NAME="Ubuntu"
2022-02-17T08:45:58.285 INFO:teuthology.orchestra.run.smithi198.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-17T08:45:58.285 INFO:teuthology.orchestra.run.smithi198.stdout:ID=ubuntu
2022-02-17T08:45:58.286 INFO:teuthology.orchestra.run.smithi198.stdout:ID_LIKE=debian
2022-02-17T08:45:58.286 INFO:teuthology.orchestra.run.smithi198.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-17T08:45:58.286 INFO:teuthology.orchestra.run.smithi198.stdout:VERSION_ID="20.04"
2022-02-17T08:45:58.286 INFO:teuthology.orchestra.run.smithi198.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-17T08:45:58.287 INFO:teuthology.orchestra.run.smithi198.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-17T08:45:58.287 INFO:teuthology.orchestra.run.smithi198.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-17T08:45:58.287 INFO:teuthology.orchestra.run.smithi198.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-17T08:45:58.287 INFO:teuthology.orchestra.run.smithi198.stdout:VERSION_CODENAME=focal
2022-02-17T08:45:58.288 INFO:teuthology.orchestra.run.smithi198.stdout:UBUNTU_CODENAME=focal
2022-02-17T08:45:58.288 INFO:teuthology.lock.ops:Updating smithi198.front.sepia.ceph.com on lock server
2022-02-17T08:45:58.306 INFO:teuthology.lock.ops:Updating [smithi201.front.sepia.ceph.com]: set os type and version on server
2022-02-17T08:45:58.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi201.front.sepia.ceph.com'
2022-02-17T08:45:58.484 INFO:teuthology.orchestra.run.smithi201.stdout:x86_64
2022-02-17T08:45:58.533 INFO:teuthology.orchestra.run.smithi201.stdout:NAME="Ubuntu"
2022-02-17T08:45:58.534 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION="20.04.3 LTS (Focal Fossa)"
2022-02-17T08:45:58.534 INFO:teuthology.orchestra.run.smithi201.stdout:ID=ubuntu
2022-02-17T08:45:58.534 INFO:teuthology.orchestra.run.smithi201.stdout:ID_LIKE=debian
2022-02-17T08:45:58.535 INFO:teuthology.orchestra.run.smithi201.stdout:PRETTY_NAME="Ubuntu 20.04.3 LTS"
2022-02-17T08:45:58.535 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION_ID="20.04"
2022-02-17T08:45:58.535 INFO:teuthology.orchestra.run.smithi201.stdout:HOME_URL="https://www.ubuntu.com/"
2022-02-17T08:45:58.535 INFO:teuthology.orchestra.run.smithi201.stdout:SUPPORT_URL="https://help.ubuntu.com/"
2022-02-17T08:45:58.536 INFO:teuthology.orchestra.run.smithi201.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2022-02-17T08:45:58.536 INFO:teuthology.orchestra.run.smithi201.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2022-02-17T08:45:58.536 INFO:teuthology.orchestra.run.smithi201.stdout:VERSION_CODENAME=focal
2022-02-17T08:45:58.536 INFO:teuthology.orchestra.run.smithi201.stdout:UBUNTU_CODENAME=focal
2022-02-17T08:45:58.537 INFO:teuthology.lock.ops:Updating smithi201.front.sepia.ceph.com on lock server
2022-02-17T08:45:58.628 INFO:teuthology.dispatcher.supervisor:Running job 6690488
2022-02-17T08:45:58.629 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c6442aa9e509aace1201390ea4c2d6a80e3d0800/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi/6690488 --name benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi --description rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} -- /home/teuthworker/archive/benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi/6690488/orig.config.yaml
2022-02-17T08:45:58.634 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi/6690488
2022-02-17T08:45:58.635 INFO:teuthology.dispatcher.supervisor:Job PID: 5997
2022-02-17T08:45:58.635 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-02-17T15:16:09.244 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-02-17T15:16:09.508 INFO:teuthology.kill:Killing Pids: {5997}
2022-02-17T15:16:10.865 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'mon.a']
2022-02-17T15:16:10.867 INFO:teuthology.task.internal:roles: ubuntu@smithi198.front.sepia.ceph.com - ['osd.4', 'osd.5', 'osd.6', 'osd.7', 'mon.b', 'mgr.x']
2022-02-17T15:16:10.868 INFO:teuthology.task.internal:roles: ubuntu@smithi201.front.sepia.ceph.com - ['osd.8', 'osd.9', 'osd.10', 'osd.11', 'mon.c']
2022-02-17T15:16:10.869 INFO:teuthology.misc:Compressing logs...
2022-02-17T15:16:10.869 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com'
2022-02-17T15:16:11.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi198.front.sepia.ceph.com'
2022-02-17T15:16:11.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi201.front.sepia.ceph.com'
2022-02-17T15:16:11.606 INFO:teuthology.misc:Compressing logs...
2022-02-17T15:16:12.345 INFO:teuthology.misc:Compressing logs...
2022-02-17T15:17:24.372 INFO:teuthology.kill:No teuthology processes running
2022-02-17T15:17:25.607 INFO:teuthology.kill:Nuking machines: ['smithi057', 'smithi198', 'smithi201']
2022-02-17T15:17:26.687 INFO:teuthology.kill:2022-02-17 15:17:26,687.687 INFO:teuthology.nuke:targets:
2022-02-17T15:17:26.688 INFO:teuthology.kill:  smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgIjAYnDboxe2nnj1K9EES6tnQPgwJZt6Z+vjfjIPTb+qVIPRejHKElYQ1652aUNqM/XmtKa4b/wIQ9ouKV1m9LxTrd4gyqmbYW2rN/bJucnUPGjGe5qfc0x5QyTk341Io4JgcpAD4B4H5GiZs2ZitB9c9BT3rDlvIHEaMAYgWvt/hAcRZ2aGYuiksBmskN2HBp+4zvmeTWpp2ffpOgLuQBN6hWgKHHN9DaowFPhUoUfd1fx1OSWfbXJ2nOhD1H4vaZR6ichLsbAL8OWv+RMy/F4iMQJ8AGIF8/rq7KyGprma19tXG3ta98MIfLRYtQ7nQEbsjSi4pVoD10mWXwTxZnfDh5EC9jxSBxVjhmMquFndfi98pCEkOyBRW3Na5COSaCZ2QRKJGtvfct/EQmKK0QikUE8qPEJ6uCj4BlsWl9thuuHiW0QGarZTSVp9CRl9VK049peIKT/SM15E70yOsJW3J0opOvJKjrWlUh4mV1MrY+c1jT9hrN3vfJPzwyFk=
2022-02-17T15:17:26.688 INFO:teuthology.kill:  smithi198.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0Kyx44PMC7tHl58Gd8TDlWWDmJmcjlHNUKLU/5Vz2GgSv80STZTtdRsOoJdgJGGPfni3Ity+T+8g77T9GkJoXF6MEf1py7MZ0LSTIAL96iHtaRwuWb9Tyq1QmmA3L4UzxS747mjR3SWej+ajXJwOfGdXy/T6YUCYBpi7jfN4UGKaAPIssJOLmx2NBQWDec7ZUS1Yat7hBFL2t/okCepKmMEa+bMT7nHiO3teRfG+6KqkyRzN0zO0lVUeNNebSRAgbjcSG2q+D4NDyHz4QZxaoA5iD5bBZVHOUWm4j/SLYQTGoVD1QJNTkx0BBWf9EdB2pgGTP8BRdD/lOgduf8oakoxG1yM7SnrB+x2jGIzsgIYKgeFJY2YPADzvOa9FQ8zAq730cLupfUwNsLXDZ4E2Uj9uxF7JmJToLdXh/lcJduxzqzifVipgo8ok4KkyDvFJI4+CAZgfVYNTnQTg12qR57ZzZGUKYAg0C49RpQvEo3+YEbGq+ekuTMwEw34npc20=
2022-02-17T15:17:26.688 INFO:teuthology.kill:  smithi201.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSDjFyMX8x2nbyHEvGTyntCp/R931P3B+PGGrb5LuOtGRWWB0gwl1sKhmWN3chAidScUEx0/wXbiaG+B2Jqji5I+/9jfyrpDrXPu1w88ERe/K81Kov7BYTlVzvvl7jxE/0Jvw43OkWPhvY8QjkflvVWJJCWpON0wAwlHdyQnKbJ1evCXbvEPAR92qCjDzTvCqv9kFuL3x5NyaxMZnBUsZ1feDJ55Aw+mbpdd9y4Cg3rmTKPcoYnGXFPDR8O0h54GkG+v6BRnRGHmSoRI3USOKk2wD8aYRZGsSGzpNlgEDuqbs78wJy2PXXUuPYbCaW16m9oafFoSID0CjNPtRiZWOHh91uejlfocoujov1+5OQV9QCnWxOM4BADnVhPN3RFfAZYSAzHxcfTWgrLwMRDkwNCCg1UoFVPEX5NC117VLEl2tGUcpz+lt45UmQv6LvaIf+GAdPK9Rf1Cekp2+YbzlY2xBFNaFzZVPjuoRANEmQIIar3r0ySVry/KEnndkM9gs=
2022-02-17T15:17:26.717 INFO:teuthology.kill:2022-02-17 15:17:26,717.717 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-17T15:17:26.722 INFO:teuthology.kill:2022-02-17 15:17:26,722.722 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-17T15:17:26.723 INFO:teuthology.kill:2022-02-17 15:17:26,723.723 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-17T15:17:26.811 INFO:teuthology.kill:2022-02-17 15:17:26,810.810 INFO:teuthology.orchestra.console:Power off smithi198
2022-02-17T15:17:26.830 INFO:teuthology.kill:2022-02-17 15:17:26,829.829 INFO:teuthology.orchestra.console:Power off smithi201
2022-02-17T15:17:26.843 INFO:teuthology.kill:2022-02-17 15:17:26,843.843 INFO:teuthology.orchestra.console:Power off smithi057
2022-02-17T15:17:35.217 INFO:teuthology.kill:2022-02-17 15:17:35,216.216 INFO:teuthology.orchestra.console:Power off for smithi201 completed
2022-02-17T15:17:35.494 INFO:teuthology.kill:2022-02-17 15:17:35,493.493 INFO:teuthology.lock.ops:unlocked: smithi201.front.sepia.ceph.com
2022-02-17T15:17:39.326 INFO:teuthology.kill:2022-02-17 15:17:39,326.326 INFO:teuthology.orchestra.console:Power off for smithi198 completed
2022-02-17T15:17:39.332 INFO:teuthology.kill:2022-02-17 15:17:39,332.332 INFO:teuthology.orchestra.console:Power off for smithi057 completed
2022-02-17T15:17:39.610 INFO:teuthology.kill:2022-02-17 15:17:39,610.610 INFO:teuthology.lock.ops:unlocked: smithi198.front.sepia.ceph.com
2022-02-17T15:17:39.805 INFO:teuthology.kill:2022-02-17 15:17:39,805.805 INFO:teuthology.lock.ops:unlocked: smithi057.front.sepia.ceph.com
2022-02-17T15:19:40.262 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-02-17T15:19:40.389 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi057 but it was locked by another job: /home/teuthworker/archive/dis-2022-02-16_22:25:18-rbd-wip-rbd-quincy-batch-3-distro-default-smithi/6689691
2022-02-17T15:19:40.390 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi198 but it was locked by another job: /home/teuthworker/archive/dis-2022-02-16_22:25:18-rbd-wip-rbd-quincy-batch-3-distro-default-smithi/6689691
2022-02-17T15:19:40.390 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi201 but it was locked by another job: /home/teuthworker/archive/dis-2022-02-16_22:25:18-rbd-wip-rbd-quincy-batch-3-distro-default-smithi/6689691