2022-02-17T00:28:33.044 INFO:root:teuthology version: 1.1.0-c6442aa9
2022-02-17T00:28:33.178 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi139.front.sepia.ceph.com' reimaging
2022-02-17T00:28:33.179 INFO:teuthology.lock.ops:Updating [ubuntu@smithi139.front.sepia.ceph.com]: reset os type and version on server
2022-02-17T00:28:33.179 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server
2022-02-17T00:28:33.202 INFO:teuthology.lock.ops:Node 'ubuntu@smithi139.front.sepia.ceph.com' reimaging is complete
2022-02-17T00:28:33.202 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi143.front.sepia.ceph.com' reimaging
2022-02-17T00:28:33.203 INFO:teuthology.lock.ops:Updating [ubuntu@smithi143.front.sepia.ceph.com]: reset os type and version on server
2022-02-17T00:28:33.203 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server
2022-02-17T00:28:33.222 INFO:teuthology.lock.ops:Node 'ubuntu@smithi143.front.sepia.ceph.com' reimaging is complete
2022-02-17T00:28:33.419 INFO:teuthology.provision.fog.smithi139:Scheduling deploy of rhel 8.4
2022-02-17T00:28:33.426 INFO:teuthology.provision.fog.smithi143:Scheduling deploy of rhel 8.4
2022-02-17T00:28:33.729 INFO:teuthology.orchestra.console:Power off smithi143
2022-02-17T00:28:33.745 INFO:teuthology.orchestra.console:Power off smithi139
2022-02-17T00:28:42.100 INFO:teuthology.orchestra.console:Power off for smithi143 completed
2022-02-17T00:28:42.201 INFO:teuthology.orchestra.console:Power on smithi143
2022-02-17T00:28:46.230 INFO:teuthology.orchestra.console:Power off for smithi139 completed
2022-02-17T00:28:46.332 INFO:teuthology.orchestra.console:Power on smithi139
2022-02-17T00:28:50.668 INFO:teuthology.orchestra.console:Power on for smithi143 completed
2022-02-17T00:28:50.769 INFO:teuthology.provision.fog.smithi143:Waiting for deploy to finish
2022-02-17T00:28:54.695 INFO:teuthology.orchestra.console:Power on for smithi139 completed
2022-02-17T00:28:54.800 INFO:teuthology.provision.fog.smithi139:Waiting for deploy to finish
2022-02-17T00:32:56.780 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-17T00:33:09.981 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-02-17T00:33:17.143 INFO:teuthology.provision.fog.smithi143:Node is ready
2022-02-17T00:33:17.163 INFO:teuthology.orchestra.run.smithi143.stdout:smithi143.front.sepia.ceph.com
2022-02-17T00:33:17.220 INFO:teuthology.orchestra.run.smithi143.stdout:172.21.15.143 smithi143.front.sepia.ceph.com smithi143
2022-02-17T00:33:17.576 INFO:teuthology.provision.fog.smithi143:Deploy complete!
2022-02-17T00:33:30.058 INFO:teuthology.provision.fog.smithi139:Node is ready
2022-02-17T00:33:30.078 INFO:teuthology.orchestra.run.smithi139.stdout:smithi139.front.sepia.ceph.com
2022-02-17T00:33:30.134 INFO:teuthology.orchestra.run.smithi139.stdout:172.21.15.139 smithi139.front.sepia.ceph.com smithi139
2022-02-17T00:33:30.472 INFO:teuthology.provision.fog.smithi139:Deploy complete!
2022-02-17T00:33:30.636 INFO:teuthology.lock.ops:Checking smithi139.front.sepia.ceph.com
2022-02-17T00:33:30.854 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-17T00:33:30.916 INFO:teuthology.lock.ops:Checking smithi143.front.sepia.ceph.com
2022-02-17T00:33:30.916 INFO:teuthology.lock.ops:New key found. Updating...
2022-02-17T00:33:30.971 INFO:teuthology.lock.ops:Updating [smithi139.front.sepia.ceph.com]: set os type and version on server
2022-02-17T00:33:30.972 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com'
2022-02-17T00:33:31.199 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64
2022-02-17T00:33:31.253 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="Red Hat Enterprise Linux"
2022-02-17T00:33:31.253 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="8.4 (Ootpa)"
2022-02-17T00:33:31.253 INFO:teuthology.orchestra.run.smithi139.stdout:ID="rhel"
2022-02-17T00:33:31.253 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="fedora"
2022-02-17T00:33:31.254 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="8.4"
2022-02-17T00:33:31.254 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el8"
2022-02-17T00:33:31.254 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-02-17T00:33:31.254 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31"
2022-02-17T00:33:31.255 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-02-17T00:33:31.255 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://www.redhat.com/"
2022-02-17T00:33:31.255 INFO:teuthology.orchestra.run.smithi139.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-02-17T00:33:31.255 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-17T00:33:31.256 INFO:teuthology.orchestra.run.smithi139.stdout:
2022-02-17T00:33:31.256 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-17T00:33:31.256 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-02-17T00:33:31.256 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-02-17T00:33:31.257 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-02-17T00:33:31.258 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server
2022-02-17T00:33:31.276 INFO:teuthology.lock.ops:Updating [smithi143.front.sepia.ceph.com]: set os type and version on server
2022-02-17T00:33:31.277 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com'
2022-02-17T00:33:31.512 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64
2022-02-17T00:33:31.568 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="Red Hat Enterprise Linux"
2022-02-17T00:33:31.568 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="8.4 (Ootpa)"
2022-02-17T00:33:31.568 INFO:teuthology.orchestra.run.smithi143.stdout:ID="rhel"
2022-02-17T00:33:31.568 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="fedora"
2022-02-17T00:33:31.569 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="8.4"
2022-02-17T00:33:31.569 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el8"
2022-02-17T00:33:31.569 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
2022-02-17T00:33:31.569 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31"
2022-02-17T00:33:31.570 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
2022-02-17T00:33:31.570 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://www.redhat.com/"
2022-02-17T00:33:31.570 INFO:teuthology.orchestra.run.smithi143.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-02-17T00:33:31.570 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-02-17T00:33:31.571 INFO:teuthology.orchestra.run.smithi143.stdout:
2022-02-17T00:33:31.571 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-02-17T00:33:31.571 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
2022-02-17T00:33:31.571 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-02-17T00:33:31.571 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
2022-02-17T00:33:31.572 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server
2022-02-17T00:33:31.667 INFO:teuthology.dispatcher.supervisor:Running job 6690132
2022-02-17T00:33:31.667 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c6442aa9e509aace1201390ea4c2d6a80e3d0800/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi/6690132 --name nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi --description rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/redirect} -- /home/teuthworker/archive/nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi/6690132/orig.config.yaml
2022-02-17T00:33:31.671 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi/6690132
2022-02-17T00:33:31.672 INFO:teuthology.dispatcher.supervisor:Job PID: 1686
2022-02-17T00:33:31.672 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-02-17T07:04:01.099 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing...
2022-02-17T07:04:01.309 INFO:teuthology.kill:Killing Pids: {1686}
2022-02-17T07:04:02.507 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2022-02-17T07:04:02.508 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2022-02-17T07:04:02.510 INFO:teuthology.misc:Compressing logs...
2022-02-17T07:04:02.511 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com'
2022-02-17T07:04:02.693 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com'
2022-02-17T07:04:03.036 INFO:teuthology.misc:Compressing logs...
2022-02-17T07:04:03.602 INFO:teuthology.orchestra.run.smithi143.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-02-17T07:04:03.722 INFO:teuthology.orchestra.run.smithi139.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-02-17T07:04:04.111 INFO:teuthology.misc:Compressing logs...
2022-02-17T07:04:19.330 INFO:teuthology.kill:No teuthology processes running
2022-02-17T07:04:20.524 INFO:teuthology.kill:Nuking machines: ['smithi139', 'smithi143']
2022-02-17T07:04:21.575 INFO:teuthology.kill:2022-02-17 07:04:21,575.575 INFO:teuthology.nuke:targets:
2022-02-17T07:04:21.576 INFO:teuthology.kill:  smithi139.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCrk4lZx8q2QM5clz1+t1lHuNItx+4tLKyTv7DZnyEY+9kWrZDk15/lplJPRXbukH0oqKd7Yzf+yTCzJUDF76Qa+Dg3Wb+Y51kBZcHOwEZrosTGbptzm4k5L/K9NsnG1HrEDYtNaH0XcEO9HlV7IkzR1O07YJfCV8R4gaPgH2xk51MFq9NCp5XlwBvX23EnEmf3VfOuPOAnuFZ7Rc3jPdtZegZdzho3aP1NjiGjwSLGGGGllBXBQI194o2rBYKByRtLqJJum1dakvtEmaooRhQWr+m51ndC40qu+iifNyHKyUVVp4kdq0VTcsVLlQgyC+IhXftr6LX0QWwQ1FYeLN9+URzkmaxAcZnETHtf5/hW2ahD/sNmBlLLt5gvTCilcvzu/YB0uzuwS/4yEQsXrl4/PpIO4XIvGKxqBH/Jj86CFeOmqEB/77NXWAWznTh/oI+vrTrlnPgtjEY22bfXbyEVieqcc3DOXaVl2fK5fT1OG9eKJwjAEJTT7q2ETrkX90=
2022-02-17T07:04:21.577 INFO:teuthology.kill:  smithi143.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTdyvzqaDXSG8eSd6ftJRbP0iJNSXYyvD0HfJ042Ismgk6puyiI0Cm0qddTYAMIHHSk4IlQpL4rlylDfzXoqrKtJl4zSCEnWVHXxsZ6WPW1OVpC6sSYOEJMpKGV6fxFPR2OBTY9z/8v3043ypDLht3Aq5rzl1Iv0a1JKDrlcSbW9IAnzzyotyJD5d9qkoiR0+r0AnWMepJIoE9V7KF0fK7Bl4LeAq8yMMXw/kKmtS3BWf++4fdS6PJTJ3DPAKpOgbI5ER6Yre4cAUUrsm14ChzyClHsF+7JfxioMui9uvSZYVMFD32yYALR4E6IvbkynuA3ViT+xHbAFY2ZNWjhQzd/WyzOTwkr+jQv7TGHjSjFPQqj2Daj59uM66M8jo9H9YNxfn+sOl3CRCsK4EXF8UqoARI6NegMS4pZwBM3PFSbvqDbyY2QqQfl9UMNyCXAvLZ9eP1A9ZRKNGx03gyZDlOZwz2lEdam1Fxvda2ZaShNVsyGgtziZehJ7lZtgj6PP0=
2022-02-17T07:04:21.601 INFO:teuthology.kill:2022-02-17 07:04:21,601.601 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-17T07:04:21.605 INFO:teuthology.kill:2022-02-17 07:04:21,604.604 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-02-17T07:04:21.682 INFO:teuthology.kill:2022-02-17 07:04:21,682.682 INFO:teuthology.orchestra.console:Power off smithi143
2022-02-17T07:04:21.697 INFO:teuthology.kill:2022-02-17 07:04:21,696.696 INFO:teuthology.orchestra.console:Power off smithi139
2022-02-17T07:04:30.052 INFO:teuthology.kill:2022-02-17 07:04:30,052.052 INFO:teuthology.orchestra.console:Power off for smithi143 completed
2022-02-17T07:04:30.058 INFO:teuthology.kill:2022-02-17 07:04:30,058.058 INFO:teuthology.orchestra.console:Power off for smithi139 completed
2022-02-17T07:04:30.204 INFO:teuthology.kill:2022-02-17 07:04:30,204.204 INFO:teuthology.lock.ops:unlocked: smithi143.front.sepia.ceph.com
2022-02-17T07:04:30.207 INFO:teuthology.kill:2022-02-17 07:04:30,207.207 INFO:teuthology.lock.ops:unlocked: smithi139.front.sepia.ceph.com
2022-02-17T07:06:30.534 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-02-17T07:06:30.650 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi139 but it was locked by another job: /home/teuthworker/archive/benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi/6690451
2022-02-17T07:06:30.651 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi143 but it was locked by another job: /home/teuthworker/archive/benhanokh-2022-02-17_04:58:07-rados-GBH_TEST_scrub_allocation_file_on_mount-distro-basic-smithi/6690451