2022-02-17T14:17:01.726 INFO:root:teuthology version: 1.1.0-c6442aa9 2022-02-17T14:17:01.866 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi008.front.sepia.ceph.com' reimaging 2022-02-17T14:17:01.867 INFO:teuthology.lock.ops:Updating [ubuntu@smithi008.front.sepia.ceph.com]: reset os type and version on server 2022-02-17T14:17:01.867 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2022-02-17T14:17:01.896 INFO:teuthology.lock.ops:Node 'ubuntu@smithi008.front.sepia.ceph.com' reimaging is complete 2022-02-17T14:17:01.896 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi067.front.sepia.ceph.com' reimaging 2022-02-17T14:17:01.897 INFO:teuthology.lock.ops:Updating [ubuntu@smithi067.front.sepia.ceph.com]: reset os type and version on server 2022-02-17T14:17:01.897 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2022-02-17T14:17:01.917 INFO:teuthology.lock.ops:Node 'ubuntu@smithi067.front.sepia.ceph.com' reimaging is complete 2022-02-17T14:17:02.114 INFO:teuthology.provision.fog.smithi008:Scheduling deploy of rhel 8.4 2022-02-17T14:17:02.130 INFO:teuthology.provision.fog.smithi067:Scheduling deploy of rhel 8.4 2022-02-17T14:17:02.413 INFO:teuthology.orchestra.console:Power off smithi008 2022-02-17T14:17:02.427 INFO:teuthology.orchestra.console:Power off smithi067 2022-02-17T14:17:02.557 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2022-02-17T14:17:02.567 INFO:teuthology.orchestra.console:Power off for smithi067 completed 2022-02-17T14:17:02.659 INFO:teuthology.orchestra.console:Power on smithi008 2022-02-17T14:17:02.669 INFO:teuthology.orchestra.console:Power on smithi067 2022-02-17T14:17:11.029 INFO:teuthology.orchestra.console:Power on for smithi008 completed 2022-02-17T14:17:11.037 INFO:teuthology.orchestra.console:Power on for smithi067 completed 2022-02-17T14:17:11.131 INFO:teuthology.provision.fog.smithi008:Waiting for deploy to finish 2022-02-17T14:17:11.138 INFO:teuthology.provision.fog.smithi067:Waiting for deploy to finish 2022-02-17T14:21:42.839 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-17T14:21:43.049 INFO:teuthology.provision.fog.smithi008:Node is ready 2022-02-17T14:21:43.066 INFO:teuthology.orchestra.run.smithi008.stdout:smithi008.front.sepia.ceph.com 2022-02-17T14:21:43.123 INFO:teuthology.orchestra.run.smithi008.stdout:172.21.15.8 smithi008.front.sepia.ceph.com smithi008 2022-02-17T14:21:43.465 INFO:teuthology.provision.fog.smithi008:Deploy complete! 2022-02-17T14:21:57.894 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-17T14:21:58.102 INFO:teuthology.provision.fog.smithi067:Node is ready 2022-02-17T14:21:58.119 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067.front.sepia.ceph.com 2022-02-17T14:21:58.177 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67 smithi067.front.sepia.ceph.com smithi067 2022-02-17T14:21:58.525 INFO:teuthology.provision.fog.smithi067:Deploy complete! 2022-02-17T14:21:58.674 INFO:teuthology.lock.ops:Checking smithi008.front.sepia.ceph.com 2022-02-17T14:21:58.675 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-17T14:21:58.695 INFO:teuthology.lock.ops:Checking smithi067.front.sepia.ceph.com 2022-02-17T14:21:58.695 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-17T14:21:58.714 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: set os type and version on server 2022-02-17T14:21:58.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2022-02-17T14:21:58.954 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2022-02-17T14:21:59.011 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="Red Hat Enterprise Linux" 2022-02-17T14:21:59.011 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="8.4 (Ootpa)" 2022-02-17T14:21:59.012 INFO:teuthology.orchestra.run.smithi008.stdout:ID="rhel" 2022-02-17T14:21:59.012 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="fedora" 2022-02-17T14:21:59.013 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="8.4" 2022-02-17T14:21:59.013 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el8" 2022-02-17T14:21:59.014 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-02-17T14:21:59.014 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31" 2022-02-17T14:21:59.015 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-02-17T14:21:59.015 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://www.redhat.com/" 2022-02-17T14:21:59.015 INFO:teuthology.orchestra.run.smithi008.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-02-17T14:21:59.016 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-17T14:21:59.016 INFO:teuthology.orchestra.run.smithi008.stdout: 2022-02-17T14:21:59.017 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-17T14:21:59.017 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-02-17T14:21:59.018 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-02-17T14:21:59.018 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-02-17T14:21:59.020 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2022-02-17T14:21:59.042 INFO:teuthology.lock.ops:Updating [smithi067.front.sepia.ceph.com]: set os type and version on server 2022-02-17T14:21:59.043 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2022-02-17T14:21:59.276 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2022-02-17T14:21:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="Red Hat Enterprise Linux" 2022-02-17T14:21:59.332 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="8.4 (Ootpa)" 2022-02-17T14:21:59.333 INFO:teuthology.orchestra.run.smithi067.stdout:ID="rhel" 2022-02-17T14:21:59.333 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="fedora" 2022-02-17T14:21:59.333 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="8.4" 2022-02-17T14:21:59.333 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el8" 2022-02-17T14:21:59.334 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-02-17T14:21:59.334 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2022-02-17T14:21:59.334 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-02-17T14:21:59.334 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.redhat.com/" 2022-02-17T14:21:59.335 INFO:teuthology.orchestra.run.smithi067.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-02-17T14:21:59.335 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-17T14:21:59.335 INFO:teuthology.orchestra.run.smithi067.stdout: 2022-02-17T14:21:59.335 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-17T14:21:59.335 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-02-17T14:21:59.336 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-02-17T14:21:59.336 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-02-17T14:21:59.336 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2022-02-17T14:21:59.434 INFO:teuthology.dispatcher.supervisor:Running job 6690273 2022-02-17T14:21:59.435 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c6442aa9e509aace1201390ea4c2d6a80e3d0800/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi/6690273 --name nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi --description rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{rhel_8} thrashers/morepggrow thrashosds-health workloads/cache-pool-snaps} -- /home/teuthworker/archive/nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi/6690273/orig.config.yaml 2022-02-17T14:21:59.438 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi/6690273 2022-02-17T14:21:59.439 INFO:teuthology.dispatcher.supervisor:Job PID: 8347 2022-02-17T14:21:59.440 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-17T20:52:30.254 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-17T20:52:30.497 INFO:teuthology.kill:Killing Pids: {8347} 2022-02-17T20:52:31.562 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-02-17T20:52:31.563 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-02-17T20:52:31.563 INFO:teuthology.misc:Compressing logs... 2022-02-17T20:52:31.564 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi008.front.sepia.ceph.com' 2022-02-17T20:52:31.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2022-02-17T20:52:32.097 INFO:teuthology.misc:Compressing logs... 2022-02-17T20:52:32.649 INFO:teuthology.orchestra.run.smithi008.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-02-17T20:52:32.838 INFO:teuthology.orchestra.run.smithi067.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-02-17T20:52:33.210 INFO:teuthology.misc:Compressing logs... 2022-02-17T20:52:34.067 INFO:teuthology.kill:No teuthology processes running 2022-02-17T20:52:35.065 INFO:teuthology.kill:Nuking machines: ['smithi008', 'smithi067'] 2022-02-17T20:52:35.931 INFO:teuthology.kill:2022-02-17 20:52:35,931.931 INFO:teuthology.nuke:targets: 2022-02-17T20:52:35.932 INFO:teuthology.kill: smithi008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCqYKgIEexQnQdmj+xw29518l53mD9t0Vt2UeAP/uL7Jrl9b3epUJz6hzmPPca+hksDWzl/ZbxZQyK5LVLTDJVU6kpDlJQlHniKZ0Z9S/GmhNPIyPurf0Jsm8OsDylJ5nHLRieVNeqFj+UMgkSsDrjX9DmcnvPSF3h7TNRLc/hgLjxfk1jjAYR6S9X5tDxpV0CInEVz6lMYKtDDDQSOeonMlw6nlYTCT3wTQT8gZcYSZT93uBK/IIp4yzBcx5yGdZ1BtFFZLJl0UKnSrE8sS7jYDi/Uc2yLEk1FK6W0Ix20I0NMCA5Jsdx7CzWuwVS43BIDdS9u0NvM8iTqADsmjc/ySSSoDxznlsttQxOvl8SRidSHeE2W0Xykq6kVkQUXbaZOKW3+enEfh5eHSBIDtJgTt00DLC11t4ZbgaBATIXoqD+ZG/5/cuwB7obqgXMLw4A36wfpY7zQwZaMumBs+o+rUm514ezm9CrMrHRolJDt+FPEFesL7Q3Gt9gNVoC5k9k= 2022-02-17T20:52:35.932 INFO:teuthology.kill: smithi067.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMl7r4t9SgG4M9duPeQ28wOY5DiUxeYQSAboXndNwX97Wv0sr3pgoVmhx7tWn3x/w+/5g3wlnz5NfoPFgemUyaaNOfrOzayk3MLuiNT8bl2S97MG/TTfHAx9IdxCSZvmpd1XiwTo6+RotdRJ1L3lWHNJHsG3I1zhG4/RmKI2JX73AfuZMxa4rraTsWI5aUDMfWe33ewkVzIkVnVsFhfx57CTlfixbQ2ehWkK7Hh7/3arRiAQg1rxvh2wBh9uQs9hVXnerFMFOIiNLTUerRDMQRZ59+LbzHI9bcAPRSzd7bTKYEqtPQoFE5z/ZNXxZhBuiupNf7zxjBpZQZxsYUOAcfnynFiSlqayAzQljAVc7Pqb4MbEfwQ5Xu1ZsfFEco1XqU7j4wmdxGXdGt8CWVl6THUWAwarKa1XfD/EIzI6d5QIavC6a3+lRAM9rU0jM4dz2BovMTNmq5PUY3qRYsBeHnQe6QwZgIhzsQGxnPBcS5C8zZw5T7Cnep/MJ8YSEOKTk= 2022-02-17T20:52:35.959 INFO:teuthology.kill:2022-02-17 20:52:35,959.959 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-17T20:52:35.961 INFO:teuthology.kill:2022-02-17 20:52:35,961.961 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-17T20:52:36.035 INFO:teuthology.kill:2022-02-17 20:52:36,035.035 INFO:teuthology.orchestra.console:Power off smithi067 2022-02-17T20:52:36.047 INFO:teuthology.kill:2022-02-17 20:52:36,047.047 INFO:teuthology.orchestra.console:Power off smithi008 2022-02-17T20:52:44.410 INFO:teuthology.kill:2022-02-17 20:52:44,410.410 INFO:teuthology.orchestra.console:Power off for smithi067 completed 2022-02-17T20:52:44.415 INFO:teuthology.kill:2022-02-17 20:52:44,414.414 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2022-02-17T20:52:44.558 INFO:teuthology.kill:2022-02-17 20:52:44,557.557 INFO:teuthology.lock.ops:unlocked: smithi067.front.sepia.ceph.com 2022-02-17T20:52:44.562 INFO:teuthology.kill:2022-02-17 20:52:44,562.562 INFO:teuthology.lock.ops:unlocked: smithi008.front.sepia.ceph.com 2022-02-17T20:54:44.838 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15