2022-02-17T14:17:32.144 INFO:root:teuthology version: 1.1.0-c6442aa9 2022-02-17T14:17:32.269 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi153.front.sepia.ceph.com' reimaging 2022-02-17T14:17:32.271 INFO:teuthology.lock.ops:Updating [ubuntu@smithi153.front.sepia.ceph.com]: reset os type and version on server 2022-02-17T14:17:32.272 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2022-02-17T14:17:32.294 INFO:teuthology.lock.ops:Node 'ubuntu@smithi153.front.sepia.ceph.com' reimaging is complete 2022-02-17T14:17:32.294 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi186.front.sepia.ceph.com' reimaging 2022-02-17T14:17:32.295 INFO:teuthology.lock.ops:Updating [ubuntu@smithi186.front.sepia.ceph.com]: reset os type and version on server 2022-02-17T14:17:32.295 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2022-02-17T14:17:32.319 INFO:teuthology.lock.ops:Node 'ubuntu@smithi186.front.sepia.ceph.com' reimaging is complete 2022-02-17T14:17:32.506 INFO:teuthology.provision.fog.smithi153:Scheduling deploy of rhel 8.4 2022-02-17T14:17:32.538 INFO:teuthology.provision.fog.smithi186:Scheduling deploy of rhel 8.4 2022-02-17T14:17:32.874 INFO:teuthology.orchestra.console:Power off smithi186 2022-02-17T14:17:32.889 INFO:teuthology.orchestra.console:Power off smithi153 2022-02-17T14:17:33.018 INFO:teuthology.orchestra.console:Power off for smithi186 completed 2022-02-17T14:17:33.029 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2022-02-17T14:17:33.119 INFO:teuthology.orchestra.console:Power on smithi186 2022-02-17T14:17:33.130 INFO:teuthology.orchestra.console:Power on smithi153 2022-02-17T14:17:37.373 INFO:teuthology.orchestra.console:Power on for smithi186 completed 2022-02-17T14:17:37.474 INFO:teuthology.provision.fog.smithi186:Waiting for deploy to finish 2022-02-17T14:17:41.500 INFO:teuthology.orchestra.console:Power on for smithi153 completed 2022-02-17T14:17:41.601 INFO:teuthology.provision.fog.smithi153:Waiting for deploy to finish 2022-02-17T14:22:33.498 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-17T14:22:33.713 INFO:teuthology.provision.fog.smithi186:Node is ready 2022-02-17T14:22:33.729 INFO:teuthology.orchestra.run.smithi186.stdout:smithi186.front.sepia.ceph.com 2022-02-17T14:22:33.785 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.15.186 smithi186.front.sepia.ceph.com smithi186 2022-02-17T14:22:34.114 INFO:teuthology.provision.fog.smithi186:Deploy complete! 2022-02-17T14:22:37.404 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-02-17T14:22:37.604 INFO:teuthology.provision.fog.smithi153:Node is ready 2022-02-17T14:22:37.619 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153.front.sepia.ceph.com 2022-02-17T14:22:37.676 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153 smithi153.front.sepia.ceph.com smithi153 2022-02-17T14:22:38.010 INFO:teuthology.provision.fog.smithi153:Deploy complete! 2022-02-17T14:22:38.159 INFO:teuthology.lock.ops:Checking smithi153.front.sepia.ceph.com 2022-02-17T14:22:38.160 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-17T14:22:38.178 INFO:teuthology.lock.ops:Checking smithi186.front.sepia.ceph.com 2022-02-17T14:22:38.179 INFO:teuthology.lock.ops:New key found. Updating... 2022-02-17T14:22:38.195 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: set os type and version on server 2022-02-17T14:22:38.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2022-02-17T14:22:38.423 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2022-02-17T14:22:38.476 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="Red Hat Enterprise Linux" 2022-02-17T14:22:38.477 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="8.4 (Ootpa)" 2022-02-17T14:22:38.477 INFO:teuthology.orchestra.run.smithi153.stdout:ID="rhel" 2022-02-17T14:22:38.477 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="fedora" 2022-02-17T14:22:38.478 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="8.4" 2022-02-17T14:22:38.478 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el8" 2022-02-17T14:22:38.478 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-02-17T14:22:38.478 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2022-02-17T14:22:38.478 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-02-17T14:22:38.479 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://www.redhat.com/" 2022-02-17T14:22:38.479 INFO:teuthology.orchestra.run.smithi153.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-02-17T14:22:38.479 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-17T14:22:38.479 INFO:teuthology.orchestra.run.smithi153.stdout: 2022-02-17T14:22:38.480 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-17T14:22:38.480 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-02-17T14:22:38.480 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-02-17T14:22:38.480 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-02-17T14:22:38.481 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2022-02-17T14:22:38.500 INFO:teuthology.lock.ops:Updating [smithi186.front.sepia.ceph.com]: set os type and version on server 2022-02-17T14:22:38.500 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-02-17T14:22:38.738 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2022-02-17T14:22:38.793 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="Red Hat Enterprise Linux" 2022-02-17T14:22:38.793 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="8.4 (Ootpa)" 2022-02-17T14:22:38.794 INFO:teuthology.orchestra.run.smithi186.stdout:ID="rhel" 2022-02-17T14:22:38.794 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="fedora" 2022-02-17T14:22:38.794 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="8.4" 2022-02-17T14:22:38.794 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el8" 2022-02-17T14:22:38.795 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-02-17T14:22:38.795 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2022-02-17T14:22:38.795 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-02-17T14:22:38.795 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://www.redhat.com/" 2022-02-17T14:22:38.795 INFO:teuthology.orchestra.run.smithi186.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-02-17T14:22:38.796 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-02-17T14:22:38.796 INFO:teuthology.orchestra.run.smithi186.stdout: 2022-02-17T14:22:38.796 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-02-17T14:22:38.796 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-02-17T14:22:38.796 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-02-17T14:22:38.797 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-02-17T14:22:38.797 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2022-02-17T14:22:38.897 INFO:teuthology.dispatcher.supervisor:Running job 6690274 2022-02-17T14:22:38.898 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_c6442aa9e509aace1201390ea4c2d6a80e3d0800/virtualenv/bin/teuthology -v --owner scheduled_nojha@teuthology --archive /home/teuthworker/archive/nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi/6690274 --name nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi --description rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{default} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/cache-snaps} -- /home/teuthworker/archive/nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi/6690274/orig.config.yaml 2022-02-17T14:22:38.901 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/nojha-2022-02-16_23:40:43-rados:thrash-GBH_scrub_allocation_file_on_mount-distro-basic-smithi/6690274 2022-02-17T14:22:38.903 INFO:teuthology.dispatcher.supervisor:Job PID: 10243 2022-02-17T14:22:38.903 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-02-17T20:53:07.987 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 23400s. Killing... 2022-02-17T20:53:08.306 INFO:teuthology.kill:Killing Pids: {10243} 2022-02-17T20:53:09.423 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-02-17T20:53:09.424 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-02-17T20:53:09.424 INFO:teuthology.misc:Compressing logs... 2022-02-17T20:53:09.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2022-02-17T20:53:09.610 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2022-02-17T20:53:09.962 INFO:teuthology.misc:Compressing logs... 2022-02-17T20:53:10.491 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-02-17T20:53:10.635 INFO:teuthology.orchestra.run.smithi186.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-02-17T20:53:11.008 INFO:teuthology.misc:Compressing logs... 2022-02-17T20:53:11.888 INFO:teuthology.kill:No teuthology processes running 2022-02-17T20:53:12.867 INFO:teuthology.kill:Nuking machines: ['smithi153', 'smithi186'] 2022-02-17T20:53:13.704 INFO:teuthology.kill:2022-02-17 20:53:13,703.703 INFO:teuthology.nuke:targets: 2022-02-17T20:53:13.705 INFO:teuthology.kill: smithi153.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVHIlGfDTS68uygvJZk9k50sguOyty+fqFVW5mFV3gTPqojcLzSlq5mfEiSDx0vnYTYkdWRTJhKxGK5C8lyKqxlwwb3uUbJI8a/86fgYimInoc8tC0BmjHWGhGoc9ky1dhkP6o8Ha67zH+7zz+7hmAq7D4/kCOn8oM8CCDMpXVapCVfOtchCutSEu+ipVrzoi0ECJYtH1PLLbMzr5JwX82Tulo9iuu3IBs7ZV1lUsAmP+MdPMtm0Bz5AWPEnuW2bC7NX9RnYE3Mno7aR4Y7YIg+H3SDDXudZ5wOXvfV1Jjd4tVyqADXlA5FmyiI4dmXG4q/Tx4aTGP664sIWiEIJRqWxv2c3X5GFmK26zwufvXRoFjIMrBib6RYJPcOGd1154vN5TCvxT0aUHFmI7Nlyen3HvlTKd7TCzoFHD3XQRT1mr12Li8zGCIhFfKiOKUT1ef+gn92WofGV5DCtHtNQDPVrY4NKfmePKi/Hj7XNNHaFZKyaLL2Nan2wY99WTKVM8= 2022-02-17T20:53:13.705 INFO:teuthology.kill: smithi186.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgdBXQAT+Ko4jUi8yl3J6QDA97bE9pW9ITc0ecmG9K2km76yOGFs32RUGh2y+pZjEplyA2QaJgWnYZJmXiaqB67fT53nwCvzm4eloZ1IW1FXg9EL+SicaLNW12aG4ziEEwEfDw0YH2Fvczd8nyBFEwkXDE59EDo10p8s4a4pHWvCSMLEuFjTY1OJeQW3J5lT2vr0N5BP1JDR2A4jl5lMpWtOVoOyt92zzK/Kml3OhP7JkVJ0q4WE9RuKYJ6lCXptmDIE/pmINg6f9P4b7ampZNn+2G7xUganH2Jg5V8cUXNEgvKwY3fBRtaNNSJfQVuC58HdTcxty54Ea4lD6jIlbgWNqBwr3xvH/cxv1ajTPC5DekSK0A8rBjOxRMAnBlRmxeK7R58UNRTnIco3sxc//KDcjIp3DNJaDLAVItgGXwFp8S2KhM5Vh7YDu7/xE/YwwTYhoQ0YVxiRPQo53F+A3KY8MwKVxuQ7c4jzI1knCzoVl7s1fNVecwFsV8kU81U78= 2022-02-17T20:53:13.732 INFO:teuthology.kill:2022-02-17 20:53:13,732.732 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-17T20:53:13.733 INFO:teuthology.kill:2022-02-17 20:53:13,733.733 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-02-17T20:53:13.807 INFO:teuthology.kill:2022-02-17 20:53:13,807.807 INFO:teuthology.orchestra.console:Power off smithi153 2022-02-17T20:53:13.818 INFO:teuthology.kill:2022-02-17 20:53:13,817.817 INFO:teuthology.orchestra.console:Power off smithi186 2022-02-17T20:53:26.289 INFO:teuthology.kill:2022-02-17 20:53:26,288.288 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2022-02-17T20:53:26.294 INFO:teuthology.kill:2022-02-17 20:53:26,294.294 INFO:teuthology.orchestra.console:Power off for smithi186 completed 2022-02-17T20:53:26.442 INFO:teuthology.kill:2022-02-17 20:53:26,441.441 INFO:teuthology.lock.ops:unlocked: smithi186.front.sepia.ceph.com 2022-02-17T20:53:26.446 INFO:teuthology.kill:2022-02-17 20:53:26,446.446 INFO:teuthology.lock.ops:unlocked: smithi153.front.sepia.ceph.com 2022-02-17T20:55:26.728 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15