2021-12-05T17:06:17.485 INFO:root:teuthology version: 1.1.0-844d1377 2021-12-05T17:06:17.606 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi037.front.sepia.ceph.com' reimaging 2021-12-05T17:06:17.607 INFO:teuthology.lock.ops:Updating [ubuntu@smithi037.front.sepia.ceph.com]: reset os type and version on server 2021-12-05T17:06:17.608 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2021-12-05T17:06:17.631 INFO:teuthology.lock.ops:Node 'ubuntu@smithi037.front.sepia.ceph.com' reimaging is complete 2021-12-05T17:06:17.631 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging 2021-12-05T17:06:17.632 INFO:teuthology.lock.ops:Updating [ubuntu@smithi062.front.sepia.ceph.com]: reset os type and version on server 2021-12-05T17:06:17.632 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2021-12-05T17:06:17.652 INFO:teuthology.lock.ops:Node 'ubuntu@smithi062.front.sepia.ceph.com' reimaging is complete 2021-12-05T17:06:17.846 INFO:teuthology.provision.fog.smithi037:Scheduling deploy of rhel 8.4 2021-12-05T17:06:17.863 INFO:teuthology.provision.fog.smithi062:Scheduling deploy of rhel 8.4 2021-12-05T17:06:18.187 INFO:teuthology.orchestra.console:Power off smithi062 2021-12-05T17:06:18.201 INFO:teuthology.orchestra.console:Power off smithi037 2021-12-05T17:06:18.354 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-12-05T17:06:18.456 INFO:teuthology.orchestra.console:Power on smithi062 2021-12-05T17:06:26.950 INFO:teuthology.orchestra.console:Power on for smithi062 completed 2021-12-05T17:06:27.052 INFO:teuthology.provision.fog.smithi062:Waiting for deploy to finish 2021-12-05T17:06:30.679 INFO:teuthology.orchestra.console:Power off for smithi037 completed 2021-12-05T17:06:30.780 INFO:teuthology.orchestra.console:Power on smithi037 2021-12-05T17:06:39.143 INFO:teuthology.orchestra.console:Power on for smithi037 completed 2021-12-05T17:06:39.244 INFO:teuthology.provision.fog.smithi037:Waiting for deploy to finish 2021-12-05T17:11:13.818 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-05T17:11:14.062 INFO:teuthology.provision.fog.smithi037:Node is ready 2021-12-05T17:11:14.118 INFO:teuthology.orchestra.run.smithi037.stdout:smithi037.front.sepia.ceph.com 2021-12-05T17:11:14.174 INFO:teuthology.orchestra.run.smithi037.stdout:172.21.15.37 smithi037.front.sepia.ceph.com smithi037 2021-12-05T17:11:14.504 INFO:teuthology.provision.fog.smithi037:Deploy complete! 2021-12-05T17:11:16.525 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-05T17:11:16.739 INFO:teuthology.provision.fog.smithi062:Node is ready 2021-12-05T17:11:16.755 INFO:teuthology.orchestra.run.smithi062.stdout:smithi062.front.sepia.ceph.com 2021-12-05T17:11:16.811 INFO:teuthology.orchestra.run.smithi062.stdout:172.21.15.62 smithi062.front.sepia.ceph.com smithi062 2021-12-05T17:11:17.131 INFO:teuthology.provision.fog.smithi062:Deploy complete! 2021-12-05T17:11:17.304 INFO:teuthology.lock.ops:Checking smithi037.front.sepia.ceph.com 2021-12-05T17:11:17.305 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-05T17:11:17.325 INFO:teuthology.lock.ops:Checking smithi062.front.sepia.ceph.com 2021-12-05T17:11:17.326 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-05T17:11:17.344 INFO:teuthology.lock.ops:Updating [smithi037.front.sepia.ceph.com]: set os type and version on server 2021-12-05T17:11:17.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2021-12-05T17:11:17.578 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2021-12-05T17:11:17.633 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="Red Hat Enterprise Linux" 2021-12-05T17:11:17.634 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="8.4 (Ootpa)" 2021-12-05T17:11:17.634 INFO:teuthology.orchestra.run.smithi037.stdout:ID="rhel" 2021-12-05T17:11:17.634 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="fedora" 2021-12-05T17:11:17.635 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="8.4" 2021-12-05T17:11:17.635 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el8" 2021-12-05T17:11:17.635 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-12-05T17:11:17.635 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31" 2021-12-05T17:11:17.636 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-12-05T17:11:17.636 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://www.redhat.com/" 2021-12-05T17:11:17.636 INFO:teuthology.orchestra.run.smithi037.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-12-05T17:11:17.636 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-05T17:11:17.637 INFO:teuthology.orchestra.run.smithi037.stdout: 2021-12-05T17:11:17.637 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-05T17:11:17.637 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-12-05T17:11:17.637 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-12-05T17:11:17.638 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-12-05T17:11:17.639 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2021-12-05T17:11:17.657 INFO:teuthology.lock.ops:Updating [smithi062.front.sepia.ceph.com]: set os type and version on server 2021-12-05T17:11:17.657 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2021-12-05T17:11:17.887 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2021-12-05T17:11:17.941 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="Red Hat Enterprise Linux" 2021-12-05T17:11:17.942 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="8.4 (Ootpa)" 2021-12-05T17:11:17.942 INFO:teuthology.orchestra.run.smithi062.stdout:ID="rhel" 2021-12-05T17:11:17.942 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="fedora" 2021-12-05T17:11:17.942 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="8.4" 2021-12-05T17:11:17.942 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el8" 2021-12-05T17:11:17.943 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-12-05T17:11:17.943 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2021-12-05T17:11:17.943 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-12-05T17:11:17.943 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://www.redhat.com/" 2021-12-05T17:11:17.944 INFO:teuthology.orchestra.run.smithi062.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-12-05T17:11:17.944 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-05T17:11:17.944 INFO:teuthology.orchestra.run.smithi062.stdout: 2021-12-05T17:11:17.944 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-05T17:11:17.945 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-12-05T17:11:17.945 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-12-05T17:11:17.945 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-12-05T17:11:17.946 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2021-12-05T17:11:18.043 INFO:teuthology.dispatcher.supervisor:Running job 6546666 2021-12-05T17:11:18.044 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_844d13778e221f3c1f3cb6626445c4bc2f71766d/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-12-05_15:06:27-rados-WIP_GBH_safe_fast_shutdown_1-distro-basic-smithi/6546666 --name benhanokh-2021-12-05_15:06:27-rados-WIP_GBH_safe_fast_shutdown_1-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} 3-scrub-overrides/{max-simultaneous-scrubs-2} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{rhel_8} thrashers/careful thrashosds-health workloads/small-objects} -- /home/teuthworker/archive/benhanokh-2021-12-05_15:06:27-rados-WIP_GBH_safe_fast_shutdown_1-distro-basic-smithi/6546666/orig.config.yaml 2021-12-05T17:11:18.047 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-12-05_15:06:27-rados-WIP_GBH_safe_fast_shutdown_1-distro-basic-smithi/6546666 2021-12-05T17:11:18.048 INFO:teuthology.dispatcher.supervisor:Job PID: 23527 2021-12-05T17:11:18.048 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-12-06T05:12:11.806 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-12-06T05:12:12.049 INFO:teuthology.kill:Killing Pids: {23527} 2021-12-06T05:12:13.224 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-12-06T05:12:13.225 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-12-06T05:12:13.225 INFO:teuthology.misc:Compressing logs... 2021-12-06T05:12:13.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2021-12-06T05:12:13.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2021-12-06T05:12:13.778 INFO:teuthology.misc:Compressing logs... 2021-12-06T05:12:14.653 INFO:teuthology.orchestra.run.smithi062.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-06T05:12:14.896 INFO:teuthology.orchestra.run.smithi037.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-06T05:12:15.425 INFO:teuthology.misc:Compressing logs... 2021-12-06T05:12:31.638 INFO:teuthology.kill:No teuthology processes running 2021-12-06T05:12:32.747 INFO:teuthology.kill:Nuking machines: ['smithi037', 'smithi062'] 2021-12-06T05:12:33.712 INFO:teuthology.kill:2021-12-06 05:12:33,712.712 INFO:teuthology.nuke:targets: 2021-12-06T05:12:33.713 INFO:teuthology.kill: smithi037.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDlq1fKtgrVh3SgyDUX++PRve5x9KJhtp/3HwFetBGdFs9QNuFs6PzmVHKNnofZ+spsKDGeiUTl1QGltamjcRjpCexOXUPpXNjV6sl9T9SbhwWyRpx66deqWvguFn64lMxi2SLwtLywX20isiBwGvEayGx4gSLDo/NFX+bMiJygDaasUp+jgSBk9olkpzVwIPM+19lXqiRGoYryelvFDRkgH8khD2BLj14w7D1+rmvowyS4OMZkmGWkibWVyU1sZPOtpJxipcJl5wvYHhW/6YaQe++Kg9+37BoZ5cSsg4IdMQiBzPrLonIS0ROe5euykBpC5hWNCNtutiDRDVakhC/Hg/7MnSJpoGI7lqcMsssUuCoLdzeZauGct3MdHRLbI2R/nl7cy5mthuJFtwu0gQPIuJ4ysUKeTkA3m1tRZmJTQLnKFBsPT9kVpWOAD7hCOo6PzBAO+d3+Ag9f5ZXlkFocyc+9SZNwcUntYRq9+/NSMbTVCAk6d7mhsUO3vTWv7/s= 2021-12-06T05:12:33.713 INFO:teuthology.kill: smithi062.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoHNt7c5TAJjQ1p9uPp12RA8WJNIHFOeX7v1Bmj7LhPb3Ayw8zMcA2LpwaeM6NpAOmiy1o0YulEijPnRZ7kIEsyKPQ3tgnk9okQBVeDd0aFlrL39TKs5PmO+iV8w3NJOv3I+AqSuStG4nFQNKAkjb1sllF73YqBUJyUhtSVBCek9ceJOnpmpFJkJhdIO/Pw4/9b8sE+/yHhdGtymHguvlriK6GKxw34dyT2gXPmntJ8zu2YCEVOpNMzccmCscunEi8rowZLc0LZ1+trumPFOfexsjdVSAkGW1Am7OJom+qtPMkmHJwKFffBu8Q8kv/nrOIvVrHCEMxFR2UxRMpQW/EZWHYEZ2yaslNYDH7D46I8+V+a2gzJaREDeWzyGVZE6Pr/pP8OuLVRV6ukmx7PH9BuPRta5YE5a7ra1mkdCStd8rgM+XkMLj2q56UrwTKCxVQiDCOvLjLLZ5Pxc0s1oPla2pD/RY5lNjoRFm44fHWOXoGqQaIq9fkyUN5xkwdZYs= 2021-12-06T05:12:33.738 INFO:teuthology.kill:2021-12-06 05:12:33,738.738 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-06T05:12:33.742 INFO:teuthology.kill:2021-12-06 05:12:33,741.741 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-06T05:12:33.815 INFO:teuthology.kill:2021-12-06 05:12:33,815.815 INFO:teuthology.orchestra.console:Power off smithi062 2021-12-06T05:12:33.828 INFO:teuthology.kill:2021-12-06 05:12:33,828.828 INFO:teuthology.orchestra.console:Power off smithi037 2021-12-06T05:12:46.298 INFO:teuthology.kill:2021-12-06 05:12:46,297.297 INFO:teuthology.orchestra.console:Power off for smithi062 completed 2021-12-06T05:12:46.303 INFO:teuthology.kill:2021-12-06 05:12:46,303.303 INFO:teuthology.orchestra.console:Power off for smithi037 completed 2021-12-06T05:12:46.454 INFO:teuthology.kill:2021-12-06 05:12:46,454.454 INFO:teuthology.lock.ops:unlocked: smithi062.front.sepia.ceph.com 2021-12-06T05:12:46.459 INFO:teuthology.kill:2021-12-06 05:12:46,459.459 INFO:teuthology.lock.ops:unlocked: smithi037.front.sepia.ceph.com 2021-12-06T05:14:46.767 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-12-06T05:14:46.867 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi037 but it was locked by another job: /home/teuthworker/archive/teuthology-2021-12-06_02:01:02-rbd-master-distro-default-smithi/6547182 2021-12-06T05:14:46.867 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi062 but it was locked by another job: /home/teuthworker/archive/teuthology-2021-12-06_02:01:02-rbd-master-distro-default-smithi/6547187