2021-12-08T23:25:31.183 INFO:root:teuthology version: 1.1.0-95a7d479 2021-12-08T23:25:31.311 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi141.front.sepia.ceph.com' reimaging 2021-12-08T23:25:31.312 INFO:teuthology.lock.ops:Updating [ubuntu@smithi141.front.sepia.ceph.com]: reset os type and version on server 2021-12-08T23:25:31.312 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server 2021-12-08T23:25:31.331 INFO:teuthology.lock.ops:Node 'ubuntu@smithi141.front.sepia.ceph.com' reimaging is complete 2021-12-08T23:25:31.332 INFO:teuthology.lock.ops:Start node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging 2021-12-08T23:25:31.333 INFO:teuthology.lock.ops:Updating [ubuntu@smithi170.front.sepia.ceph.com]: reset os type and version on server 2021-12-08T23:25:31.333 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2021-12-08T23:25:31.353 INFO:teuthology.lock.ops:Node 'ubuntu@smithi170.front.sepia.ceph.com' reimaging is complete 2021-12-08T23:25:31.545 INFO:teuthology.provision.fog.smithi141:Scheduling deploy of rhel 8.4 2021-12-08T23:25:31.563 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of rhel 8.4 2021-12-08T23:25:31.899 INFO:teuthology.orchestra.console:Power off smithi141 2021-12-08T23:25:31.919 INFO:teuthology.orchestra.console:Power off smithi170 2021-12-08T23:25:44.401 INFO:teuthology.orchestra.console:Power off for smithi141 completed 2021-12-08T23:25:44.404 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2021-12-08T23:25:44.502 INFO:teuthology.orchestra.console:Power on smithi141 2021-12-08T23:25:44.510 INFO:teuthology.orchestra.console:Power on smithi170 2021-12-08T23:25:48.752 INFO:teuthology.orchestra.console:Power on for smithi141 completed 2021-12-08T23:25:48.854 INFO:teuthology.provision.fog.smithi141:Waiting for deploy to finish 2021-12-08T23:25:52.868 INFO:teuthology.orchestra.console:Power on for smithi170 completed 2021-12-08T23:25:52.969 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish 2021-12-08T23:28:20.165 ERROR:teuthology.orchestra.connection:Error authenticating with smithi141.front.sepia.ceph.com: Authentication failed. 2021-12-08T23:30:33.409 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-08T23:30:33.629 INFO:teuthology.provision.fog.smithi141:Node is ready 2021-12-08T23:30:33.645 INFO:teuthology.orchestra.run.smithi141.stdout:smithi141.front.sepia.ceph.com 2021-12-08T23:30:33.701 INFO:teuthology.orchestra.run.smithi141.stdout:172.21.15.141 smithi141.front.sepia.ceph.com smithi141 2021-12-08T23:30:34.025 INFO:teuthology.provision.fog.smithi141:Deploy complete! 2021-12-08T23:30:42.573 INFO:teuthology.orchestra.run:Running command with timeout 600 2021-12-08T23:30:42.779 INFO:teuthology.provision.fog.smithi170:Node is ready 2021-12-08T23:30:42.795 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com 2021-12-08T23:30:42.850 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170 2021-12-08T23:30:43.168 INFO:teuthology.provision.fog.smithi170:Deploy complete! 2021-12-08T23:30:43.355 INFO:teuthology.lock.ops:Checking smithi141.front.sepia.ceph.com 2021-12-08T23:30:43.356 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-08T23:30:43.378 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com 2021-12-08T23:30:43.379 INFO:teuthology.lock.ops:New key found. Updating... 2021-12-08T23:30:43.396 INFO:teuthology.lock.ops:Updating [smithi141.front.sepia.ceph.com]: set os type and version on server 2021-12-08T23:30:43.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2021-12-08T23:30:43.633 INFO:teuthology.orchestra.run.smithi141.stdout:x86_64 2021-12-08T23:30:43.689 INFO:teuthology.orchestra.run.smithi141.stdout:NAME="Red Hat Enterprise Linux" 2021-12-08T23:30:43.689 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION="8.4 (Ootpa)" 2021-12-08T23:30:43.689 INFO:teuthology.orchestra.run.smithi141.stdout:ID="rhel" 2021-12-08T23:30:43.689 INFO:teuthology.orchestra.run.smithi141.stdout:ID_LIKE="fedora" 2021-12-08T23:30:43.690 INFO:teuthology.orchestra.run.smithi141.stdout:VERSION_ID="8.4" 2021-12-08T23:30:43.690 INFO:teuthology.orchestra.run.smithi141.stdout:PLATFORM_ID="platform:el8" 2021-12-08T23:30:43.690 INFO:teuthology.orchestra.run.smithi141.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-12-08T23:30:43.691 INFO:teuthology.orchestra.run.smithi141.stdout:ANSI_COLOR="0;31" 2021-12-08T23:30:43.691 INFO:teuthology.orchestra.run.smithi141.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-12-08T23:30:43.691 INFO:teuthology.orchestra.run.smithi141.stdout:HOME_URL="https://www.redhat.com/" 2021-12-08T23:30:43.691 INFO:teuthology.orchestra.run.smithi141.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-12-08T23:30:43.692 INFO:teuthology.orchestra.run.smithi141.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-08T23:30:43.692 INFO:teuthology.orchestra.run.smithi141.stdout: 2021-12-08T23:30:43.692 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-08T23:30:43.692 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-12-08T23:30:43.693 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-12-08T23:30:43.693 INFO:teuthology.orchestra.run.smithi141.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-12-08T23:30:43.695 INFO:teuthology.lock.ops:Updating smithi141.front.sepia.ceph.com on lock server 2021-12-08T23:30:43.713 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server 2021-12-08T23:30:43.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2021-12-08T23:30:43.935 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2021-12-08T23:30:43.992 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="Red Hat Enterprise Linux" 2021-12-08T23:30:43.992 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8.4 (Ootpa)" 2021-12-08T23:30:43.993 INFO:teuthology.orchestra.run.smithi170.stdout:ID="rhel" 2021-12-08T23:30:43.993 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="fedora" 2021-12-08T23:30:43.993 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8.4" 2021-12-08T23:30:43.993 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8" 2021-12-08T23:30:43.994 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2021-12-08T23:30:43.994 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2021-12-08T23:30:43.994 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2021-12-08T23:30:43.994 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://www.redhat.com/" 2021-12-08T23:30:43.995 INFO:teuthology.orchestra.run.smithi170.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2021-12-08T23:30:43.995 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2021-12-08T23:30:43.995 INFO:teuthology.orchestra.run.smithi170.stdout: 2021-12-08T23:30:43.995 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2021-12-08T23:30:43.995 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2021-12-08T23:30:43.996 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2021-12-08T23:30:43.996 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2021-12-08T23:30:43.997 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2021-12-08T23:30:44.095 INFO:teuthology.dispatcher.supervisor:Running job 6552943 2021-12-08T23:30:44.095 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_95a7d4799b562f3bbb5ec66107094963abd62fa1/virtualenv/bin/teuthology -v --owner scheduled_benhanokh@teuthology --archive /home/teuthworker/archive/benhanokh-2021-12-08_20:57:58-rados-WIP_GBH_safe_fast_shutdown_6-distro-basic-smithi/6552943 --name benhanokh-2021-12-08_20:57:58-rados-WIP_GBH_safe_fast_shutdown_6-distro-basic-smithi --description rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/dedup-io-snaps} -- /home/teuthworker/archive/benhanokh-2021-12-08_20:57:58-rados-WIP_GBH_safe_fast_shutdown_6-distro-basic-smithi/6552943/orig.config.yaml 2021-12-08T23:30:44.099 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/benhanokh-2021-12-08_20:57:58-rados-WIP_GBH_safe_fast_shutdown_6-distro-basic-smithi/6552943 2021-12-08T23:30:44.100 INFO:teuthology.dispatcher.supervisor:Job PID: 15791 2021-12-08T23:30:44.100 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2021-12-09T11:31:04.948 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2021-12-09T11:31:05.112 INFO:teuthology.kill:Killing Pids: {15791} 2021-12-09T11:31:06.316 INFO:teuthology.task.internal:roles: ubuntu@smithi141.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2021-12-09T11:31:06.317 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2021-12-09T11:31:06.317 INFO:teuthology.misc:Compressing logs... 2021-12-09T11:31:06.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi141.front.sepia.ceph.com' 2021-12-09T11:31:06.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2021-12-09T11:31:06.849 INFO:teuthology.misc:Compressing logs... 2021-12-09T11:31:07.629 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-09T11:31:07.831 INFO:teuthology.orchestra.run.smithi141.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2021-12-09T11:31:08.352 INFO:teuthology.misc:Compressing logs... 2021-12-09T11:31:35.322 INFO:teuthology.kill:No teuthology processes running 2021-12-09T11:31:36.542 INFO:teuthology.kill:Nuking machines: ['smithi141', 'smithi170'] 2021-12-09T11:31:37.517 INFO:teuthology.kill:2021-12-09 11:31:37,516.516 INFO:teuthology.nuke:targets: 2021-12-09T11:31:37.518 INFO:teuthology.kill: smithi141.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAMdGzHdYcAygYte6bdLKGkxF6TmIufsn6TbAmy5W4zl/+YPLUBXawzDMDNVKEu0pdRmoZXHZ8V803Rg8dIpNlywDTFvafJgOu1gJPWnmQ6U8XFBOszwLhsQSogAXmzMLaCn37nh7xO8hVdFki8bYVGMZHHjkF3y+Ff4G2+Yj0vDQALhLuqdrz5d5kRhzgKB+xZTaLQm+xRgUqWALZWxxiW2NDTHzPJwJV0X1fB/iEC5tLJuDUVJj3WvVBRu9DNylzVqaL58Gr8+6MVJYv0uuOux2tiJAuZO39sUR7LwnJ6KlmnpqbovrbVnONO0fxSDEw6im5WsUcmUUuAf0osoc76kmc9ABFtwC96yZs4gKFiFU6WdBMR5X8awmYnMxq0TpTkbN2J0B0oLqIi99Yn+xuXl9+EftLn6gXAmKdhWOftswIeK1SW7D0lPlpjo7aptKeAgQye7zDkUBwXDy52/n4pxE+BTcmq8/OX9PlE7KMv9gzA6G/2VlTYTYumClYYhM= 2021-12-09T11:31:37.518 INFO:teuthology.kill: smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDeBg04rKbBqZusfHOPGWwNAYvkfs/XXwgSuQV/aHdAV9IScQV95sX6nB1onFDDjyTwRNJUeQzneJG6kFbRYgceWL615uReWsRiFgen7xibA6EZlKHZoMrYN0MCI80FF1Qk9QGNJWZ89mZqB7gSLb4a64DADwq3liLv+7qNMwwGbNOU54pkNnjUyD0Z2MhcQX3zf7s9nSz7hML0nsBt9EkFqayPgmjxiYB531UP6YM2oV6ayAhb9VYBW9s/1lZQnEBiXmNCzCV8P2MIEdFhznw/wxeLdRgcKddfBlboYPz1wmh8liEo0+GPaY7D/fPyYy04QzdvqGFxFMvO8WIju9+RU2aBQ+AabaCCaTGeAAOdLuqrHAdRRLVvtR2xbfHGAqw47EZPl8WcGrQbGbLe5zYkgBG+wh3MutZ2wQZ/MxY0MD0PbAT9BUWZjGEQNifgpgyCxkXzkR3LGzugE3XlL00FTE2EmMzrk05ATCNgdcVmfRqDjGdM8Ghb6ePNjjvlvCs= 2021-12-09T11:31:37.544 INFO:teuthology.kill:2021-12-09 11:31:37,543.543 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-09T11:31:37.547 INFO:teuthology.kill:2021-12-09 11:31:37,547.547 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-12-09T11:31:37.623 INFO:teuthology.kill:2021-12-09 11:31:37,623.623 INFO:teuthology.orchestra.console:Power off smithi141 2021-12-09T11:31:37.634 INFO:teuthology.kill:2021-12-09 11:31:37,634.634 INFO:teuthology.orchestra.console:Power off smithi170 2021-12-09T11:31:45.992 INFO:teuthology.kill:2021-12-09 11:31:45,992.992 INFO:teuthology.orchestra.console:Power off for smithi141 completed 2021-12-09T11:31:45.997 INFO:teuthology.kill:2021-12-09 11:31:45,997.997 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2021-12-09T11:31:46.143 INFO:teuthology.kill:2021-12-09 11:31:46,143.143 INFO:teuthology.lock.ops:unlocked: smithi170.front.sepia.ceph.com 2021-12-09T11:31:46.147 INFO:teuthology.kill:2021-12-09 11:31:46,146.146 INFO:teuthology.lock.ops:unlocked: smithi141.front.sepia.ceph.com 2021-12-09T11:33:46.365 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2021-12-09T11:33:46.467 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi141 but it was locked by another job: /home/teuthworker/archive/ideepika-2021-12-09_11:27:25-rbd-wip-deepika2-testing-pacific-distro-default-smithi/6554663