2023-06-03T14:24:43.320 INFO:root:teuthology version: 0.0.1.dev116+g62814ab
2023-06-03T14:24:43.475 INFO:teuthology.lock.ops:Start node 'smithi070.front.sepia.ceph.com' reimaging
2023-06-03T14:24:43.476 INFO:teuthology.lock.ops:Updating [smithi070.front.sepia.ceph.com]: reset os type and version on server
2023-06-03T14:24:43.476 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server
2023-06-03T14:24:43.498 INFO:teuthology.lock.ops:Start node 'smithi139.front.sepia.ceph.com' reimaging
2023-06-03T14:24:43.498 INFO:teuthology.lock.ops:Updating [smithi139.front.sepia.ceph.com]: reset os type and version on server
2023-06-03T14:24:43.498 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server
2023-06-03T14:24:43.665 INFO:teuthology.provision.fog.smithi070:Scheduling deploy of rhel 8.6
2023-06-03T14:24:44.124 INFO:teuthology.orchestra.console:Power off smithi070
2023-06-03T14:24:44.765 INFO:teuthology.provision.fog.smithi139:Scheduling deploy of rhel 8.6
2023-06-03T14:24:45.207 INFO:teuthology.orchestra.console:Power off smithi139
2023-06-03T14:24:52.493 INFO:teuthology.orchestra.console:Power off for smithi070 completed
2023-06-03T14:24:52.594 INFO:teuthology.orchestra.console:Power on smithi070
2023-06-03T14:24:53.573 INFO:teuthology.orchestra.console:Power off for smithi139 completed
2023-06-03T14:24:53.674 INFO:teuthology.orchestra.console:Power on smithi139
2023-06-03T14:24:57.925 INFO:teuthology.orchestra.console:Power on for smithi139 completed
2023-06-03T14:24:58.027 INFO:teuthology.provision.fog.smithi139:Waiting for deploy to finish
2023-06-03T14:25:00.977 INFO:teuthology.orchestra.console:Power on for smithi070 completed
2023-06-03T14:25:01.079 INFO:teuthology.provision.fog.smithi070:Waiting for deploy to finish
2023-06-03T14:29:24.528 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-06-03T14:29:29.104 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.70
2023-06-03T14:29:33.584 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-06-03T14:29:38.160 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.70
2023-06-03T14:29:42.640 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-06-03T14:29:47.216 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.70
2023-06-03T14:29:51.696 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-06-03T14:29:56.272 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.70
2023-06-03T14:30:00.752 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-06-03T14:30:05.332 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.70
2023-06-03T14:30:09.808 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-06-03T14:30:14.384 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.70
2023-06-03T14:30:34.352 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-06-03T14:30:43.408 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.139
2023-06-03T14:30:50.697 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-06-03T14:30:51.002 INFO:teuthology.provision.fog.smithi139:Node is ready
2023-06-03T14:30:51.017 INFO:teuthology.orchestra.run.smithi139.stdout:smithi022
2023-06-03T14:30:51.073 INFO:teuthology.orchestra.run.smithi139.stdout:172.21.15.22 smithi022.front.sepia.ceph.com smithi022
2023-06-03T14:30:51.467 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="Red Hat Enterprise Linux"
2023-06-03T14:30:51.467 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="8.6 (Ootpa)"
2023-06-03T14:30:51.467 INFO:teuthology.orchestra.run.smithi139.stdout:ID="rhel"
2023-06-03T14:30:51.467 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="fedora"
2023-06-03T14:30:51.467 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="8.6"
2023-06-03T14:30:51.468 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el8"
2023-06-03T14:30:51.468 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-06-03T14:30:51.468 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31"
2023-06-03T14:30:51.468 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-06-03T14:30:51.468 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://www.redhat.com/"
2023-06-03T14:30:51.468 INFO:teuthology.orchestra.run.smithi139.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-06-03T14:30:51.468 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-06-03T14:30:51.469 INFO:teuthology.orchestra.run.smithi139.stdout:
2023-06-03T14:30:51.469 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-06-03T14:30:51.469 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-06-03T14:30:51.469 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-06-03T14:30:51.469 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-06-03T14:30:51.470 INFO:teuthology.provision.fog.smithi139:Deploy complete!
2023-06-03T14:31:20.420 WARNING:teuthology.provision.fog:timed out
2023-06-03T14:31:26.749 INFO:teuthology.orchestra.run:Running command with timeout 600
2023-06-03T14:31:27.033 INFO:teuthology.provision.fog.smithi070:Node is ready
2023-06-03T14:31:27.051 INFO:teuthology.orchestra.run.smithi070.stdout:smithi070
2023-06-03T14:31:27.109 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.70 smithi070.front.sepia.ceph.com smithi070
2023-06-03T14:31:27.519 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="Red Hat Enterprise Linux"
2023-06-03T14:31:27.519 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="8.6 (Ootpa)"
2023-06-03T14:31:27.519 INFO:teuthology.orchestra.run.smithi070.stdout:ID="rhel"
2023-06-03T14:31:27.519 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="fedora"
2023-06-03T14:31:27.520 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="8.6"
2023-06-03T14:31:27.520 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el8"
2023-06-03T14:31:27.520 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-06-03T14:31:27.520 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31"
2023-06-03T14:31:27.520 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-06-03T14:31:27.520 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://www.redhat.com/"
2023-06-03T14:31:27.520 INFO:teuthology.orchestra.run.smithi070.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-06-03T14:31:27.521 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-06-03T14:31:27.521 INFO:teuthology.orchestra.run.smithi070.stdout:
2023-06-03T14:31:27.521 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-06-03T14:31:27.521 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-06-03T14:31:27.521 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-06-03T14:31:27.521 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-06-03T14:31:27.521 INFO:teuthology.provision.fog.smithi070:Deploy complete!
2023-06-03T14:31:27.693 INFO:teuthology.lock.ops:Checking smithi070.front.sepia.ceph.com
2023-06-03T14:31:27.694 INFO:teuthology.lock.ops:New key found. Updating...
2023-06-03T14:31:27.716 INFO:teuthology.lock.ops:Checking smithi139.front.sepia.ceph.com
2023-06-03T14:31:27.717 INFO:teuthology.lock.ops:New key found. Updating...
2023-06-03T14:31:27.742 INFO:teuthology.lock.ops:Updating [smithi070.front.sepia.ceph.com]: set os type and version on server
2023-06-03T14:31:27.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com'
2023-06-03T14:31:27.927 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64
2023-06-03T14:31:27.983 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="Red Hat Enterprise Linux"
2023-06-03T14:31:27.983 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="8.6 (Ootpa)"
2023-06-03T14:31:27.983 INFO:teuthology.orchestra.run.smithi070.stdout:ID="rhel"
2023-06-03T14:31:27.983 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="fedora"
2023-06-03T14:31:27.983 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="8.6"
2023-06-03T14:31:27.983 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el8"
2023-06-03T14:31:27.984 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-06-03T14:31:27.984 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31"
2023-06-03T14:31:27.984 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-06-03T14:31:27.984 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://www.redhat.com/"
2023-06-03T14:31:27.984 INFO:teuthology.orchestra.run.smithi070.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-06-03T14:31:27.984 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-06-03T14:31:27.984 INFO:teuthology.orchestra.run.smithi070.stdout:
2023-06-03T14:31:27.985 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-06-03T14:31:27.985 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-06-03T14:31:27.985 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-06-03T14:31:27.985 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-06-03T14:31:27.985 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server
2023-06-03T14:31:28.008 INFO:teuthology.lock.ops:Updating [smithi139.front.sepia.ceph.com]: set os type and version on server
2023-06-03T14:31:28.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com'
2023-06-03T14:31:28.260 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64
2023-06-03T14:31:28.316 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="Red Hat Enterprise Linux"
2023-06-03T14:31:28.316 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="8.6 (Ootpa)"
2023-06-03T14:31:28.317 INFO:teuthology.orchestra.run.smithi139.stdout:ID="rhel"
2023-06-03T14:31:28.317 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="fedora"
2023-06-03T14:31:28.317 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="8.6"
2023-06-03T14:31:28.317 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el8"
2023-06-03T14:31:28.317 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2023-06-03T14:31:28.317 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31"
2023-06-03T14:31:28.317 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2023-06-03T14:31:28.318 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://www.redhat.com/"
2023-06-03T14:31:28.318 INFO:teuthology.orchestra.run.smithi139.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2023-06-03T14:31:28.318 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2023-06-03T14:31:28.318 INFO:teuthology.orchestra.run.smithi139.stdout:
2023-06-03T14:31:28.318 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2023-06-03T14:31:28.318 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2023-06-03T14:31:28.318 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2023-06-03T14:31:28.319 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2023-06-03T14:31:28.319 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server
2023-06-03T14:31:28.448 INFO:teuthology.dispatcher.supervisor:Running job 7295487
2023-06-03T14:31:28.449 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_62814aba234085f597115b9ba9bdc56af117de9e/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2023-06-03_14:23:09-upgrade:quincy-x-reef-distro-default-smithi/7295487 --name teuthology-2023-06-03_14:23:09-upgrade:quincy-x-reef-distro-default-smithi --description upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} -- /home/teuthworker/archive/teuthology-2023-06-03_14:23:09-upgrade:quincy-x-reef-distro-default-smithi/7295487/orig.config.yaml
2023-06-03T14:31:28.451 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2023-06-03_14:23:09-upgrade:quincy-x-reef-distro-default-smithi/7295487
2023-06-03T14:31:28.452 INFO:teuthology.dispatcher.supervisor:Job PID: 13151
2023-06-03T14:31:28.452 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2023-06-04T02:32:29.993 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2023-06-04T02:32:30.173 INFO:teuthology.kill:Killing Pids: {13151}
2023-06-04T02:32:31.203 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2023-06-04T02:32:31.204 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
2023-06-04T02:32:31.204 INFO:teuthology.misc:Compressing logs...
2023-06-04T02:32:31.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com'
2023-06-04T02:32:31.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com'
2023-06-04T02:32:31.451 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2023-06-04T02:32:31.452 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2023-06-04T02:32:31.452 INFO:teuthology.orchestra.run.smithi070.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2023-06-04T02:32:31.453 INFO:teuthology.orchestra.run.smithi070.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:	 84.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2023-06-04T02:32:31.592 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log
2023-06-04T02:32:31.592 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log
2023-06-04T02:32:31.593 INFO:teuthology.orchestra.run.smithi139.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log:	  0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2023-06-04T02:32:31.594 INFO:teuthology.orchestra.run.smithi139.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log:	 84.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2023-06-04T02:32:32.098 INFO:teuthology.kill:No teuthology processes running
2023-06-04T02:32:32.911 INFO:teuthology.kill:Nuking machines: ['smithi070', 'smithi139']
2023-06-04T02:32:33.601 INFO:teuthology.kill:2023-06-04 02:32:33,601.601 INFO:teuthology.nuke:targets:
2023-06-04T02:32:33.602 ERROR:teuthology.dispatcher.supervisor:Failed to kill job and unlock machines
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_62814aba234085f597115b9ba9bdc56af117de9e/teuthology/dispatcher/supervisor.py", line 320, in run_with_watchdog
    kill.kill_job(
  File "/home/teuthworker/src/git.ceph.com_teuthology_62814aba234085f597115b9ba9bdc56af117de9e/teuthology/kill.py", line 101, in kill_job
    nuke_targets(targets, owner)
  File "/home/teuthworker/src/git.ceph.com_teuthology_62814aba234085f597115b9ba9bdc56af117de9e/teuthology/kill.py", line 276, in nuke_targets
    sys.stdout.flush()
OSError: [Errno 5] Input/output error
2023-06-04T02:34:33.874 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2023-06-04T02:34:33.997 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi070 but it was locked by another job: /home/teuthworker/archive/dis-2023-06-03_23:51:22-krbd-main-wip-exclusive-lock-snapc-default-smithi/7295899
2023-06-04T02:34:33.997 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi139 but it was locked by another job: /home/teuthworker/archive/dis-2023-06-03_23:51:22-krbd-main-wip-exclusive-lock-snapc-default-smithi/7295899