2022-10-22T14:25:56.096 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-22T14:25:56.256 INFO:teuthology.lock.ops:Start node 'smithi160.front.sepia.ceph.com' reimaging
2022-10-22T14:25:56.257 INFO:teuthology.lock.ops:Updating [smithi160.front.sepia.ceph.com]: reset os type and version on server
2022-10-22T14:25:56.258 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server
2022-10-22T14:25:56.288 INFO:teuthology.lock.ops:Node 'smithi160.front.sepia.ceph.com' reimaging is complete
2022-10-22T14:25:56.289 INFO:teuthology.lock.ops:Start node 'smithi161.front.sepia.ceph.com' reimaging
2022-10-22T14:25:56.289 INFO:teuthology.lock.ops:Updating [smithi161.front.sepia.ceph.com]: reset os type and version on server
2022-10-22T14:25:56.290 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server
2022-10-22T14:25:56.315 INFO:teuthology.lock.ops:Node 'smithi161.front.sepia.ceph.com' reimaging is complete
2022-10-22T14:25:56.503 INFO:teuthology.provision.fog.smithi160:Scheduling deploy of rhel 8.6
2022-10-22T14:25:56.520 INFO:teuthology.provision.fog.smithi161:Scheduling deploy of rhel 8.6
2022-10-22T14:25:56.858 INFO:teuthology.orchestra.console:Power off smithi160
2022-10-22T14:25:56.877 INFO:teuthology.orchestra.console:Power off smithi161
2022-10-22T14:26:05.237 INFO:teuthology.orchestra.console:Power off for smithi160 completed
2022-10-22T14:26:05.339 INFO:teuthology.orchestra.console:Power on smithi160
2022-10-22T14:26:09.359 INFO:teuthology.orchestra.console:Power off for smithi161 completed
2022-10-22T14:26:09.461 INFO:teuthology.orchestra.console:Power on smithi161
2022-10-22T14:26:09.593 INFO:teuthology.orchestra.console:Power on for smithi160 completed
2022-10-22T14:26:09.696 INFO:teuthology.provision.fog.smithi160:Waiting for deploy to finish
2022-10-22T14:26:13.721 INFO:teuthology.orchestra.console:Power on for smithi161 completed
2022-10-22T14:26:13.822 INFO:teuthology.provision.fog.smithi161:Waiting for deploy to finish
2022-10-22T14:31:03.074 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-10-22T14:31:03.282 INFO:teuthology.provision.fog.smithi160:Node is ready
2022-10-22T14:31:03.342 INFO:teuthology.orchestra.run.smithi160.stdout:smithi160.front.sepia.ceph.com
2022-10-22T14:31:03.407 INFO:teuthology.orchestra.run.smithi160.stdout:172.21.15.160 smithi160.front.sepia.ceph.com smithi160
2022-10-22T14:31:03.765 INFO:teuthology.provision.fog.smithi160:Deploy complete!
2022-10-22T14:31:07.031 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-10-22T14:31:07.290 INFO:teuthology.provision.fog.smithi161:Node is ready
2022-10-22T14:31:07.310 INFO:teuthology.orchestra.run.smithi161.stdout:smithi161.front.sepia.ceph.com
2022-10-22T14:31:07.369 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.15.161 smithi161.front.sepia.ceph.com smithi161
2022-10-22T14:31:07.720 INFO:teuthology.provision.fog.smithi161:Deploy complete!
2022-10-22T14:31:07.935 INFO:teuthology.lock.ops:Checking smithi160.front.sepia.ceph.com
2022-10-22T14:31:07.936 INFO:teuthology.lock.ops:New key found. Updating...
2022-10-22T14:31:07.962 INFO:teuthology.lock.ops:Checking smithi161.front.sepia.ceph.com
2022-10-22T14:31:07.962 INFO:teuthology.lock.ops:New key found. Updating...
2022-10-22T14:31:07.992 INFO:teuthology.lock.ops:Updating [smithi160.front.sepia.ceph.com]: set os type and version on server
2022-10-22T14:31:07.993 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com'
2022-10-22T14:31:08.174 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64
2022-10-22T14:31:08.239 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="Red Hat Enterprise Linux"
2022-10-22T14:31:08.239 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="8.6 (Ootpa)"
2022-10-22T14:31:08.239 INFO:teuthology.orchestra.run.smithi160.stdout:ID="rhel"
2022-10-22T14:31:08.240 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE="fedora"
2022-10-22T14:31:08.240 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="8.6"
2022-10-22T14:31:08.240 INFO:teuthology.orchestra.run.smithi160.stdout:PLATFORM_ID="platform:el8"
2022-10-22T14:31:08.240 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2022-10-22T14:31:08.241 INFO:teuthology.orchestra.run.smithi160.stdout:ANSI_COLOR="0;31"
2022-10-22T14:31:08.241 INFO:teuthology.orchestra.run.smithi160.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2022-10-22T14:31:08.241 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://www.redhat.com/"
2022-10-22T14:31:08.242 INFO:teuthology.orchestra.run.smithi160.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-10-22T14:31:08.242 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-10-22T14:31:08.242 INFO:teuthology.orchestra.run.smithi160.stdout:
2022-10-22T14:31:08.243 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-10-22T14:31:08.243 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2022-10-22T14:31:08.243 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-10-22T14:31:08.243 INFO:teuthology.orchestra.run.smithi160.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2022-10-22T14:31:08.245 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server
2022-10-22T14:31:08.267 INFO:teuthology.lock.ops:Updating [smithi161.front.sepia.ceph.com]: set os type and version on server
2022-10-22T14:31:08.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com'
2022-10-22T14:31:08.549 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64
2022-10-22T14:31:08.623 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="Red Hat Enterprise Linux"
2022-10-22T14:31:08.623 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="8.6 (Ootpa)"
2022-10-22T14:31:08.624 INFO:teuthology.orchestra.run.smithi161.stdout:ID="rhel"
2022-10-22T14:31:08.624 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="fedora"
2022-10-22T14:31:08.624 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="8.6"
2022-10-22T14:31:08.624 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el8"
2022-10-22T14:31:08.625 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2022-10-22T14:31:08.625 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31"
2022-10-22T14:31:08.625 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2022-10-22T14:31:08.625 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://www.redhat.com/"
2022-10-22T14:31:08.626 INFO:teuthology.orchestra.run.smithi161.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-10-22T14:31:08.626 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-10-22T14:31:08.627 INFO:teuthology.orchestra.run.smithi161.stdout:
2022-10-22T14:31:08.627 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-10-22T14:31:08.627 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2022-10-22T14:31:08.627 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-10-22T14:31:08.627 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2022-10-22T14:31:08.628 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server
2022-10-22T14:31:08.761 INFO:teuthology.dispatcher.supervisor:Running job 7077360
2022-10-22T14:31:08.762 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-10-22_14:23:03-upgrade:quincy-x-main-distro-default-smithi/7077360 --name teuthology-2022-10-22_14:23:03-upgrade:quincy-x-main-distro-default-smithi --description upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0 0-roles 1-start 2-first-half-tasks/radosbench 3-stress-tasks/{radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 4-second-half-tasks/radosbench mon_election/classic} -- /home/teuthworker/archive/teuthology-2022-10-22_14:23:03-upgrade:quincy-x-main-distro-default-smithi/7077360/orig.config.yaml
2022-10-22T14:31:08.770 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-10-22_14:23:03-upgrade:quincy-x-main-distro-default-smithi/7077360
2022-10-22T14:31:08.771 INFO:teuthology.dispatcher.supervisor:Job PID: 30342
2022-10-22T14:31:08.772 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-10-23T02:32:03.016 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-10-23T02:32:03.268 INFO:teuthology.kill:Killing Pids: {30342}
2022-10-23T02:32:04.694 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2022-10-23T02:32:04.695 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
2022-10-23T02:32:04.695 INFO:teuthology.misc:Compressing logs...
2022-10-23T02:32:04.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com'
2022-10-23T02:32:04.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com'
2022-10-23T02:32:05.406 INFO:teuthology.misc:Compressing logs...
2022-10-23T02:32:05.602 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-10-23T02:32:05.715 INFO:teuthology.orchestra.run.smithi160.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-10-23T02:32:05.868 INFO:teuthology.misc:Compressing logs...
2022-10-23T02:33:07.035 INFO:teuthology.orchestra.run.smithi160.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-10-23T02:33:11.780 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-10-23T02:33:59.976 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-10-23T02:34:22.460 INFO:teuthology.orchestra.run.smithi160.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-10-23T02:34:56.975 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-10-23T02:35:38.220 INFO:teuthology.orchestra.run.smithi160.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-10-23T02:35:50.555 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-10-23T02:35:59.664 INFO:teuthology.orchestra.run.smithi161.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2022-10-23T02:36:41.786 INFO:teuthology.orchestra.run.smithi160.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-10-23T02:36:55.370 INFO:teuthology.orchestra.run.smithi160.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2022-10-23T02:37:24.224 INFO:teuthology.orchestra.run.smithi160.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2022-10-23T02:37:30.451 INFO:teuthology.orchestra.run.smithi160.stderr:gzip: /var/log/ceph/fa1af2f4-5217-11ed-8438-001a4aab830c/ceph-mgr.y.log: file size changed while zipping
2022-10-23T02:39:27.090 INFO:teuthology.kill:No teuthology processes running
2022-10-23T02:39:28.383 INFO:teuthology.kill:Nuking machines: ['smithi160', 'smithi161']
2022-10-23T02:39:29.483 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
2022-10-23T02:39:29.484 INFO:teuthology.kill:  from cryptography.hazmat.backends import default_backend
2022-10-23T02:39:29.484 INFO:teuthology.kill:2022-10-23 02:39:29,483.483 INFO:teuthology.nuke:targets:
2022-10-23T02:39:29.485 INFO:teuthology.kill:  smithi160.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCji6kEI+TMiBhHWdg1XhwIom6C/peht88zsMYuAgCLYpjw+LpZOhQYVm53ZhFbfzNt40+kBMh2pmXAB68qrHi5E1e6o9MrRsdiShwFJUccSACscy63ZfxonVnexGmMAh+qtZwFdQ78qfbN0cuKh5YhLX9s7cj5/XVjGOIfqQrRCfpcy0wl9YThU3Hbv2ZNMrgneU4TwNh9/t8ZWClIbHvsQBsyP1YkI5ixzyS9EOj04nOT4pM60mJDEGVZgZWUrCPpPJYsAlsgMQmtYUic6Jq2SOaJyzzwUstGTb//pzGf+DsECoNp1QmbLgnLLllJR3xE9LrTFq+5co/+ZugUT4EJ3+JZxMH5Zt434K1tdTY6slTdZ/WaBY1cabYEuCPHhqfWRCsYwG+ui9HiVKyH9s0alfACLOY1uRJ69CuEy77Jx8gin1s/1G3zdVuNeLGJmJBDV7dmCyXc6+ehm35ZNyUcio0APGmPUg3nagsDqVl2frvSjccSKWizeOz/WzfQghc=
2022-10-23T02:39:29.485 INFO:teuthology.kill:  smithi161.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4VPhXzijCRnBu5gu7mmFYtPZQtPHpX7pbGvWfSuyotbWMbj2VhezCf+I472GrCHOUfXjMe/n99hs4n6ds5jnuoLiOJdDhzPSfXaf9pl+syaGqxcWSskqo/3pdEbOWNBxvo0319BgtKpny9czlte5smSGaFJVgUxcuZzgI9exIVXq59H0dOH03L1qtq3sFYPgsmU6jcDDzIoGRDkDAgUzdrWM/BGrhM6ZXsVFVsKhrJ+Qs7ZjhBKAEZs+5/iOkjFqtAdXniT8E1V+aHEQKjnHrJZY7hxx1JA3uzYXjxjlKQmET5oiTyGwQlpSemxpQJOV63rQT43nEakw6gDhyMqcttMPfHMwNfOYzKwOk4I127zt+fWHWCVhlt9s6Inh0UkHNoQwdBSbw/P7Q14gyPxZQANwqLcnkVT+ceg0TFy7XuSkDiS5IozvO8otXu9FQ2U7FsKPbmpCaEzMCHkiukXCjJoQhgS9mr0GAqTxZzf96W8bFQTzsDKuJkmQxJJHEwFk=
2022-10-23T02:39:29.485 INFO:teuthology.kill:2022-10-23 02:39:29,483.483 INFO:teuthology.nuke:Checking targets against current locks
2022-10-23T02:39:29.534 INFO:teuthology.kill:2022-10-23 02:39:29,534.534 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-10-23T02:39:29.564 INFO:teuthology.kill:2022-10-23 02:39:29,564.564 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-10-23T02:39:29.619 INFO:teuthology.kill:2022-10-23 02:39:29,618.618 INFO:teuthology.orchestra.console:Power off smithi160
2022-10-23T02:39:29.658 INFO:teuthology.kill:2022-10-23 02:39:29,657.657 INFO:teuthology.orchestra.console:Power off smithi161
2022-10-23T02:39:38.016 INFO:teuthology.kill:2022-10-23 02:39:38,016.016 INFO:teuthology.orchestra.console:Power off for smithi161 completed
2022-10-23T02:39:38.169 INFO:teuthology.kill:2022-10-23 02:39:38,168.168 INFO:teuthology.lock.ops:unlocked: smithi161.front.sepia.ceph.com
2022-10-23T02:39:42.098 INFO:teuthology.kill:2022-10-23 02:39:42,098.098 INFO:teuthology.orchestra.console:Power off for smithi160 completed
2022-10-23T02:39:42.267 INFO:teuthology.kill:2022-10-23 02:39:42,266.266 INFO:teuthology.lock.ops:unlocked: smithi160.front.sepia.ceph.com
2022-10-23T02:41:42.700 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-10-23T02:41:42.844 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi160 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-10-17_02:01:03-rbd-quincy-distro-default-smithi/7069336
2022-10-23T02:41:42.844 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi161 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-10-17_02:01:03-rbd-quincy-distro-default-smithi/7069336