2022-09-30T04:38:05.447 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-09-30T04:38:05.597 INFO:teuthology.lock.ops:Start node 'smithi179.front.sepia.ceph.com' reimaging
2022-09-30T04:38:05.598 INFO:teuthology.lock.ops:Updating [smithi179.front.sepia.ceph.com]: reset os type and version on server
2022-09-30T04:38:05.599 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server
2022-09-30T04:38:05.624 INFO:teuthology.lock.ops:Node 'smithi179.front.sepia.ceph.com' reimaging is complete
2022-09-30T04:38:05.625 INFO:teuthology.lock.ops:Start node 'smithi203.front.sepia.ceph.com' reimaging
2022-09-30T04:38:05.625 INFO:teuthology.lock.ops:Updating [smithi203.front.sepia.ceph.com]: reset os type and version on server
2022-09-30T04:38:05.626 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server
2022-09-30T04:38:05.648 INFO:teuthology.lock.ops:Node 'smithi203.front.sepia.ceph.com' reimaging is complete
2022-09-30T04:38:05.839 INFO:teuthology.provision.fog.smithi179:Scheduling deploy of rhel 8.6
2022-09-30T04:38:05.860 INFO:teuthology.provision.fog.smithi203:Scheduling deploy of rhel 8.6
2022-09-30T04:38:06.215 INFO:teuthology.orchestra.console:Power off smithi179
2022-09-30T04:38:06.264 INFO:teuthology.orchestra.console:Power off smithi203
2022-09-30T04:38:18.709 INFO:teuthology.orchestra.console:Power off for smithi179 completed
2022-09-30T04:38:18.738 INFO:teuthology.orchestra.console:Power off for smithi203 completed
2022-09-30T04:38:18.811 INFO:teuthology.orchestra.console:Power on smithi179
2022-09-30T04:38:18.839 INFO:teuthology.orchestra.console:Power on smithi203
2022-09-30T04:38:23.098 INFO:teuthology.orchestra.console:Power on for smithi203 completed
2022-09-30T04:38:23.200 INFO:teuthology.provision.fog.smithi203:Waiting for deploy to finish
2022-09-30T04:38:27.178 INFO:teuthology.orchestra.console:Power on for smithi179 completed
2022-09-30T04:38:27.279 INFO:teuthology.provision.fog.smithi179:Waiting for deploy to finish
2022-09-30T04:42:56.287 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-09-30T04:42:56.571 INFO:teuthology.provision.fog.smithi179:Node is ready
2022-09-30T04:42:56.631 INFO:teuthology.orchestra.run.smithi179.stdout:smithi179.front.sepia.ceph.com
2022-09-30T04:42:56.692 INFO:teuthology.orchestra.run.smithi179.stdout:172.21.15.179 smithi179.front.sepia.ceph.com smithi179
2022-09-30T04:42:57.062 INFO:teuthology.provision.fog.smithi179:Deploy complete!
2022-09-30T04:43:16.559 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-09-30T04:43:16.809 INFO:teuthology.provision.fog.smithi203:Node is ready
2022-09-30T04:43:16.829 INFO:teuthology.orchestra.run.smithi203.stdout:smithi203
2022-09-30T04:43:16.887 INFO:teuthology.orchestra.run.smithi203.stdout:172.21.15.203 smithi203.front.sepia.ceph.com smithi203
2022-09-30T04:43:17.251 INFO:teuthology.provision.fog.smithi203:Deploy complete!
2022-09-30T04:43:17.467 INFO:teuthology.lock.ops:Checking smithi179.front.sepia.ceph.com
2022-09-30T04:43:17.468 INFO:teuthology.lock.ops:New key found. Updating...
2022-09-30T04:43:17.491 INFO:teuthology.lock.ops:Checking smithi203.front.sepia.ceph.com
2022-09-30T04:43:17.492 INFO:teuthology.lock.ops:New key found. Updating...
2022-09-30T04:43:17.513 INFO:teuthology.lock.ops:Updating [smithi179.front.sepia.ceph.com]: set os type and version on server
2022-09-30T04:43:17.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com'
2022-09-30T04:43:17.791 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64
2022-09-30T04:43:17.850 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="Red Hat Enterprise Linux"
2022-09-30T04:43:17.851 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="8.6 (Ootpa)"
2022-09-30T04:43:17.851 INFO:teuthology.orchestra.run.smithi179.stdout:ID="rhel"
2022-09-30T04:43:17.851 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="fedora"
2022-09-30T04:43:17.852 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="8.6"
2022-09-30T04:43:17.852 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el8"
2022-09-30T04:43:17.852 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2022-09-30T04:43:17.852 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31"
2022-09-30T04:43:17.853 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2022-09-30T04:43:17.853 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://www.redhat.com/"
2022-09-30T04:43:17.853 INFO:teuthology.orchestra.run.smithi179.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-09-30T04:43:17.853 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-09-30T04:43:17.854 INFO:teuthology.orchestra.run.smithi179.stdout:
2022-09-30T04:43:17.854 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-09-30T04:43:17.854 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2022-09-30T04:43:17.854 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-09-30T04:43:17.855 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2022-09-30T04:43:17.856 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server
2022-09-30T04:43:17.876 INFO:teuthology.lock.ops:Updating [smithi203.front.sepia.ceph.com]: set os type and version on server
2022-09-30T04:43:17.877 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com'
2022-09-30T04:43:18.105 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64
2022-09-30T04:43:18.167 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="Red Hat Enterprise Linux"
2022-09-30T04:43:18.167 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="8.6 (Ootpa)"
2022-09-30T04:43:18.168 INFO:teuthology.orchestra.run.smithi203.stdout:ID="rhel"
2022-09-30T04:43:18.168 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="fedora"
2022-09-30T04:43:18.169 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="8.6"
2022-09-30T04:43:18.169 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el8"
2022-09-30T04:43:18.170 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2022-09-30T04:43:18.171 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31"
2022-09-30T04:43:18.171 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2022-09-30T04:43:18.172 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://www.redhat.com/"
2022-09-30T04:43:18.172 INFO:teuthology.orchestra.run.smithi203.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-09-30T04:43:18.173 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-09-30T04:43:18.174 INFO:teuthology.orchestra.run.smithi203.stdout:
2022-09-30T04:43:18.174 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-09-30T04:43:18.174 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2022-09-30T04:43:18.175 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-09-30T04:43:18.176 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2022-09-30T04:43:18.187 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server
2022-09-30T04:43:18.330 INFO:teuthology.dispatcher.supervisor:Running job 7048614
2022-09-30T04:43:18.331 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2022-09-29_19:32:46-rados-wip-yuri6-testing-2022-09-29-0908-distro-default-smithi/7048614 --name yuriw-2022-09-29_19:32:46-rados-wip-yuri6-testing-2022-09-29-0908-distro-default-smithi --description rados/upgrade/parallel/{0-random-distro$/{rhel_8.6_container_tools_3.0} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} -- /home/teuthworker/archive/yuriw-2022-09-29_19:32:46-rados-wip-yuri6-testing-2022-09-29-0908-distro-default-smithi/7048614/orig.config.yaml
2022-09-30T04:43:18.335 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2022-09-29_19:32:46-rados-wip-yuri6-testing-2022-09-29-0908-distro-default-smithi/7048614
2022-09-30T04:43:18.337 INFO:teuthology.dispatcher.supervisor:Job PID: 29655
2022-09-30T04:43:18.337 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-09-30T16:44:17.443 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-09-30T16:44:17.662 INFO:teuthology.kill:Killing Pids: {29655}
2022-09-30T16:44:19.100 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2022-09-30T16:44:19.101 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
2022-09-30T16:44:19.102 INFO:teuthology.misc:Compressing logs...
2022-09-30T16:44:19.102 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com'
2022-09-30T16:44:19.304 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com'
2022-09-30T16:44:19.631 INFO:teuthology.misc:Compressing logs...
2022-09-30T16:44:19.989 INFO:teuthology.misc:Compressing logs...
2022-09-30T16:44:45.767 INFO:teuthology.orchestra.run.smithi179.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-09-30T16:44:49.543 INFO:teuthology.orchestra.run.smithi203.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-09-30T16:45:17.691 INFO:teuthology.orchestra.run.smithi179.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-09-30T16:45:18.181 INFO:teuthology.orchestra.run.smithi203.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-09-30T16:45:45.297 INFO:teuthology.orchestra.run.smithi203.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-09-30T16:45:56.867 INFO:teuthology.orchestra.run.smithi179.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-09-30T16:46:16.678 INFO:teuthology.orchestra.run.smithi203.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-09-30T16:46:24.375 INFO:teuthology.orchestra.run.smithi203.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2022-09-30T16:46:37.403 INFO:teuthology.orchestra.run.smithi179.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-09-30T16:46:46.919 INFO:teuthology.orchestra.run.smithi179.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2022-09-30T16:47:10.867 INFO:teuthology.orchestra.run.smithi179.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2022-09-30T16:47:17.523 INFO:teuthology.orchestra.run.smithi179.stderr:gzip: /var/log/ceph/2f5495e0-407c-11ed-8432-001a4aab830c/ceph-mgr.y.log: file size changed while zipping
2022-09-30T16:48:17.511 INFO:teuthology.kill:No teuthology processes running
2022-09-30T16:48:18.842 INFO:teuthology.kill:Nuking machines: ['smithi179', 'smithi203']
2022-09-30T16:48:20.014 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
2022-09-30T16:48:20.015 INFO:teuthology.kill:  from cryptography.hazmat.backends import default_backend
2022-09-30T16:48:20.015 INFO:teuthology.kill:2022-09-30 16:48:20,014.014 INFO:teuthology.nuke:targets:
2022-09-30T16:48:20.015 INFO:teuthology.kill:  smithi179.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+RLwp/6abHaVhuB3jc3Id+xxF84fCZx0H1vQX6Cqgl9kXURv1Ho9sOejKEFDvOVGCiI2k4KwRDw09KdC1TCl+Wz7GYIN+D83et3yvpQykwTh9OSI8BZLQL0LGbmANO26FBZFmxeuhNc3/Vua5fugERxk+F4fPMkRq041nI1MAfykOcabNroP8JwjJOoD71h+KbEvas2rCbc8M3gKLNjX6d6jPseD/aXe+K4DcJ1Qzpl+zSwNVICrcTv4W1Cnk4v8o5BPaSPO0qHB3mfT8CJgpQuSpcHmmvpHCJF4NN25LDga3Dddg9YgRx0sYC62485am1npef3K4eCqMGrnBBTPMX1zFhNO4QYFea8AAJFNBH7F/5OFzl32GC7xRAa093w4k0qsBXtXqS7qEeNO27g7cHyzHds1nAsQi5T3zcG0SFpkgwX6La6d7z00x8VAmrYDslQG/VkjYaaTv+hZuKl9e4DO7cNRcRGk9EDheO4dzneMlk8XAkFWlMvj6I4GVQQU=
2022-09-30T16:48:20.016 INFO:teuthology.kill:  smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEFydu8WZcykiPSqh42A1CemSc9pZmmf/rsP2VBrGt7GP7qWj2UbP5fVfAcKQT0surcOUKC1W2Ae9l7J272YULE=
2022-09-30T16:48:20.016 INFO:teuthology.kill:2022-09-30 16:48:20,014.014 INFO:teuthology.nuke:Checking targets against current locks
2022-09-30T16:48:20.067 INFO:teuthology.kill:2022-09-30 16:48:20,067.067 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-09-30T16:48:20.101 INFO:teuthology.kill:2022-09-30 16:48:20,100.100 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-09-30T16:48:20.173 INFO:teuthology.kill:2022-09-30 16:48:20,172.172 INFO:teuthology.orchestra.console:Power off smithi179
2022-09-30T16:48:20.192 INFO:teuthology.kill:2022-09-30 16:48:20,192.192 INFO:teuthology.orchestra.console:Power off smithi203
2022-09-30T16:48:28.545 INFO:teuthology.kill:2022-09-30 16:48:28,544.544 INFO:teuthology.orchestra.console:Power off for smithi179 completed
2022-09-30T16:48:28.700 INFO:teuthology.kill:2022-09-30 16:48:28,699.699 INFO:teuthology.lock.ops:unlocked: smithi179.front.sepia.ceph.com
2022-09-30T16:48:32.665 INFO:teuthology.kill:2022-09-30 16:48:32,664.664 INFO:teuthology.orchestra.console:Power off for smithi203 completed
2022-09-30T16:48:32.819 INFO:teuthology.kill:2022-09-30 16:48:32,819.819 INFO:teuthology.lock.ops:unlocked: smithi203.front.sepia.ceph.com
2022-09-30T16:50:33.132 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-09-30T16:50:33.260 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi179 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-09-30_03:20:04-krbd-quincy-testing-default-smithi/7048939
2022-09-30T16:50:33.260 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi203 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-09-30_03:20:04-krbd-quincy-testing-default-smithi/7048939