2022-10-05T17:10:13.007 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9
2022-10-05T17:10:13.133 INFO:teuthology.lock.ops:Start node 'smithi142.front.sepia.ceph.com' reimaging
2022-10-05T17:10:13.135 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: reset os type and version on server
2022-10-05T17:10:13.135 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2022-10-05T17:10:13.154 INFO:teuthology.lock.ops:Node 'smithi142.front.sepia.ceph.com' reimaging is complete
2022-10-05T17:10:13.155 INFO:teuthology.lock.ops:Start node 'smithi164.front.sepia.ceph.com' reimaging
2022-10-05T17:10:13.155 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: reset os type and version on server
2022-10-05T17:10:13.156 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server
2022-10-05T17:10:13.175 INFO:teuthology.lock.ops:Node 'smithi164.front.sepia.ceph.com' reimaging is complete
2022-10-05T17:10:13.391 INFO:teuthology.provision.fog.smithi164:Scheduling deploy of rhel 8.6
2022-10-05T17:10:13.394 INFO:teuthology.provision.fog.smithi142:Scheduling deploy of rhel 8.6
2022-10-05T17:10:13.779 INFO:teuthology.orchestra.console:Power off smithi164
2022-10-05T17:10:13.793 INFO:teuthology.orchestra.console:Power off smithi142
2022-10-05T17:10:22.143 INFO:teuthology.orchestra.console:Power off for smithi164 completed
2022-10-05T17:10:22.245 INFO:teuthology.orchestra.console:Power on smithi164
2022-10-05T17:10:26.262 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2022-10-05T17:10:26.363 INFO:teuthology.orchestra.console:Power on smithi142
2022-10-05T17:10:30.603 INFO:teuthology.orchestra.console:Power on for smithi164 completed
2022-10-05T17:10:30.613 INFO:teuthology.orchestra.console:Power on for smithi142 completed
2022-10-05T17:10:30.705 INFO:teuthology.provision.fog.smithi164:Waiting for deploy to finish
2022-10-05T17:10:30.715 INFO:teuthology.provision.fog.smithi142:Waiting for deploy to finish
2022-10-05T17:14:43.132 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-10-05T17:15:08.332 INFO:teuthology.provision.fog.smithi164:Node is ready
2022-10-05T17:15:08.351 INFO:teuthology.orchestra.run.smithi164.stdout:smithi164.front.sepia.ceph.com
2022-10-05T17:15:08.408 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.15.164 smithi164.front.sepia.ceph.com smithi164
2022-10-05T17:15:08.757 INFO:teuthology.provision.fog.smithi164:Deploy complete!
2022-10-05T17:15:14.656 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-10-05T17:15:14.917 INFO:teuthology.provision.fog.smithi142:Node is ready
2022-10-05T17:15:14.935 INFO:teuthology.orchestra.run.smithi142.stdout:smithi142
2022-10-05T17:15:14.995 INFO:teuthology.orchestra.run.smithi142.stdout:172.21.15.142 smithi142.front.sepia.ceph.com smithi142
2022-10-05T17:15:15.338 INFO:teuthology.provision.fog.smithi142:Deploy complete!
2022-10-05T17:15:15.527 INFO:teuthology.lock.ops:Checking smithi142.front.sepia.ceph.com
2022-10-05T17:15:15.528 INFO:teuthology.lock.ops:New key found. Updating...
2022-10-05T17:15:15.550 INFO:teuthology.lock.ops:Checking smithi164.front.sepia.ceph.com
2022-10-05T17:15:15.550 INFO:teuthology.lock.ops:New key found. Updating...
2022-10-05T17:15:15.570 INFO:teuthology.lock.ops:Updating [smithi142.front.sepia.ceph.com]: set os type and version on server
2022-10-05T17:15:15.570 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2022-10-05T17:15:15.771 INFO:teuthology.orchestra.run.smithi142.stdout:x86_64
2022-10-05T17:15:15.828 INFO:teuthology.orchestra.run.smithi142.stdout:NAME="Red Hat Enterprise Linux"
2022-10-05T17:15:15.828 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION="8.6 (Ootpa)"
2022-10-05T17:15:15.828 INFO:teuthology.orchestra.run.smithi142.stdout:ID="rhel"
2022-10-05T17:15:15.829 INFO:teuthology.orchestra.run.smithi142.stdout:ID_LIKE="fedora"
2022-10-05T17:15:15.829 INFO:teuthology.orchestra.run.smithi142.stdout:VERSION_ID="8.6"
2022-10-05T17:15:15.829 INFO:teuthology.orchestra.run.smithi142.stdout:PLATFORM_ID="platform:el8"
2022-10-05T17:15:15.829 INFO:teuthology.orchestra.run.smithi142.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2022-10-05T17:15:15.830 INFO:teuthology.orchestra.run.smithi142.stdout:ANSI_COLOR="0;31"
2022-10-05T17:15:15.830 INFO:teuthology.orchestra.run.smithi142.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2022-10-05T17:15:15.830 INFO:teuthology.orchestra.run.smithi142.stdout:HOME_URL="https://www.redhat.com/"
2022-10-05T17:15:15.830 INFO:teuthology.orchestra.run.smithi142.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-10-05T17:15:15.831 INFO:teuthology.orchestra.run.smithi142.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-10-05T17:15:15.831 INFO:teuthology.orchestra.run.smithi142.stdout:
2022-10-05T17:15:15.831 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-10-05T17:15:15.831 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2022-10-05T17:15:15.832 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-10-05T17:15:15.832 INFO:teuthology.orchestra.run.smithi142.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2022-10-05T17:15:15.833 INFO:teuthology.lock.ops:Updating smithi142.front.sepia.ceph.com on lock server
2022-10-05T17:15:15.853 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: set os type and version on server
2022-10-05T17:15:15.854 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com'
2022-10-05T17:15:16.062 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64
2022-10-05T17:15:16.122 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Red Hat Enterprise Linux"
2022-10-05T17:15:16.122 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="8.6 (Ootpa)"
2022-10-05T17:15:16.122 INFO:teuthology.orchestra.run.smithi164.stdout:ID="rhel"
2022-10-05T17:15:16.123 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="fedora"
2022-10-05T17:15:16.123 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="8.6"
2022-10-05T17:15:16.123 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el8"
2022-10-05T17:15:16.123 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
2022-10-05T17:15:16.123 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31"
2022-10-05T17:15:16.124 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
2022-10-05T17:15:16.124 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.redhat.com/"
2022-10-05T17:15:16.124 INFO:teuthology.orchestra.run.smithi164.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
2022-10-05T17:15:16.124 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-10-05T17:15:16.125 INFO:teuthology.orchestra.run.smithi164.stdout:
2022-10-05T17:15:16.125 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
2022-10-05T17:15:16.125 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
2022-10-05T17:15:16.126 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
2022-10-05T17:15:16.126 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
2022-10-05T17:15:16.127 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server
2022-10-05T17:15:16.229 INFO:teuthology.dispatcher.supervisor:Running job 7054959
2022-10-05T17:15:16.229 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/bin/teuthology -v --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2022-10-05_14:50:39-upgrade:quincy-x-main-distro-default-smithi/7054959 --name teuthology-2022-10-05_14:50:39-upgrade:quincy-x-main-distro-default-smithi --description upgrade:quincy-x/parallel/{0-random-distro$/{rhel_8.6_container_tools_rhel8} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} -- /home/teuthworker/archive/teuthology-2022-10-05_14:50:39-upgrade:quincy-x-main-distro-default-smithi/7054959/orig.config.yaml
2022-10-05T17:15:16.234 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/teuthology-2022-10-05_14:50:39-upgrade:quincy-x-main-distro-default-smithi/7054959
2022-10-05T17:15:16.235 INFO:teuthology.dispatcher.supervisor:Job PID: 3936
2022-10-05T17:15:16.235 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-10-06T05:16:12.047 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-10-06T05:16:12.213 INFO:teuthology.kill:Killing Pids: {3936}
2022-10-06T05:16:13.752 INFO:teuthology.task.internal:roles: ubuntu@smithi142.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a']
2022-10-06T05:16:13.754 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mds.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b']
2022-10-06T05:16:13.754 INFO:teuthology.misc:Compressing logs...
2022-10-06T05:16:13.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi142.front.sepia.ceph.com'
2022-10-06T05:16:13.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com'
2022-10-06T05:16:14.321 INFO:teuthology.misc:Compressing logs...
2022-10-06T05:16:14.660 INFO:teuthology.misc:Compressing logs...
2022-10-06T05:16:16.357 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-osd.2.log: file size changed while zipping
2022-10-06T05:16:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-osd.5.log: file size changed while zipping
2022-10-06T05:16:18.516 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-osd.0.log: file size changed while zipping
2022-10-06T05:16:20.140 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-osd.6.log: file size changed while zipping
2022-10-06T05:16:20.595 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-osd.3.log: file size changed while zipping
2022-10-06T05:16:22.532 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-osd.4.log: file size changed while zipping
2022-10-06T05:16:22.671 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-osd.1.log: file size changed while zipping
2022-10-06T05:16:24.354 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-mon.c.log: file size changed while zipping
2022-10-06T05:16:25.462 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-osd.7.log: file size changed while zipping
2022-10-06T05:16:26.482 INFO:teuthology.orchestra.run.smithi164.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-mon.b.log: file size changed while zipping
2022-10-06T05:16:27.192 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-mon.a.log: file size changed while zipping
2022-10-06T05:16:27.896 INFO:teuthology.orchestra.run.smithi142.stderr:gzip: /var/log/ceph/f7c1ffcc-44d2-11ed-8433-001a4aab830c/ceph-mgr.y.log: file size changed while zipping
2022-10-06T05:17:04.321 INFO:teuthology.orchestra.run.smithi142.stderr:tar: .: file changed as we read it
2022-10-06T05:17:30.433 INFO:teuthology.kill:No teuthology processes running
2022-10-06T05:17:32.028 INFO:teuthology.kill:Nuking machines: ['smithi142', 'smithi164']
2022-10-06T05:17:33.334 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_35ea38a9840006713a3d42472a2c536a25e88c15/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
2022-10-06T05:17:33.335 INFO:teuthology.kill:  from cryptography.hazmat.backends import default_backend
2022-10-06T05:17:33.335 INFO:teuthology.kill:2022-10-06 05:17:33,334.334 INFO:teuthology.nuke:targets:
2022-10-06T05:17:33.336 INFO:teuthology.kill:  smithi142.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCazmzE4oMdzlJAMn+t3gJ11HrhVrsvPyjFLgUKjZMZewAjaV7K65InUbN3zqKVT3bMvwacDrrQMdvl5LD3T/+RcZE/w4aolQ7DQp4UpRtFtRqTyVi95braICFdaHIbXh5tBF4G4BRCDyeIV2V3wAggtuTvz5iulVSowIBjHlwAClXhrl1ZcnapojZvX7b+jCmC3fr3bG7U70sm19A2KqRfksaHpHg5l+iaKsxyll6ZYvJ0QB9+5RdnHqKBQyvQ/62Sn+sDpBsVyQf0ApO/L77dMrv/GV8WVe46oAZnxR8DaJmNzwviWhwk9B7wFZJzrOtXdIodGS6trUZ+KoPXW4WQfmkePmJKaQpbjaFmXOqGxZjO1gMHtO6artYz52mnj+w+kdgbXuuA4XZdMebogK9Drm9APY8dN/uM2P8QvOV58djlb2EGaXvX75QSsQnBfoggsjjwMID2plUMX50QXjo8Z83CwrXmXw2AObDORDxU1TxPB+XFLmCwa7+VxzX6/MU=
2022-10-06T05:17:33.336 INFO:teuthology.kill:  smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSBYMG7YxAaB432XJJ1pDq/z3Vnq2xh6Vwcb81Z88EdFkmTp6t14naDXBPBa/ArlBwi7IHP478C2hW9hTU/jMMFd6ZzJEES6t50Ap5eo/0rSXcuxTI13BpdzopfSet2BnjAsd1h1NyRRS+snT931O1OnDJiiBXdKpifuT5nwx//8h+NybVv0UGfJ1Z8aY5NVvGlPoueVqkQ/VwSLx2MV71WQw/zlQscDdULQNdp2arqSwTGw8qd/Ywt1N9lqxV1z/wJ7YRvk9TnKiB9N8SoeWIwLvZHOWEvpDUFX+RIT5hq1rSlXpUev27nRIcgxJSKqRidyZT20d+S+E4vxM0bw4Mn1rEuyjm3sYLVq4iM9JETy+tUBghl2BnnZ2JK+TrBIiHVo5YVTsVqlCADj9VLRKmZWyJrcj3Lfqv3dR6SaE+buA+3NRPFNyP7NhhVJ57YCs5WmeUw4k3Ohqa8nYV4nU2x4Iz9g7KvYF6LBYMb5eb8bF6gJZGJpe9MI4DB0NwTcE=
2022-10-06T05:17:33.336 INFO:teuthology.kill:2022-10-06 05:17:33,334.334 INFO:teuthology.nuke:Checking targets against current locks
2022-10-06T05:17:33.395 INFO:teuthology.kill:2022-10-06 05:17:33,395.395 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-10-06T05:17:33.431 INFO:teuthology.kill:2022-10-06 05:17:33,429.429 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-10-06T05:17:33.496 INFO:teuthology.kill:2022-10-06 05:17:33,496.496 INFO:teuthology.orchestra.console:Power off smithi142
2022-10-06T05:17:33.562 INFO:teuthology.kill:2022-10-06 05:17:33,561.561 INFO:teuthology.orchestra.console:Power off smithi164
2022-10-06T05:17:41.934 INFO:teuthology.kill:2022-10-06 05:17:41,933.933 INFO:teuthology.orchestra.console:Power off for smithi164 completed
2022-10-06T05:17:42.084 INFO:teuthology.kill:2022-10-06 05:17:42,083.083 INFO:teuthology.lock.ops:unlocked: smithi164.front.sepia.ceph.com
2022-10-06T05:17:46.019 INFO:teuthology.kill:2022-10-06 05:17:46,018.018 INFO:teuthology.orchestra.console:Power off for smithi142 completed
2022-10-06T05:17:46.171 INFO:teuthology.kill:2022-10-06 05:17:46,171.171 INFO:teuthology.lock.ops:unlocked: smithi142.front.sepia.ceph.com
2022-10-06T05:19:46.543 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-10-06T05:19:46.736 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi142 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-10-06_05:00:03-smoke-main-distro-default-smithi/7057023
2022-10-06T05:19:46.736 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi164 but it was locked by another job: /home/teuthworker/archive/teuthology-2022-10-06_05:00:03-smoke-main-distro-default-smithi/7057023