2022-12-09T05:35:29.207 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf
2022-12-09T05:35:29.402 INFO:teuthology.lock.ops:Start node 'smithi149.front.sepia.ceph.com' reimaging
2022-12-09T05:35:29.403 INFO:teuthology.lock.ops:Updating [smithi149.front.sepia.ceph.com]: reset os type and version on server
2022-12-09T05:35:29.403 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server
2022-12-09T05:35:29.457 INFO:teuthology.lock.ops:Node 'smithi149.front.sepia.ceph.com' reimaging is complete
2022-12-09T05:35:29.457 INFO:teuthology.lock.ops:Start node 'smithi170.front.sepia.ceph.com' reimaging
2022-12-09T05:35:29.467 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: reset os type and version on server
2022-12-09T05:35:29.467 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2022-12-09T05:35:29.519 INFO:teuthology.lock.ops:Node 'smithi170.front.sepia.ceph.com' reimaging is complete
2022-12-09T05:35:29.736 INFO:teuthology.provision.fog.smithi149:Scheduling deploy of centos 8.stream
2022-12-09T05:35:29.786 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of centos 8.stream
2022-12-09T05:35:30.105 INFO:teuthology.orchestra.console:Power off smithi149
2022-12-09T05:35:30.195 INFO:teuthology.orchestra.console:Power off smithi170
2022-12-09T05:35:38.600 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2022-12-09T05:35:38.706 INFO:teuthology.orchestra.console:Power on smithi170
2022-12-09T05:35:42.662 INFO:teuthology.orchestra.console:Power off for smithi149 completed
2022-12-09T05:35:42.783 INFO:teuthology.orchestra.console:Power on smithi149
2022-12-09T05:35:47.058 INFO:teuthology.orchestra.console:Power on for smithi149 completed
2022-12-09T05:35:47.111 INFO:teuthology.orchestra.console:Power on for smithi170 completed
2022-12-09T05:35:47.160 INFO:teuthology.provision.fog.smithi149:Waiting for deploy to finish
2022-12-09T05:35:47.219 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish
2022-12-09T05:38:18.440 ERROR:teuthology.orchestra.connection:Error authenticating with smithi170.front.sepia.ceph.com: Authentication failed.
2022-12-09T05:39:52.943 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-12-09T05:40:18.088 INFO:teuthology.provision.fog.smithi170:Node is ready
2022-12-09T05:40:18.128 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com
2022-12-09T05:40:18.223 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170
2022-12-09T05:40:18.581 INFO:teuthology.provision.fog.smithi170:Deploy complete!
2022-12-09T05:40:31.274 INFO:teuthology.orchestra.run:Running command with timeout 600
2022-12-09T05:40:31.537 ERROR:paramiko.transport:Unknown exception: The socket has already been closed by another greenlet
2022-12-09T05:40:31.541 ERROR:paramiko.transport:Traceback (most recent call last):
2022-12-09T05:40:31.542 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/virtualenv/lib/python3.6/site-packages/paramiko/transport.py", line 2110, in run
2022-12-09T05:40:31.542 ERROR:paramiko.transport:    ptype, m = self.packetizer.read_message()
2022-12-09T05:40:31.542 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 459, in read_message
2022-12-09T05:40:31.543 ERROR:paramiko.transport:    header = self.read_all(self.__block_size_in, check_rekey=True)
2022-12-09T05:40:31.543 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/virtualenv/lib/python3.6/site-packages/paramiko/packet.py", line 301, in read_all
2022-12-09T05:40:31.543 ERROR:paramiko.transport:    x = self.__socket.recv(n)
2022-12-09T05:40:31.544 ERROR:paramiko.transport:  File "/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/virtualenv/lib/python3.6/site-packages/gevent/_socketcommon.py", line 663, in recv
2022-12-09T05:40:31.545 ERROR:paramiko.transport:    self._wait(self._read_event)
2022-12-09T05:40:31.545 ERROR:paramiko.transport:  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
2022-12-09T05:40:31.545 ERROR:paramiko.transport:  File "src/gevent/_hub_primitives.py", line 321, in gevent._gevent_c_hub_primitives.wait_on_socket
2022-12-09T05:40:31.545 ERROR:paramiko.transport:gevent.exceptions.ConcurrentObjectUseError: The socket has already been closed by another greenlet
2022-12-09T05:40:31.545 ERROR:paramiko.transport:
2022-12-09T05:40:31.548 INFO:teuthology.provision.fog.smithi149:Node is ready
2022-12-09T05:40:31.567 INFO:teuthology.orchestra.run.smithi149.stdout:smithi149.front.sepia.ceph.com
2022-12-09T05:40:31.626 INFO:teuthology.orchestra.run.smithi149.stdout:172.21.15.149 smithi149.front.sepia.ceph.com smithi149
2022-12-09T05:40:31.972 INFO:teuthology.provision.fog.smithi149:Deploy complete!
2022-12-09T05:40:32.253 INFO:teuthology.lock.ops:Checking smithi149.front.sepia.ceph.com
2022-12-09T05:40:32.254 INFO:teuthology.lock.ops:New key found. Updating...
2022-12-09T05:40:32.275 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com
2022-12-09T05:40:32.275 INFO:teuthology.lock.ops:New key found. Updating...
2022-12-09T05:40:32.296 INFO:teuthology.lock.ops:Updating [smithi149.front.sepia.ceph.com]: set os type and version on server
2022-12-09T05:40:32.296 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com'
2022-12-09T05:40:32.524 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64
2022-12-09T05:40:32.582 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="CentOS Stream"
2022-12-09T05:40:32.583 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8"
2022-12-09T05:40:32.583 INFO:teuthology.orchestra.run.smithi149.stdout:ID="centos"
2022-12-09T05:40:32.583 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="rhel fedora"
2022-12-09T05:40:32.583 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8"
2022-12-09T05:40:32.583 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8"
2022-12-09T05:40:32.584 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="CentOS Stream 8"
2022-12-09T05:40:32.584 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31"
2022-12-09T05:40:32.584 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-12-09T05:40:32.584 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://centos.org/"
2022-12-09T05:40:32.584 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-12-09T05:40:32.585 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-12-09T05:40:32.585 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-12-09T05:40:32.587 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server
2022-12-09T05:40:32.609 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server
2022-12-09T05:40:32.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2022-12-09T05:40:32.839 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64
2022-12-09T05:40:32.897 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream"
2022-12-09T05:40:32.899 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8"
2022-12-09T05:40:32.899 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos"
2022-12-09T05:40:32.899 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora"
2022-12-09T05:40:32.899 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8"
2022-12-09T05:40:32.899 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8"
2022-12-09T05:40:32.900 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 8"
2022-12-09T05:40:32.900 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31"
2022-12-09T05:40:32.900 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:8"
2022-12-09T05:40:32.900 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/"
2022-12-09T05:40:32.900 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/"
2022-12-09T05:40:32.900 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
2022-12-09T05:40:32.901 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
2022-12-09T05:40:32.901 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server
2022-12-09T05:40:33.025 INFO:teuthology.dispatcher.supervisor:Running job 7108280
2022-12-09T05:40:33.025 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/virtualenv/bin/teuthology -v --owner scheduled_ksirivad@teuthology --archive /home/teuthworker/archive/ksirivad-2022-12-08_14:50:59-rados-wip-ksirivad-fix-bz-2121452-distro-default-smithi/7108280 --name ksirivad-2022-12-08_14:50:59-rados-wip-ksirivad-fix-bz-2121452-distro-default-smithi --description rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-hybrid rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} -- /home/teuthworker/archive/ksirivad-2022-12-08_14:50:59-rados-wip-ksirivad-fix-bz-2121452-distro-default-smithi/7108280/orig.config.yaml
2022-12-09T05:40:33.032 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/ksirivad-2022-12-08_14:50:59-rados-wip-ksirivad-fix-bz-2121452-distro-default-smithi/7108280
2022-12-09T05:40:33.032 INFO:teuthology.dispatcher.supervisor:Job PID: 8184
2022-12-09T05:40:33.033 INFO:teuthology.dispatcher.supervisor:Running with watchdog
2022-12-09T17:41:00.930 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing...
2022-12-09T17:41:01.153 INFO:teuthology.kill:Killing Pids: {8184}
2022-12-09T17:41:03.793 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
2022-12-09T17:41:03.793 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
2022-12-09T17:41:03.793 INFO:teuthology.misc:Compressing logs...
2022-12-09T17:41:03.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com'
2022-12-09T17:41:04.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com'
2022-12-09T17:41:04.438 INFO:teuthology.misc:Compressing logs...
2022-12-09T17:41:05.514 INFO:teuthology.orchestra.run.smithi170.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-12-09T17:41:05.565 INFO:teuthology.orchestra.run.smithi149.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping
2022-12-09T17:41:06.202 INFO:teuthology.misc:Compressing logs...
2022-12-09T17:44:34.299 INFO:teuthology.kill:No teuthology processes running
2022-12-09T17:44:35.657 INFO:teuthology.kill:Nuking machines: ['smithi149', 'smithi170']
2022-12-09T17:44:36.837 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_teuthology_4da97cf64e542f347ec47b7bdbe5eca99759f9b7/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
2022-12-09T17:44:36.838 INFO:teuthology.kill:  from cryptography.hazmat.backends import default_backend
2022-12-09T17:44:36.838 INFO:teuthology.kill:2022-12-09 17:44:36,837.837 INFO:teuthology.nuke:targets:
2022-12-09T17:44:36.838 INFO:teuthology.kill:  smithi149.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ1HdsA6RjuD9Yx6w0moD6AlPXgMoaAB4atzVnh8Vz6GOQ3nvcrDZkIRhUh+nmt0W20lZqBs0TGm42FtyoTPixPUHnQDX/yo9whmHCk11PYCdaKONUgDPuT5fCwUrgD2hP1eYwfUZ1CrRwbEHK2+ZhwaWDJXK5tTEBlWEkyVRq1V5p/zVKnTl9t01PGZjWb0Mtg92m/7kXT+gxxU6zT4sO+kW1RZOCQ+/ofODSvfGzP8XhaKeZ8MkpWALr5WMBhjAy+oalyUihmkb2wjc+69skj3lFb9tydW8G+bSkK0UEpaRfdKkaTeVbI3ztR/UwabL5ocJATJARLWai9tPPAuodJQn2uF8wcq9d4KUA79LBOqUw8XSkbGupkwpP4U66k5lenX57Ff1ZyYdhJzZ70Gpdrr11i+87hhL95ZVfp9mZIeSoPPBSvRAtLTk1JxnwWBhhE6GAE9lTQ+Eds72voVIF8evfYsXp8Z+zIa1uzWX8Hulz0WGoss1d01ghAV023CU=
2022-12-09T17:44:36.838 INFO:teuthology.kill:  smithi170.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgvTLi26gk6HIiCWfAUvokJ505VLWPkwg3cTRgotTxNnjoF1PXzZcOmQjCDweSs/kB4w/5mvqJSBkRcnF+K5qrD1VKlieMXi2IkXup8IKReCeSOQqgH0taKqdGWR4Z1yOryWb5gLHpRLwzGYmFv/iG+JiCSfNSHcjR1DwXMfgd7aUATU6p4KMppnEa08VacYB5oFffOORMCLk2QT95w2W8aiF+iikrSFtog5yz1M6kyef4QzwXTSrOi7rxqpayqksguGAxRKvl38602V0K8qC7CD+P6BaOmNBFmZ5TDevpABAbPg39V9NlhWAJsYySF1YaBHs6xJfF68kLZW0mCBNtrRq686/QyciNtFEThgH6HMNpzauqqVU6tHaKDJZKyKJjJi8U03wfLjQRs3L6n3/r274XbhL6iM89iXw1yptyr5Z4c7LslLg8w3kuBsnZ6tTHzVlwUakMyTeUQIGCCQ/3wc+vwcsGdxm8aqLqWSyYmlaIXqFUoyQKaMhKMCXOIW8=
2022-12-09T17:44:36.838 INFO:teuthology.kill:2022-12-09 17:44:36,837.837 INFO:teuthology.nuke:Checking targets against current locks
2022-12-09T17:44:36.899 INFO:teuthology.kill:2022-12-09 17:44:36,898.898 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-12-09T17:44:36.926 INFO:teuthology.kill:2022-12-09 17:44:36,926.926 INFO:teuthology.task.internal.check_lock:Checking locks...
2022-12-09T17:44:36.997 INFO:teuthology.kill:2022-12-09 17:44:36,996.996 INFO:teuthology.orchestra.console:Power off smithi149
2022-12-09T17:44:37.041 INFO:teuthology.kill:2022-12-09 17:44:37,040.040 INFO:teuthology.orchestra.console:Power off smithi170
2022-12-09T17:44:45.377 INFO:teuthology.kill:2022-12-09 17:44:45,376.376 INFO:teuthology.orchestra.console:Power off for smithi149 completed
2022-12-09T17:44:45.399 INFO:teuthology.kill:2022-12-09 17:44:45,399.399 INFO:teuthology.orchestra.console:Power off for smithi170 completed
2022-12-09T17:44:45.531 INFO:teuthology.kill:2022-12-09 17:44:45,530.530 INFO:teuthology.lock.ops:unlocked: smithi149.front.sepia.ceph.com
2022-12-09T17:44:45.552 INFO:teuthology.kill:2022-12-09 17:44:45,552.552 INFO:teuthology.lock.ops:unlocked: smithi170.front.sepia.ceph.com
2022-12-09T17:46:45.812 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15
2022-12-09T17:46:45.918 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi149 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-12-09_17:11:01-fs-wip-yuri11-testing-2022-11-18-1506-distro-default-smithi/7109851
2022-12-09T17:46:45.919 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi170 but it was locked by another job: /home/teuthworker/archive/yuriw-2022-12-09_17:11:01-fs-wip-yuri11-testing-2022-11-18-1506-distro-default-smithi/7109851