2022-06-28T19:17:28.649 INFO:root:teuthology version: 1.1.1.dev376+g082ae7ef 2022-06-28T19:17:28.831 INFO:teuthology.lock.ops:Start node 'smithi119.front.sepia.ceph.com' reimaging 2022-06-28T19:17:28.834 INFO:teuthology.lock.ops:Updating [smithi119.front.sepia.ceph.com]: reset os type and version on server 2022-06-28T19:17:28.835 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2022-06-28T19:17:28.861 INFO:teuthology.lock.ops:Node 'smithi119.front.sepia.ceph.com' reimaging is complete 2022-06-28T19:17:28.862 INFO:teuthology.lock.ops:Start node 'smithi153.front.sepia.ceph.com' reimaging 2022-06-28T19:17:28.862 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: reset os type and version on server 2022-06-28T19:17:28.866 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2022-06-28T19:17:28.900 INFO:teuthology.lock.ops:Node 'smithi153.front.sepia.ceph.com' reimaging is complete 2022-06-28T19:17:29.105 INFO:teuthology.provision.fog.smithi119:Scheduling deploy of centos 8.stream 2022-06-28T19:17:29.122 INFO:teuthology.provision.fog.smithi153:Scheduling deploy of centos 8.stream 2022-06-28T19:17:29.495 INFO:teuthology.orchestra.console:Power off smithi119 2022-06-28T19:17:29.536 INFO:teuthology.orchestra.console:Power off smithi153 2022-06-28T19:17:37.947 INFO:teuthology.orchestra.console:Power off for smithi119 completed 2022-06-28T19:17:38.049 INFO:teuthology.orchestra.console:Power on smithi119 2022-06-28T19:17:42.089 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2022-06-28T19:17:42.195 INFO:teuthology.orchestra.console:Power on smithi153 2022-06-28T19:17:46.441 INFO:teuthology.orchestra.console:Power on for smithi119 completed 2022-06-28T19:17:46.467 INFO:teuthology.orchestra.console:Power on for smithi153 completed 2022-06-28T19:17:46.544 INFO:teuthology.provision.fog.smithi119:Waiting for deploy to finish 2022-06-28T19:17:46.568 INFO:teuthology.provision.fog.smithi153:Waiting for deploy to finish 2022-06-28T19:22:18.808 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-06-28T19:22:19.127 INFO:teuthology.provision.fog.smithi119:Node is ready 2022-06-28T19:22:19.151 INFO:teuthology.orchestra.run.smithi119.stdout:smithi119.front.sepia.ceph.com 2022-06-28T19:22:19.222 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119 smithi119.front.sepia.ceph.com smithi119 2022-06-28T19:22:19.586 INFO:teuthology.provision.fog.smithi119:Deploy complete! 2022-06-28T19:22:33.901 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-06-28T19:22:34.210 INFO:teuthology.provision.fog.smithi153:Node is ready 2022-06-28T19:22:34.234 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153.front.sepia.ceph.com 2022-06-28T19:22:34.296 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.15.153 smithi153.front.sepia.ceph.com smithi153 2022-06-28T19:22:34.679 INFO:teuthology.provision.fog.smithi153:Deploy complete! 2022-06-28T19:22:34.868 INFO:teuthology.lock.ops:Checking smithi119.front.sepia.ceph.com 2022-06-28T19:22:34.869 INFO:teuthology.lock.ops:New key found. Updating... 2022-06-28T19:22:34.894 INFO:teuthology.lock.ops:Checking smithi153.front.sepia.ceph.com 2022-06-28T19:22:34.894 INFO:teuthology.lock.ops:New key found. Updating... 2022-06-28T19:22:34.917 INFO:teuthology.lock.ops:Updating [smithi119.front.sepia.ceph.com]: set os type and version on server 2022-06-28T19:22:34.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2022-06-28T19:22:35.186 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2022-06-28T19:22:35.245 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2022-06-28T19:22:35.246 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="8" 2022-06-28T19:22:35.246 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2022-06-28T19:22:35.246 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2022-06-28T19:22:35.246 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="8" 2022-06-28T19:22:35.247 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el8" 2022-06-28T19:22:35.247 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 8" 2022-06-28T19:22:35.247 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2022-06-28T19:22:35.247 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-06-28T19:22:35.248 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2022-06-28T19:22:35.248 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-06-28T19:22:35.248 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-06-28T19:22:35.248 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-06-28T19:22:35.250 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2022-06-28T19:22:35.284 INFO:teuthology.lock.ops:Updating [smithi153.front.sepia.ceph.com]: set os type and version on server 2022-06-28T19:22:35.284 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2022-06-28T19:22:35.453 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2022-06-28T19:22:35.511 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2022-06-28T19:22:35.511 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="8" 2022-06-28T19:22:35.512 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2022-06-28T19:22:35.512 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2022-06-28T19:22:35.512 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="8" 2022-06-28T19:22:35.512 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el8" 2022-06-28T19:22:35.513 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 8" 2022-06-28T19:22:35.513 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2022-06-28T19:22:35.513 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2022-06-28T19:22:35.514 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2022-06-28T19:22:35.514 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-06-28T19:22:35.514 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2022-06-28T19:22:35.515 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2022-06-28T19:22:35.516 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2022-06-28T19:22:35.632 INFO:teuthology.dispatcher.supervisor:Running job 6901723 2022-06-28T19:22:35.633 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/bin/teuthology -v --owner scheduled_lflores@teuthology --archive /home/teuthworker/archive/lflores-2022-06-27_23:44:12-rados:thrash-erasure-code-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6901723 --name lflores-2022-06-27_23:44:12-rados:thrash-erasure-code-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi --description rados:thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast msgr-failures/osd-delay objectstore/bluestore-bitmap rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_8} thrashers/minsize_recovery thrashosds-health workloads/ec-small-objects-balanced} -- /home/teuthworker/archive/lflores-2022-06-27_23:44:12-rados:thrash-erasure-code-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6901723/orig.config.yaml 2022-06-28T19:22:35.639 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/lflores-2022-06-27_23:44:12-rados:thrash-erasure-code-wip-yuri2-testing-2022-04-26-1132-octopus-distro-default-smithi/6901723 2022-06-28T19:22:35.641 INFO:teuthology.dispatcher.supervisor:Job PID: 29049 2022-06-28T19:22:35.641 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-06-29T07:23:42.503 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-06-29T07:23:43.182 INFO:teuthology.kill:Killing Pids: {29049} 2022-06-29T07:23:45.239 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-06-29T07:23:45.242 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-06-29T07:23:45.248 INFO:teuthology.misc:Compressing logs... 2022-06-29T07:23:45.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2022-06-29T07:23:46.308 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2022-06-29T07:23:47.129 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-06-29T07:23:47.341 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-06-29T07:23:48.123 INFO:teuthology.kill:No teuthology processes running 2022-06-29T07:23:49.830 INFO:teuthology.kill:Nuking machines: ['smithi119', 'smithi153'] 2022-06-29T07:23:51.003 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_082ae7ef4302fa54665ed0a2535e8e254118dcfd/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-06-29T07:23:51.004 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-06-29T07:23:51.005 INFO:teuthology.kill:2022-06-29 07:23:51,003.003 INFO:teuthology.nuke:targets: 2022-06-29T07:23:51.005 INFO:teuthology.kill: smithi119.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC7jdOxsUHE3i+Q2Qj2vqAEd1ii1/GG6O7/IiJ9MKbeRObf6OzcHvCEk2dcrYiUIfK8EVRJrKcs9RoXUvuqrfP7RYsbigftXx2b8a8ONWoqIHuln51sqe8jH2clkuTt2X8GRXao3xPHnURX62Ba9QlYZaFcai23oNs6CdSZXaSvJh9f9gKXGZbDNBqvJcA4nKWqQ4rvQTPXI5ecqPPOk/Selw0FunxOv2YRkQphpGjaqgF5xE5rk/oDjRAgZ/YtwWp3T/C1K9xMtJBUdrtxHM/imH6mDuxi4TZi2kzo4yJS5SPndc3Q7zWDSjKGjz9U3xq8pfWQGKt5RoGQoaMZlLhvu5wJTUde1BNjm9mzk92Obwu6f95SoMx8rTFpOkvo0KNAk/+J97sFpVRD3prTlhPHfxJ5OfSWFjENEipMsl17kmjoLsDSAk6Qz6nRawBrIiw/FxclBENxn2Sr6CumEqq/YOHPDNZeCYmSJBvxUwxeSIkKFvY2buZ+LInWJQfyGq0= 2022-06-29T07:23:51.005 INFO:teuthology.kill: smithi153.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEfPXqKPEYdMbaXDqw3SvVfSq9KpcYwbq/vnugfOlXZmnHehn15ay2ByVm80EeUqeh/zheIPquwWh2acxkIGXNQEBm6gAZv8cCJLb9GoX4IblM8eXVCAJiokzQ8BzDg3TycZHD5XpjKr30484kdUWGfP+Qu8+lQ46H8hfRrRCDFz/Eq203XEQIG7fNidBRUiEfdsjaNUk6bCDe4NJ20ROLkB4ZeLw0X4ijTg9VMxSH4vJ5mCgQ5c80QlTOqTgSUdPkPOE4/8wRhgPsfUZ0DpttChcs8fON3YFKW3MpH0YwQgcIiPKdLjN0EUaqctbjKb0KfNxcokYMk8wsZJ1yJBSfQ0gf4pizpN+/LSKdZlrHOQ64ocsmPRwfrf3fsG4ES63aeYY8gwVLnepWJJfB3xlytBSEyMhk+tAubzL/gwWQ+QlfIiaMuUtgZp4fGWQCz5kktP3y1Al2c7+bQnbbbbHhTkLo9FqKB5UgiXTuOgjK4w4Oz7w7S4IZm9NRkY6/0gU= 2022-06-29T07:23:51.006 INFO:teuthology.kill:2022-06-29 07:23:51,003.003 INFO:teuthology.nuke:Checking targets against current locks 2022-06-29T07:23:51.073 INFO:teuthology.kill:2022-06-29 07:23:51,072.072 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-06-29T07:23:51.089 INFO:teuthology.kill:2022-06-29 07:23:51,089.089 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-06-29T07:23:51.175 INFO:teuthology.kill:2022-06-29 07:23:51,174.174 INFO:teuthology.orchestra.console:Power off smithi119 2022-06-29T07:23:51.191 INFO:teuthology.kill:2022-06-29 07:23:51,190.190 INFO:teuthology.orchestra.console:Power off smithi153 2022-06-29T07:23:59.561 INFO:teuthology.kill:2022-06-29 07:23:59,560.560 INFO:teuthology.orchestra.console:Power off for smithi153 completed 2022-06-29T07:23:59.715 INFO:teuthology.kill:2022-06-29 07:23:59,715.715 INFO:teuthology.lock.ops:unlocked: smithi153.front.sepia.ceph.com 2022-06-29T07:24:03.668 INFO:teuthology.kill:2022-06-29 07:24:03,668.668 INFO:teuthology.orchestra.console:Power off for smithi119 completed 2022-06-29T07:24:03.828 INFO:teuthology.kill:2022-06-29 07:24:03,828.828 INFO:teuthology.lock.ops:unlocked: smithi119.front.sepia.ceph.com 2022-06-29T07:26:04.234 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-06-29T07:26:04.456 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi119 but it was locked by another job: /home/teuthworker/archive/dis-2022-06-29_07:19:12-rbd-wip-dis-testing-distro-default-smithi/6904949 2022-06-29T07:26:04.457 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi153 but it was locked by another job: /home/teuthworker/archive/dis-2022-06-29_07:19:12-rbd-wip-dis-testing-distro-default-smithi/6904949