2022-09-14T17:39:22.551 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-14T17:39:22.685 INFO:teuthology.lock.ops:Start node 'smithi066.front.sepia.ceph.com' reimaging 2022-09-14T17:39:22.687 INFO:teuthology.lock.ops:Updating [smithi066.front.sepia.ceph.com]: reset os type and version on server 2022-09-14T17:39:22.688 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2022-09-14T17:39:22.708 INFO:teuthology.lock.ops:Node 'smithi066.front.sepia.ceph.com' reimaging is complete 2022-09-14T17:39:22.709 INFO:teuthology.lock.ops:Start node 'smithi149.front.sepia.ceph.com' reimaging 2022-09-14T17:39:22.710 INFO:teuthology.lock.ops:Updating [smithi149.front.sepia.ceph.com]: reset os type and version on server 2022-09-14T17:39:22.711 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2022-09-14T17:39:22.733 INFO:teuthology.lock.ops:Node 'smithi149.front.sepia.ceph.com' reimaging is complete 2022-09-14T17:39:22.920 INFO:teuthology.provision.fog.smithi066:Scheduling deploy of rhel 8.4 2022-09-14T17:39:22.944 INFO:teuthology.provision.fog.smithi149:Scheduling deploy of rhel 8.4 2022-09-14T17:39:23.225 INFO:teuthology.orchestra.console:Power off smithi066 2022-09-14T17:39:23.244 INFO:teuthology.orchestra.console:Power off smithi149 2022-09-14T17:39:23.385 INFO:teuthology.orchestra.console:Power off for smithi149 completed 2022-09-14T17:39:23.486 INFO:teuthology.orchestra.console:Power on smithi149 2022-09-14T17:39:31.595 INFO:teuthology.orchestra.console:Power off for smithi066 completed 2022-09-14T17:39:31.698 INFO:teuthology.orchestra.console:Power on smithi066 2022-09-14T17:39:31.861 INFO:teuthology.orchestra.console:Power on for smithi149 completed 2022-09-14T17:39:31.964 INFO:teuthology.provision.fog.smithi149:Waiting for deploy to finish 2022-09-14T17:39:40.080 INFO:teuthology.orchestra.console:Power on for smithi066 completed 2022-09-14T17:39:40.182 INFO:teuthology.provision.fog.smithi066:Waiting for deploy to finish 2022-09-14T17:42:41.372 ERROR:teuthology.orchestra.connection:Error authenticating with smithi066.front.sepia.ceph.com: Authentication failed. 2022-09-14T17:44:24.865 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-14T17:44:25.085 INFO:teuthology.provision.fog.smithi149:Node is ready 2022-09-14T17:44:25.100 INFO:teuthology.orchestra.run.smithi149.stdout:smithi149.front.sepia.ceph.com 2022-09-14T17:44:25.155 INFO:teuthology.orchestra.run.smithi149.stdout:172.21.15.149 smithi149.front.sepia.ceph.com smithi149 2022-09-14T17:44:25.468 INFO:teuthology.provision.fog.smithi149:Deploy complete! 2022-09-14T17:45:14.481 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-14T17:45:14.724 INFO:teuthology.provision.fog.smithi066:Node is ready 2022-09-14T17:45:14.780 INFO:teuthology.orchestra.run.smithi066.stdout:smithi066.front.sepia.ceph.com 2022-09-14T17:45:14.836 INFO:teuthology.orchestra.run.smithi066.stdout:172.21.15.66 smithi066.front.sepia.ceph.com smithi066 2022-09-14T17:45:15.159 INFO:teuthology.provision.fog.smithi066:Deploy complete! 2022-09-14T17:45:15.436 INFO:teuthology.lock.ops:Checking smithi066.front.sepia.ceph.com 2022-09-14T17:45:15.437 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-14T17:45:15.459 INFO:teuthology.lock.ops:Checking smithi149.front.sepia.ceph.com 2022-09-14T17:45:15.461 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-14T17:45:15.482 INFO:teuthology.lock.ops:Updating [smithi066.front.sepia.ceph.com]: set os type and version on server 2022-09-14T17:45:15.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-09-14T17:45:15.718 INFO:teuthology.orchestra.run.smithi066.stdout:x86_64 2022-09-14T17:45:15.773 INFO:teuthology.orchestra.run.smithi066.stdout:NAME="Red Hat Enterprise Linux" 2022-09-14T17:45:15.774 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION="8.4 (Ootpa)" 2022-09-14T17:45:15.774 INFO:teuthology.orchestra.run.smithi066.stdout:ID="rhel" 2022-09-14T17:45:15.775 INFO:teuthology.orchestra.run.smithi066.stdout:ID_LIKE="fedora" 2022-09-14T17:45:15.775 INFO:teuthology.orchestra.run.smithi066.stdout:VERSION_ID="8.4" 2022-09-14T17:45:15.775 INFO:teuthology.orchestra.run.smithi066.stdout:PLATFORM_ID="platform:el8" 2022-09-14T17:45:15.775 INFO:teuthology.orchestra.run.smithi066.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-09-14T17:45:15.775 INFO:teuthology.orchestra.run.smithi066.stdout:ANSI_COLOR="0;31" 2022-09-14T17:45:15.776 INFO:teuthology.orchestra.run.smithi066.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-09-14T17:45:15.776 INFO:teuthology.orchestra.run.smithi066.stdout:HOME_URL="https://www.redhat.com/" 2022-09-14T17:45:15.776 INFO:teuthology.orchestra.run.smithi066.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-14T17:45:15.776 INFO:teuthology.orchestra.run.smithi066.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-14T17:45:15.777 INFO:teuthology.orchestra.run.smithi066.stdout: 2022-09-14T17:45:15.777 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-14T17:45:15.778 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-09-14T17:45:15.778 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-14T17:45:15.778 INFO:teuthology.orchestra.run.smithi066.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-09-14T17:45:15.780 INFO:teuthology.lock.ops:Updating smithi066.front.sepia.ceph.com on lock server 2022-09-14T17:45:15.809 INFO:teuthology.lock.ops:Updating [smithi149.front.sepia.ceph.com]: set os type and version on server 2022-09-14T17:45:15.810 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2022-09-14T17:45:16.056 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2022-09-14T17:45:16.111 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="Red Hat Enterprise Linux" 2022-09-14T17:45:16.111 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="8.4 (Ootpa)" 2022-09-14T17:45:16.111 INFO:teuthology.orchestra.run.smithi149.stdout:ID="rhel" 2022-09-14T17:45:16.112 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="fedora" 2022-09-14T17:45:16.112 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="8.4" 2022-09-14T17:45:16.112 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el8" 2022-09-14T17:45:16.113 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2022-09-14T17:45:16.113 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2022-09-14T17:45:16.114 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2022-09-14T17:45:16.114 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://www.redhat.com/" 2022-09-14T17:45:16.115 INFO:teuthology.orchestra.run.smithi149.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-14T17:45:16.115 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-14T17:45:16.115 INFO:teuthology.orchestra.run.smithi149.stdout: 2022-09-14T17:45:16.115 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-14T17:45:16.116 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2022-09-14T17:45:16.116 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-14T17:45:16.116 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2022-09-14T17:45:16.117 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2022-09-14T17:45:16.227 INFO:teuthology.dispatcher.supervisor:Running job 7032619 2022-09-14T17:45:16.228 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/bin/teuthology -v --owner scheduled_lflores@teuthology --archive /home/teuthworker/archive/lflores-2022-09-14_15:11:39-rados-quincy-release-distro-default-smithi/7032619 --name lflores-2022-09-14_15:11:39-rados-quincy-release-distro-default-smithi --description rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/bluestore-comp-zstd rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2} -- /home/teuthworker/archive/lflores-2022-09-14_15:11:39-rados-quincy-release-distro-default-smithi/7032619/orig.config.yaml 2022-09-14T17:45:16.232 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/lflores-2022-09-14_15:11:39-rados-quincy-release-distro-default-smithi/7032619 2022-09-14T17:45:16.233 INFO:teuthology.dispatcher.supervisor:Job PID: 15867 2022-09-14T17:45:16.234 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-09-15T05:45:40.167 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-09-15T05:45:40.395 INFO:teuthology.kill:Killing Pids: {15867} 2022-09-15T05:45:41.789 INFO:teuthology.task.internal:roles: ubuntu@smithi066.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-09-15T05:45:41.789 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-09-15T05:45:41.790 INFO:teuthology.misc:Compressing logs... 2022-09-15T05:45:41.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi066.front.sepia.ceph.com' 2022-09-15T05:45:42.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2022-09-15T05:45:42.460 INFO:teuthology.misc:Compressing logs... 2022-09-15T05:45:43.526 INFO:teuthology.orchestra.run.smithi149.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-15T05:45:43.701 INFO:teuthology.orchestra.run.smithi066.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-15T05:45:44.354 INFO:teuthology.misc:Compressing logs... 2022-09-15T05:56:30.204 INFO:teuthology.kill:No teuthology processes running 2022-09-15T05:56:32.319 INFO:teuthology.kill:Nuking machines: ['smithi066', 'smithi149'] 2022-09-15T05:56:33.973 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-09-15T05:56:33.974 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-09-15T05:56:33.975 INFO:teuthology.kill:2022-09-15 05:56:33,973.973 INFO:teuthology.nuke:targets: 2022-09-15T05:56:33.975 INFO:teuthology.kill: smithi066.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDop+CXNOSoFjliEjVD2e1mkmCGuxV0nQRCaZPpT/J02qRvhvWbRvkRgDL2CPn4uyjGtFNqzOECUwZO1g3z3M/HkIJmIV082UlWFrNhI6fQdSkZ+Dq0nGQN3o+LkXGEY8MWJyW+fYLHHQuqaMewATx0tH3MMhboMRuyOy/+CSu0qnYiu8Iljj7UKuLdLOBdXzTLY6HPfilmGLciuPwM/8FCLEWB8qcSAQjVVD/NZKPCurtfRjtxC4bWJOewuTMOie7zKwuZK5m4kGN9/ZXNrUU3+Jb4ZO2NWW3rUsiOwC1OSId+ctbdbvJVidACYB49HszTFl3HU7Bx8NUwG/xD923FMLfvAUNElvW0uLGTm0zPzvvLWs1PnvrKwNnVfnUZPjJIqQciqSdr/da35AY6nMaVFKGzjmWR0ZoE7CufidS0CMuLUA5lOzzMVYYz27ovEJUfeqq+7Dza5k+rVJhun8fRc8kQrlNF7gGYR4esTZblH6u0clUMoBoBymQK7h8dgJs= 2022-09-15T05:56:33.975 INFO:teuthology.kill: smithi149.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQClQuRyfQMzfLeHNCc8JU1IVWpF8yWqKCkK+SWsI2yJva5HsYwssBznUeBGSrdTWBDRIPfNE7QQesawELl2BAtGn8RL+7VykdDYdW7hH5Omaf3IYvrC8uJ6kg3pO49fRXYjlksqs+zQKXn6f0XIA2mdHtgj0LfnKhdMuXxZcs6dOjN1dNdwmc6NNlNaUCKJOGPBUoWsTzwQZ3S6bgj6tGxNy7iR3c1HK8sbh9FPcjgc3CsXNnjj3kWaJZPAryYLwBSwsX4SbLWjcjyMdXHRodtAoz4ZhBslfXnJDY5LZNncRosl/iYT++Fa9oKijZwEsmDs2IbpkzLpgyoSmlF1YbPRWY2JPkLLJqoBtayiY+wMkcGV0p2X6jXrQDNarSBfYUQAx+FZ6IorcxwfwHDfqyJa5SqZytV/RdHLYJM81s/BNxFxBPWvWJm1jD5Gnz+9X4z+II2ex0zyGZazdA5Jn8ln99nVnmIHUScOLOus/kkBuVTBTDmUKyeCoDjAJOocPWM= 2022-09-15T05:56:33.975 INFO:teuthology.kill:2022-09-15 05:56:33,974.974 INFO:teuthology.nuke:Checking targets against current locks 2022-09-15T05:56:34.037 INFO:teuthology.kill:2022-09-15 05:56:34,037.037 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-15T05:56:34.056 INFO:teuthology.kill:2022-09-15 05:56:34,056.056 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-15T05:56:34.131 INFO:teuthology.kill:2022-09-15 05:56:34,130.130 INFO:teuthology.orchestra.console:Power off smithi066 2022-09-15T05:56:34.155 INFO:teuthology.kill:2022-09-15 05:56:34,155.155 INFO:teuthology.orchestra.console:Power off smithi149 2022-09-15T05:56:46.638 INFO:teuthology.kill:2022-09-15 05:56:46,638.638 INFO:teuthology.orchestra.console:Power off for smithi066 completed 2022-09-15T05:56:46.654 INFO:teuthology.kill:2022-09-15 05:56:46,653.653 INFO:teuthology.orchestra.console:Power off for smithi149 completed 2022-09-15T05:56:46.813 INFO:teuthology.kill:2022-09-15 05:56:46,811.811 INFO:teuthology.lock.ops:unlocked: smithi066.front.sepia.ceph.com 2022-09-15T05:56:46.823 INFO:teuthology.kill:2022-09-15 05:56:46,815.815 INFO:teuthology.lock.ops:unlocked: smithi149.front.sepia.ceph.com 2022-09-15T05:58:47.193 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-09-15T05:58:47.376 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi066 but it was locked by another job: /home/teuthworker/archive/pdhange-2022-09-15_04:10:46-rados-wip-pdhange-testing-distro-default-smithi/7033546 2022-09-15T05:58:47.377 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi149 but it was locked by another job: /home/teuthworker/archive/pdhange-2022-09-15_04:10:46-rados-wip-pdhange-testing-distro-default-smithi/7033546