2022-09-21T07:16:34.110 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-21T07:16:34.288 INFO:teuthology.lock.ops:Start node 'smithi104.front.sepia.ceph.com' reimaging 2022-09-21T07:16:34.289 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: reset os type and version on server 2022-09-21T07:16:34.290 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2022-09-21T07:16:34.322 INFO:teuthology.lock.ops:Node 'smithi104.front.sepia.ceph.com' reimaging is complete 2022-09-21T07:16:34.322 INFO:teuthology.lock.ops:Start node 'smithi137.front.sepia.ceph.com' reimaging 2022-09-21T07:16:34.323 INFO:teuthology.lock.ops:Updating [smithi137.front.sepia.ceph.com]: reset os type and version on server 2022-09-21T07:16:34.323 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2022-09-21T07:16:34.342 INFO:teuthology.lock.ops:Node 'smithi137.front.sepia.ceph.com' reimaging is complete 2022-09-21T07:16:34.544 INFO:teuthology.provision.fog.smithi104:Scheduling deploy of rhel 8.6 2022-09-21T07:16:34.566 INFO:teuthology.provision.fog.smithi137:Scheduling deploy of rhel 8.6 2022-09-21T07:16:34.951 INFO:teuthology.orchestra.console:Power off smithi104 2022-09-21T07:16:34.974 INFO:teuthology.orchestra.console:Power off smithi137 2022-09-21T07:16:47.470 INFO:teuthology.orchestra.console:Power off for smithi104 completed 2022-09-21T07:16:47.471 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2022-09-21T07:16:47.572 INFO:teuthology.orchestra.console:Power on smithi104 2022-09-21T07:16:47.580 INFO:teuthology.orchestra.console:Power on smithi137 2022-09-21T07:16:51.854 INFO:teuthology.orchestra.console:Power on for smithi137 completed 2022-09-21T07:16:51.955 INFO:teuthology.provision.fog.smithi137:Waiting for deploy to finish 2022-09-21T07:16:55.953 INFO:teuthology.orchestra.console:Power on for smithi104 completed 2022-09-21T07:16:56.054 INFO:teuthology.provision.fog.smithi104:Waiting for deploy to finish 2022-09-21T07:20:55.881 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-21T07:21:20.975 INFO:teuthology.provision.fog.smithi137:Node is ready 2022-09-21T07:21:20.997 INFO:teuthology.orchestra.run.smithi137.stdout:smithi137.front.sepia.ceph.com 2022-09-21T07:21:21.056 INFO:teuthology.orchestra.run.smithi137.stdout:172.21.15.137 smithi137.front.sepia.ceph.com smithi137 2022-09-21T07:21:21.424 INFO:teuthology.provision.fog.smithi137:Deploy complete! 2022-09-21T07:21:34.223 INFO:teuthology.orchestra.run:Running command with timeout 600 2022-09-21T07:21:34.508 INFO:teuthology.provision.fog.smithi104:Node is ready 2022-09-21T07:21:34.567 INFO:teuthology.orchestra.run.smithi104.stdout:smithi104.front.sepia.ceph.com 2022-09-21T07:21:34.628 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.15.104 smithi104.front.sepia.ceph.com smithi104 2022-09-21T07:21:34.984 INFO:teuthology.provision.fog.smithi104:Deploy complete! 2022-09-21T07:21:35.289 INFO:teuthology.lock.ops:Checking smithi104.front.sepia.ceph.com 2022-09-21T07:21:35.290 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-21T07:21:35.338 INFO:teuthology.lock.ops:Checking smithi137.front.sepia.ceph.com 2022-09-21T07:21:35.339 INFO:teuthology.lock.ops:New key found. Updating... 2022-09-21T07:21:35.390 INFO:teuthology.lock.ops:Updating [smithi104.front.sepia.ceph.com]: set os type and version on server 2022-09-21T07:21:35.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2022-09-21T07:21:35.635 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2022-09-21T07:21:35.704 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="Red Hat Enterprise Linux" 2022-09-21T07:21:35.705 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="8.6 (Ootpa)" 2022-09-21T07:21:35.705 INFO:teuthology.orchestra.run.smithi104.stdout:ID="rhel" 2022-09-21T07:21:35.706 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="fedora" 2022-09-21T07:21:35.706 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="8.6" 2022-09-21T07:21:35.706 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el8" 2022-09-21T07:21:35.707 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-21T07:21:35.707 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2022-09-21T07:21:35.708 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-21T07:21:35.708 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://www.redhat.com/" 2022-09-21T07:21:35.708 INFO:teuthology.orchestra.run.smithi104.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-21T07:21:35.709 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-21T07:21:35.709 INFO:teuthology.orchestra.run.smithi104.stdout: 2022-09-21T07:21:35.709 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-21T07:21:35.710 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-21T07:21:35.710 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-21T07:21:35.711 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-21T07:21:35.713 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2022-09-21T07:21:35.777 INFO:teuthology.lock.ops:Updating [smithi137.front.sepia.ceph.com]: set os type and version on server 2022-09-21T07:21:35.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2022-09-21T07:21:36.132 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2022-09-21T07:21:36.198 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="Red Hat Enterprise Linux" 2022-09-21T07:21:36.199 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="8.6 (Ootpa)" 2022-09-21T07:21:36.200 INFO:teuthology.orchestra.run.smithi137.stdout:ID="rhel" 2022-09-21T07:21:36.200 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="fedora" 2022-09-21T07:21:36.201 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="8.6" 2022-09-21T07:21:36.201 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el8" 2022-09-21T07:21:36.202 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2022-09-21T07:21:36.202 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2022-09-21T07:21:36.203 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2022-09-21T07:21:36.203 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://www.redhat.com/" 2022-09-21T07:21:36.204 INFO:teuthology.orchestra.run.smithi137.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2022-09-21T07:21:36.204 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2022-09-21T07:21:36.205 INFO:teuthology.orchestra.run.smithi137.stdout: 2022-09-21T07:21:36.205 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2022-09-21T07:21:36.207 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2022-09-21T07:21:36.207 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2022-09-21T07:21:36.207 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2022-09-21T07:21:36.209 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2022-09-21T07:21:36.367 INFO:teuthology.dispatcher.supervisor:Running job 7039818 2022-09-21T07:21:36.367 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/bin/teuthology -v --owner scheduled_rfriedma@teuthology --archive /home/teuthworker/archive/rfriedma-2022-09-21_06:47:34-rados:thrash-wip-rf-tout-distro-default-smithi/7039818 --name rfriedma-2022-09-21_06:47:34-rados:thrash-wip-rf-tout-distro-default-smithi --description rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat mon_election/classic msgr-failures/osd-delay msgr/async objectstore/bluestore-comp-snappy rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/cache-agent-small} -- /home/teuthworker/archive/rfriedma-2022-09-21_06:47:34-rados:thrash-wip-rf-tout-distro-default-smithi/7039818/orig.config.yaml 2022-09-21T07:21:36.374 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/rfriedma-2022-09-21_06:47:34-rados:thrash-wip-rf-tout-distro-default-smithi/7039818 2022-09-21T07:21:36.375 INFO:teuthology.dispatcher.supervisor:Job PID: 5615 2022-09-21T07:21:36.375 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2022-09-21T19:22:35.163 WARNING:teuthology.dispatcher.supervisor:Job ran longer than 43200s. Killing... 2022-09-21T19:22:35.356 INFO:teuthology.kill:Killing Pids: {5615} 2022-09-21T19:22:36.874 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2022-09-21T19:22:36.875 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2022-09-21T19:22:36.875 INFO:teuthology.misc:Compressing logs... 2022-09-21T19:22:36.876 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2022-09-21T19:22:37.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2022-09-21T19:22:37.593 INFO:teuthology.misc:Compressing logs... 2022-09-21T19:22:38.661 INFO:teuthology.orchestra.run.smithi104.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-21T19:22:38.682 INFO:teuthology.orchestra.run.smithi137.stderr:gzip: /home/ubuntu/cephtest/archive/syslog/misc.log: file size changed while zipping 2022-09-21T19:27:17.781 INFO:teuthology.misc:Compressing logs... 2022-09-21T19:27:25.896 INFO:teuthology.kill:No teuthology processes running 2022-09-21T19:27:28.474 INFO:teuthology.kill:Nuking machines: ['smithi104', 'smithi137'] 2022-09-21T19:27:30.502 INFO:teuthology.kill:/home/teuthworker/src/git.ceph.com_git_teuthology_b58f2c18636eb10faa77ed3614abd00cb85dfc2c/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release. 2022-09-21T19:27:30.504 INFO:teuthology.kill: from cryptography.hazmat.backends import default_backend 2022-09-21T19:27:30.504 INFO:teuthology.kill:2022-09-21 19:27:30,502.502 INFO:teuthology.nuke:targets: 2022-09-21T19:27:30.505 INFO:teuthology.kill: smithi104.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVneE61LQnRHWT7NDfPWgzzm4siySkC3+YjMP4K9Dhg9D9tuqSFICuNtT7OwJCJdTSY2ERu1SuvHEnhvWajcjPC4NrCT0VyLNzRIWhNLRhV+xPGi61G/ZLKyfOc8t9rzwGPCkftzJUpxrEBNngt0OGN4QLm+XZlGt9CiTo3YFyyujp0ixhahRqRXqtKKYsmbXe/1HqQ2gXJWFrgZoFuUCGFWO2r4KLgYorZ2JV3uuQx1MDp/wSvLPb7/6lT11MfxQ/nK4bHD4KEe6ddAMhFfTs6PAv1KTBGuAZkn5DAkHtmwasJMJSVjP2nKdoVypUNqdGFL8Am9NWKbPbJtE96BUU0UDsdyYvJsfXUYsEbzQNCzE8AamKEjcQ/vOxbxHTM4ZMSehUZAAdMWqmmVRVyzEt9L6n7jj9+aQbnXaALM43EW/xCkclq8pWxrJAky5Vdpd1e/WzhbiCmozDmoL9fqgFNc+ypGnJDgLWuXZslLUwGRwGv+B20fM324SnCt9TYVc= 2022-09-21T19:27:30.505 INFO:teuthology.kill: smithi137.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDaieprmUo1vfqZybeaT4p/g2XHIucTdhk+rn8k/A5tWWYGoaaxYMDFDliD6hXITZhanfTwwv9vmhCNBFTBWsdiq9+djM5e4jY82lV+JDWMGk3i16E5v0bFGNC5+lOR+aI5lGu609Frt1YdXiTHd6G1mBtoAAG1F5O5IFuPurV/6gw/utQHrJ3xGCELr5bIxVTPWjCv2Lmdj0yiYoikZu2pZDp2Td/drjaSeMJGzooQNOfQrLhQ2xesMsg2RH9T8JOrbKfhgFe0MbZsH/M2RClPwFM4+l1z/QHhmFmQsxE7MafLA4zxmoKorNHOEYH9flktiVk1MYVoYRBRQMz6dGpXXrUFk832zwtXUuo1TBl2im1pCu4SHy7U0XAJqM4LZX+468iX8mYiBiskOujaSNBhFSVEaJgaCFSMQs+NOYvzSsRerR76gIX9VbmnkMGWK7WA14J8QzDaR8MceQhFW0DJk6imLhtnkjYFyv28r0A/wo0Bu8EoqE5V2vG0MH0T1EE= 2022-09-21T19:27:30.505 INFO:teuthology.kill:2022-09-21 19:27:30,502.502 INFO:teuthology.nuke:Checking targets against current locks 2022-09-21T19:27:30.574 INFO:teuthology.kill:2022-09-21 19:27:30,573.573 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-21T19:27:30.617 INFO:teuthology.kill:2022-09-21 19:27:30,617.617 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-09-21T19:27:30.718 INFO:teuthology.kill:2022-09-21 19:27:30,718.718 INFO:teuthology.orchestra.console:Power off smithi104 2022-09-21T19:27:30.788 INFO:teuthology.kill:2022-09-21 19:27:30,787.787 INFO:teuthology.orchestra.console:Power off smithi137 2022-09-21T19:27:39.162 INFO:teuthology.kill:2022-09-21 19:27:39,161.161 INFO:teuthology.orchestra.console:Power off for smithi104 completed 2022-09-21T19:27:39.230 INFO:teuthology.kill:2022-09-21 19:27:39,229.229 INFO:teuthology.orchestra.console:Power off for smithi137 completed 2022-09-21T19:27:39.331 INFO:teuthology.kill:2022-09-21 19:27:39,325.325 INFO:teuthology.lock.ops:unlocked: smithi104.front.sepia.ceph.com 2022-09-21T19:27:39.392 INFO:teuthology.kill:2022-09-21 19:27:39,392.392 INFO:teuthology.lock.ops:unlocked: smithi137.front.sepia.ceph.com 2022-09-21T19:29:39.875 ERROR:teuthology.dispatcher.supervisor:Child exited with code -15 2022-09-21T19:29:40.074 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi104 but it was locked by another job: /home/teuthworker/archive/ivancich-2022-09-21_19:14:37-rgw-wip-eric-testing-1-distro-default-smithi/7040335 2022-09-21T19:29:40.074 WARNING:teuthology.dispatcher.supervisor:Was going to unlock smithi137 but it was locked by another job: /home/teuthworker/archive/ivancich-2022-09-21_19:14:37-rgw-wip-eric-testing-1-distro-default-smithi/7040335