2023-02-12T03:03:10.038 INFO:root:teuthology version: 0.0.1.dev37+g00690ec 2023-02-12T03:03:10.210 INFO:teuthology.lock.ops:Start node 'smithi072.front.sepia.ceph.com' reimaging 2023-02-12T03:03:10.211 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: reset os type and version on server 2023-02-12T03:03:10.211 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2023-02-12T03:03:10.228 INFO:teuthology.lock.ops:Node 'smithi072.front.sepia.ceph.com' reimaging is complete 2023-02-12T03:03:10.228 INFO:teuthology.lock.ops:Start node 'smithi129.front.sepia.ceph.com' reimaging 2023-02-12T03:03:10.229 INFO:teuthology.lock.ops:Updating [smithi129.front.sepia.ceph.com]: reset os type and version on server 2023-02-12T03:03:10.229 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2023-02-12T03:03:10.249 INFO:teuthology.lock.ops:Node 'smithi129.front.sepia.ceph.com' reimaging is complete 2023-02-12T03:03:10.249 INFO:teuthology.lock.ops:Start node 'smithi170.front.sepia.ceph.com' reimaging 2023-02-12T03:03:10.249 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: reset os type and version on server 2023-02-12T03:03:10.250 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2023-02-12T03:03:10.268 INFO:teuthology.lock.ops:Node 'smithi170.front.sepia.ceph.com' reimaging is complete 2023-02-12T03:03:10.375 INFO:teuthology.provision.fog.smithi072:Scheduling deploy of centos 8.stream 2023-02-12T03:03:10.402 INFO:teuthology.provision.fog.smithi129:Scheduling deploy of centos 8.stream 2023-02-12T03:03:10.438 INFO:teuthology.provision.fog.smithi170:Scheduling deploy of centos 8.stream 2023-02-12T03:03:10.609 INFO:teuthology.orchestra.console:Power off smithi072 2023-02-12T03:03:10.653 INFO:teuthology.orchestra.console:Power off smithi129 2023-02-12T03:03:10.669 INFO:teuthology.orchestra.console:Power off smithi170 2023-02-12T03:03:23.241 INFO:teuthology.orchestra.console:Power off for smithi170 completed 2023-02-12T03:03:23.248 INFO:teuthology.orchestra.console:Power off for smithi129 completed 2023-02-12T03:03:23.255 INFO:teuthology.orchestra.console:Power off for smithi072 completed 2023-02-12T03:03:23.343 INFO:teuthology.orchestra.console:Power on smithi170 2023-02-12T03:03:23.352 INFO:teuthology.orchestra.console:Power on smithi129 2023-02-12T03:03:23.360 INFO:teuthology.orchestra.console:Power on smithi072 2023-02-12T03:03:27.609 INFO:teuthology.orchestra.console:Power on for smithi170 completed 2023-02-12T03:03:27.709 INFO:teuthology.provision.fog.smithi170:Waiting for deploy to finish 2023-02-12T03:03:31.729 INFO:teuthology.orchestra.console:Power on for smithi129 completed 2023-02-12T03:03:31.736 INFO:teuthology.orchestra.console:Power on for smithi072 completed 2023-02-12T03:03:31.830 INFO:teuthology.provision.fog.smithi129:Waiting for deploy to finish 2023-02-12T03:03:31.837 INFO:teuthology.provision.fog.smithi072:Waiting for deploy to finish 2023-02-12T03:06:32.069 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-02-12T03:06:35.909 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-02-12T03:06:41.157 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-02-12T03:06:44.965 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-02-12T03:06:50.213 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-02-12T03:06:54.021 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-02-12T03:06:59.269 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.170 2023-02-12T03:07:03.077 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-02-12T03:07:06.117 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-02-12T03:07:12.133 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.129 2023-02-12T03:07:15.173 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-02-12T03:07:24.229 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-02-12T03:07:33.285 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-02-12T03:07:42.345 WARNING:teuthology.provision.fog:[Errno None] Unable to connect to port 22 on 172.21.15.72 2023-02-12T03:08:05.287 WARNING:teuthology.provision.fog:timed out 2023-02-12T03:08:11.644 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-02-12T03:08:11.928 INFO:teuthology.provision.fog.smithi170:Node is ready 2023-02-12T03:08:11.987 INFO:teuthology.orchestra.run.smithi170.stdout:smithi170.front.sepia.ceph.com 2023-02-12T03:08:12.047 INFO:teuthology.orchestra.run.smithi170.stdout:172.21.15.170 smithi170.front.sepia.ceph.com smithi170 2023-02-12T03:08:12.408 INFO:teuthology.provision.fog.smithi170:Deploy complete! 2023-02-12T03:08:18.139 WARNING:teuthology.provision.fog:timed out 2023-02-12T03:08:24.467 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-02-12T03:08:24.723 INFO:teuthology.provision.fog.smithi129:Node is ready 2023-02-12T03:08:24.739 INFO:teuthology.orchestra.run.smithi129.stdout:smithi129.front.sepia.ceph.com 2023-02-12T03:08:24.796 INFO:teuthology.orchestra.run.smithi129.stdout:172.21.15.129 smithi129.front.sepia.ceph.com smithi129 2023-02-12T03:08:25.145 INFO:teuthology.provision.fog.smithi129:Deploy complete! 2023-02-12T03:08:48.373 WARNING:teuthology.provision.fog:timed out 2023-02-12T03:08:54.690 INFO:teuthology.orchestra.run:Running command with timeout 600 2023-02-12T03:08:54.999 INFO:teuthology.provision.fog.smithi072:Node is ready 2023-02-12T03:08:55.017 INFO:teuthology.orchestra.run.smithi072.stdout:smithi072 2023-02-12T03:08:55.075 INFO:teuthology.orchestra.run.smithi072.stdout:172.21.15.72 smithi072.front.sepia.ceph.com smithi072 2023-02-12T03:08:55.437 INFO:teuthology.provision.fog.smithi072:Deploy complete! 2023-02-12T03:08:55.643 INFO:teuthology.lock.ops:Checking smithi072.front.sepia.ceph.com 2023-02-12T03:08:55.643 INFO:teuthology.lock.ops:New key found. Updating... 2023-02-12T03:08:55.665 INFO:teuthology.lock.ops:Checking smithi129.front.sepia.ceph.com 2023-02-12T03:08:55.665 INFO:teuthology.lock.ops:New key found. Updating... 2023-02-12T03:08:55.681 INFO:teuthology.lock.ops:Checking smithi170.front.sepia.ceph.com 2023-02-12T03:08:55.682 INFO:teuthology.lock.ops:New key found. Updating... 2023-02-12T03:08:55.703 INFO:teuthology.lock.ops:Updating [smithi072.front.sepia.ceph.com]: set os type and version on server 2023-02-12T03:08:55.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2023-02-12T03:08:55.890 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2023-02-12T03:08:55.947 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="8" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="8" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el8" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 8" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-02-12T03:08:55.948 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-02-12T03:08:55.949 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2023-02-12T03:08:55.966 INFO:teuthology.lock.ops:Updating [smithi129.front.sepia.ceph.com]: set os type and version on server 2023-02-12T03:08:55.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi129.front.sepia.ceph.com' 2023-02-12T03:08:56.217 INFO:teuthology.orchestra.run.smithi129.stdout:x86_64 2023-02-12T03:08:56.275 INFO:teuthology.orchestra.run.smithi129.stdout:NAME="CentOS Stream" 2023-02-12T03:08:56.275 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION="8" 2023-02-12T03:08:56.276 INFO:teuthology.orchestra.run.smithi129.stdout:ID="centos" 2023-02-12T03:08:56.276 INFO:teuthology.orchestra.run.smithi129.stdout:ID_LIKE="rhel fedora" 2023-02-12T03:08:56.276 INFO:teuthology.orchestra.run.smithi129.stdout:VERSION_ID="8" 2023-02-12T03:08:56.276 INFO:teuthology.orchestra.run.smithi129.stdout:PLATFORM_ID="platform:el8" 2023-02-12T03:08:56.276 INFO:teuthology.orchestra.run.smithi129.stdout:PRETTY_NAME="CentOS Stream 8" 2023-02-12T03:08:56.276 INFO:teuthology.orchestra.run.smithi129.stdout:ANSI_COLOR="0;31" 2023-02-12T03:08:56.276 INFO:teuthology.orchestra.run.smithi129.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-02-12T03:08:56.276 INFO:teuthology.orchestra.run.smithi129.stdout:HOME_URL="https://centos.org/" 2023-02-12T03:08:56.277 INFO:teuthology.orchestra.run.smithi129.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-02-12T03:08:56.277 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-02-12T03:08:56.277 INFO:teuthology.orchestra.run.smithi129.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-02-12T03:08:56.277 INFO:teuthology.lock.ops:Updating smithi129.front.sepia.ceph.com on lock server 2023-02-12T03:08:56.292 INFO:teuthology.lock.ops:Updating [smithi170.front.sepia.ceph.com]: set os type and version on server 2023-02-12T03:08:56.292 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2023-02-12T03:08:56.544 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2023-02-12T03:08:56.602 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="8" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="8" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el8" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 8" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-02-12T03:08:56.603 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-02-12T03:08:56.604 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2023-02-12T03:08:56.694 INFO:teuthology.dispatcher.supervisor:Running job 7161041 2023-02-12T03:08:56.694 DEBUG:teuthology.dispatcher.supervisor:Running: /home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/virtualenv/bin/teuthology -v --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7161041 --name yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi --description rados/thrash-old-clients/{0-distro$/{centos_8.stream_container_tools} 0-size-min-size-overrides/2-size-2-min-size 1-install/nautilus-v2only backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} -- /home/teuthworker/archive/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7161041/orig.config.yaml 2023-02-12T03:08:56.701 INFO:teuthology.dispatcher.supervisor:Job archive: /home/teuthworker/archive/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/7161041 2023-02-12T03:08:56.701 INFO:teuthology.dispatcher.supervisor:Job PID: 10238 2023-02-12T03:08:56.702 INFO:teuthology.dispatcher.supervisor:Running with watchdog 2023-02-12T10:19:34.541 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-02-12T10:19:35.623 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-02-12T10:19:37.065 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-02-12T10:19:38.876 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-02-12T10:19:41.077 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-02-12T10:19:43.643 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-02-12T10:19:46.593 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-02-12T10:19:49.917 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-02-12T10:19:53.618 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-02-12T10:19:57.695 ERROR:teuthology.report:POST to http://paddles.front.sepia.ceph.com/runs/yuriw-2023-02-08_19:43:15-rados-wip-yuri3-testing-2023-02-07-0852-quincy-distro-default-smithi/jobs/ failed with status 500: Internal Server Error

Internal Server Error

2023-02-12T10:19:57.695 ERROR:teuthology.dispatcher.supervisor:run_with_watchdog had an unhandled exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/dispatcher/supervisor.py", line 140, in run_job run_with_watchdog(p, job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog report.try_push_job_info(job_info) File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/report.py", line 508, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/report.py", line 473, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/report.py", line 298, in report_job while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'report job 7161041' reached maximum tries (10) after waiting for 26.997824844334637 seconds 2023-02-12T10:19:57.697 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/virtualenv/bin/teuthology-dispatcher", line 8, in sys.exit(main()) File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/scripts/dispatcher.py", line 35, in main sys.exit(teuthology.dispatcher.main(args)) File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/dispatcher/__init__.py", line 63, in main return supervisor.main(args) File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/dispatcher/supervisor.py", line 57, in main return run_job( File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/dispatcher/supervisor.py", line 140, in run_job run_with_watchdog(p, job_config) File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/dispatcher/supervisor.py", line 302, in run_with_watchdog report.try_push_job_info(job_info) File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/report.py", line 508, in try_push_job_info push_job_info(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/report.py", line 473, in push_job_info reporter.report_job(run_name, job_id, job_info) File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/report.py", line 298, in report_job while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_00690eca1aa0bd56527bb28e9f9e35d734d406bf/teuthology/contextutil.py", line 133, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: 'report job 7161041' reached maximum tries (10) after waiting for 26.997824844334637 seconds