2026-02-24T16:17:58.304 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-24T16:17:58.305 INFO:teuthology.results:subset: '111/120000' 2026-02-24T16:17:58.305 INFO:teuthology.results:seed: '4497' 2026-02-24T16:24:32.715 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-24T16:24:32.723 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2026-02-24T22:31:12.433 INFO:teuthology.results:Tests finished! gathering results... 2026-02-24T22:31:12.740 INFO:scrape:Found 9 jobs 2026-02-24T22:31:13.985 INFO:scrape:Found 4 distinct failure reasons 2026-02-24T22:31:13.985 INFO:scrape:Dead: b'2026-02-24T22:30:24.077 DEBUG:teuthology.exit:Finished running handlers' 2026-02-24T22:31:13.985 INFO:scrape:1 jobs: ['68327'] 2026-02-24T22:31:13.985 INFO:scrape:suites: ['1-start-distro/1-start-centos_9.stream-squid', '2-setup_subsystem', '3-upgrade/simple', '4-wait', '5-workloads/nvmeof-thrasher}', 'nvmeof/upgrade/{0-clusters/4-gateways-1-initiator'] 2026-02-24T22:31:13.985 INFO:scrape: 2026-02-24T22:31:13.985 INFO:scrape:Failure: "2026-02-24T17:09:20.242584+0000 mon.c (mon.1) 341 : cluster [WRN] Health check failed: Failed to apply 1 service(s): nvmeof.mypool.mygroup0 (CEPHADM_APPLY_SPEC_FAIL)" in cluster log 2026-02-24T22:31:13.985 INFO:scrape:1 jobs: ['68326'] 2026-02-24T22:31:13.986 INFO:scrape:suites: ['centos_latest', 'clusters/4-gateways-1-initiator', 'conf/{disable-pool-app}', 'gateway-initiator-setup/10-subsys-90-namespace-no_huge_pages', 'nvmeof/thrash/{base/install', 'thrashers/nvmeof_mon_thrash', 'workloads/fio}'] 2026-02-24T22:31:13.986 INFO:scrape: 2026-02-24T22:31:13.986 INFO:scrape:Failure: "2026-02-24T16:55:04.299704+0000 mon.a (mon.0) 1280 : cluster [WRN] Health check failed: 1 gateway(s) are in unavailable state; gateway might be down, try to redeploy. (NVMEOF_GATEWAY_DOWN)" in cluster log 2026-02-24T22:31:13.986 INFO:scrape:1 jobs: ['68332'] 2026-02-24T22:31:13.986 INFO:scrape:suites: ['centos_latest', 'clusters/4-gateways-2-initiator', 'conf/{disable-pool-app}', 'nvmeof/basic/{base/install', 'workloads/nvmeof_scalability}'] 2026-02-24T22:31:13.986 INFO:scrape: 2026-02-24T22:31:13.986 INFO:scrape:Failure: Command failed (workunit test nvmeof/fio_test.sh) on trial013 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=618c318a7a0b287ab5aeff207b7bf1d73ef45044 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 IOSTAT_INTERVAL=10 NVMEOF_GROUP=mygroup0 RBD_POOL=mypool RUNTIME=1200 timeout 60m /home/ubuntu/cephtest/clone.client.0/qa/workunits/nvmeof/fio_test.sh --random_devices 32' 2026-02-24T22:31:13.986 INFO:scrape:1 jobs: ['68331'] 2026-02-24T22:31:13.986 INFO:scrape:suites: ['centos_latest', 'clusters/4-gateways-1-initiator', 'conf/{disable-pool-app}', 'gateway-initiator-setup/16-subsys-4-namespace', 'nvmeof/thrash/{base/install', 'thrashers/nvmeof_mon_thrash', 'workloads/fio}'] 2026-02-24T22:31:13.986 INFO:scrape: 2026-02-24T22:31:13.986 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial ================================================================= info: https://pulpito.ceph.com/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/ logs: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/ failed: 3 dead: 1 running: 0 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [68326] nvmeof/thrash/{base/install centos_latest clusters/4-gateways-1-initiator conf/{disable-pool-app} gateway-initiator-setup/10-subsys-90-namespace-no_huge_pages thrashers/nvmeof_mon_thrash workloads/fio} ----------------------------------------------------------------- time: 00:57:47 info: https://pulpito.ceph.com/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68326/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68326/ "2026-02-24T17:09:20.242584+0000 mon.c (mon.1) 341 : cluster [WRN] Health check failed: Failed to apply 1 service(s): nvmeof.mypool.mygroup0 (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [68331] nvmeof/thrash/{base/install centos_latest clusters/4-gateways-1-initiator conf/{disable-pool-app} gateway-initiator-setup/16-subsys-4-namespace thrashers/nvmeof_mon_thrash workloads/fio} ----------------------------------------------------------------- time: 00:42:27 info: https://pulpito.ceph.com/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68331/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68331/ Command failed (workunit test nvmeof/fio_test.sh) on trial013 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=618c318a7a0b287ab5aeff207b7bf1d73ef45044 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 IOSTAT_INTERVAL=10 NVMEOF_GROUP=mygroup0 RBD_POOL=mypool RUNTIME=1200 timeout 60m /home/ubuntu/cephtest/clone.client.0/qa/workunits/nvmeof/fio_test.sh --random_devices 32' [68332] nvmeof/basic/{base/install centos_latest clusters/4-gateways-2-initiator conf/{disable-pool-app} workloads/nvmeof_scalability} ----------------------------------------------------------------- time: 01:01:54 info: https://pulpito.ceph.com/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68332/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68332/ "2026-02-24T16:55:04.299704+0000 mon.a (mon.0) 1280 : cluster [WRN] Health check failed: 1 gateway(s) are in unavailable state; gateway might be down, try to redeploy. (NVMEOF_GATEWAY_DOWN)" in cluster log Dead ================================================================= [68327] nvmeof/upgrade/{0-clusters/4-gateways-1-initiator 1-start-distro/1-start-centos_9.stream-squid 2-setup_subsystem 3-upgrade/simple 4-wait 5-workloads/nvmeof-thrasher} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68327/ log: https://qa-proxy.ceph.com/teuthology/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68327/ killed Pass ================================================================= [68325] nvmeof/basic/{base/install centos_latest clusters/4-gateways-2-initiator conf/{disable-pool-app} workloads/nvmeof_initiator} time: 00:26:43 info: https://pulpito.ceph.com/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68325/ [68328] nvmeof/basic/{base/install centos_latest clusters/4-gateways-2-initiator conf/{disable-pool-app} workloads/nvmeof_mtls} time: 00:16:38 info: https://pulpito.ceph.com/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68328/ [68329] nvmeof/thrash/{base/install centos_latest clusters/4-gateways-1-initiator conf/{disable-pool-app} gateway-initiator-setup/16-subsys-4-namespace thrashers/nvmeof_thrash workloads/fio} time: 00:48:56 info: https://pulpito.ceph.com/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68329/ [68330] nvmeof/basic/{base/install centos_latest clusters/4-gateways-2-initiator conf/{disable-pool-app} workloads/nvmeof_namespaces} time: 00:19:33 info: https://pulpito.ceph.com/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68330/ [68333] nvmeof/thrash/{base/install centos_latest clusters/4-gateways-1-initiator conf/{disable-pool-app} gateway-initiator-setup/10-subsys-90-namespace-no_huge_pages thrashers/nvmeof_thrash workloads/fio} time: 00:49:50 info: https://pulpito.ceph.com/yuriw-2026-02-24_16:03:54-nvmeof-wip-yuri8-testing-2026-02-23-2128-distro-default-trial/68333/ 2026-02-24T22:31:13.996 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 78, in results email_results( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 95, in email_results smtp = smtplib.SMTP('localhost') ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 255, in __init__ (code, msg) = self.connect(host, port) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 341, in connect self.sock = self._get_socket(host, port, self.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket return socket.create_connection((host, port), timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection sock.connect(sa) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect self._internal_connect(address) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect raise _SocketError(err, strerror(err)) ConnectionRefusedError: [Errno 111] Connection refused