2026-02-19T13:47:37.118 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-19T13:47:37.118 INFO:teuthology.results:subset: '919/1024' 2026-02-19T13:47:37.118 INFO:teuthology.results:seed: '5980' 2026-02-19T13:50:37.998 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-19T13:50:38.006 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2026-02-19T16:27:51.051 INFO:teuthology.results:Tests finished! gathering results... 2026-02-19T16:27:51.676 INFO:scrape:Found 16 jobs 2026-02-19T16:27:52.575 INFO:scrape:Found 3 distinct failure reasons 2026-02-19T16:27:52.575 INFO:scrape:Failure: 2026-02-19T16:27:52.575 INFO:scrape:3 jobs: ['58883', '58873', '58885'] 2026-02-19T16:27:52.575 INFO:scrape:suites intersection: ['1-client', '2-upgrade', 'centos_9.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-from/reef'] 2026-02-19T16:27:52.576 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/yes}}', '4-compat_client', '5-client-sanity}}', 'centos_9.stream', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs:upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'multimds/no', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health}', 'tasks/{0-from/reef'] 2026-02-19T16:27:52.576 INFO:scrape: 2026-02-19T16:27:52.576 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2026-02-19T16:27:52.576 INFO:scrape:3 jobs: ['58881', '58887', '58874'] 2026-02-19T16:27:52.576 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/squid'] 2026-02-19T16:27:52.576 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/yes', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/squid'] 2026-02-19T16:27:52.576 INFO:scrape: 2026-02-19T16:27:52.576 INFO:scrape:Failure: Command failed on trial110 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c85a218-0d9b-11f1-a67b-d404e6e7d460 -e sha1=7544a95646bb0bb6c0298abd0c9716403553949c -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2026-02-19T16:27:52.576 INFO:scrape:1 jobs: ['58878'] 2026-02-19T16:27:52.576 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client/kclient', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_9.stream', 'conf/{client', 'fail_fs/no', 'fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'kernel', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}'] 2026-02-19T16:27:52.576 INFO:scrape: 2026-02-19T16:27:52.577 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial ================================================================= info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/ logs: https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [58873] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 02:30:18 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58873/ log: https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58873/ [58874] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:24:56 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58874/ log: https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58874/ reached maximum tries (50) after waiting for 300 seconds [58878] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:15:17 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58878/ log: https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58878/ Command failed on trial110 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9c85a218-0d9b-11f1-a67b-d404e6e7d460 -e sha1=7544a95646bb0bb6c0298abd0c9716403553949c -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' [58881] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:24:03 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58881/ log: https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58881/ reached maximum tries (50) after waiting for 300 seconds [58883] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/yes}} ----------------------------------------------------------------- time: 01:42:28 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58883/ log: https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58883/ [58885] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/reef 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 01:40:58 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58885/ log: https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58885/ [58887] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/yes kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/squid 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:24:27 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58887/ log: https://qa-proxy.ceph.com/teuthology/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58887/ reached maximum tries (50) after waiting for 300 seconds Pass ================================================================= [58872] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{v18.2.1} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:21:53 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58872/ [58875] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/yes}} time: 00:12:59 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58875/ [58876] fs:upgrade/featureful_client/old_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-compat_client/no}} time: 00:14:41 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58876/ [58877] fs:upgrade/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/squid 1-upgrade}} time: 00:07:53 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58877/ [58879] fs:upgrade/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/squid 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/stress_tests/kernel_untar_build}} time: 01:20:30 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58879/ [58880] fs:upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_9.stream clusters/1-mds-2-client-micro conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no multimds/yes pg-warn pg_health} tasks/{0-from/squid 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:14:29 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58880/ [58882] fs:upgrade/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/squid 1-mount/mount/fuse 2-clients/fuse-upgrade 3-workload/new_ops/{newops}}} time: 00:08:19 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58882/ [58884] fs:upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no kernel overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:21:50 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58884/ [58886] fs:upgrade/nofs/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} kernel no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-from/reef 1-upgrade}} time: 00:07:20 info: https://pulpito.ceph.com/vshankar-2026-02-19_12:57:29-fs:upgrade-wip-vshankar-testing-20260219.051425-testing-default-trial/58886/ 2026-02-19T16:27:52.583 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 78, in results email_results( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/results.py", line 95, in email_results smtp = smtplib.SMTP('localhost') ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 255, in __init__ (code, msg) = self.connect(host, port) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 341, in connect self.sock = self._get_socket(host, port, self.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket return socket.create_connection((host, port), timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection sock.connect(sa) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect self._internal_connect(address) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect raise _SocketError(err, strerror(err)) ConnectionRefusedError: [Errno 111] Connection refused