2026-02-13T23:25:48.726 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71 2026-02-13T23:25:48.726 INFO:teuthology.results:subset: '466/1024' 2026-02-13T23:25:48.726 INFO:teuthology.results:seed: '1318' 2026-02-13T23:33:28.779 INFO:root:teuthology version: 1.2.4.dev5+g8bec0da71 2026-02-13T23:33:28.785 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2026-02-14T05:37:56.005 INFO:teuthology.results:Tests finished! gathering results... 2026-02-14T05:37:56.453 INFO:scrape:Found 10 jobs 2026-02-14T05:37:57.426 INFO:scrape:Found 3 distinct failure reasons 2026-02-14T05:37:57.426 INFO:scrape:Failure: "2026-02-13T23:47:30.090053+0000 mon.a (mon.0) 412 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log 2026-02-14T05:37:57.426 INFO:scrape:7 jobs: ['50029', '50028', '50027', '50021', '50023', '50026', '50022'] 2026-02-14T05:37:57.427 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-kernel}', '3-snaps/yes', '4-flush/no', '5-quiesce/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'session_timeout}', 'standby-replay', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'wsync/no}'] 2026-02-14T05:37:57.427 INFO:scrape:suites union: ['0-subvolume/{with-namespace-isolated-and-quota}', '0-subvolume/{with-namespace-isolated}', '0-subvolume/{with-no-extra-options}', '0-subvolume/{with-quota}', '0-use-snaprealm/{global}', '0-use-snaprealm/{subvol}', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-kernel}', '3-snaps/yes', '4-flush/no', '5-quiesce/no', '6-workunit/fs/misc}}', '6-workunit/fs/norstats}}', '6-workunit/fs/test_o_trunc}}', '6-workunit/postgres}}', '6-workunit/suites/dbench}}', '6-workunit/suites/fsx}}', '6-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'wsync/no}'] 2026-02-14T05:37:57.427 INFO:scrape: 2026-02-14T05:37:57.427 INFO:scrape:Dead: b'2026-02-14T05:37:30.430 DEBUG:teuthology.exit:Finished running handlers' 2026-02-14T05:37:57.427 INFO:scrape:1 jobs: ['50030'] 2026-02-14T05:37:57.427 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'begin/{0-install', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{0-use-snaprealm/{global}', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/truncate_delay}'] 2026-02-14T05:37:57.427 INFO:scrape: 2026-02-14T05:37:57.427 INFO:scrape:Failure: Command failed on trial182 with status 32: 'sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=a' 2026-02-14T05:37:57.427 INFO:scrape:1 jobs: ['50024'] 2026-02-14T05:37:57.427 INFO:scrape:suites: ['1-ceph', '2-logrotate', '2-workunit/iozone}}', '3-kernel}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'pg_health', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2026-02-14T05:37:57.427 INFO:scrape: 2026-02-14T05:37:57.427 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial ================================================================= info: https://pulpito.ceph.com/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/ logs: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/ failed: 8 dead: 1 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [50021] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:16:20 info: https://pulpito.ceph.com/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50021/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50021/ "2026-02-13T23:41:22.872933+0000 mon.a (mon.0) 375 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [50022] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:15:29 info: https://pulpito.ceph.com/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50022/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50022/ "2026-02-13T23:42:33.499666+0000 mon.a (mon.0) 408 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [50023] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated-and-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/postgres}} ----------------------------------------------------------------- time: 00:15:55 info: https://pulpito.ceph.com/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50023/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50023/ "2026-02-13T23:42:49.819587+0000 mon.a (mon.0) 409 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [50024] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:10:20 info: https://pulpito.ceph.com/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50024/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50024/ Command failed on trial182 with status 32: 'sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph. conf,norbytes,name=0,mds_namespace=a' [50026] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:16:17 info: https://pulpito.ceph.com/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50026/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50026/ "2026-02-13T23:46:27.477597+0000 mon.a (mon.0) 408 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [50027] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:25:52 info: https://pulpito.ceph.com/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50027/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50027/ "2026-02-14T00:01:02.067095+0000 mon.a (mon.0) 406 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [50028] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:16:10 info: https://pulpito.ceph.com/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50028/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50028/ "2026-02-13T23:46:44.783711+0000 mon.a (mon.0) 371 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log [50029] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:26:45 info: https://pulpito.ceph.com/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50029/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50029/ "2026-02-13T23:47:30.090053+0000 mon.a (mon.0) 412 : cluster [WRN] Health check failed: Failed to apply 1 service(s): osd.default (CEPHADM_APPLY_SPEC_FAIL)" in cluster log Dead ================================================================= [50030] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50030/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50030/ killed Pass ================================================================= [50025] fs/upgrade/upgraded_client/{bluestore-bitmap centos_9.stream clusters/{1-mds-1-client-micro} conf/{client mds mgr mon osd} kernel overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-install/squid 1-mount/mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} 2-clients/kclient 3-workload/stress_tests/kernel_untar_build}} time: 00:18:24 info: https://pulpito.ceph.com/pdonnell-2026-02-13_19:09:57-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/50025/ 2026-02-14T05:37:57.435 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/results.py", line 78, in results email_results( File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/teuthology/results.py", line 95, in email_results smtp = smtplib.SMTP('localhost') ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 255, in __init__ (code, msg) = self.connect(host, port) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 341, in connect self.sock = self._get_socket(host, port, self.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket return socket.create_connection((host, port), timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection sock.connect(sa) File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect self._internal_connect(address) File "/home/teuthworker/src/git.ceph.com_teuthology_8bec0da71becad44414c54979f64c9ef0e7099c6/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect raise _SocketError(err, strerror(err)) ConnectionRefusedError: [Errno 111] Connection refused