2026-02-12T04:12:28.712 INFO:root:teuthology version: 0.0.1.dev29+gc433f1062 2026-02-12T04:12:28.712 INFO:teuthology.results:subset: '227/1024' 2026-02-12T04:12:28.712 INFO:teuthology.results:seed: '4065' 2026-02-12T04:21:28.554 INFO:root:teuthology version: 0.0.1.dev29+gc433f1062 2026-02-12T04:21:28.561 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2026-02-12T07:38:42.046 INFO:teuthology.results:Tests finished! gathering results... 2026-02-12T07:38:42.457 INFO:scrape:Found 10 jobs 2026-02-12T07:38:43.065 INFO:scrape:Found 3 distinct failure reasons 2026-02-12T07:38:43.065 INFO:scrape:Failure: Failed to reconnect to trial140 2026-02-12T07:38:43.065 INFO:scrape:6 jobs: ['45904', '45907', '45910', '45908', '45902', '45905'] 2026-02-12T07:38:43.065 INFO:scrape:suites intersection: ['2-logrotate', '3-kernel}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing'] 2026-02-12T07:38:43.066 INFO:scrape:suites union: ['0-subvolume/{with-namespace-isolated}', '0-subvolume/{with-no-extra-options}', '0-use-snaprealm/{global}', '1-ceph', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/no', '3-kernel}', '3-snaps/yes', '4-flush/no', '5-quiesce/with-quiesce', '6-workunit/suites/ffsb}}', '6-workunit/suites/fsync-tester}}', '6-workunit/suites/pjd}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'export-check', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{0-use-snaprealm/{subvol}', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'no_client_pidfile', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'pg_health}', 'ranks/1', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/cap-flush}', 'tasks/data-scan}', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'tasks/{1-tests/fscrypt-common}}', 'wsync/no}'] 2026-02-12T07:38:43.066 INFO:scrape: 2026-02-12T07:38:43.066 INFO:scrape:Failure: Command failed on trial172 with status 32: 'sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback' 2026-02-12T07:38:43.066 INFO:scrape:2 jobs: ['45903', '45911'] 2026-02-12T07:38:43.066 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-kernel}', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'pg_health', 'session_timeout', 'thrashosds-health}'] 2026-02-12T07:38:43.066 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '2-workunit/ffsb}}', '2-workunit/suites/ffsb}}', '3-kernel}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'pg_health', 'prefetch_entire_dirfrags/yes', 'prefetch_oft_dirfrags/no', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2026-02-12T07:38:43.066 INFO:scrape: 2026-02-12T07:38:43.066 INFO:scrape:Failure: Fuse mount failed to populate/sys/ after 31 seconds 2026-02-12T07:38:43.066 INFO:scrape:2 jobs: ['45909', '45906'] 2026-02-12T07:38:43.066 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-kernel}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg_health}'] 2026-02-12T07:38:43.066 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '3-kernel}', 'begin/{0-install', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu/{overrides', 'distro/{ubuntu_latest}', 'fs/functional/{0-use-snaprealm/{subvol}', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/client-readahead}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'ubuntu_latest}}'] 2026-02-12T07:38:43.066 INFO:scrape: 2026-02-12T07:38:43.066 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial ================================================================= info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/ logs: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/ failed: 10 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [45902] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/1 standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45902/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45902/ Failed to reconnect to trial014 [45903] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_entire_dirfrags/yes prefetch_oft_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:07:54 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45903/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45903/ Command failed on trial172 with status 32: 'sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback' [45904] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/cap-flush} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45904/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45904/ Failed to reconnect to trial140 [45905] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45905/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45905/ Failed to reconnect to trial007 [45906] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/client-readahead} ----------------------------------------------------------------- time: 00:04:47 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45906/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45906/ Fuse mount failed to populate/sys/ after 31 seconds [45907] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{global} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-quiesce/with-quiesce 6-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45907/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45907/ Failed to reconnect to trial043 [45908] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-kernel} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45908/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45908/ Failed to reconnect to trial047 [45909] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu/{overrides ubuntu_latest}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 03:12:26 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45909/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45909/ Fuse mount failed to populate/sys/ after 31 seconds [45910] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/data-scan} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45910/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45910/ Failed to reconnect to trial055 [45911] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:08:55 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45911/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:47:42-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45911/ Command failed on trial056 with status 32: 'sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.a=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback' 2026-02-12T07:38:43.073 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/results.py", line 78, in results email_results( File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/results.py", line 95, in email_results smtp = smtplib.SMTP('localhost') ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 255, in __init__ (code, msg) = self.connect(host, port) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 341, in connect self.sock = self._get_socket(host, port, self.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket return socket.create_connection((host, port), timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection sock.connect(sa) File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect self._internal_connect(address) File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect raise _SocketError(err, strerror(err)) ConnectionRefusedError: [Errno 111] Connection refused