2026-02-12T00:53:04.560 INFO:root:teuthology version: 0.0.1.dev29+gc433f1062 2026-02-12T00:53:04.560 INFO:teuthology.results:subset: '320/1024' 2026-02-12T00:53:04.560 INFO:teuthology.results:seed: '9688' 2026-02-12T01:01:17.915 INFO:root:teuthology version: 0.0.1.dev29+gc433f1062 2026-02-12T01:01:17.921 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2026-02-12T07:03:39.961 INFO:teuthology.results:Tests finished! gathering results... 2026-02-12T07:03:40.411 INFO:scrape:Found 10 jobs 2026-02-12T07:03:40.830 INFO:scrape:Found 3 distinct failure reasons 2026-02-12T07:03:40.830 INFO:scrape:Failure: Failed to reconnect to trial086 2026-02-12T07:03:40.830 INFO:scrape:7 jobs: ['45920', '45914', '45923', '45918', '45919', '45915', '45916'] 2026-02-12T07:03:40.831 INFO:scrape:suites intersection: ['2-logrotate', '3-kernel}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing'] 2026-02-12T07:03:40.831 INFO:scrape:suites union: ['0-subvolume/{with-namespace-isolated}', '0-subvolume/{with-no-extra-options}', '0-subvolume/{with-quota}', '0-use-snaprealm/{global}', '0-use-snaprealm/{subvol}', '1-ceph', '1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '2-workunit/cfuse_workunit_suites_pjd}}', '3-kernel}', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-quiesce/no', '6-workunit/kernel_untar_build}}', '6-workunit/suites/dbench}}', '6-workunit/suites/fsx}}', '6-workunit/suites/iozone}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rpm_latest}', 'export-check', 'frag', 'fs/functional/{0-use-snaprealm/{global}', 'fs/functional/{0-use-snaprealm/{subvol}', 'fs/thrash/multifs/{begin/{0-install', 'fs/workload/{0-distro', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'msgr-failures/none', 'multifs', 'n/5', 'no_client_pidfile', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health', 'pg_health}', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/referent}', 'tasks/snap_schedule_snapdir}', 'tasks/{0-fscrypt/{with-fscrypt-dummy}', 'tasks/{1-thrash/mon', 'thrashosds-health}', 'wsync/no}'] 2026-02-12T07:03:40.831 INFO:scrape: 2026-02-12T07:03:40.831 INFO:scrape:Failure: Command failed on trial089 with status 32: 'sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.a=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback' 2026-02-12T07:03:40.831 INFO:scrape:2 jobs: ['45922', '45917'] 2026-02-12T07:03:40.831 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate', '3-kernel}', 'conf/{client', 'distro/{ubuntu_latest}', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing'] 2026-02-12T07:03:40.831 INFO:scrape:suites union: ['1-ceph', '2-logrotate', '2-workunit/iozone}}', '3-kernel}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn', 'pg_health', 'pg_health}', 'session_timeout', 'tasks/{1-tests/fscrypt-dbench}}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2026-02-12T07:03:40.831 INFO:scrape: 2026-02-12T07:03:40.831 INFO:scrape:Dead: b'2026-02-12T07:03:16.934 DEBUG:teuthology.exit:Finished running handlers' 2026-02-12T07:03:40.831 INFO:scrape:1 jobs: ['45921'] 2026-02-12T07:03:40.831 INFO:scrape:suites: ['1-ceph', '2-logrotate', '3-kernel}', 'begin/{0-install', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{0-use-snaprealm/{subvol}', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg_health}', 'tasks/test_journal_migration}'] 2026-02-12T07:03:40.831 INFO:scrape: 2026-02-12T07:03:40.831 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial ================================================================= info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/ logs: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/ failed: 9 dead: 1 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [45914] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45914/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45914/ Failed to reconnect to trial116 [45915] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45915/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45915/ Failed to reconnect to trial140 [45916] fs/functional/{0-use-snaprealm/{global} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/referent} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45916/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45916/ Failed to reconnect to trial021 [45917] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate 3-kernel} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down osd pg-warn pg_health} tasks/{1-tests/fscrypt-dbench}} ----------------------------------------------------------------- time: 00:05:30 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45917/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45917/ Command failed on trial128 with status 32: 'sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph. conf,norbytes,name=0,mds_namespace=cephfs' [45918] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-namespace-isolated} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45918/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45918/ Failed to reconnect to trial079 [45919] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{rpm_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/snap_schedule_snapdir} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45919/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45919/ Failed to reconnect to trial157 [45920] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-quota} 0-use-snaprealm/{global} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-quiesce/no 6-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45920/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45920/ Failed to reconnect to trial086 [45922] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-2c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs pg_health session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:08:26 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45922/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45922/ Command failed on trial089 with status 32: 'sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.a=/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,nofallback' [45923] fs/workload/{0-distro begin/{0-install 1-cephadm 2-logrotate 3-kernel} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-fscrypt/{with-fscrypt-dummy} 0-subvolume/{with-no-extra-options} 0-use-snaprealm/{subvol} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-quiesce/no 6-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45923/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45923/ Failed to reconnect to trial036 Dead ================================================================= [45921] fs/functional/{0-use-snaprealm/{subvol} begin/{0-install 1-ceph 2-logrotate 3-kernel} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45921/ log: https://qa-proxy.ceph.com/teuthology/pdonnell-2026-02-12_00:50:38-fs-wip-pdonnell-testing-20260210.171210-distro-default-trial/45921/ killed 2026-02-12T07:03:40.838 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/results.py", line 78, in results email_results( File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/teuthology/results.py", line 95, in email_results smtp = smtplib.SMTP('localhost') ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 255, in __init__ (code, msg) = self.connect(host, port) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 341, in connect self.sock = self._get_socket(host, port, self.timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/smtplib.py", line 312, in _get_socket return socket.create_connection((host, port), timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/virtualenv/lib/python3.12/site-packages/gevent/socket.py", line 115, in create_connection sock.connect(sa) File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 590, in connect self._internal_connect(address) File "/home/teuthworker/src/git.ceph.com_teuthology_c433f1062990a0488dc29a553589bc609a460691/virtualenv/lib/python3.12/site-packages/gevent/_socketcommon.py", line 634, in _internal_connect raise _SocketError(err, strerror(err)) ConnectionRefusedError: [Errno 111] Connection refused