2023-08-04T13:33:29.617 INFO:root:teuthology version: 0.0.1.dev179+gce5f6d5 2023-08-04T13:33:29.618 INFO:teuthology.results:seed: '3291' 2023-08-04T13:43:33.400 INFO:root:teuthology version: 0.0.1.dev179+gce5f6d5 2023-08-04T13:43:33.404 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-05T01:44:23.907 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-08-05T01:44:24.649 INFO:scrape:Found 6 jobs 2023-08-05T01:44:25.541 INFO:scrape:Found 2 distinct failure reasons 2023-08-05T01:44:25.542 INFO:scrape:Dead: b'2023-08-04T14:09:45.142 INFO:teuthology.orchestra.run.smithi187.stdout: 263645 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 4 14:09 /home/ubuntu/cephtest' 2023-08-05T01:44:25.542 INFO:scrape:4 jobs: ['7360330', '7360331', '7360332', '7360329'] 2023-08-05T01:44:25.542 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/fragment}'] 2023-08-05T01:44:25.543 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/fragment}'] 2023-08-05T01:44:25.543 INFO:scrape: 2023-08-05T01:44:25.543 INFO:scrape:Failure: Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation) 2023-08-05T01:44:25.544 INFO:scrape:2 jobs: ['7360328', '7360327'] 2023-08-05T01:44:25.544 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/fragment}'] 2023-08-05T01:44:25.544 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/fragment}'] 2023-08-05T01:44:25.545 INFO:scrape: 2023-08-05T01:44:25.545 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-08-04_13:31:01-fs-wip-batrick-testing-20230803.140718-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-08-04_13:31:01-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-04_13:31:01-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/ failed: 2 dead: 0 running: 4 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7360327] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:21:09 info: http://pulpito.ceph.com/pdonnell-2023-08-04_13:31:01-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360327/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-04_13:31:01-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360327/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c1ee7d568bd04e1e816a296826c574fb Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation) [7360328] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:13:55 info: http://pulpito.ceph.com/pdonnell-2023-08-04_13:31:01-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360328/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-04_13:31:01-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360328/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b092d971ef894f8ca821766e6b074978 Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation) Running ================================================================= [7360329] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} info: http://pulpito.ceph.com/pdonnell-2023-08-04_13:31:01-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360329/ [7360330] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} info: http://pulpito.ceph.com/pdonnell-2023-08-04_13:31:01-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360330/ [7360331] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} info: http://pulpito.ceph.com/pdonnell-2023-08-04_13:31:01-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360331/ [7360332] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} info: http://pulpito.ceph.com/pdonnell-2023-08-04_13:31:01-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360332/ 2023-08-05T01:44:25.927 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_ce5f6d589a888dfc88977261bfa1ed63441de8fb/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_ce5f6d589a888dfc88977261bfa1ed63441de8fb/teuthology/results.py", line 78, in results email_results( File "/home/teuthworker/src/git.ceph.com_teuthology_ce5f6d589a888dfc88977261bfa1ed63441de8fb/teuthology/results.py", line 96, in email_results smtp.sendmail(msg['From'], [msg['To']], msg.as_string()) File "/usr/lib/python3.8/smtplib.py", line 871, in sendmail raise SMTPSenderRefused(code, resp, from_addr) smtplib.SMTPSenderRefused: (452, b'4.3.1 Insufficient system storage', 'ceph-qa@ceph.com')