2023-08-04T14:31:23.567 INFO:root:teuthology version: 0.0.1.dev179+gce5f6d5 2023-08-04T14:31:23.568 INFO:teuthology.results:seed: '6695' 2023-08-04T14:53:39.134 INFO:root:teuthology version: 0.0.1.dev179+gce5f6d5 2023-08-04T14:53:39.138 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-05T02:54:29.253 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-08-05T02:54:29.688 INFO:scrape:Found 6 jobs 2023-08-05T02:54:30.679 INFO:scrape:Found 3 distinct failure reasons 2023-08-05T02:54:30.680 INFO:scrape:Dead: b'2023-08-04T15:12:28.570 INFO:teuthology.orchestra.run.smithi129.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 4 15:12 /home/ubuntu/cephtest' 2023-08-05T02:54:30.680 INFO:scrape:2 jobs: ['7360380', '7360379'] 2023-08-05T02:54:30.681 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/fragment}'] 2023-08-05T02:54:30.681 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/fragment}'] 2023-08-05T02:54:30.681 INFO:scrape: 2023-08-05T02:54:30.682 INFO:scrape:Failure: Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation) 2023-08-05T02:54:30.682 INFO:scrape:2 jobs: ['7360375', '7360376'] 2023-08-05T02:54:30.682 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/fragment}'] 2023-08-05T02:54:30.683 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/fragment}'] 2023-08-05T02:54:30.683 INFO:scrape: 2023-08-05T02:54:30.683 INFO:scrape:Dead: b'2023-08-05T02:51:54.410 DEBUG:teuthology.exit:Finished running handlers' 2023-08-05T02:54:30.683 INFO:scrape:2 jobs: ['7360377', '7360378'] 2023-08-05T02:54:30.684 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/fragment}'] 2023-08-05T02:54:30.684 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/fragment}'] 2023-08-05T02:54:30.684 INFO:scrape: 2023-08-05T02:54:30.685 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/ failed: 2 dead: 1 running: 3 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7360375] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:16:37 info: http://pulpito.ceph.com/pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360375/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360375/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e41e955f7ae4e20a3f3cf77bf4a3bed Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation) [7360376] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:17:44 info: http://pulpito.ceph.com/pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360376/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360376/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f2e0e59bf1647c685df37a246ce6850 Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation) Dead ================================================================= [7360377] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360377/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360377/ hit max job timeout Running ================================================================= [7360378] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} info: http://pulpito.ceph.com/pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360378/ [7360379] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} info: http://pulpito.ceph.com/pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360379/ [7360380] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} info: http://pulpito.ceph.com/pdonnell-2023-08-04_14:30:08-fs-wip-batrick-testing-20230803.140718-distro-default-smithi/7360380/ 2023-08-05T02:54:30.718 ERROR:teuthology.results:error generating memo/results Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_ce5f6d589a888dfc88977261bfa1ed63441de8fb/teuthology/results.py", line 33, in main results(args['--archive-dir'], args['--name'], args['--email'], File "/home/teuthworker/src/git.ceph.com_teuthology_ce5f6d589a888dfc88977261bfa1ed63441de8fb/teuthology/results.py", line 78, in results email_results( File "/home/teuthworker/src/git.ceph.com_teuthology_ce5f6d589a888dfc88977261bfa1ed63441de8fb/teuthology/results.py", line 96, in email_results smtp.sendmail(msg['From'], [msg['To']], msg.as_string()) File "/usr/lib/python3.8/smtplib.py", line 871, in sendmail raise SMTPSenderRefused(code, resp, from_addr) smtplib.SMTPSenderRefused: (452, b'4.3.1 Insufficient system storage', 'ceph-qa@ceph.com')