2023-07-16T10:31:01.501 INFO:root:teuthology version: 0.0.1.dev147+gff9991b 2023-07-16T10:31:01.502 INFO:teuthology.results:subset: '359/1024' 2023-07-16T10:31:01.503 INFO:teuthology.results:seed: '1015' 2023-07-16T10:40:13.649 INFO:root:teuthology version: 0.0.1.dev147+gff9991b 2023-07-16T10:40:13.690 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-07-16T12:58:24.683 INFO:teuthology.results:Tests finished! gathering results... 2023-07-16T12:58:25.364 INFO:scrape:Found 8 jobs 2023-07-16T12:58:26.749 WARNING:scrape:Misunderstood line: 2023-07-16T12:58:26.750 INFO:scrape:Found 2 distinct failure reasons 2023-07-16T12:58:26.750 INFO:scrape:Failure: Command failed on smithi092 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-07-16T12:58:26.751 INFO:scrape:4 jobs: ['7341024', '7341018', '7341025', '7341020'] 2023-07-16T12:58:26.751 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-07-16T12:58:26.751 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-07-16T12:58:26.752 INFO:scrape: 2023-07-16T12:58:26.752 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-07-16T12:58:26.753 INFO:scrape:1 jobs: ['7341023'] 2023-07-16T12:58:26.753 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2023-07-16T12:58:26.754 INFO:scrape: 2023-07-16T12:58:26.756 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 3 Fail ================================================================= [7341018] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:16:27 info: http://pulpito.ceph.com/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341018/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341018/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9405938dc1ef4608ad7dee5843362b8c Command failed on smithi175 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7341020] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:16:39 info: http://pulpito.ceph.com/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341020/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341020/ sentry: https://sentry.ceph.com/organizations/ceph/?query=21465cad28ca490697cee0c60e216602 Command failed on smithi160 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7341023] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 02:09:33 info: http://pulpito.ceph.com/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341023/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341023/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22f47acb60c74e7e86d535a77e1b9241 saw valgrind issues [7341024] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:15:54 info: http://pulpito.ceph.com/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341024/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341024/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73a50bd3de6e4b5a80e6210e77dd773a Command failed on smithi092 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7341025] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:16:32 info: http://pulpito.ceph.com/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341025/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341025/ sentry: https://sentry.ceph.com/organizations/ceph/?query=623728822a6f4e7188b1e9efc826b1b2 Command failed on smithi142 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' Pass ================================================================= [7341019] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:43:46 info: http://pulpito.ceph.com/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341019/ [7341021] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:43:56 info: http://pulpito.ceph.com/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341021/ [7341022] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:49:32 info: http://pulpito.ceph.com/vshankar-2023-07-16_09:52:25-fs-main-testing-default-smithi/7341022/