2022-04-29T16:34:07.974 INFO:root:teuthology version: 1.1.1.dev331+gfd5c3166 2022-04-29T16:34:07.974 INFO:teuthology.results:subset: '22/32' 2022-04-29T16:34:07.974 INFO:teuthology.results:seed: '4470' 2022-04-29T16:42:13.865 INFO:root:teuthology version: 1.1.1.dev331+gfd5c3166 2022-04-29T16:42:14.076 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-29T18:42:22.996 INFO:teuthology.results:Tests finished! gathering results... 2022-04-29T18:42:23.688 INFO:scrape:Found 8 jobs 2022-04-29T18:42:24.847 INFO:scrape:Found 2 distinct failure reasons 2022-04-29T18:42:24.847 INFO:scrape:Failure: Command failed on smithi059 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" 2022-04-29T18:42:24.847 INFO:scrape:3 jobs: ['6814151', '6814149', '6814153'] 2022-04-29T18:42:24.848 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'prefetch_entire_dirfrags/yes', 'races', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-29T18:42:24.848 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-29T18:42:24.848 INFO:scrape: 2022-04-29T18:42:24.848 INFO:scrape:Failure: Timed out waiting for MDS daemons to become healthy 2022-04-29T18:42:24.848 INFO:scrape:1 jobs: ['6814152'] 2022-04-29T18:42:24.848 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-29T18:42:24.848 INFO:scrape: 2022-04-29T18:42:24.849 INFO:teuthology.results:Sending results to jlayton@redhat.com: Test Run: jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi ================================================================= info: http://pulpito.ceph.com/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 4 Fail ================================================================= [6814149] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:10:02 info: http://pulpito.ceph.com/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814149/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814149/ sentry: https://sentry.ceph.com/organizations/ceph/?query=627b44c2e119436199206946ecf3b023 Command failed on smithi121 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6814151] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:15:01 info: http://pulpito.ceph.com/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814151/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814151/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5ba4b6c391b49b182d33ae9ff118672 Command failed on smithi059 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6814152] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:23:31 info: http://pulpito.ceph.com/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814152/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814152/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75f678fff932432a856799833363b019 Timed out waiting for MDS daemons to become healthy [6814153] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:11:55 info: http://pulpito.ceph.com/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814153/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814153/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fdcbf91ca77b47b6be447e04dc960150 Command failed on smithi110 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" Pass ================================================================= [6814146] fs:thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:13:39 info: http://pulpito.ceph.com/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814146/ [6814147] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:53:50 info: http://pulpito.ceph.com/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814147/ [6814148] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:46:46 info: http://pulpito.ceph.com/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814148/ [6814150] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:16:55 info: http://pulpito.ceph.com/jlayton-2022-04-29_16:30:59-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814150/