2022-04-28T10:27:41.482 INFO:root:teuthology version: 1.1.1.dev328+g004701de 2022-04-28T10:27:41.483 INFO:teuthology.results:subset: '15/32' 2022-04-28T10:27:41.483 INFO:teuthology.results:seed: '4249' 2022-04-28T10:41:30.066 INFO:root:teuthology version: 1.1.1.dev328+g004701de 2022-04-28T10:41:30.070 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-28T11:20:33.095 INFO:teuthology.results:Tests finished! gathering results... 2022-04-28T11:20:34.132 INFO:scrape:Found 16 jobs 2022-04-28T11:20:34.736 INFO:scrape:Found 1 distinct failure reasons 2022-04-28T11:20:34.737 INFO:scrape:Failure: Command failed on smithi115 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" 2022-04-28T11:20:34.737 INFO:scrape:6 jobs: ['6810748', '6810742', '6810740', '6810750', '6810738', '6810736'] 2022-04-28T11:20:34.738 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'msgr-failures/none', 'osd}', 'overrides/{frag', 'prefetch_entire_dirfrags/yes', 'races', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-28T11:20:34.738 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-28T11:20:34.739 INFO:scrape: 2022-04-28T11:20:34.740 INFO:teuthology.results:Sending results to jlayton@redhat.com: Test Run: jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi ================================================================= info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 10 Fail ================================================================= [6810736] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:10:32 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810736/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810736/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ef4116a585743379d63d81af370b7ce Command failed on smithi082 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6810738] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:12:20 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810738/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810738/ sentry: https://sentry.ceph.com/organizations/ceph/?query=29c01540f73e48909df824e43f1e86b8 Command failed on smithi022 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6810740] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:11:39 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810740/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810740/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b729add1616444e0a588d9432da03fcc Command failed on smithi063 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6810742] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:10:10 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810742/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810742/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac188f9690f24fb78db5b8a173a765ba Command failed on smithi123 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6810748] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:10:10 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810748/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=309d5f3ba79e42a7b8c9df01880ae14c Command failed on smithi115 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6810750] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:14:49 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810750/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810750/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ac0ec7851c24913bdd44e78b4554759 Command failed on smithi125 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" Pass ================================================================= [6810735] fs:thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:15:09 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810735/ [6810737] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:31:13 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810737/ [6810739] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:13:07 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810739/ [6810741] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:25:00 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810741/ [6810743] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:38:46 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810743/ [6810744] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:24:46 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810744/ [6810745] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:12:53 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810745/ [6810746] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:21:25 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810746/ [6810747] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:19:01 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810747/ [6810749] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:29:20 info: http://pulpito.ceph.com/jlayton-2022-04-28_10:26:43-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810749/