2022-04-28T13:38:51.486 INFO:root:teuthology version: 1.1.1.dev328+g004701de 2022-04-28T13:38:51.487 INFO:teuthology.results:subset: '27/32' 2022-04-28T13:38:51.487 INFO:teuthology.results:seed: '7125' 2022-04-28T14:07:25.322 INFO:root:teuthology version: 1.1.1.dev328+g004701de 2022-04-28T14:07:25.337 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-28T16:14:34.972 INFO:teuthology.results:Tests finished! gathering results... 2022-04-28T16:14:36.281 INFO:scrape:Found 16 jobs 2022-04-28T16:14:37.883 INFO:scrape:Found 2 distinct failure reasons 2022-04-28T16:14:37.884 INFO:scrape:Failure: Command failed on smithi035 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" 2022-04-28T16:14:37.884 INFO:scrape:6 jobs: ['6810770', '6810772', '6810780', '6810776', '6810768', '6810774'] 2022-04-28T16:14:37.884 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'msgr-failures/none', 'osd}', 'overrides/{frag', 'prefetch_entire_dirfrags/no', 'races', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-28T16:14:37.884 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-28T16:14:37.885 INFO:scrape: 2022-04-28T16:14:37.885 INFO:scrape:Failure: Timed out waiting for MDS daemons to become healthy 2022-04-28T16:14:37.885 INFO:scrape:1 jobs: ['6810778'] 2022-04-28T16:14:37.885 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-28T16:14:37.886 INFO:scrape: 2022-04-28T16:14:37.887 INFO:teuthology.results:Sending results to jlayton@redhat.com: Test Run: jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi ================================================================= info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [6810768] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:14:31 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810768/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810768/ sentry: https://sentry.ceph.com/organizations/ceph/?query=81e37bf840ec4e7dac4f4137d45cd149 Command failed on smithi176 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6810770] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:10:12 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810770/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810770/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d1ffb7c67a644c29f06f5add87ea1f8 Command failed on smithi035 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6810772] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:13:26 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810772/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810772/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc136d7a46984e4587e42577e125e481 Command failed on smithi103 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6810774] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:14:58 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810774/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810774/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2fa02fac4d364a12817d5321a7f68a92 Command failed on smithi003 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6810776] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:10:17 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810776/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810776/ sentry: https://sentry.ceph.com/organizations/ceph/?query=137751fd212b4afab0f4a24de9a6597b Command failed on smithi123 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6810778] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:29:54 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810778/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810778/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50c6a92d081c469798de7775e29d7522 Timed out waiting for MDS daemons to become healthy [6810780] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:12:09 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810780/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810780/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fec0035e40ac447f92a5e0f5ee8463ad Command failed on smithi062 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" Pass ================================================================= [6810766] fs:thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:14:16 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810766/ [6810767] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:26:09 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810767/ [6810769] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:30:40 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810769/ [6810771] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:20:38 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810771/ [6810773] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:14:06 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810773/ [6810775] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:28:20 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810775/ [6810777] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:24:14 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810777/ [6810779] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:00:36 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810779/ [6810781] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:28:13 info: http://pulpito.ceph.com/jlayton-2022-04-28_13:37:57-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6810781/