2022-04-29T21:49:13.056 INFO:root:teuthology version: 1.1.1.dev331+gfd5c3166 2022-04-29T21:49:13.057 INFO:teuthology.results:subset: '1/32' 2022-04-29T21:49:13.057 INFO:teuthology.results:seed: '3567' 2022-04-29T21:55:17.824 INFO:root:teuthology version: 1.1.1.dev331+gfd5c3166 2022-04-29T21:55:17.828 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-30T02:11:36.701 INFO:teuthology.results:Tests finished! gathering results... 2022-04-30T02:11:37.542 INFO:scrape:Found 8 jobs 2022-04-30T02:11:37.923 INFO:scrape:Found 2 distinct failure reasons 2022-04-30T02:11:37.923 INFO:scrape:Failure: Command failed on smithi039 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" 2022-04-30T02:11:37.923 INFO:scrape:1 jobs: ['6814243'] 2022-04-30T02:11:37.923 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/pjd}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-30T02:11:37.923 INFO:scrape: 2022-04-30T02:11:37.924 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-04-30T02:11:37.924 INFO:scrape:1 jobs: ['6814241'] 2022-04-30T02:11:37.924 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/trivial_sync}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs:thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-30T02:11:37.924 INFO:scrape: 2022-04-30T02:11:37.925 INFO:teuthology.results:Sending results to jlayton@redhat.com: Test Run: jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi ================================================================= info: http://pulpito.ceph.com/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [6814241] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:23:43 info: http://pulpito.ceph.com/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814241/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814241/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e2be8737b20d4479a74e2935782fdc64 reached maximum tries (90) after waiting for 540 seconds [6814243] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:12:18 info: http://pulpito.ceph.com/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814243/ log: http://qa-proxy.ceph.com/teuthology/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814243/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cefeffc261e04c17a63d66101aee74e9 Command failed on smithi039 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" Pass ================================================================= [6814237] fs:thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:05 info: http://pulpito.ceph.com/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814237/ [6814238] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:18:31 info: http://pulpito.ceph.com/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814238/ [6814239] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:34:58 info: http://pulpito.ceph.com/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814239/ [6814240] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:26:30 info: http://pulpito.ceph.com/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814240/ [6814242] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:14:59 info: http://pulpito.ceph.com/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814242/ [6814244] fs:thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 04:06:10 info: http://pulpito.ceph.com/jlayton-2022-04-29_21:48:47-fs:thrash-wip-vshankar-testing1-20220427-145314-wip-55258-default-smithi/6814244/