2022-04-27T14:19:06.514 INFO:root:teuthology version: 1.1.1.dev326+g135488ac 2022-04-27T14:19:06.515 INFO:teuthology.results:subset: '7/32' 2022-04-27T14:19:06.515 INFO:teuthology.results:seed: '6004' 2022-04-27T15:21:43.446 INFO:root:teuthology version: 1.1.1.dev326+g135488ac 2022-04-27T15:21:43.475 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-27T21:49:12.813 INFO:teuthology.results:Tests finished! gathering results... 2022-04-27T21:49:17.098 INFO:scrape:Found 40 jobs 2022-04-27T21:49:26.770 INFO:scrape:Found 9 distinct failure reasons 2022-04-27T21:49:26.770 INFO:scrape:Crash: Found coredumps on ubuntu@smithi116.front.sepia.ceph.com 2022-04-27T21:49:26.770 INFO:scrape:ceph version 17.0.0-11796-g04b6d965 (04b6d965e55961a6cb690e35b6f77d0192e1fe45) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f8a3bf4cc20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f8a3cf6d112] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2852d5) [0x7f8a3cf6d2d5] 6: ceph-mds(+0x28ede3) [0x559b42f18de3] 7: (MDCache::dispatch_fragment_dir(boost::intrusive_ptr&)+0x4f9) [0x559b42fa7a49] 8: (MDCache::fragment_frozen(boost::intrusive_ptr&, int)+0x2d8) [0x559b42fa8888] 9: (MDCache::fragment_mark_and_complete(boost::intrusive_ptr&)+0xa5c) [0x559b42fba5ac] 10: (MDCache::merge_dir(CInode*, frag_t)+0x5ab) [0x559b42fbb3eb] 11: ceph-mds(+0x41b5b2) [0x559b430a55b2] 12: (Context::complete(int)+0xd) [0x559b42e572bd] 13: (CommonSafeTimer::timer_thread()+0x18b) [0x7f8a3d0807bb] 14: (CommonSafeTimerThread::entry()+0x11) [0x7f8a3d0827c1] 15: /lib64/libpthread.so.0(+0x817a) [0x7f8a3bf4217a] 16: clone() 2022-04-27T21:49:26.771 INFO:scrape:1 jobs: ['6808775'] 2022-04-27T21:49:26.771 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T21:49:26.771 INFO:scrape: 2022-04-27T21:49:26.771 INFO:scrape:Failure: "2022-04-27T15:38:12.237171+0000 mds.e (mds.0) 1 : cluster [WRN] client.4661 isn't responding to mclientcaps(revoke), ino 0x1000000b532 pending pAsLsXsFsc issued pAsLsXsFscb, sent 300.016786 seconds ago" in cluster log 2022-04-27T21:49:26.771 INFO:scrape:2 jobs: ['6808792', '6808798'] 2022-04-27T21:49:26.771 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T21:49:26.772 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T21:49:26.772 INFO:scrape: 2022-04-27T21:49:26.772 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-04-27T21:49:26.772 INFO:scrape:3 jobs: ['6808808', '6808777', '6808803'] 2022-04-27T21:49:26.772 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{frag', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T21:49:26.772 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount/fuse', 'ms_mode/{legacy}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{no}}', 'wsync/{yes}}'] 2022-04-27T21:49:26.772 INFO:scrape: 2022-04-27T21:49:26.773 INFO:scrape:Failure: Extra data: line 2 column 555 (char 555) 2022-04-27T21:49:26.773 INFO:scrape:2 jobs: ['6808799', '6808780'] 2022-04-27T21:49:26.773 INFO:scrape:suites intersection: ['1-client', '2-upgrade', '3-compat_client/pacific}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/yes', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T21:49:26.773 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-compat_client/pacific}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/yes', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T21:49:26.773 INFO:scrape: 2022-04-27T21:49:26.774 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2022-04-27T21:49:26.774 INFO:scrape:1 jobs: ['6808783'] 2022-04-27T21:49:26.774 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T21:49:26.774 INFO:scrape: 2022-04-27T21:49:26.774 INFO:scrape:Dead: b'2022-04-27T21:46:14.831 DEBUG:teuthology.exit:Finished running handlers' 2022-04-27T21:49:26.774 INFO:scrape:2 jobs: ['6808800', '6808778'] 2022-04-27T21:49:26.775 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T21:49:26.775 INFO:scrape:suites union: ['1-cephadm', '1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{legacy}', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{pg-warn', 'ranks/3', 'roles', 'scrub/yes', 'session_timeout', 'standby-replay', 'syntax', 'tasks/{0-check-counter', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/blogbench}', 'wsync/{no}}'] 2022-04-27T21:49:26.775 INFO:scrape: 2022-04-27T21:49:26.775 INFO:scrape:Failure: "2022-04-27T18:37:09.136426+0000 osd.3 (osd.3) 35 : cluster [ERR] 5.1s0 deep-scrub : stat mismatch, got 0/12 objects, 0/0 clones, 0/12 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/114630 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log 2022-04-27T21:49:26.775 INFO:scrape:1 jobs: ['6808770'] 2022-04-27T21:49:26.775 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T21:49:26.776 INFO:scrape: 2022-04-27T21:49:26.776 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi151.front.sepia.ceph.com: ['type=AVC msg=audit(1651071612.513:202): avc: denied { node_bind } for pid=1541 comm="ping" saddr=172.21.15.151 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] 2022-04-27T21:49:26.776 INFO:scrape:1 jobs: ['6808793'] 2022-04-27T21:49:26.776 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T21:49:26.776 INFO:scrape: 2022-04-27T21:49:26.776 INFO:scrape:Failure: "2022-04-27T15:09:50.656887+0000 mon.a (mon.0) 405 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log 2022-04-27T21:49:26.776 INFO:scrape:1 jobs: ['6808796'] 2022-04-27T21:49:26.777 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/volumes/{overrides', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-27T21:49:26.777 INFO:scrape: 2022-04-27T21:49:26.779 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/ failed: 12 dead: 2 running: 0 waiting: 0 queued: 0 passed: 26 Fail ================================================================= [6808770] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 04:34:30 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808770/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808770/ "2022-04-27T18:37:09.136426+0000 osd.3 (osd.3) 35 : cluster [ERR] 5.1s0 deep-scrub : stat mismatch, got 0/12 objects, 0/0 clones, 0/12 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/114630 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes." in cluster log [6808775] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:54:57 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808775/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808775/ Found coredumps on ubuntu@smithi116.front.sepia.ceph.com [6808777] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 02:51:48 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808777/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808777/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6808780] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:30:34 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808780/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808780/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4e6a5ea788f431084783f3e96b48163 Extra data: line 2 column 560 (char 560) [6808783] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 03:36:57 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808783/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808783/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64e64a10aa4d4b9fab16c49f017dfaee Command failed (workunit test suites/fsstress.sh) on smithi059 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=04b6d965e55961a6cb690e35b6f77d0192e1fe45 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [6808792] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 02:11:17 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808792/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808792/ "2022-04-27T15:38:12.237171+0000 mds.e (mds.0) 1 : cluster [WRN] client.4661 isn't responding to mclientcaps(revoke), ino 0x1000000b532 pending pAsLsXsFsc issued pAsLsXsFscb, sent 300.016786 seconds ago" in cluster log [6808793] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:40:53 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808793/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808793/ SELinux denials found on ubuntu@smithi151.front.sepia.ceph.com: ['type=AVC msg=audit(1651071612.513:202): avc: denied { node_bind } for pid=1541 comm="ping" saddr=172.21.15.151 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [6808796] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:16:43 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808796/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808796/ "2022-04-27T15:09:50.656887+0000 mon.a (mon.0) 405 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [6808798] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 01:43:14 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808798/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808798/ "2022-04-27T15:45:54.005855+0000 mds.e (mds.0) 29 : cluster [WRN] client.4816 isn't responding to mclientcaps(revoke), ino 0x10000004298 pending pAsLsXsFscr issued pAsLsXsFscrb, sent 300.042178 seconds ago" in cluster log [6808799] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:30:06 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808799/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808799/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aff5de62c97348f8932ab5aac32bf589 Extra data: line 2 column 555 (char 555) [6808803] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} ----------------------------------------------------------------- time: 02:52:05 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808803/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808803/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [6808808] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:23:35 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808808/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808808/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ce4be22d16014bcbb1568177a43f7623 reached maximum tries (90) after waiting for 540 seconds Dead ================================================================= [6808778] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808778/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808778/ hit max job timeout [6808800] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808800/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808800/ hit max job timeout Pass ================================================================= [6808769] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:40:15 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808769/ [6808771] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:01:29 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808771/ [6808772] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:15:20 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808772/ [6808773] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:53:41 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808773/ [6808774] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:28:39 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808774/ [6808776] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports} time: 00:55:01 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808776/ [6808779] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:42:59 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808779/ [6808781] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:22:44 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808781/ [6808782] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:05:31 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808782/ [6808784] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:21:23 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808784/ [6808785] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:39:27 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808785/ [6808786] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:40:10 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808786/ [6808787] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:15:31 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808787/ [6808788] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:40:30 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808788/ [6808789] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:51:29 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808789/ [6808790] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} time: 00:54:20 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808790/ [6808791] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:15:24 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808791/ [6808794] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} time: 00:37:37 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808794/ [6808795] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:15:02 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808795/ [6808797] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:49:37 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808797/ [6808801] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:30:55 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808801/ [6808802] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:18:53 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808802/ [6808804] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:34:52 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808804/ [6808805] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} time: 00:45:42 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808805/ [6808806] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:45:43 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808806/ [6808807] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:20:50 info: http://pulpito.ceph.com/vshankar-2022-04-27_14:16:20-fs-wip-55170-20220425-134100-testing-default-smithi/6808807/