2022-06-03T04:44:55.202 INFO:root:teuthology version: 1.1.1.dev360+g53fc7fef 2022-06-03T04:44:55.203 INFO:teuthology.results:seed: '2697' 2022-06-03T04:46:56.292 INFO:root:teuthology version: 1.1.1.dev360+g53fc7fef 2022-06-03T04:46:56.298 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-03T05:57:02.234 INFO:teuthology.results:Tests finished! gathering results... 2022-06-03T05:57:02.716 INFO:scrape:Found 6 jobs 2022-06-03T05:57:03.994 INFO:scrape:Found 2 distinct failure reasons 2022-06-03T05:57:03.994 INFO:scrape:Crash: Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) 2022-06-03T05:57:03.995 INFO:scrape:ceph version 17.0.0-12513-g11df58ac (11df58acf2435c70639a65f781a743218ae38732) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f29ee2b2c20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f29ef2d3772] 5: /usr/lib64/ceph/libceph-common.so.2(+0x285935) [0x7f29ef2d3935] 6: (MDCache::uninline_data_work(boost::intrusive_ptr)+0x1262) [0x55abd3696ad2] 7: (MDCache::dispatch_request(boost::intrusive_ptr&)+0xc4) [0x55abd36a3e54] 8: (MDSContext::complete(int)+0x203) [0x55abd38751b3] 9: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x55abd3548cbd] 10: (Locker::eval(CInode*, int, bool)+0x4d8) [0x55abd3742a88] 11: (Locker::handle_client_caps(boost::intrusive_ptr const&)+0x1c9b) [0x55abd375280b] 12: (Locker::dispatch(boost::intrusive_ptr const&)+0x114) [0x55abd3754604] 13: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xd54) [0x55abd354fde4] 14: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x7cb) [0x55abd3552a1b] 15: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55abd355303c] 16: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x108) [0x55abd3541f38] 17: (DispatchQueue::entry()+0x14fa) [0x7f29ef55d9ca] 18: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f29ef615f21] 19: /lib64/libpthread.so.0(+0x817a) [0x7f29ee2a817a] 20: clone() 2022-06-03T05:57:03.995 INFO:scrape:4 jobs: ['6861708', '6861707', '6861704', '6861705'] 2022-06-03T05:57:03.995 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/uninlining}'] 2022-06-03T05:57:03.995 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/uninlining}'] 2022-06-03T05:57:03.995 INFO:scrape: 2022-06-03T05:57:03.995 INFO:scrape:Crash: Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining), test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) 2022-06-03T05:57:03.996 INFO:scrape:ceph version 17.0.0-12513-g11df58ac (11df58acf2435c70639a65f781a743218ae38732) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f63f1babce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f63f2bcc772] 5: /usr/lib64/ceph/libceph-common.so.2(+0x285935) [0x7f63f2bcc935] 6: ceph-mds(+0x49cc80) [0x55ffb4644c80] 7: (Continuation::_continue_function(int, int)+0x163) [0x55ffb4650603] 8: (CInode::validate_disk_state(CInode::validated_data*, MDSContext*)+0x4dc) [0x55ffb4642dbc] 9: (MDSContext::complete(int)+0x203) [0x55ffb46a71b3] 10: (C_MDC_DataUninlinedSubmitted::finish(int)+0x8f) [0x55ffb453b0ff] 11: (MDSContext::complete(int)+0x203) [0x55ffb46a71b3] 12: (MDSIOContextBase::complete(int)+0x69c) [0x55ffb46a794c] 13: (MDSLogContextBase::complete(int)+0x44) [0x55ffb46a7bc4] 14: (Finisher::finisher_thread_entry()+0x19d) [0x7f63f2c783bd] 15: /lib64/libpthread.so.0(+0x81ca) [0x7f63f1ba11ca] 16: clone() 2022-06-03T05:57:03.996 INFO:scrape:1 jobs: ['6861709'] 2022-06-03T05:57:03.997 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/uninlining}'] 2022-06-03T05:57:03.997 INFO:scrape: 2022-06-03T05:57:03.997 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [6861704] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} ----------------------------------------------------------------- time: 00:31:53 info: http://pulpito.ceph.com/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6861704/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6861704/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d4061d2d385445cb9a2e063a1fe8460d Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining), test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) [6861705] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} ----------------------------------------------------------------- time: 00:15:28 info: http://pulpito.ceph.com/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6861705/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6861705/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d4f39cd4ad0340cb8334d61ec1d865a4 Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) [6861707] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} ----------------------------------------------------------------- time: 00:19:47 info: http://pulpito.ceph.com/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6861707/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6861707/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3f6f4e78b7ac4def941702fa7abc8cd2 Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) [6861708] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} ----------------------------------------------------------------- time: 00:14:53 info: http://pulpito.ceph.com/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6861708/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6861708/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eef9a1a40ab4434b903d9dd81e49a282 Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) [6861709] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} ----------------------------------------------------------------- time: 00:26:04 info: http://pulpito.ceph.com/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6861709/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6861709/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea95ed7f0a3d47bd84a2a4070566262c Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining), test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) Pass ================================================================= [6861706] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} time: 01:02:08 info: http://pulpito.ceph.com/mchangir-2022-06-03_04:43:56-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6861706/