2022-05-31T14:48:11.910 INFO:root:teuthology version: 1.1.1.dev348+g8df7ff2b 2022-05-31T14:48:11.911 INFO:teuthology.results:seed: '5936' 2022-05-31T14:59:09.730 INFO:root:teuthology version: 1.1.1.dev348+g8df7ff2b 2022-05-31T14:59:09.752 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-05-31T16:42:17.515 INFO:teuthology.results:Tests finished! gathering results... 2022-05-31T16:42:18.010 INFO:scrape:Found 6 jobs 2022-05-31T16:42:19.167 INFO:scrape:Found 2 distinct failure reasons 2022-05-31T16:42:19.168 INFO:scrape:Failure: Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) 2022-05-31T16:42:19.168 INFO:scrape:1 jobs: ['6855653'] 2022-05-31T16:42:19.168 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/uninlining}'] 2022-05-31T16:42:19.168 INFO:scrape: 2022-05-31T16:42:19.168 INFO:scrape:Crash: Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining), test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) 2022-05-31T16:42:19.168 INFO:scrape:ceph version 17.0.0-12513-gd8fda3d7 (d8fda3d7eba77a49f17bd41db1955341219ffed8) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fa63d012ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7fa63e885772] 5: /usr/lib64/ceph/libceph-common.so.2(+0x285935) [0x7fa63e885935] 6: ceph-mds(+0x49bf70) [0x55d21da9bf70] 7: (Continuation::_continue_function(int, int)+0x163) [0x55d21daa78e3] 8: (CInode::validate_disk_state(CInode::validated_data*, MDSContext*)+0x4dc) [0x55d21da9a0ac] 9: (MDSContext::complete(int)+0x203) [0x55d21dafe303] 10: (MDCache::uninline_data_work(boost::intrusive_ptr)+0x2e3) [0x55d21d91fa23] 11: (MDCache::dispatch_request(boost::intrusive_ptr&)+0xc4) [0x55d21d92e0b4] 12: (MDSContext::complete(int)+0x203) [0x55d21dafe303] 13: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x55d21d7d2b7d] 14: (Locker::eval(CInode*, int, bool)+0x4d8) [0x55d21d9cccf8] 15: (Locker::handle_client_caps(boost::intrusive_ptr const&)+0x1c9b) [0x55d21d9dca7b] 16: (Locker::dispatch(boost::intrusive_ptr const&)+0x114) [0x55d21d9de874] 17: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xd54) [0x55d21d7d9ca4] 18: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x7cb) [0x55d21d7dc8db] 19: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d21d7dcefc] 20: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x108) [0x55d21d7cbdf8] 21: (DispatchQueue::entry()+0x14fa) [0x7fa63eb0f9ca] 22: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa63ebc7f21] 23: /lib64/libpthread.so.0(+0x81ca) [0x7fa63d0081ca] 24: clone() 2022-05-31T16:42:19.168 INFO:scrape:3 jobs: ['6855652', '6855654', '6855650'] 2022-05-31T16:42:19.169 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/uninlining}'] 2022-05-31T16:42:19.169 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/uninlining}'] 2022-05-31T16:42:19.169 INFO:scrape: 2022-05-31T16:42:19.169 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [6855650] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} ----------------------------------------------------------------- time: 00:23:38 info: http://pulpito.ceph.com/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6855650/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6855650/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0acde84157744e02b9ffadb9b9823b18 Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining), test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) [6855652] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} ----------------------------------------------------------------- time: 00:31:30 info: http://pulpito.ceph.com/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6855652/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6855652/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bfb2dcc414344e078fdb4df076969acc Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining), test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) [6855653] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} ----------------------------------------------------------------- time: 00:17:57 info: http://pulpito.ceph.com/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6855653/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6855653/ sentry: https://sentry.ceph.com/organizations/ceph/?query=322b3421c4be4a308666f321d5a4b3ce Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) [6855654] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} ----------------------------------------------------------------- time: 00:25:09 info: http://pulpito.ceph.com/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6855654/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6855654/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2aa6c78a33b84c2db8b440d98764bec0 Test failure: test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining), test_data_uninlining (tasks.cephfs.test_uninlining.TestDataUninlining) Pass ================================================================= [6855649] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} time: 01:38:56 info: http://pulpito.ceph.com/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6855649/ [6855651] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/uninlining} time: 00:58:16 info: http://pulpito.ceph.com/mchangir-2022-05-31_14:46:34-fs-wip-mchangir-mds-uninline-file-during-scrub-distro-default-smithi/6855651/