2022-09-20T18:34:22.040 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-20T18:34:22.042 INFO:teuthology.results:seed: '4877' 2022-09-20T18:43:33.263 INFO:root:teuthology version: 1.1.1.dev418+gb58f2c18 2022-09-20T18:43:33.300 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-09-20T19:10:35.311 INFO:teuthology.results:Tests finished! gathering results... 2022-09-20T19:10:35.663 INFO:scrape:Found 6 jobs 2022-09-20T19:10:37.225 INFO:scrape:Found 1 distinct failure reasons 2022-09-20T19:10:37.226 INFO:scrape:Crash: Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) 2022-09-20T19:10:37.226 INFO:scrape:ceph version 17.0.0-15134-gada1013c (ada1013c357729fc3bf91fdc0bbe1d411f5af166) quincy (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fd17b2a4ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd17c2e6e35] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a6fa1) [0x7fd17c2e6fa1] 6: (CDir::check_rstats(bool)+0x1257) [0x5573459084f7] 7: (MDCache::predirty_journal_parents(boost::intrusive_ptr, EMetaBlob*, CInode*, CDir*, int, int, snapid_t)+0xc20) [0x5573457d21f0] 8: (MDCache::_create_system_file(CDir*, std::basic_string_view >, CInode*, MDSContext*)+0x538) [0x5573457d3b78] 9: (MDCache::populate_mydir()+0x8f1) [0x5573457ee921] 10: (MDSContext::complete(int)+0x5f) [0x5573459d50af] 11: (MDSRank::_advance_queues()+0xaa) [0x55734568d1fa] 12: (MDSRank::ProgressThread::entry()+0xb8) [0x55734568d798] 13: /lib64/libpthread.so.0(+0x81cf) [0x7fd17b29a1cf] 14: clone() 2022-09-20T19:10:37.227 INFO:scrape:6 jobs: ['7039466', '7039465', '7039463', '7039467', '7039464', '7039462'] 2022-09-20T19:10:37.227 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/alternate-pool}'] 2022-09-20T19:10:37.228 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs:functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/alternate-pool}'] 2022-09-20T19:10:37.228 INFO:scrape: 2022-09-20T19:10:37.229 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7039462] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:17:24 info: http://pulpito.ceph.com/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039462/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70ab211a77bc4852b22046b2ae5b363f Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) [7039463] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:13:34 info: http://pulpito.ceph.com/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039463/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039463/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a871fdc7757c4671a35880ab4cdf5b62 Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) [7039464] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:13:55 info: http://pulpito.ceph.com/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039464/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039464/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aebbe803ba424c26a2325c2d08d4d76a Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) [7039465] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:18:03 info: http://pulpito.ceph.com/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039465/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ccbb912a3cd4210ae4628b7ba59293a Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) [7039466] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:13:46 info: http://pulpito.ceph.com/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039466/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039466/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5ae65e0339af48efa152b5160db2459f Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) [7039467] fs:functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:12:17 info: http://pulpito.ceph.com/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039467/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2022-09-20_18:30:11-fs:functional-wip-pdonnell-testing-20220920.151129-distro-default-smithi/7039467/ sentry: https://sentry.ceph.com/organizations/ceph/?query=77cef77361ab4c90a5482aaa53989cf4 Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool)