2023-03-24T23:13:17.530 INFO:root:teuthology version: 0.0.1.dev1+ge372916 2023-03-24T23:13:17.531 INFO:teuthology.results:subset: '103/128' 2023-03-24T23:13:17.531 INFO:teuthology.results:seed: '138' 2023-03-25T00:55:54.139 INFO:root:teuthology version: 0.0.1.dev1+ge372916 2023-03-25T00:55:54.166 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-25T12:56:50.507 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-03-25T12:56:53.025 INFO:scrape:Found 31 jobs 2023-03-25T12:57:13.931 INFO:scrape:Found 9 distinct failure reasons 2023-03-25T12:57:13.932 INFO:scrape:Dead: b'2023-03-25T12:33:03.563 DEBUG:teuthology.exit:Finished running handlers' 2023-03-25T12:57:13.932 INFO:scrape:4 jobs: ['7219649', '7219647', '7219637', '7219633'] 2023-03-25T12:57:13.933 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}'] 2023-03-25T12:57:13.933 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_20.04}', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/fscrypt-common}', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-03-25T12:57:13.933 INFO:scrape: 2023-03-25T12:57:13.934 INFO:scrape:Failure: Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-03-25T12:57:13.934 INFO:scrape:1 jobs: ['7219654'] 2023-03-25T12:57:13.934 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}'] 2023-03-25T12:57:13.934 INFO:scrape: 2023-03-25T12:57:13.935 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds 2023-03-25T12:57:13.935 INFO:scrape:5 jobs: ['7219631', '7219639', '7219638', '7219660', '7219655'] 2023-03-25T12:57:13.935 INFO:scrape:suites intersection: ['2-logrotate}'] 2023-03-25T12:57:13.936 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', 'begin/{0-install', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/mirror-ha/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{whitelist_health}', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'supported-random-distro$/{rhel_8}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'workloads/cephfs-mirror-ha-workunit}', 'wsync/no}'] 2023-03-25T12:57:13.936 INFO:scrape: 2023-03-25T12:57:13.936 INFO:scrape:Failure: Test failure: test_rebuild_missing_zeroth (tasks.cephfs.test_data_scan.TestDataScan) 2023-03-25T12:57:13.937 INFO:scrape:2 jobs: ['7219656', '7219632'] 2023-03-25T12:57:13.937 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/data-scan}'] 2023-03-25T12:57:13.937 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/data-scan}'] 2023-03-25T12:57:13.938 INFO:scrape: 2023-03-25T12:57:13.938 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1e7de2f05230c7419acf5ca7f5fd22964e30ab77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-03-25T12:57:13.938 INFO:scrape:1 jobs: ['7219651'] 2023-03-25T12:57:13.939 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-03-25T12:57:13.939 INFO:scrape: 2023-03-25T12:57:13.939 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1e7de2f05230c7419acf5ca7f5fd22964e30ab77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2023-03-25T12:57:13.940 INFO:scrape:5 jobs: ['7219650', '7219659', '7219640', '7219645', '7219636'] 2023-03-25T12:57:13.940 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2023-03-25T12:57:13.941 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2023-03-25T12:57:13.941 INFO:scrape: 2023-03-25T12:57:13.941 INFO:scrape:Crash: Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage) 2023-03-25T12:57:13.941 INFO:scrape:ceph version 18.0.0-3124-gc19e8f2a (c19e8f2a497e95f75abc43ec14bc6095e30dd2cb) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f40cb2c6420] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x190) [0x7f40ca71e95c] 5: (CDentry::check_corruption(bool)+0x613) [0x56400289e223] 6: ceph-mds(+0x44186a) [0x5640028b986a] 7: (CDir::_omap_commit(int)+0x30a) [0x5640028b9e3a] 8: (CDir::commit(unsigned long, MDSContext*, bool, int)+0x12d) [0x5640028bb05d] 9: (LogSegment::try_to_expire(MDSRank*, C_GatherBuilderBase >&, int)+0x2576) [0x564002a01986] 10: (MDLog::try_expire(LogSegment*, int)+0x6a) [0x56400298e8da] 11: (MDLog::trim_all()+0x6e4) [0x564002992bc4] 12: (C_Flush_Journal::trim_mdlog()+0x14b) [0x564002667b8b] 13: (Context::complete(int)+0xd) [0x564002624a0d] 14: (MDSContext::complete(int)+0x5f) [0x564002977eef] 15: (C_IO_Wrapper::finish(int)+0x12) [0x564002978d22] 16: (MDSContext::complete(int)+0x5f) [0x564002977eef] 17: (MDSIOContextBase::complete(int)+0x514) [0x564002978654] 18: (Finisher::finisher_thread_entry()+0x18d) [0x7f40ca7c4e9d] 19: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f40cb2ba609] 20: clone() 2023-03-25T12:57:13.942 INFO:scrape:1 jobs: ['7219630'] 2023-03-25T12:57:13.942 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/damage}'] 2023-03-25T12:57:13.942 INFO:scrape: 2023-03-25T12:57:13.943 INFO:scrape:Failure: Command failed on smithi146 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-03-25T12:57:13.943 INFO:scrape:5 jobs: ['7219646', '7219643', '7219657', '7219634', '7219652'] 2023-03-25T12:57:13.943 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2023-03-25T12:57:13.944 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distros/ubuntu_latest', 'frag', 'fs/multiclient/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/ior-shared-file}', 'tasks/mdtest}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2023-03-25T12:57:13.944 INFO:scrape: 2023-03-25T12:57:13.944 INFO:scrape:Crash: Test failure: test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2023-03-25T12:57:13.944 INFO:scrape:ceph version 18.0.0-3124-gc19e8f2a (c19e8f2a497e95f75abc43ec14bc6095e30dd2cb) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fb20a387ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb20b5d4e19] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7fb20b5d4f85] 6: (EMetaBlob::replay(MDSRank*, LogSegment*, MDPeerUpdate*)+0x521e) [0x563845836e4e] 7: (EUpdate::replay(MDSRank*)+0x40) [0x563845838640] 8: (MDLog::_replay_thread()+0x66d) [0x5638457bf27d] 9: (MDLog::ReplayThread::entry()+0x11) [0x563845472f21] 10: /lib64/libpthread.so.0(+0x81cf) [0x7fb20a37d1cf] 11: clone() 2023-03-25T12:57:13.945 INFO:scrape:1 jobs: ['7219658'] 2023-03-25T12:57:13.945 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/forward-scrub}'] 2023-03-25T12:57:13.945 INFO:scrape: 2023-03-25T12:57:13.948 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/ failed: 21 dead: 2 running: 2 waiting: 0 queued: 0 passed: 6 Fail ================================================================= [7219630] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:33:20 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219630/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219630/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0de6dd47d3f44ca5a5d16cf5cdc55fc6 Test failure: test_object_deletion (tasks.cephfs.test_damage.TestDamage) [7219631] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:32:50 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219631/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219631/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7219632] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:31:59 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219632/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219632/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a12759da98a94649a2c6e5bb5e81a596 Test failure: test_rebuild_missing_zeroth (tasks.cephfs.test_data_scan.TestDataScan) [7219634] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:14:15 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219634/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219634/ sentry: https://sentry.ceph.com/organizations/ceph/?query=28ea5c36cffc40b5b23b4d384cfccd09 Command failed on smithi179 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7219636] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:26:51 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219636/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219636/ sentry: https://sentry.ceph.com/organizations/ceph/?query=66c4253d785d4ba1a1b6588e51a1663d Command failed (workunit test suites/pjd.sh) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1e7de2f05230c7419acf5ca7f5fd22964e30ab77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7219638] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:25:39 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219638/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219638/ reached maximum tries (50) after waiting for 300 seconds [7219639] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:12:22 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219639/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219639/ error during scrub thrashing: rank damage found: {'backtrace'} [7219640] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:32 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219640/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219640/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f507dcc782c41e08653c7e1bcdb4333 Command failed (workunit test suites/pjd.sh) on smithi107 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1e7de2f05230c7419acf5ca7f5fd22964e30ab77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7219643] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:14:07 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219643/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219643/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89166ad7e6cc488baa491ee57c5bd320 Command failed on smithi179 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7219645] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:25:09 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219645/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219645/ sentry: https://sentry.ceph.com/organizations/ceph/?query=82f13d8ab0ec494d99767287d8210e60 Command failed (workunit test suites/pjd.sh) on smithi123 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1e7de2f05230c7419acf5ca7f5fd22964e30ab77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7219646] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:14:11 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219646/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219646/ sentry: https://sentry.ceph.com/organizations/ceph/?query=21b75889fb954b7da39dcf6d382de86f Command failed on smithi146 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7219650] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:11 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219650/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219650/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8e8dd24f5ae4d4c80bf29a642294caf Command failed (workunit test suites/pjd.sh) on smithi057 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1e7de2f05230c7419acf5ca7f5fd22964e30ab77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7219651] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:47:16 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219651/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219651/ sentry: https://sentry.ceph.com/organizations/ceph/?query=85ebc3878512417d9f0b486962dc9b05 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi132 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1e7de2f05230c7419acf5ca7f5fd22964e30ab77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7219652] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:12:51 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219652/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219652/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e71d047cdea46ad9292590a7f7b675b Command failed on smithi099 with status 5: 'sudo systemctl stop ceph-3361c7f2-caa6-11ed-9afe-001a4aab830c@mon.b' [7219654] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:28:32 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219654/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219654/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1de6331be2e2491681f4502d8d63ebdd Test failure: test_reconnect_after_blocklisted (tasks.cephfs.test_client_recovery.TestClientRecovery) [7219655] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:56:41 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219655/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219655/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7219656] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:36:34 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219656/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219656/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ba92c7ff027045c68245dd0556409bd3 Test failure: test_rebuild_missing_zeroth (tasks.cephfs.test_data_scan.TestDataScan) [7219657] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:14:25 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219657/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219657/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49cb5cb6e3d7458095b54f8828ba056a Command failed on smithi146 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7219658] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:16:21 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219658/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219658/ sentry: https://sentry.ceph.com/organizations/ceph/?query=708db6ec6831498893b474a5543188af Test failure: test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) [7219659] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:25:45 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219659/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219659/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a84133f37dfb4fd285701cbfb165806e Command failed (workunit test suites/pjd.sh) on smithi073 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1e7de2f05230c7419acf5ca7f5fd22964e30ab77 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7219660] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:04:33 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219660/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219660/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds Dead ================================================================= [7219633] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219633/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219633/ hit max job timeout [7219637] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219637/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219637/ hit max job timeout Running ================================================================= [7219647] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219647/ [7219649] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219649/ Pass ================================================================= [7219635] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:21:20 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219635/ [7219641] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 01:34:21 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219641/ [7219642] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:23:30 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219642/ [7219644] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:26:27 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219644/ [7219648] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:15:36 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219648/ [7219653] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 02:08:35 info: http://pulpito.ceph.com/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219653/