2023-03-14T04:07:43.782 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-14T04:07:43.783 INFO:teuthology.results:subset: '78/1024' 2023-03-14T04:07:43.783 INFO:teuthology.results:seed: '8081' 2023-03-14T05:17:04.054 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-14T05:17:04.097 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-14T17:18:03.697 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2023-03-14T17:18:08.275 INFO:scrape:Found 46 jobs 2023-03-14T17:18:20.903 WARNING:scrape:Misunderstood line: 2023-03-14T17:18:33.018 INFO:scrape:Found 18 distinct failure reasons 2023-03-14T17:18:33.019 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi140 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-03-14T17:18:33.020 INFO:scrape:1 jobs: ['7206211'] 2023-03-14T17:18:33.020 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2023-03-14T17:18:33.020 INFO:scrape: 2023-03-14T17:18:33.021 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-03-14T17:18:33.021 INFO:scrape:2 jobs: ['7206210', '7206175'] 2023-03-14T17:18:33.021 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2023-03-14T17:18:33.022 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-03-14T17:18:33.022 INFO:scrape: 2023-03-14T17:18:33.022 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-03-14T17:18:33.023 INFO:scrape:5 jobs: ['7206185', '7206219', '7206184', '7206202', '7206177'] 2023-03-14T17:18:33.023 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}'] 2023-03-14T17:18:33.023 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/suites/fsstress}}', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'ranks/5', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout', 'session_timeout}', 'standby-replay', 'tasks/fsstress', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}', 'validater/valgrind}', 'wsync/yes}'] 2023-03-14T17:18:33.024 INFO:scrape: 2023-03-14T17:18:33.024 INFO:scrape:Dead: b'2023-03-14T16:25:42.529 DEBUG:teuthology.exit:Finished running handlers' 2023-03-14T17:18:33.024 INFO:scrape:5 jobs: ['7206181', '7206206', '7206221', '7206174', '7206212'] 2023-03-14T17:18:33.025 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}'] 2023-03-14T17:18:33.025 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu/{latest', 'distro/{ubuntu_latest}', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'overrides}}', 'pg-warn}', 'ranks/3', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/data-scan}', 'tasks/fscrypt-common}', 'tasks/fsstress', 'tasks/volumes/{overrides', 'tasks/{0-subvolume/{with-no-extra-options}', 'test/basic}}', 'validater/lockdep}', 'wsync/no}'] 2023-03-14T17:18:33.025 INFO:scrape: 2023-03-14T17:18:33.026 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/dbench.sh 2023-03-14T17:18:33.026 INFO:scrape:1 jobs: ['7206222'] 2023-03-14T17:18:33.026 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'wsync/yes}'] 2023-03-14T17:18:33.027 INFO:scrape: 2023-03-14T17:18:33.027 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi046 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2023-03-14T17:18:33.027 INFO:scrape:1 jobs: ['7206204'] 2023-03-14T17:18:33.028 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2023-03-14T17:18:33.028 INFO:scrape: 2023-03-14T17:18:33.028 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi101 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' 2023-03-14T17:18:33.029 INFO:scrape:1 jobs: ['7206195'] 2023-03-14T17:18:33.029 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}'] 2023-03-14T17:18:33.030 INFO:scrape: 2023-03-14T17:18:33.030 INFO:scrape:Failure: Test failure: test_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize) 2023-03-14T17:18:33.030 INFO:scrape:1 jobs: ['7206199'] 2023-03-14T17:18:33.030 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/admin}'] 2023-03-14T17:18:33.031 INFO:scrape: 2023-03-14T17:18:33.031 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2023-03-14T17:18:33.031 INFO:scrape:1 jobs: ['7206183'] 2023-03-14T17:18:33.031 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-03-14T17:18:33.032 INFO:scrape: 2023-03-14T17:18:33.033 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-03-14T17:18:33.034 INFO:scrape:1 jobs: ['7206217'] 2023-03-14T17:18:33.034 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-03-14T17:18:33.034 INFO:scrape: 2023-03-14T17:18:33.034 INFO:scrape:Crash: Command failed on smithi174 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp' 2023-03-14T17:18:33.035 INFO:scrape:ceph version 18.0.0-2942-g3727df45 (3727df45d557f35807d41b1592494b2fbb9bdfec) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f12e6212ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f12e76b6e19] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2b6f85) [0x7f12e76b6f85] 6: (Client::add_update_cap(Inode*, MetaSession*, unsigned long, unsigned int, unsigned int, unsigned int, unsigned int, inodeno_t, int, UserPerm const&)+0xc53) [0x56417ce5ee83] 7: (Client::add_update_inode(InodeStat*, utime_t, MetaSession*, UserPerm const&)+0x943) [0x56417ce60253] 8: (Client::insert_readdir_results(MetaRequest*, MetaSession*, Inode*)+0x75c) [0x56417ce6133c] 9: (Client::insert_trace(MetaRequest*, MetaSession*)+0xf58) [0x56417ce63758] 10: (Client::handle_client_reply(boost::intrusive_ptr const&)+0x3a0) [0x56417ce642a0] 11: (Client::ms_dispatch2(boost::intrusive_ptr const&)+0x58d) [0x56417ce8658d] 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f12e792ebc8] 13: (DispatchQueue::entry()+0x50f) [0x7f12e792bd6f] 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f12e79f38a1] 15: /lib64/libpthread.so.0(+0x81cf) [0x7f12e62081cf] 16: clone() 2023-03-14T17:18:33.035 INFO:scrape:1 jobs: ['7206215'] 2023-03-14T17:18:33.035 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-03-14T17:18:33.036 INFO:scrape: 2023-03-14T17:18:33.036 INFO:scrape:Timeout 120 running ceph --cluster ceph tell mds.1:0 scrub status 2023-03-14T17:18:33.036 INFO:scrape:1 jobs: ['7206178'] 2023-03-14T17:18:33.037 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2023-03-14T17:18:33.037 INFO:scrape: 2023-03-14T17:18:33.037 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/dbench.sh 2023-03-14T17:18:33.038 INFO:scrape:2 jobs: ['7206176', '7206194'] 2023-03-14T17:18:33.038 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/kernel_cfuse_workunits_dbench_iozone}'] 2023-03-14T17:18:33.038 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'rhel_8}', 'tasks/kernel_cfuse_workunits_dbench_iozone}'] 2023-03-14T17:18:33.039 INFO:scrape: 2023-03-14T17:18:33.039 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/fs/fscrypt.sh none dbench 2023-03-14T17:18:33.039 INFO:scrape:1 jobs: ['7206193'] 2023-03-14T17:18:33.039 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-dbench}'] 2023-03-14T17:18:33.040 INFO:scrape: 2023-03-14T17:18:33.040 INFO:scrape:Failure: Test failure: test_client_release_bug (tasks.cephfs.test_client_limits.TestClientLimits) 2023-03-14T17:18:33.040 INFO:scrape:1 jobs: ['7206208'] 2023-03-14T17:18:33.041 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}'] 2023-03-14T17:18:33.041 INFO:scrape: 2023-03-14T17:18:33.041 INFO:scrape:Failure: Command failed on smithi143 with status 5: 'sudo systemctl stop ceph-14625448-c227-11ed-9af2-001a4aab830c@mon.b' 2023-03-14T17:18:33.042 INFO:scrape:1 jobs: ['7206214'] 2023-03-14T17:18:33.042 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-03-14T17:18:33.042 INFO:scrape: 2023-03-14T17:18:33.043 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2023-03-14T17:18:33.043 INFO:scrape:1 jobs: ['7206182'] 2023-03-14T17:18:33.044 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror-ha/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/cephfs-mirror-ha-workunit}'] 2023-03-14T17:18:33.044 INFO:scrape: 2023-03-14T17:18:33.044 INFO:scrape:Crash: Command failed on smithi064 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' 2023-03-14T17:18:33.045 INFO:scrape:ceph version 18.0.0-2942-g3727df45 (3727df45d557f35807d41b1592494b2fbb9bdfec) reef (dev) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fee94012cf0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fee95cb6e19] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2b6f85) [0x7fee95cb6f85] 6: ceph-mds(+0x2aa33f) [0x564385c7e33f] 7: (MDCache::request_cleanup(boost::intrusive_ptr&)+0xe6) [0x564385ce6d26] 8: (Server::reply_client_request(boost::intrusive_ptr&, boost::intrusive_ptr const&)+0x724) [0x564385bf2f84] 9: (Server::respond_to_request(boost::intrusive_ptr&, int)+0x350) [0x564385bf5750] 10: (Server::handle_client_getattr(boost::intrusive_ptr&, bool)+0x6de) [0x564385c136de] 11: (MDSContext::complete(int)+0x5f) [0x564385ed8fcf] 12: (void finish_contexts > >(ceph::common::CephContext*, std::vector >&, int)+0x8d) [0x564385b7d05d] 13: (MDSCacheObject::finish_waiting(unsigned long, int)+0x5c) [0x564385ee9f4c] 14: (Locker::handle_file_lock(ScatterLock*, boost::intrusive_ptr const&)+0x840) [0x564385d95fd0] 15: (Locker::dispatch(boost::intrusive_ptr const&)+0x174) [0x564385da5674] 16: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a3) [0x564385b8bc53] 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x564385b8c26c] 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x564385b75c1f] 19: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fee95f2ebc8] 20: (DispatchQueue::entry()+0x50f) [0x7fee95f2bd6f] 21: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fee95ff38a1] 22: /lib64/libpthread.so.0(+0x81ca) [0x7fee940081ca] 23: clone() 2023-03-14T17:18:33.045 INFO:scrape:3 jobs: ['7206213', '7206197', '7206179'] 2023-03-14T17:18:33.045 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'msgr-failures/none', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'races', 'ranks/3', 'session_timeout', 'thrashosds-health}'] 2023-03-14T17:18:33.046 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2023-03-14T17:18:33.046 INFO:scrape: 2023-03-14T17:18:33.050 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/ failed: 25 dead: 4 running: 1 waiting: 0 queued: 0 passed: 16 Fail ================================================================= [7206175] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:54:35 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206175/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206175/ error during scrub thrashing: rank damage found: {'backtrace'} [7206176] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 03:15:46 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206176/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206176/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5556c95e32941869b6563cea48eb398 Command failed (workunit test suites/dbench.sh) on smithi187 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' [7206177] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 06:27:35 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206177/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206177/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4907b8ca86454af982357c57e792cb75 Command failed (workunit test suites/fsstress.sh) on smithi161 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7206178] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:00:52 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206178/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206178/ error during scrub thrashing: Command failed on smithi035 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' [7206179] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:29:45 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206179/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206179/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd11b1181bc04da38f220a52c460ab2b Command failed (workunit test fs/snaps/snaptest-1.sh) on smithi103 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh' [7206182] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:48:09 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206182/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206182/ reached maximum tries (50) after waiting for 300 seconds [7206183] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:41:47 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206183/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206183/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0bd315afca5c4c7d8f4b90a54cc7a45c Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7206184] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 06:38:40 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206184/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206184/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e7462caf767943ad97c5681216c8adca Command failed (workunit test suites/fsstress.sh) on smithi145 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7206185] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:44:06 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206185/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206185/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0cb2bd9d2284ad3a60ba92f465feefa Command failed (workunit test suites/fsstress.sh) on smithi053 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7206193] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} ----------------------------------------------------------------- time: 06:20:07 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206193/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206193/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c5c626473f3b47dabeeb811a0ed4fba4 Command failed (workunit test fs/fscrypt.sh) on smithi138 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none dbench' [7206194] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 03:14:15 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206194/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206194/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e4fe5892cfc4b04bb6db655d6579094 Command failed (workunit test suites/dbench.sh) on smithi136 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/dbench.sh' [7206195] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:14:34 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206195/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206195/ sentry: https://sentry.ceph.com/organizations/ceph/?query=80f83d6e1ae145aa9dbbde3e51190dc1 Command failed (workunit test fs/quota/quota.sh) on smithi101 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' [7206197] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:25:08 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206197/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206197/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9565d6be826c4f2eade3949754c58a68 Command failed on smithi033 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' [7206199] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:49:51 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206199/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206199/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d8d685480c8d404789e6385e9377a3df Test failure: test_single_path_rootsquash (tasks.cephfs.test_admin.TestFsAuthorize) [7206202] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 06:28:18 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206202/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206202/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f08ab9038f244154a191f72d9245c7fb Command failed (workunit test suites/fsstress.sh) on smithi017 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7206204] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:46 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206204/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206204/ sentry: https://sentry.ceph.com/organizations/ceph/?query=493b04840af54853addd599cdfa00d22 Command failed (workunit test suites/pjd.sh) on smithi046 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7206208] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:32:44 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206208/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206208/ sentry: https://sentry.ceph.com/organizations/ceph/?query=09a0aa50885b4d4d894b395ca2e6d7a6 Test failure: test_client_release_bug (tasks.cephfs.test_client_limits.TestClientLimits) [7206210] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:50:11 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206210/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206210/ error during scrub thrashing: rank damage found: {'backtrace'} [7206211] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:20:21 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206211/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206211/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1cf206f8287146f29a6434d9bc50f43c Command failed (workunit test libcephfs/test.sh) on smithi140 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7206213] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:29:58 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206213/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206213/ sentry: https://sentry.ceph.com/organizations/ceph/?query=072bbdbcd2ad497d9c2a0f4002458726 Command failed on smithi064 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp' [7206214] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:12:37 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206214/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206214/ sentry: https://sentry.ceph.com/organizations/ceph/?query=48592b6051b8434a938b24320c9aa383 Command failed on smithi143 with status 5: 'sudo systemctl stop ceph-14625448-c227-11ed-9af2-001a4aab830c@mon.b' [7206215] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:25:39 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206215/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206215/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22d5f8cff6894d0bb07ee96b3599f1bd Command failed on smithi174 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp' [7206217] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 02:02:05 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206217/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206217/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7afd442c51ca4591a92ec316982a0689 saw valgrind issues [7206219] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 06:20:49 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206219/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206219/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e044daed9a64633a356612539def9e8 Command failed (workunit test suites/fsstress.sh) on smithi047 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7206222] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 03:27:49 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206222/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206222/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d589f4e4174c47a5809f1156d86b11ef Command failed (workunit test suites/dbench.sh) on smithi016 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3727df45d557f35807d41b1592494b2fbb9bdfec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' Dead ================================================================= [7206174] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206174/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206174/ hit max job timeout [7206181] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206181/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206181/ hit max job timeout [7206206] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206206/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206206/ hit max job timeout [7206212] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206212/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206212/ hit max job timeout Running ================================================================= [7206221] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206221/ Pass ================================================================= [7206173] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:17:53 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206173/ [7206180] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:25:35 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206180/ [7206186] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 01:00:20 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206186/ [7206187] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} time: 00:39:36 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206187/ [7206188] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:36:28 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206188/ [7206189] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:23:11 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206189/ [7206190] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:19:45 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206190/ [7206191] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:39:54 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206191/ [7206192] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:17:57 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206192/ [7206196] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 00:54:50 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206196/ [7206198] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:03:23 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206198/ [7206200] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} time: 00:32:55 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206200/ [7206201] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:32:48 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206201/ [7206216] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 00:49:16 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206216/ [7206218] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:16:48 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206218/ [7206220] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:37:50 info: http://pulpito.ceph.com/vshankar-2023-03-14_04:05:49-fs-wip-vshankar-testing-20230310.111733-testing-default-smithi/7206220/