2023-05-01T20:50:53.870 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-01T20:50:53.871 INFO:teuthology.results:subset: '93/128' 2023-05-01T20:50:53.871 INFO:teuthology.results:seed: '9800' 2023-05-02T07:10:53.699 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-02T07:10:53.736 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-02T16:56:52.391 INFO:teuthology.results:Tests finished! gathering results... 2023-05-02T16:57:22.756 INFO:scrape:Found 344 jobs 2023-05-02T16:58:46.983 INFO:scrape:Found 22 distinct failure reasons 2023-05-02T16:58:46.984 INFO:scrape:Failure: Command failed on smithi100 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 28849702-e882-11ed-9b00-001a4aab830c -- ceph daemon mds.e perf dump' 2023-05-02T16:58:46.985 INFO:scrape:73 jobs 2023-05-02T16:58:46.986 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2023-05-02T16:58:46.987 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/direct_io}}', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/fs/test_o_trunc}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/fsync-tester}}', '5-workunit/suites/iogen}}', '5-workunit/suites/iozone}}', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/1', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2023-05-02T16:58:46.987 INFO:scrape: 2023-05-02T16:58:46.987 INFO:scrape:Failure: "1682984190.0175424 osd.3 (osd.3) 3 : cluster [WRN] OSD bench result of 150124.227799 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.3. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-05-02T16:58:46.988 INFO:scrape:1 jobs: ['7259970'] 2023-05-02T16:58:46.988 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/mds-full}'] 2023-05-02T16:58:46.988 INFO:scrape: 2023-05-02T16:58:46.989 INFO:scrape:Failure: Command failed on smithi170 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' 2023-05-02T16:58:46.989 INFO:scrape:31 jobs 2023-05-02T16:58:46.990 INFO:scrape:suites intersection: [] 2023-05-02T16:58:46.990 INFO:scrape:suites union: ['1-ceph', '1-client-sanity}}}', '1-mount/mount/fuse', '1-tests/blogbench}}}', '1-tests/dbench}}}', '1-tests/fsstress}}}', '1-tests/iozone}}}', '1-tests/kernel_untar_build}}}', '2-logrotate}', '2-workload/new_ops/{0-clients/fuse-upgrade', '2-workload/stress_tests/{0-client-upgrade', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/fs/snaps}}', '2-workunit/fs/trivial_sync}}', '2-workunit/iozone}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/iozone}}', 'bluestore-bitmap', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1-mds-1-client', 'clusters/1-mds-1-client-coloc', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'clusters/small-cluster', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'debug', 'distro/{centos_8}', 'distro/{centos_latest}', 'fs/32bits/{begin/{0-install', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'fs/valgrind/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/v1', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/whitelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{distro/testing/k-testing}}', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'session_timeout', 'tasks/alternate-pool}', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/failover}', 'tasks/forward-scrub}', 'tasks/fragment}', 'tasks/fscrypt-common}', 'tasks/fscrypt-dbench}', 'tasks/fscrypt-ffsb}', 'tasks/fscrypt-pjd}', 'tasks/libcephfs/{frag', 'tasks/mirror}}', 'tasks/multifs-auth}', 'tasks/trim-i24137}', 'tasks/volumes/{overrides', 'tasks/workunit/snaps}', 'tasks/{0-from/nautilus', 'tasks/{0-from/pacific', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/snapshot}}', 'test}}', 'thrashosds-health}'] 2023-05-02T16:58:46.990 INFO:scrape: 2023-05-02T16:58:46.991 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-05-02T16:58:46.991 INFO:scrape:1 jobs: ['7260189'] 2023-05-02T16:58:46.991 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-05-02T16:58:46.992 INFO:scrape: 2023-05-02T16:58:46.992 INFO:scrape:Crash: Found coredumps on ubuntu@smithi148.front.sepia.ceph.com 2023-05-02T16:58:46.992 INFO:scrape:ceph version 18.0.0-3684-g936d612f (936d612fc76e1365b1db1735747c696ec357d376) reef (dev) 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fb8fc768ce0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb8fd9b5e19] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9f85) [0x7fb8fd9b5f85] 6: (interval_set::erase(inodeno_t, inodeno_t, std::function)+0x2a9) [0x559cddcc1979] 7: (EPurged::replay(MDSRank*)+0xde) [0x559cddc9bade] 8: (MDLog::_replay_thread()+0x61c) [0x559cddfdbc2c] 9: (MDLog::ReplayThread::entry()+0x11) [0x559cddfe5621] 10: /lib64/libpthread.so.0(+0x81cf) [0x7fb8fc75e1cf] 11: clone() 2023-05-02T16:58:46.992 INFO:scrape:6 jobs: ['7260117', '7260169', '7260017', '7259963', '7260204', '7259894'] 2023-05-02T16:58:46.993 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-05-02T16:58:46.993 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/fs/snaps}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-05-02T16:58:46.993 INFO:scrape: 2023-05-02T16:58:46.994 INFO:scrape:Failure: Test failure: test_filelock (tasks.cephfs.test_client_recovery.TestClientRecovery) 2023-05-02T16:58:46.994 INFO:scrape:2 jobs: ['7260186', '7259901'] 2023-05-02T16:58:46.994 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-recovery}'] 2023-05-02T16:58:46.995 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/client-recovery}'] 2023-05-02T16:58:46.995 INFO:scrape: 2023-05-02T16:58:46.995 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi087 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' 2023-05-02T16:58:46.996 INFO:scrape:1 jobs: ['7260112'] 2023-05-02T16:58:46.996 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}'] 2023-05-02T16:58:46.996 INFO:scrape: 2023-05-02T16:58:46.996 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-05-02T16:58:46.997 INFO:scrape:15 jobs: ['7259902', '7260185', '7259997', '7260180', '7260217', '7260010', '7259898', '7259944', '7260140', '7259987', '7259918', '7259973', '7259954', '7260152', '7259958'] 2023-05-02T16:58:46.997 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay'] 2023-05-02T16:58:46.998 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/ffsb}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2023-05-02T16:58:46.998 INFO:scrape: 2023-05-02T16:58:46.998 INFO:scrape:Failure: Test failure: test_version_splitting (tasks.cephfs.test_sessionmap.TestSessionMap) 2023-05-02T16:58:46.998 INFO:scrape:1 jobs: ['7260039'] 2023-05-02T16:58:46.999 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/sessionmap}'] 2023-05-02T16:58:46.999 INFO:scrape: 2023-05-02T16:58:46.999 INFO:scrape:Dead: b'2023-05-02T12:22:39.481 DEBUG:teuthology.exit:Finished running handlers' 2023-05-02T16:58:47.000 INFO:scrape:9 jobs: ['7260014', '7260080', '7260012', '7260145', '7260099', '7260076', '7260113', '7259941', '7259948'] 2023-05-02T16:58:47.000 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}'] 2023-05-02T16:58:47.000 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/functional/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'ms_mode/secure', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2023-05-02T16:58:47.001 INFO:scrape: 2023-05-02T16:58:47.001 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-05-02T16:58:47.001 INFO:scrape:1 jobs: ['7260107'] 2023-05-02T16:58:47.002 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-05-02T16:58:47.002 INFO:scrape: 2023-05-02T16:58:47.002 INFO:scrape:Failure: Test failure: test_files_throttle (tasks.cephfs.test_strays.TestStrays) 2023-05-02T16:58:47.002 INFO:scrape:1 jobs: ['7260073'] 2023-05-02T16:58:47.003 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/strays}'] 2023-05-02T16:58:47.003 INFO:scrape: 2023-05-02T16:58:47.003 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh 2023-05-02T16:58:47.004 INFO:scrape:2 jobs: ['7260023', '7260115'] 2023-05-02T16:58:47.004 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'osd}', 'session_timeout}'] 2023-05-02T16:58:47.004 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'frag', 'fs/verify/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'ranks/5', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/fsstress', 'tasks/{0-subvolume/{with-no-extra-options}', 'validater/valgrind}', 'wsync/no}'] 2023-05-02T16:58:47.005 INFO:scrape: 2023-05-02T16:58:47.005 INFO:scrape:Failure: Command failed on smithi130 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' 2023-05-02T16:58:47.005 INFO:scrape:6 jobs: ['7259915', '7260214', '7260046', '7260159', '7259975', '7259990'] 2023-05-02T16:58:47.005 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}'] 2023-05-02T16:58:47.006 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'frag', 'fs/functional/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/data-scan}', 'tasks/ior-shared-file}', 'tasks/mdtest}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}'] 2023-05-02T16:58:47.006 INFO:scrape: 2023-05-02T16:58:47.006 INFO:scrape:Failure: Command failed on smithi177 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 2023-05-02T16:58:47.007 INFO:scrape:2 jobs: ['7260130', '7260196'] 2023-05-02T16:58:47.007 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-05-02T16:58:47.007 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'frag', 'fs/mixed-clients/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount/fuse', 'ms-die-on-skipped}', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10', 'osd-asserts', 'osd-asserts}', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/{0-subvolume/{with-namespace-isolated}'] 2023-05-02T16:58:47.008 INFO:scrape: 2023-05-02T16:58:47.008 INFO:scrape:Failure: Test failure: test_client_pin (tasks.cephfs.test_client_limits.TestClientLimits) 2023-05-02T16:58:47.008 INFO:scrape:1 jobs: ['7260172'] 2023-05-02T16:58:47.008 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/client-limits}'] 2023-05-02T16:58:47.009 INFO:scrape: 2023-05-02T16:58:47.009 INFO:scrape:Failure: Test failure: test_shutdown_killpoint_SHUTDOWN_POSTALLEXPORTS (tasks.cephfs.test_failover.TestShutdownKillpoints) 2023-05-02T16:58:47.009 INFO:scrape:2 jobs: ['7260160', '7260047'] 2023-05-02T16:58:47.010 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{ignorelist_health', 'tasks/failover}'] 2023-05-02T16:58:47.010 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/failover}'] 2023-05-02T16:58:47.010 INFO:scrape: 2023-05-02T16:58:47.011 INFO:scrape:Crash: Command failed (workunit test suites/ffsb.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2023-05-02T16:58:47.011 INFO:scrape:ceph version 18.0.0-3684-g936d612f (936d612fc76e1365b1db1735747c696ec357d376) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff27db40520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7ff27e49073b] 6: /usr/lib/ceph/libceph-common.so.2(+0x161895) [0x7ff27e490895] 7: (Monitor::~Monitor()+0xd56) [0x55fc8d80f2c6] 8: (Monitor::~Monitor()+0xd) [0x55fc8d80f31d] 9: main() 10: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7ff27db27d90] 11: __libc_start_main() 12: _start() 2023-05-02T16:58:47.011 INFO:scrape:4 jobs: ['7260110', '7260053', '7259905', '7260078'] 2023-05-02T16:58:47.011 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_latest}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-05-02T16:58:47.012 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/ffsb}}', '2-workunit/fs/snaps}}', '2-workunit/suites/ffsb}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/5', 'session_timeout', 'tasks/test_journal_migration}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-05-02T16:58:47.012 INFO:scrape: 2023-05-02T16:58:47.012 INFO:scrape:Failure: Test failure: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable) 2023-05-02T16:58:47.013 INFO:scrape:1 jobs: ['7260002'] 2023-05-02T16:58:47.013 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/openfiletable}'] 2023-05-02T16:58:47.013 INFO:scrape: 2023-05-02T16:58:47.014 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-05-02T16:58:47.014 INFO:scrape:1 jobs: ['7260018'] 2023-05-02T16:58:47.014 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}'] 2023-05-02T16:58:47.014 INFO:scrape: 2023-05-02T16:58:47.015 INFO:scrape:Failure: Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) 2023-05-02T16:58:47.015 INFO:scrape:1 jobs: ['7259920'] 2023-05-02T16:58:47.015 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/exports}'] 2023-05-02T16:58:47.015 INFO:scrape: 2023-05-02T16:58:47.016 INFO:scrape:Failure: Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation) 2023-05-02T16:58:47.016 INFO:scrape:1 jobs: ['7260225'] 2023-05-02T16:58:47.016 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/fragment}'] 2023-05-02T16:58:47.017 INFO:scrape: 2023-05-02T16:58:47.038 INFO:teuthology.results:Sending results to pdonnell@redhat.com: Test Run: pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/ failed: 154 dead: 9 running: 0 waiting: 0 queued: 0 passed: 181 Fail ================================================================= [7259894] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:37:59 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259894/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=80a34ae666e141e888f7cba86b5b69ef Command failed (workunit test fs/snaps/snaptest-double-null.sh) on smithi032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double- null.sh' [7259897] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:50:45 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259897/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259897/ Command failed on smithi195 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid a7390366-e865-11ed-9b00-001a4aab830c -- ceph daemon mds.i perf dump' [7259898] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:19:22 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259898/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259898/ error during scrub thrashing: rank damage found: {'backtrace'} [7259900] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:55:31 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259900/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259900/ Command failed on smithi084 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid fb25f376-e865-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7259901] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:24:02 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259901/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259901/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bec94ac11a5b4ba0a873324c6b1104b8 Test failure: test_filelock (tasks.cephfs.test_client_recovery.TestClientRecovery) [7259902] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:39:51 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259902/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259902/ error during scrub thrashing: rank damage found: {'backtrace'} [7259904] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:03:30 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259904/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259904/ Command failed on smithi118 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 90b29430-e866-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7259905] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:24:45 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259905/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259905/ sentry: https://sentry.ceph.com/organizations/ceph/?query=67a48ac51b874cd29f5be313e5d8af35 Command failed (workunit test suites/ffsb.sh) on smithi083 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7259908] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:25:23 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259908/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259908/ Command failed on smithi172 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid ef92d7d6-e86a-11ed-9b00-001a4aab830c -- ceph daemon mds.c perf dump' [7259910] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:39:53 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259910/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259910/ Command failed on smithi032 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 60945d6e-e86c-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7259913] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:35:04 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259913/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259913/ Command failed on smithi067 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 760dc620-e86d-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7259915] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 04:00:05 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259915/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259915/ Command failed on smithi130 with status 3: 'sudo logrotate /etc/logrotate.d/ceph-test.conf' [7259918] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:38:28 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259918/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259918/ error during scrub thrashing: rank damage found: {'backtrace'} [7259920] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:42:27 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259920/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259920/ sentry: https://sentry.ceph.com/organizations/ceph/?query=11215a6cce9b491bb870ea56a0972dc8 Test failure: test_export_pin_many (tasks.cephfs.test_exports.TestExportPin) [7259923] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:06:55 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259923/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259923/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fcc0291fd7e64552ab5b1839c2de9669 Command failed on smithi173 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7259924] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:07:07 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259924/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259924/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6bf9130c541142b38a73352c28f5c7ea Command failed on smithi046 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7259926] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:09:46 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259926/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259926/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7ba3c7bdd52d4fd1b389651457be0425 Command failed on smithi039 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7259932] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:11:15 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259932/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259932/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50855736649b470badad61791f7a4082 Command failed on smithi083 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7259935] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:07:05 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259935/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259935/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dc5d0be96b9648389eec3d1f5634311c Command failed on smithi150 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7259938] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:07:08 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259938/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259938/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58b157a0cc8a4cdf9fcacc017eb80f3a Command failed on smithi151 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph- immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon- debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs- debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw- debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio- devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib' [7259939] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:43:55 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259939/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259939/ Command failed on smithi103 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 07c51336-e876-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7259942] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:09:33 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259942/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259942/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c47c05725e674209b3f4f1fb2487fa18 Command failed on smithi170 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7259944] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:09:40 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259944/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259944/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7259950] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:07:11 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259950/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259950/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dd0f08c04fd14f11bde9bde421d92955 Command failed on smithi039 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7259952] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:54:09 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259952/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259952/ Command failed on smithi161 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 214df182-e877-11ed-9b00-001a4aab830c -- ceph daemon mds.b perf dump' [7259954] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:31:24 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259954/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259954/ error during scrub thrashing: rank damage found: {'backtrace'} [7259955] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:34:03 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259955/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259955/ Command failed on smithi096 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 60cb439c-e876-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7259956] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:09:23 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259956/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259956/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e61b30cdaa5c41429a6e157b855a2c03 Command failed on smithi117 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7259958] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:42:04 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259958/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259958/ error during scrub thrashing: rank damage found: {'backtrace'} [7259959] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} ----------------------------------------------------------------- time: 00:07:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259959/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259959/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9f7d0ed912844cea371959ac0955c3a Command failed on smithi177 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7259961] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:33:28 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259961/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259961/ Command failed on smithi050 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 64762dda-e877-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7259963] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:23:24 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259963/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259963/ sentry: https://sentry.ceph.com/organizations/ceph/?query=497042f61c174e9b9a498a0f418fa7ce Command failed (workunit test suites/ffsb.sh) on smithi006 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7259964] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:48:37 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259964/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259964/ Command failed on smithi066 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid e55cf8b6-e877-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7259967] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:23 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259967/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259967/ Command failed on smithi138 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 7aaf6728-e878-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7259970] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:23:29 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259970/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259970/ "1682984190.0175424 osd.3 (osd.3) 3 : cluster [WRN] OSD bench result of 150124.227799 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.3. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7259971] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:27:47 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259971/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259971/ Command failed on smithi163 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 9506d402-e879-11ed-9b00-001a4aab830c -- ceph daemon mds.e perf dump' [7259973] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:43:49 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259973/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259973/ error during scrub thrashing: rank damage found: {'backtrace'} [7259975] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:15:44 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259975/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259975/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc1536c4ddb04e4087f0d521c3eccb72 Command failed on smithi077 with status 5: 'sudo systemctl stop ceph-c65cc534-e879-11ed-9b00-001a4aab830c@mon.b' [7259981] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:06:40 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259981/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259981/ Command failed on smithi134 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 7d5f41ca-e87b-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7259982] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:11:25 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259982/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259982/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b98d4a329dc048c4bd7bf5c8c885e026 Command failed on smithi150 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7259985] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:33:58 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259985/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259985/ Command failed on smithi033 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid e34f2842-e87b-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7259987] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:32:29 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259987/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259987/ error during scrub thrashing: rank damage found: {'backtrace'} [7259989] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:36:01 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259989/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259989/ Command failed on smithi029 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid e216bcb4-e87c-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7259990] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:15:09 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259990/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259990/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b9bd7507fd1741c8bb596ca534ad7667 Command failed on smithi156 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7259994] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:38:29 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259994/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259994/ Command failed on smithi031 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid fa6b89b0-e87d-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7259996] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} ----------------------------------------------------------------- time: 00:07:16 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259996/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259996/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4b0f63563d8b412781cba5b60383c3ce Command failed on smithi046 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7259997] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:40:42 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259997/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259997/ error during scrub thrashing: rank damage found: {'backtrace'} [7260002] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} ----------------------------------------------------------------- time: 00:13:46 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260002/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260002/ sentry: https://sentry.ceph.com/organizations/ceph/?query=711e8bd709c24dba9828801212164276 Test failure: test_max_items_per_obj (tasks.cephfs.test_openfiletable.OpenFileTable) [7260003] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} ----------------------------------------------------------------- time: 00:09:15 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260003/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260003/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1fbbb934ab784f5fa9247429692ff66e Command failed on smithi067 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7260004] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:07:10 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260004/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260004/ sentry: https://sentry.ceph.com/organizations/ceph/?query=efd31bfbacd8455a8e68398d88efedf4 Command failed on smithi159 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260005] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:29:32 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260005/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260005/ Command failed on smithi148 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid f704c02e-e87e-11ed-9b00-001a4aab830c -- ceph daemon mds.k perf dump' [7260010] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:46:37 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260010/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260010/ error during scrub thrashing: rank damage found: {'backtrace'} [7260013] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:06:56 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260013/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260013/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee450ce3e935496787e47d1ce707d19d Command failed on smithi177 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260016] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:49:15 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260016/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260016/ Command failed on smithi100 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 28849702-e882-11ed-9b00-001a4aab830c -- ceph daemon mds.e perf dump' [7260017] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:30:56 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260017/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260017/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6611d653a4ea43808d69e46763d3d5cd Command failed (workunit test suites/fsstress.sh) on smithi060 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7260018] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:17:41 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260018/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260018/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d5590376bfe4ad3ad87c63c1892abdf Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7260023] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 07:26:35 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260023/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260023/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dee44d366fbd41858b156e60449e09dd Command failed (workunit test suites/fsstress.sh) on smithi120 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7260024] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:36:18 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260024/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260024/ Command failed on smithi005 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 945ed258-e882-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260025] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:09:16 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260025/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260025/ sentry: https://sentry.ceph.com/organizations/ceph/?query=511f09f7f0b04b858f386eb50c43c80e Command failed on smithi177 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260027] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:30:17 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260027/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260027/ Command failed on smithi093 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 5087ed02-e883-11ed-9b00-001a4aab830c -- ceph daemon mds.k perf dump' [7260030] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:09:28 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260030/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260030/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8abe1c6496cf4b5daef6acaf15e0e40e Command failed on smithi078 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260034] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:27:11 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260034/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260034/ Command failed on smithi031 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid c73f32ca-e883-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260038] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:47:54 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260038/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260038/ Command failed on smithi081 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 9d4ff98a-e884-11ed-9b00-001a4aab830c -- ceph daemon mds.e perf dump' [7260039] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} ----------------------------------------------------------------- time: 00:17:08 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260039/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260039/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b2698781af2c4ac09c9fc686b495ef0d Test failure: test_version_splitting (tasks.cephfs.test_sessionmap.TestSessionMap) [7260040] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 02:08:16 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260040/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260040/ Command failed on smithi032 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 38140236-e885-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260041] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:59:03 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260041/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260041/ Command failed on smithi087 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid dc81b854-e885-11ed-9b00-001a4aab830c -- ceph daemon mds.g perf dump' [7260043] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} ----------------------------------------------------------------- time: 00:06:50 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260043/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260043/ sentry: https://sentry.ceph.com/organizations/ceph/?query=42db800e0e4e4faf978ca13867fe7eeb Command failed on smithi134 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7260044] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:43:22 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260044/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260044/ Command failed on smithi060 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 888924ce-e887-11ed-9b00-001a4aab830c -- ceph daemon mds.g perf dump' [7260046] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:16:08 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260046/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260046/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9de740d13a7447ba9c9ac2bff33d0f09 Command failed on smithi078 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7260047] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:52:43 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260047/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260047/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bab975f5d9e4e63b13f281ba072f42d Test failure: test_shutdown_killpoint_SHUTDOWN_POSTALLEXPORTS (tasks.cephfs.test_failover.TestShutdownKillpoints) [7260051] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:08:31 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260051/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260051/ Command failed on smithi012 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 83809312-e888-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260053] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:31:02 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260053/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260053/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33b38e698bc04f3ab4fb6b5ea5d79bd5 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7260054] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:30:40 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260054/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260054/ Command failed on smithi123 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 1ed2d078-e889-11ed-9b00-001a4aab830c -- ceph daemon mds.h perf dump' [7260056] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:37:19 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260056/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260056/ Command failed on smithi031 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 332dfdcc-e889-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260058] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:37:09 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260058/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260058/ Command failed on smithi033 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 46c8e91e-e889-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260067] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:28:33 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260067/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260067/ Command failed on smithi067 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 76aad416-e88a-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260068] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:58 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260068/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260068/ Command failed on smithi027 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 26c9c082-e88b-11ed-9b00-001a4aab830c -- ceph daemon mds.j perf dump' [7260071] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} ----------------------------------------------------------------- time: 00:09:15 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260071/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260071/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7554fa8ceae247769c5e1716e857fc5c Command failed on smithi083 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7260072] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:40:17 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260072/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260072/ Command failed on smithi026 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 4de3a41a-e88d-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260073] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:17:42 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260073/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260073/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ecf46a4a49c4269b75b8ad7449ab2a5 Test failure: test_files_throttle (tasks.cephfs.test_strays.TestStrays) [7260075] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:49:53 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260075/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260075/ Command failed on smithi121 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid cd68102c-e88d-11ed-9b00-001a4aab830c -- ceph daemon mds.i perf dump' [7260078] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:17:25 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260078/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260078/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1d002b4c593e44c0a333369c4ac18f07 Test failure: test_journal_migration (tasks.cephfs.test_journal_migration.TestJournalMigration) [7260083] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:46:59 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260083/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260083/ Command failed on smithi173 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 82c0b992-e88e-11ed-9b00-001a4aab830c -- ceph daemon mds.e perf dump' [7260087] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:40:42 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260087/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260087/ Command failed on smithi031 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid fc6becc0-e88f-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260088] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} ----------------------------------------------------------------- time: 00:07:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260088/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260088/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94215f683be843a4908123e072cbcf79 Command failed on smithi099 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7260089] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:42:35 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260089/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260089/ Command failed on smithi042 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 513b0abe-e891-11ed-9b00-001a4aab830c -- ceph daemon mds.e perf dump' [7260092] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:33:38 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260092/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260092/ Command failed on smithi067 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid ccb9f7d8-e88f-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260096] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:55:35 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260096/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260096/ Command failed on smithi029 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 051b1f70-e891-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260100] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:42:36 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260100/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260100/ Command failed on smithi066 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 44ddc292-e892-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260103] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:06:52 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260103/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260103/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ce5fe0134834de6ac1c0fa56f0157ca Command failed on smithi170 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260107] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:34:06 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260107/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260107/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f03cc5a8d024b08972f707c5b2ee6e3 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7260109] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:25:17 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260109/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260109/ Command failed on smithi170 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 0403cd4c-e893-11ed-9b00-001a4aab830c -- ceph daemon mds.e perf dump' [7260110] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:12:15 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260110/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260110/ sentry: https://sentry.ceph.com/organizations/ceph/?query=044814283b8d49308272e92a23eb4c52 Command failed (workunit test suites/ffsb.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' [7260111] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:44:54 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260111/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260111/ Command failed on smithi023 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid bf191384-e894-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260112] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:13:14 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260112/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260112/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f7081ea785554087955df810bc86050d Command failed (workunit test fs/quota/quota.sh) on smithi087 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' [7260115] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 06:36:58 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260115/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260115/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2b32c19d24544a8844274a20aeb61c7 Command failed (workunit test suites/fsstress.sh) on smithi084 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7260117] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:15:08 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260117/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260117/ Found coredumps on ubuntu@smithi148.front.sepia.ceph.com [7260118] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:17:13 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260118/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260118/ Command failed on smithi031 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 90797958-e897-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260124] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:35:37 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260124/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260124/ Command failed on smithi027 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid a8cab3f0-e897-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260125] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:35:57 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260125/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260125/ Command failed on smithi017 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid b25ece16-e896-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260129] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:34:22 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260129/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260129/ Command failed on smithi066 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 05f80eb0-e898-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260130] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:40:35 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260130/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260130/ Command failed on smithi177 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [7260133] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:09:48 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260133/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260133/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4146f9dfa2184383b38a0b2420b26ee5 Command failed on smithi072 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260134] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:38:36 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260134/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260134/ Command failed on smithi067 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 2fb5e654-e899-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260136] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:40 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260136/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260136/ Command failed on smithi150 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 3fa56dd6-e89a-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260139] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:32:01 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260139/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260139/ Command failed on smithi039 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 07f94412-e8a3-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260140] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:06:56 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260140/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260140/ error during scrub thrashing: rank damage found: {'backtrace'} [7260142] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} ----------------------------------------------------------------- time: 00:11:33 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260142/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260142/ sentry: https://sentry.ceph.com/organizations/ceph/?query=586baa097b904f5e94258b7eb7c04688 Command failed on smithi082 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7260144] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:48:33 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260144/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260144/ Command failed on smithi169 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 819dc7d4-e8a3-11ed-9b00-001a4aab830c -- ceph daemon mds.c perf dump' [7260146] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:07:22 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260146/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260146/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15ce123965f24db5b36c58a7e7e4cdf7 Command failed on smithi044 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260148] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:49:53 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260148/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260148/ Command failed on smithi016 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 1e26a446-e8a8-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260152] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:00:27 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260152/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260152/ error during scrub thrashing: rank damage found: {'backtrace'} [7260153] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:48:54 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260153/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260153/ Command failed on smithi093 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 434ebc40-e8a8-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260154] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:44:03 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260154/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260154/ Command failed on smithi149 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 2fa29940-e8a9-11ed-9b00-001a4aab830c -- ceph daemon mds.b perf dump' [7260157] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:31:48 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260157/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260157/ Command failed on smithi134 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid a87cb16c-e8a8-11ed-9b00-001a4aab830c -- ceph daemon mds.e perf dump' [7260158] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:34:55 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260158/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260158/ Command failed on smithi082 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 67ac0ed0-e8a8-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260159] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:14:22 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260159/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260159/ sentry: https://sentry.ceph.com/organizations/ceph/?query=513b32d61663449c98f702717ab8e5da Command failed on smithi101 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7260160] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:55:41 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260160/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260160/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dc7558d4edcd47bd86a25f1b8f0e76b5 Test failure: test_shutdown_killpoint_SHUTDOWN_POSTALLEXPORTS (tasks.cephfs.test_failover.TestShutdownKillpoints) [7260162] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} ----------------------------------------------------------------- time: 00:29:29 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260162/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260162/ Command failed on smithi057 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 325b19a4-e8aa-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260169] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:27:52 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260169/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260169/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e256a68ce804f8c9351a8551a4f13c0 Command failed (workunit test suites/fsstress.sh) on smithi026 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7260170] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:30:39 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260170/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260170/ Command failed on smithi012 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid c6a02fb2-e8ac-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260172] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:31:55 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260172/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260172/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2dff87b5d5b4454b51bb5418eb6c3de Test failure: test_client_pin (tasks.cephfs.test_client_limits.TestClientLimits) [7260174] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:43:53 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260174/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260174/ Command failed on smithi115 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 494e1bc2-e8ad-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260178] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:01:42 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260178/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260178/ Command failed on smithi082 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 54657a3a-e8af-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260180] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:44:40 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260180/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260180/ error during scrub thrashing: rank damage found: {'backtrace'} [7260183] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:01:47 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260183/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260183/ Command failed on smithi016 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid dce1114a-e8ae-11ed-9b00-001a4aab830c -- ceph daemon mds.d perf dump' [7260184] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} ----------------------------------------------------------------- time: 00:07:04 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260184/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260184/ sentry: https://sentry.ceph.com/organizations/ceph/?query=31c7ceb226a447928feda84144595793 Command failed on smithi112 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base ceph-mgr ceph-mgr-dashboard ceph-mgr- diskprediction-local ceph-mgr-rook ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 rbd-fuse rbd-mirror rbd-nbd python-rados python-rgw python-rbd python-cephfs bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux bison flex elfutils- libelf-devel openssl-devel NetworkManager iproute util-linux bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux' [7260185] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:51:22 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260185/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260185/ error during scrub thrashing: rank damage found: {'backtrace'} [7260186] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:24:12 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260186/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260186/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3d7905c04fa48a7b7fd21e196087e6e Test failure: test_filelock (tasks.cephfs.test_client_recovery.TestClientRecovery) [7260188] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:11:44 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260188/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260188/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33e61d6d7d3543dc8a06191b2ac53052 Command failed on smithi157 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260189] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:12:12 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260189/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260189/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9be1d449fea4f7e92e0dba90307d1c0 Command failed (workunit test fs/test_python.sh) on smithi026 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7260191] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:07:16 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260191/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260191/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f93904f106924dcbb22796268100dccc Command failed on smithi153 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260192] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:44:23 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260192/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260192/ Command failed on smithi031 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 08a53f96-e8b3-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260193] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:35:07 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260193/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260193/ Command failed on smithi012 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid f41e3fec-e8b1-11ed-9b00-001a4aab830c -- ceph daemon mds.j perf dump' [7260196] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:40:33 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260196/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260196/ Command failed on smithi114 with status 123: "time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose --" [7260199] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} ----------------------------------------------------------------- time: 00:40:42 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260199/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260199/ Command failed on smithi029 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 3da9b6fe-e8b3-11ed-9b00-001a4aab830c -- ceph daemon mds.a perf dump' [7260200] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:39:14 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260200/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260200/ Command failed on smithi097 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 60dba272-e8b3-11ed-9b00-001a4aab830c -- ceph daemon mds.i perf dump' [7260202] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:30:21 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260202/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260202/ Command failed on smithi153 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 79b32b62-e8b3-11ed-9b00-001a4aab830c -- ceph daemon mds.k perf dump' [7260204] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:38:47 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260204/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260204/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c00c437a5344fd0b32be6aecfdf798f Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=936d612fc76e1365b1db1735747c696ec357d376 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7260206] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} ----------------------------------------------------------------- time: 00:09:23 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260206/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260206/ sentry: https://sentry.ceph.com/organizations/ceph/?query=735992036ef947da9a54d7ae9e4d4434 Command failed on smithi115 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib' [7260207] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/direct_io}} ----------------------------------------------------------------- time: 00:33:33 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260207/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260207/ Command failed on smithi090 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid fb2d5550-e8b3-11ed-9b00-001a4aab830c -- ceph daemon mds.j perf dump' [7260210] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 01:43:10 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260210/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260210/ Command failed on smithi038 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid c3788f60-e8b5-11ed-9b00-001a4aab830c -- ceph daemon mds.h perf dump' [7260211] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:07:04 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260211/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260211/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0601df6f062f4603a7dc64c6501624e6 Command failed on smithi103 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260213] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:53:44 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260213/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260213/ Command failed on smithi066 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 0227f9e8-e8b7-11ed-9b01-001a4aab830c -- ceph daemon mds.d perf dump' [7260214] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:14:58 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260214/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260214/ sentry: https://sentry.ceph.com/organizations/ceph/?query=302dee40229f40e1a96c3d8594437693 Command failed on smithi101 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7260215] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:14:15 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260215/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260215/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3bcb722a29a4f4f93e943741f5b7db4 Command failed on smithi177 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260217] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:07:23 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260217/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260217/ error during scrub thrashing: rank damage found: {'backtrace'} [7260220] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:52:46 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260220/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260220/ Command failed on smithi112 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 8670272e-e8b8-11ed-9b01-001a4aab830c -- ceph daemon mds.a perf dump' [7260223] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:45:11 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260223/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260223/ Command failed on smithi067 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid bd655f1e-e8b9-11ed-9b01-001a4aab830c -- ceph daemon mds.a perf dump' [7260225] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:14:04 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260225/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260225/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22f6586fc09145e9a18dab897fbd670f Test failure: test_deep_split (tasks.cephfs.test_fragment.TestFragmentation) [7260226] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:39:26 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260226/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260226/ Command failed on smithi114 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 398772b2-e8ba-11ed-9b01-001a4aab830c -- ceph daemon mds.j perf dump' [7260231] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:07:09 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260231/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260231/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1410003ce1094366b87ecee6a8495c71 Command failed on smithi099 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7260234] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:48:21 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260234/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260234/ Command failed on smithi026 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid 8895875e-e8ba-11ed-9b01-001a4aab830c -- ceph daemon mds.a perf dump' [7260237] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:33:09 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260237/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260237/ Command failed on smithi090 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay- quay.apps.os.sepia.ceph.com/ceph- ci/ceph:936d612fc76e1365b1db1735747c696ec357d376 shell --fsid b885f070-e8ba-11ed-9b01-001a4aab830c -- ceph daemon mds.d perf dump' Dead ================================================================= [7259941] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259941/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259941/ hit max job timeout [7259948] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259948/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259948/ hit max job timeout [7260012] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260012/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260012/ hit max job timeout [7260014] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260014/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260014/ hit max job timeout [7260076] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260076/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260076/ hit max job timeout [7260080] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260080/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260080/ hit max job timeout [7260099] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260099/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260099/ hit max job timeout [7260113] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260113/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260113/ hit max job timeout [7260145] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260145/ log: http://qa-proxy.ceph.com/teuthology/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260145/ hit max job timeout Pass ================================================================= [7259895] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:37:51 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259895/ [7259896] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:13:48 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259896/ [7259899] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:51:50 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259899/ [7259903] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:15:18 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259903/ [7259906] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} time: 00:48:08 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259906/ [7259907] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:28:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259907/ [7259909] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/lockdep} time: 00:56:44 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259909/ [7259911] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:33:33 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259911/ [7259912] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:30:44 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259912/ [7259914] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:45:25 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259914/ [7259916] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:33:12 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259916/ [7259917] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:18:15 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259917/ [7259919] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iozone}} time: 00:32:31 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259919/ [7259921] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} time: 00:40:23 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259921/ [7259922] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:17:19 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259922/ [7259925] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:55:07 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259925/ [7259927] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:18:41 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259927/ [7259928] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:13:48 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259928/ [7259929] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} time: 02:07:03 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259929/ [7259930] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:23:35 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259930/ [7259931] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:57:54 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259931/ [7259933] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:14:27 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259933/ [7259934] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:41:14 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259934/ [7259936] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_20.04} tasks/fstop} time: 00:13:41 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259936/ [7259937] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:13 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259937/ [7259940] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} time: 00:51:41 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259940/ [7259943] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:18:30 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259943/ [7259945] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:48 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259945/ [7259946] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:29:55 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259946/ [7259947] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:23:55 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259947/ [7259949] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:09:38 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259949/ [7259951] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:18:53 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259951/ [7259953] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:15:56 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259953/ [7259957] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:24:06 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259957/ [7259960] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:23:05 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259960/ [7259962] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:37:42 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259962/ [7259965] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 00:14:17 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259965/ [7259966] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:30:03 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259966/ [7259968] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:30:16 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259968/ [7259969] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:28:07 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259969/ [7259972] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:19:54 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259972/ [7259974] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:16:46 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259974/ [7259976] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:02 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259976/ [7259977] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:29 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259977/ [7259978] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:43:15 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259978/ [7259979] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:13:42 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259979/ [7259980] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:33:36 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259980/ [7259983] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:42:29 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259983/ [7259984] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} time: 01:39:03 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259984/ [7259986] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:38:43 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259986/ [7259988] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:18:55 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259988/ [7259991] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:31:54 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259991/ [7259992] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:48:24 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259992/ [7259993] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:35:44 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259993/ [7259995] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:19:48 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259995/ [7259998] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:19:26 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259998/ [7259999] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:34:34 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7259999/ [7260000] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:30:47 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260000/ [7260001] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:33:46 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260001/ [7260006] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:51:46 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260006/ [7260007] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} time: 01:26:30 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260007/ [7260008] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:48:29 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260008/ [7260009] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:41:44 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260009/ [7260011] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:14:37 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260011/ [7260015] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} time: 00:59:06 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260015/ [7260019] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:39:15 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260019/ [7260020] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:16:06 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260020/ [7260021] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:14:06 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260021/ [7260022] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:14:42 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260022/ [7260026] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:22:02 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260026/ [7260028] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:40:37 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260028/ [7260029] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} time: 00:53:55 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260029/ [7260031] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:01 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260031/ [7260032] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:34:35 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260032/ [7260033] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:30:37 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260033/ [7260035] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:43:55 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260035/ [7260036] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:17:38 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260036/ [7260037] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:36:08 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260037/ [7260042] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:22:46 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260042/ [7260045] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 00:50:24 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260045/ [7260048] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:44:52 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260048/ [7260049] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 00:49:08 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260049/ [7260050] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:42:39 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260050/ [7260052] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} time: 00:49:18 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260052/ [7260055] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:24:06 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260055/ [7260057] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:14:22 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260057/ [7260059] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:45:25 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260059/ [7260060] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:38:46 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260060/ [7260061] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:38:01 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260061/ [7260062] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:19 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260062/ [7260063] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:26 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260063/ [7260064] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:34:43 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260064/ [7260065] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iogen}} time: 00:41:32 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260065/ [7260066] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:30:13 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260066/ [7260069] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:27:48 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260069/ [7260070] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 01:04:59 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260070/ [7260074] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:19:21 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260074/ [7260077] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 01:03:10 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260077/ [7260079] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:15:53 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260079/ [7260081] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:03:27 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260081/ [7260082] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:39 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260082/ [7260084] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:15:33 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260084/ [7260085] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:27:04 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260085/ [7260086] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:15:07 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260086/ [7260090] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:24:00 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260090/ [7260091] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:07:34 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260091/ [7260093] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:14:48 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260093/ [7260094] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:36:39 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260094/ [7260095] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:33:44 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260095/ [7260097] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:03:48 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260097/ [7260098] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:29:24 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260098/ [7260101] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:20:35 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260101/ [7260102] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:24:35 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260102/ [7260104] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:58:40 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260104/ [7260105] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:35:19 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260105/ [7260106] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:24 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260106/ [7260108] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:31:13 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260108/ [7260114] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:54 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260114/ [7260116] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:38:31 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260116/ [7260119] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:14:53 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260119/ [7260120] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:42:15 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260120/ [7260121] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:12:07 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260121/ [7260122] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:39:03 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260122/ [7260123] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:21:59 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260123/ [7260126] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} time: 01:06:09 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260126/ [7260127] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:33:06 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260127/ [7260128] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:16:24 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260128/ [7260131] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:58:31 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260131/ [7260132] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:17:37 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260132/ [7260135] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:22:59 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260135/ [7260137] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} time: 02:30:17 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260137/ [7260138] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:12:02 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260138/ [7260141] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:21:36 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260141/ [7260143] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:18:56 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260143/ [7260147] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:20:39 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260147/ [7260149] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:15:02 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260149/ [7260150] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:26:47 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260150/ [7260151] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:32 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260151/ [7260155] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:53:07 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260155/ [7260156] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:14:39 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260156/ [7260161] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:42:16 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260161/ [7260163] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:50:22 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260163/ [7260164] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:46:23 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260164/ [7260165] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:25 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260165/ [7260166] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:13:39 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260166/ [7260167] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/iogen}} time: 00:56:36 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260167/ [7260168] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}} time: 00:34:51 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260168/ [7260171] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:37:53 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260171/ [7260173] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/direct_io}} time: 00:45:57 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260173/ [7260175] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:17:05 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260175/ [7260176] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/kernel_untar_build}} time: 02:26:03 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260176/ [7260177] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 01:00:10 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260177/ [7260179] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:14:28 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260179/ [7260181] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:18:55 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260181/ [7260182] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:21:24 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260182/ [7260187] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/fs/norstats}} time: 00:31:10 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260187/ [7260190] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:30 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260190/ [7260194] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:28:50 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260194/ [7260195] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:36:28 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260195/ [7260197] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:29:01 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260197/ [7260198] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:23:02 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260198/ [7260201] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:30 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260201/ [7260203] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:35:36 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260203/ [7260205] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} time: 00:42:40 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260205/ [7260208] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/misc}} time: 00:48:02 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260208/ [7260209] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:49:29 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260209/ [7260212] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 00:58:59 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260212/ [7260216] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:11:37 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260216/ [7260218] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:26:31 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260218/ [7260219] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:18:13 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260219/ [7260221] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:39:01 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260221/ [7260222] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:40:03 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260222/ [7260224] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:15:31 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260224/ [7260227] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:34:07 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260227/ [7260228] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:39:06 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260228/ [7260229] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:28:43 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260229/ [7260230] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:41:50 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260230/ [7260232] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:22:14 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260232/ [7260233] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:04 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260233/ [7260235] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:19:25 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260235/ [7260236] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:36:04 info: http://pulpito.ceph.com/pdonnell-2023-05-01_20:47:26-fs-wip-pdonnell-testing-20230501.190312-distro-default-smithi/7260236/