2022-04-25T08:43:26.967 INFO:root:teuthology version: 1.1.1.dev313+g1b11219b 2022-04-25T08:43:26.968 INFO:teuthology.results:subset: '26/32' 2022-04-25T08:43:26.968 INFO:teuthology.results:seed: '6778' 2022-04-25T15:51:32.347 INFO:root:teuthology version: 1.1.1.dev313+g1b11219b 2022-04-25T15:51:32.381 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-04-25T22:27:07.020 INFO:teuthology.results:Tests finished! gathering results... 2022-04-25T22:27:41.867 INFO:scrape:Found 373 jobs 2022-04-25T22:30:31.535 INFO:scrape:Found 45 distinct failure reasons 2022-04-25T22:30:31.536 INFO:scrape:Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_ino_release_cb' 2022-04-25T22:30:31.536 INFO:scrape:1 jobs: ['6804856'] 2022-04-25T22:30:31.536 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'mds', 'mon', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/ino_release_cb}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.537 INFO:scrape: 2022-04-25T22:30:31.537 INFO:scrape:Failure: Command failed on smithi142 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" 2022-04-25T22:30:31.537 INFO:scrape:74 jobs 2022-04-25T22:30:31.538 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.539 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/32bits/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/verify/{begin/{0-install', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{mon-debug', 'overrides/{no_client_pidfile', 'overrides/{osd-asserts', 'overrides/{whitelist_health', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/auto-repair}', 'tasks/backtrace}', 'tasks/cap-flush}', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-readahead}', 'tasks/exports}', 'tasks/failover}', 'tasks/fragment}', 'tasks/fsstress', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/recovery-fs}', 'tasks/test_journal_migration}', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'traceless/50pc}', 'validater/lockdep}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.539 INFO:scrape: 2022-04-25T22:30:31.539 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi071.front.sepia.ceph.com: ['type=AVC msg=audit(1650880844.362:202): avc: denied { node_bind } for pid=1545 comm="ping" saddr=172.21.15.71 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] 2022-04-25T22:30:31.539 INFO:scrape:4 jobs: ['6804794', '6805094', '6804813', '6804752'] 2022-04-25T22:30:31.539 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.539 INFO:scrape:suites union: ['1-ceph', '1-upgrade}}', '2-logrotate}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/ffsb}}', 'centos_latest', 'clusters/1a5s-mds-1c-client', 'clusters/small-cluster', 'conf/{client', 'distro/{centos_8}', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/upgrade/nofs/{bluestore-bitmap', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'no-mds-cluster', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{no_client_pidfile', 'overrides/{pg-warn', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/trim-i24137}', 'tasks/{0-octopus', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.540 INFO:scrape: 2022-04-25T22:30:31.540 INFO:scrape:Failure: Timed out waiting for MDS daemons to become healthy 2022-04-25T22:30:31.540 INFO:scrape:78 jobs 2022-04-25T22:30:31.541 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{frag', 'session_timeout', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.541 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/fs/trivial_sync}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/no', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/direct_io}', 'workunit/fs/misc}', 'workunit/fs/norstats}', 'workunit/fs/test_o_trunc}', 'workunit/kernel_untar_build}', 'workunit/suites/blogbench}', 'workunit/suites/dbench}', 'workunit/suites/ffsb}', 'workunit/suites/fsstress}', 'workunit/suites/fsx}', 'workunit/suites/fsync-tester}', 'workunit/suites/iogen}', 'workunit/suites/iozone}', 'workunit/suites/pjd}', 'wsync/{no}}', 'wsync/{yes}}'] 2022-04-25T22:30:31.541 INFO:scrape: 2022-04-25T22:30:31.541 INFO:scrape:Failure: No module named 'tasks' 2022-04-25T22:30:31.541 INFO:scrape:1 jobs: ['6804924'] 2022-04-25T22:30:31.542 INFO:scrape:suites: ['1-client', '2-upgrade', '3-compat_client/pacific}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/no', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.542 INFO:scrape: 2022-04-25T22:30:31.542 INFO:scrape:Failure: Test failure: test_add_directory_path_normalization (tasks.cephfs.test_mirroring.TestMirroring) 2022-04-25T22:30:31.542 INFO:scrape:1 jobs: ['6804756'] 2022-04-25T22:30:31.542 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror/{begin/{0-install', 'mount/fuse', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distros$/{centos_8}', 'tasks/mirror}'] 2022-04-25T22:30:31.542 INFO:scrape: 2022-04-25T22:30:31.542 INFO:scrape:Failure: Command failed on smithi038 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' 2022-04-25T22:30:31.543 INFO:scrape:3 jobs: ['6804917', '6804916', '6804918'] 2022-04-25T22:30:31.543 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.543 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'msgr-failures/none', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{no_client_pidfile', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/quota}', 'tasks/{0-check-counter', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/suites/iozone}', 'wsync/{yes}}'] 2022-04-25T22:30:31.543 INFO:scrape: 2022-04-25T22:30:31.543 INFO:scrape:Failure: Test failure: test_mount_mon_and_osd_caps_present_mds_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) 2022-04-25T22:30:31.543 INFO:scrape:1 jobs: ['6804760'] 2022-04-25T22:30:31.543 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{mon-debug', 'tasks/multifs-auth}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.544 INFO:scrape: 2022-04-25T22:30:31.544 INFO:scrape:Failure: Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring) 2022-04-25T22:30:31.544 INFO:scrape:1 jobs: ['6804768'] 2022-04-25T22:30:31.544 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2022-04-25T22:30:31.544 INFO:scrape: 2022-04-25T22:30:31.544 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi097 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2022-04-25T22:30:31.544 INFO:scrape:1 jobs: ['6805105'] 2022-04-25T22:30:31.544 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.545 INFO:scrape: 2022-04-25T22:30:31.545 INFO:scrape:Timeout 3h running clone.client.1/qa/workunits/suites/fsstress.sh 2022-04-25T22:30:31.545 INFO:scrape:2 jobs: ['6805102', '6804913'] 2022-04-25T22:30:31.545 INFO:scrape:suites intersection: ['1-ranks/1', '1-volume/{0-create', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.545 INFO:scrape:suites union: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.545 INFO:scrape: 2022-04-25T22:30:31.545 INFO:scrape:Dead: b'2022-04-25T22:23:24.896 DEBUG:teuthology.exit:Finished running handlers' 2022-04-25T22:30:31.546 INFO:scrape:8 jobs: ['6805097', '6804759', '6804883', '6804821', '6805015', '6804926', '6804879', '6805073'] 2022-04-25T22:30:31.546 INFO:scrape:suites intersection: ['conf/{client', 'mds', 'mon', 'osd}', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.546 INFO:scrape:suites union: ['1-ceph', '1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-workunit/ffsb}}', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'distros/ubuntu_latest', 'fs/mixed-clients/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'mount/fuse', 'ms-die-on-skipped}', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'overrides/{osd-asserts', 'overrides/{pg-warn', 'overrides/{whitelist_health', 'rhel_8}', 'roles', 'session_timeout', 'syntax', 'tasks/cephfs_misc_tests}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/mdtest}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4', 'tasks/{1-thrash/mon', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.546 INFO:scrape: 2022-04-25T22:30:31.546 INFO:scrape:Failure: Test failure: test_default_uid_gid_subvolume_group (tasks.cephfs.test_volumes.TestSubvolumeGroups) 2022-04-25T22:30:31.546 INFO:scrape:1 jobs: ['6804770'] 2022-04-25T22:30:31.547 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.547 INFO:scrape: 2022-04-25T22:30:31.547 INFO:scrape:Failure: error during scrub thrashing: Command failed on smithi007 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' 2022-04-25T22:30:31.547 INFO:scrape:11 jobs: ['6804935', '6805046', '6805012', '6804907', '6805079', '6804799', '6804975', '6804869', '6804771', '6804836', '6805032'] 2022-04-25T22:30:31.547 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.547 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs/workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'ranks/3', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/direct_io}', 'workunit/fs/norstats}', 'workunit/fs/test_o_trunc}', 'workunit/kernel_untar_build}', 'workunit/suites/dbench}', 'workunit/suites/fsx}', 'workunit/suites/iozone}', 'wsync/{no}}', 'wsync/{yes}}'] 2022-04-25T22:30:31.548 INFO:scrape: 2022-04-25T22:30:31.548 INFO:scrape:Failure: Test failure: test_ephemeral_pin_dist_failover (tasks.cephfs.test_exports.TestEphemeralPins) 2022-04-25T22:30:31.548 INFO:scrape:1 jobs: ['6805106'] 2022-04-25T22:30:31.548 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/exports}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.548 INFO:scrape: 2022-04-25T22:30:31.548 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2022-04-25T22:30:31.549 INFO:scrape:1 jobs: ['6805040'] 2022-04-25T22:30:31.549 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_8}', 'fs/libcephfs/{begin/{0-install', 'mds', 'mon', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/libcephfs_python}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.549 INFO:scrape: 2022-04-25T22:30:31.549 INFO:scrape:Failure: Extra data: line 2 column 473 (char 473) 2022-04-25T22:30:31.549 INFO:scrape:7 jobs: ['6805088', '6804840', '6804998', '6805028', '6804896', '6804806', '6804740'] 2022-04-25T22:30:31.549 INFO:scrape:suites intersection: ['1-client', '2-upgrade', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'mds', 'mon', 'osd}', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.549 INFO:scrape:suites union: ['1-client', '2-upgrade', '3-client-upgrade', '3-compat_client/pacific}}', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{multimds/no', 'overrides/{multimds/yes', 'pg-warn', 'tasks/{0-octopus', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.550 INFO:scrape: 2022-04-25T22:30:31.550 INFO:scrape:Failure: Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) 2022-04-25T22:30:31.550 INFO:scrape:2 jobs: ['6805100', '6804817'] 2022-04-25T22:30:31.550 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/data-scan}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.550 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/data-scan}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.550 INFO:scrape: 2022-04-25T22:30:31.550 INFO:scrape:Failure: Test failure: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2022-04-25T22:30:31.551 INFO:scrape:2 jobs: ['6805071', '6804827'] 2022-04-25T22:30:31.551 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.551 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/volumes/{overrides', 'test/clone}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.551 INFO:scrape: 2022-04-25T22:30:31.551 INFO:scrape:Failure: Command failed (workunit test suites/fsstress.sh) on smithi122 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 2022-04-25T22:30:31.551 INFO:scrape:1 jobs: ['6804937'] 2022-04-25T22:30:31.551 INFO:scrape:suites: ['1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'mds', 'mon', 'osd}', 'overrides/{pg-warn', 'roles', 'syntax', 'tasks/{0-from/v16.2.4', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.552 INFO:scrape: 2022-04-25T22:30:31.552 INFO:scrape:Failure: reached maximum tries (90) after waiting for 540 seconds 2022-04-25T22:30:31.552 INFO:scrape:9 jobs: ['6804843', '6805101', '6804915', '6804954', '6804822', '6804831', '6804776', '6805075', '6804908'] 2022-04-25T22:30:31.552 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{frag', 'races', 'session_timeout', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.552 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.552 INFO:scrape: 2022-04-25T22:30:31.552 INFO:scrape:Failure: Test failure: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) 2022-04-25T22:30:31.553 INFO:scrape:1 jobs: ['6804939'] 2022-04-25T22:30:31.553 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{no_client_pidfile', 'tasks/sessionmap}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.553 INFO:scrape: 2022-04-25T22:30:31.553 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh 2022-04-25T22:30:31.553 INFO:scrape:1 jobs: ['6804861'] 2022-04-25T22:30:31.553 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/workloads/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/osd', 'thrashosds-health', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.553 INFO:scrape: 2022-04-25T22:30:31.554 INFO:scrape:Failure: Test failure: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2022-04-25T22:30:31.554 INFO:scrape:1 jobs: ['6804955'] 2022-04-25T22:30:31.554 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/snap-schedule}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.554 INFO:scrape: 2022-04-25T22:30:31.554 INFO:scrape:Failure: Test failure: test_add_already_in_use_metadata_pool (tasks.cephfs.test_admin.TestAddDataPool) 2022-04-25T22:30:31.554 INFO:scrape:1 jobs: ['6805018'] 2022-04-25T22:30:31.554 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/admin}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.554 INFO:scrape: 2022-04-25T22:30:31.555 INFO:scrape:Crash: Command failed (workunit test client/test.sh) on smithi080 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/client/test.sh' 2022-04-25T22:30:31.555 INFO:scrape:ceph version 17.0.0-11827-g9bc2ff46 (9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0) [0x7f4b72ef23c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b7) [0x7f4b7247c834] 5: /usr/lib/ceph/libceph-common.so.2(+0x27c9de) [0x7f4b7247c9de] 6: (Client::path_walk(filepath const&, Client::walk_dentry_result*, UserPerm const&, bool, int, boost::intrusive_ptr)+0x1381) [0x55f54c775341] 7: (Client::path_walk(filepath const&, boost::intrusive_ptr*, UserPerm const&, bool, int, boost::intrusive_ptr)+0x8d) [0x55f54c77550d] 8: (Client::create_and_open(int, char const*, int, UserPerm const&, unsigned int, int, int, int, char const*, std::__cxx11::basic_string, std::allocator >)+0x16e) [0x55f54c7a54de] 9: (Client::openat(int, char const*, int, UserPerm const&, unsigned int, int, int, int, char const*, std::__cxx11::basic_string, std::allocator >)+0x2b5) [0x55f54c7a60e5] 10: (Client::open(char const*, int, UserPerm const&, unsigned int, int, int, int, char const*, std::__cxx11::basic_string, std::allocator >)+0x9f) [0x55f54c7a661f] 11: (TestClient_AlternateNameCreat_Test::TestBody()+0x248) [0x55f54c701d68] 12: (void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*)+0x51) [0x55f54c852821] 13: (testing::Test::Run()+0xd6) [0x55f54c845516] 14: (testing::TestInfo::Run()+0x135) [0x55f54c845675] 15: (testing::TestSuite::Run()+0xc9) [0x55f54c845c09] 16: (testing::internal::UnitTestImpl::RunAllTests()+0x487) [0x55f54c846237] 17: (bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x51) [0x55f54c852d91] 18: (testing::UnitTest::Run()+0xa1) [0x55f54c845741] 19: main() 20: __libc_start_main() 21: _start() 2022-04-25T22:30:31.555 INFO:scrape:1 jobs: ['6804755'] 2022-04-25T22:30:31.555 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'mds', 'mon', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/client}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.555 INFO:scrape: 2022-04-25T22:30:31.555 INFO:scrape:Failure: Test failure: test_apply_tag (tasks.cephfs.test_multimds_misc.TestScrub2) 2022-04-25T22:30:31.556 INFO:scrape:1 jobs: ['6804894'] 2022-04-25T22:30:31.556 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{no_client_pidfile', 'tasks/multimds_misc}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.556 INFO:scrape: 2022-04-25T22:30:31.556 INFO:scrape:Failure: Test failure: test_client_min_caps_working_set (tasks.cephfs.test_client_limits.TestClientLimits) 2022-04-25T22:30:31.556 INFO:scrape:1 jobs: ['6805063'] 2022-04-25T22:30:31.556 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/client-limits}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.556 INFO:scrape: 2022-04-25T22:30:31.556 INFO:scrape:Failure: Test failure: test_delayed_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics) 2022-04-25T22:30:31.557 INFO:scrape:1 jobs: ['6804882'] 2022-04-25T22:30:31.557 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/metrics}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.557 INFO:scrape: 2022-04-25T22:30:31.557 INFO:scrape:Failure: Test failure: test_subvolume_group_snapshot_unsupported_status (tasks.cephfs.test_volumes.TestSubvolumeGroupSnapshots) 2022-04-25T22:30:31.557 INFO:scrape:1 jobs: ['6804951'] 2022-04-25T22:30:31.557 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/snapshot}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.557 INFO:scrape: 2022-04-25T22:30:31.557 INFO:scrape:Failure: "2022-04-25T15:44:06.267737+0000 mon.a (mon.0) 298 : cluster [WRN] Health check failed: Degraded data redundancy: 5/48 objects degraded (10.417%), 4 pgs degraded (PG_DEGRADED)" in cluster log 2022-04-25T22:30:31.558 INFO:scrape:4 jobs: ['6805068', '6805030', '6805005', '6804736'] 2022-04-25T22:30:31.558 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.558 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{mon-debug', 'overrides/{no_client_pidfile', 'overrides/{osd-asserts', 'tasks/acls}', 'tasks/alternate-pool}', 'tasks/failover}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.558 INFO:scrape: 2022-04-25T22:30:31.558 INFO:scrape:Failure: Test failure: test_subvolume_group_quota_mds_path_restriction_to_group_path (tasks.cephfs.test_volumes.TestSubvolumeGroups) 2022-04-25T22:30:31.558 INFO:scrape:1 jobs: ['6805011'] 2022-04-25T22:30:31.558 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/volumes/{overrides', 'test/basic}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.559 INFO:scrape: 2022-04-25T22:30:31.559 INFO:scrape:Failure: Command failed (workunit test fs/full/subvolume_rm.sh) on smithi192 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_rm.sh' 2022-04-25T22:30:31.559 INFO:scrape:1 jobs: ['6804754'] 2022-04-25T22:30:31.559 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-node-1-mds-1-osd', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/full/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides', 'overrides/{no_client_pidfile', 'tasks/mgr-osd-full}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.559 INFO:scrape: 2022-04-25T22:30:31.559 INFO:scrape:Failure: Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) 2022-04-25T22:30:31.559 INFO:scrape:2 jobs: ['6805092', '6804812'] 2022-04-25T22:30:31.560 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{no_client_pidfile', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.560 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/damage}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.560 INFO:scrape: 2022-04-25T22:30:31.560 INFO:scrape:Crash: Test failure: test_snapclient_cache (tasks.cephfs.test_snapshots.TestSnapshots) 2022-04-25T22:30:31.560 INFO:scrape:ceph version 17.0.0-11827-g9bc2ff46 (9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c) quincy (dev) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f9a4fc12c20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x7f9a51485192] 5: /usr/lib64/ceph/libceph-common.so.2(+0x285355) [0x7f9a51485355] 6: (MDSTableClient::handle_request(boost::intrusive_ptr const&)+0x12b0) [0x55c7c70b55f0] 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xfed) [0x55c7c6dd8fdd] 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x7cb) [0x55c7c6ddb97b] 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c7c6ddbf9c] 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x108) [0x55c7c6dcae98] 11: (DispatchQueue::entry()+0x14fa) [0x7f9a5170df8a] 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f9a517c6081] 13: /lib64/libpthread.so.0(+0x817a) [0x7f9a4fc0817a] 14: clone() 2022-04-25T22:30:31.560 INFO:scrape:1 jobs: ['6804962'] 2022-04-25T22:30:31.560 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/snapshots}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.561 INFO:scrape: 2022-04-25T22:30:31.561 INFO:scrape:Failure: Test failure: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2022-04-25T22:30:31.561 INFO:scrape:1 jobs: ['6804969'] 2022-04-25T22:30:31.561 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/strays}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.561 INFO:scrape: 2022-04-25T22:30:31.561 INFO:scrape:Failure: Test failure: test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) 2022-04-25T22:30:31.561 INFO:scrape:1 jobs: ['6804946'] 2022-04-25T22:30:31.562 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{whitelist_health', 'tasks/cephfs_misc_tests}', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.562 INFO:scrape: 2022-04-25T22:30:31.562 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2022-04-25T22:30:31.562 INFO:scrape:1 jobs: ['6804945'] 2022-04-25T22:30:31.562 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'mds', 'mon', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/libcephfs/{frag', 'test}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.562 INFO:scrape: 2022-04-25T22:30:31.562 INFO:scrape:Failure: Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) 2022-04-25T22:30:31.562 INFO:scrape:1 jobs: ['6805076'] 2022-04-25T22:30:31.563 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/client-readahead}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.563 INFO:scrape: 2022-04-25T22:30:31.563 INFO:scrape:Failure: Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2022-04-25T22:30:31.563 INFO:scrape:1 jobs: ['6804838'] 2022-04-25T22:30:31.563 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/forward-scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.563 INFO:scrape: 2022-04-25T22:30:31.563 INFO:scrape:Failure: Test failure: test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks) 2022-04-25T22:30:31.564 INFO:scrape:1 jobs: ['6804932'] 2022-04-25T22:30:31.564 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/scrub}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.564 INFO:scrape: 2022-04-25T22:30:31.564 INFO:scrape:Failure: Command failed (workunit test fs/cephfs_mirror_ha_gen.sh) on smithi124 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/mirror && cd -- /home/ubuntu/cephtest/mnt.1/mirror && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/cephfs_mirror_ha_gen.sh' 2022-04-25T22:30:31.564 INFO:scrape:1 jobs: ['6804757'] 2022-04-25T22:30:31.564 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'fs/mirror-ha/{begin/{0-install', 'objectstore/bluestore-bitmap', 'overrides/{whitelist_health}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/cephfs-mirror-ha-workunit}'] 2022-04-25T22:30:31.564 INFO:scrape: 2022-04-25T22:30:31.564 INFO:scrape:Failure: Test failure: test_cd_with_no_args (tasks.cephfs.test_cephfs_shell.TestCD) 2022-04-25T22:30:31.565 INFO:scrape:1 jobs: ['6804762'] 2022-04-25T22:30:31.565 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_8}', 'fs/shell/{begin/{0-install', 'mds', 'mon', 'mount/fuse', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{no_client_pidfile', 'tasks/cephfs-shell}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.565 INFO:scrape: 2022-04-25T22:30:31.565 INFO:scrape:Failure: Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) 2022-04-25T22:30:31.565 INFO:scrape:1 jobs: ['6805024'] 2022-04-25T22:30:31.565 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{no_client_pidfile', 'rhel_8}', 'tasks/alternate-pool}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.565 INFO:scrape: 2022-04-25T22:30:31.565 INFO:scrape:Failure: Test failure: test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) 2022-04-25T22:30:31.566 INFO:scrape:1 jobs: ['6804888'] 2022-04-25T22:30:31.566 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/volumes/{begin/{0-install', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{no_client_pidfile', 'tasks/volumes/{overrides', 'test/misc}}', 'whitelist_health', 'whitelist_wrongly_marked_down}'] 2022-04-25T22:30:31.566 INFO:scrape: 2022-04-25T22:30:31.585 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi ================================================================= info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/ failed: 228 dead: 10 running: 0 waiting: 0 queued: 0 passed: 135 Fail ================================================================= [6804735] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:11:01 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804735/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804735/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7021c87e26a945b29fd7ceb8e6d201de Command failed on smithi103 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804736] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:08:26 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804736/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804736/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d2b629f3e9234176bac009cb8355a255 "2022-04-25T09:03:57.946612+0000 mon.a (mon.0) 194 : cluster [WRN] Health check failed: Degraded data redundancy: 10/48 objects degraded (20.833%), 5 pgs degraded (PG_DEGRADED)" in cluster log [6804738] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:25:13 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804738/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804738/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03738f398f65493994000b51741666f8 Timed out waiting for MDS daemons to become healthy [6804739] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:08:39 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804739/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804739/ sentry: https://sentry.ceph.com/organizations/ceph/?query=968493da1bfb455980dea96b7212a927 Command failed on smithi007 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804740] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:23:47 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804740/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804740/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3131a9fc15a740a38b662adae187dc79 Extra data: line 2 column 556 (char 556) [6804742] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} ----------------------------------------------------------------- time: 00:26:28 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804742/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804742/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73b77db8cf9041a5af421f1de3eccc74 Timed out waiting for MDS daemons to become healthy [6804743] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:08:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804743/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804743/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a8fe6ae700e48a690b82ed28b58a92a Command failed on smithi016 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804746] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:22:06 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804746/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804746/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7696dc0b12374c35960a985d8e1dd8e7 Timed out waiting for MDS daemons to become healthy [6804747] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} ----------------------------------------------------------------- time: 00:32:31 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804747/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804747/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d8896ae2093415984b22922b112d175 Timed out waiting for MDS daemons to become healthy [6804748] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:22:07 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804748/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804748/ sentry: https://sentry.ceph.com/organizations/ceph/?query=812dbe7233564b9eb0058e231e0af6f3 Timed out waiting for MDS daemons to become healthy [6804749] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:08:36 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804749/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804749/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e698745fbce4773be98df63ddea3712 Command failed on smithi019 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804750] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair} ----------------------------------------------------------------- time: 00:07:30 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804750/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804750/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76eb327d4b71458782cdb6121ee3971c Command failed on smithi007 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804752] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:10:21 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804752/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804752/ SELinux denials found on ubuntu@smithi006.front.sepia.ceph.com: ['type=AVC msg=audit(1650877890.603:203): avc: denied { node_bind } for pid=1527 comm="ping" saddr=172.21.15.6 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [6804754] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:13:06 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804754/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804754/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f101bcedb1e0454b9b906a080cc80b85 Command failed (workunit test fs/full/subvolume_rm.sh) on smithi192 with status 110: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/full/subvolume_rm.sh' [6804755] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client} ----------------------------------------------------------------- time: 00:10:16 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804755/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804755/ sentry: https://sentry.ceph.com/organizations/ceph/?query=320eb9feee5c4c6592b41abe9bc65d82 Command failed (workunit test client/test.sh) on smithi080 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/client/test.sh' [6804756] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{centos_8} tasks/mirror} ----------------------------------------------------------------- time: 00:11:10 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804756/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804756/ sentry: https://sentry.ceph.com/organizations/ceph/?query=40c2b3b051dc4bb580fa4cbd3c7bae16 Test failure: test_add_directory_path_normalization (tasks.cephfs.test_mirroring.TestMirroring) [6804757] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:15:11 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804757/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804757/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b9570d113c8543c6ad89ce6549c03d2b Command failed (workunit test fs/cephfs_mirror_ha_gen.sh) on smithi124 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/mirror && cd -- /home/ubuntu/cephtest/mnt.1/mirror && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 1h /home/ubuntu/cephtest/clo ne.client.1/qa/workunits/fs/cephfs_mirror_ha_gen.sh' [6804758] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:08:31 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804758/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804758/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1a4f2de791a43dd9b65e02740725400 Command failed on smithi103 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804760] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:13:56 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804760/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804760/ sentry: https://sentry.ceph.com/organizations/ceph/?query=81209d7b24374570a264fad75c4bcdb7 Test failure: test_mount_mon_and_osd_caps_present_mds_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) [6804762] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:10:52 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804762/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804762/ sentry: https://sentry.ceph.com/organizations/ceph/?query=958f8768a0f0434a8ba9d78ff5199e14 Test failure: test_cd_with_no_args (tasks.cephfs.test_cephfs_shell.TestCD) [6804768] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:21:36 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804768/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804768/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df21e1f668ad4af59b66885f7da54bbb Test failure: test_add_ancestor_and_child_directory (tasks.cephfs.test_mirroring.TestMirroring) [6804770] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:20:52 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804770/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804770/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6e7aa6d316ca44e5ab2067dfe523abf7 Test failure: test_default_uid_gid_subvolume_group (tasks.cephfs.test_volumes.TestSubvolumeGroups) [6804771] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} ----------------------------------------------------------------- time: 00:33:10 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804771/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804771/ error during scrub thrashing: Command failed on smithi025 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' [6804773] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:25:26 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804773/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804773/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dd456e246969475f87cb8f9d02b12e4b Timed out waiting for MDS daemons to become healthy [6804774] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 00:34:38 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804774/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804774/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b90270ce5da946a8bb9eb5eba12ec22e Timed out waiting for MDS daemons to become healthy [6804775] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} ----------------------------------------------------------------- time: 00:11:20 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804775/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804775/ sentry: https://sentry.ceph.com/organizations/ceph/?query=66af42e7ca304068ab5e32aa8c3cbd9c Command failed on smithi006 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0)" [6804776] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:18:54 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804776/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804776/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f54aa15ac9214f90ae8ed41d8f0f2d5b reached maximum tries (90) after waiting for 540 seconds [6804777] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:07:42 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804777/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804777/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3eb832b7c6b4bb9947e416b7b791a8a Command failed on smithi008 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804778] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:24:58 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804778/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804778/ sentry: https://sentry.ceph.com/organizations/ceph/?query=640d3967f187496eb345603924a17810 Timed out waiting for MDS daemons to become healthy [6804779] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} ----------------------------------------------------------------- time: 00:31:25 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804779/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804779/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eddab167e7fe44158704cc01d7217308 Timed out waiting for MDS daemons to become healthy [6804780] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:08:42 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804780/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804780/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8c9b4055348404185161f1b76fcdb23 Command failed on smithi016 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804781] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush} ----------------------------------------------------------------- time: 00:08:29 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804781/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804781/ sentry: https://sentry.ceph.com/organizations/ceph/?query=873c40704ec2428689f82a27ade89fcd Command failed on smithi155 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804783] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:24:38 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804783/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804783/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f7b90e5b70c4ed8b068e4deb380d96e Timed out waiting for MDS daemons to become healthy [6804787] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:08:06 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804787/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804787/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e95122d56a444b2c8d0d2d1a3dba5d5a Command failed on smithi093 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804789] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:43 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804789/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804789/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7dc51c3e3a814132beaf582aa0bb6bb9 Timed out waiting for MDS daemons to become healthy [6804791] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:08:43 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804791/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804791/ sentry: https://sentry.ceph.com/organizations/ceph/?query=839295c2b6184bf2a8437aa4f102ff92 Command failed on smithi007 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804793] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} ----------------------------------------------------------------- time: 00:31:33 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804793/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804793/ sentry: https://sentry.ceph.com/organizations/ceph/?query=842990386385470aa52ee9f3a3e31c8f Timed out waiting for MDS daemons to become healthy [6804794] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} ----------------------------------------------------------------- time: 00:09:15 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804794/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804794/ SELinux denials found on ubuntu@smithi071.front.sepia.ceph.com: ['type=AVC msg=audit(1650880844.362:202): avc: denied { node_bind } for pid=1545 comm="ping" saddr=172.21.15.71 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [6804795] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:11:33 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804795/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804795/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e9b7e7519a9b4761ba891225943e56e9 Command failed on smithi005 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804796] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead} ----------------------------------------------------------------- time: 00:07:38 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804796/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804796/ sentry: https://sentry.ceph.com/organizations/ceph/?query=15f5ca1800174453a12c85d74737ada0 Command failed on smithi022 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804798] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:07:10 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804798/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804798/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e5701a055c94f7fb60a7450eb206263 Command failed on smithi066 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804799] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} ----------------------------------------------------------------- time: 00:29:40 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804799/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804799/ error during scrub thrashing: Command failed on smithi080 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' [6804800] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:21:37 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804800/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804800/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f12dbef3c0494d58b12efe618ce7679d Timed out waiting for MDS daemons to become healthy [6804801] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:08:36 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804801/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804801/ sentry: https://sentry.ceph.com/organizations/ceph/?query=baadeea2554c4cd9b54d407eddc89159 Command failed on smithi110 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804804] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} ----------------------------------------------------------------- time: 00:32:15 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804804/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804804/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57cc6777635445db9ebc1d5effa4dc64 Timed out waiting for MDS daemons to become healthy [6804806] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:14:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804806/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804806/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de21dc1974e64a74ae1851ea0e25857a Extra data: line 2 column 475 (char 475) [6804807] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:08:32 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804807/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804807/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9abd7351b16c43cc9a7e29f638eab76a Command failed on smithi093 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804808] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:11:15 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804808/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804808/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb0604b0e3134715bfd5c37ea7f00fa1 Command failed on smithi174 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804809] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:15:58 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804809/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804809/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d2fd7bbcaf547ca95e5130f1a933031 Command failed on smithi049 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0)" [6804810] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} ----------------------------------------------------------------- time: 00:31:07 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804810/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd8e239b63d24733ab77d18262759da5 Timed out waiting for MDS daemons to become healthy [6804812] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage} ----------------------------------------------------------------- time: 00:10:14 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804812/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804812/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ede95960b5ea42e084a0281fc61a4b04 Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) [6804813] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:13:52 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804813/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804813/ SELinux denials found on ubuntu@smithi154.front.sepia.ceph.com: ['type=AVC msg=audit(1650882202.512:191): avc: denied { node_bind } for pid=1528 comm="ping" saddr=172.21.15.154 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [6804814] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:25:23 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804814/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804814/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e6d86c210384372a7f0a36dc104b172 Timed out waiting for MDS daemons to become healthy [6804817] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} ----------------------------------------------------------------- time: 00:14:34 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804817/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804817/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00ead035bcd14945b5991d9ce804014d Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) [6804819] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:21:53 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804819/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804819/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1699376ee3e34a26ab46594118f77939 Timed out waiting for MDS daemons to become healthy [6804820] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} ----------------------------------------------------------------- time: 00:26:52 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804820/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804820/ sentry: https://sentry.ceph.com/organizations/ceph/?query=14e04d8ca31b46699f2e4c9106d8ac45 Timed out waiting for MDS daemons to become healthy [6804822] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:18:50 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804822/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804822/ sentry: https://sentry.ceph.com/organizations/ceph/?query=691d49578ee9433c8749d7b81d51d242 reached maximum tries (90) after waiting for 540 seconds [6804824] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} ----------------------------------------------------------------- time: 00:08:46 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804824/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804824/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dddeb32eed4e410088dd64c92177a3a5 Command failed on smithi047 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804826] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:11:40 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804826/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804826/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a435388db9a4e51a1ee8d85c5070cf3 Command failed on smithi029 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804827] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:10:08 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804827/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804827/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f9e6812685343d295f84c8afda6582f Test failure: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [6804828] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:25:22 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804828/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804828/ sentry: https://sentry.ceph.com/organizations/ceph/?query=65f70ddf99d44e10874164a07c7aed9c Timed out waiting for MDS daemons to become healthy [6804829] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports} ----------------------------------------------------------------- time: 00:08:33 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804829/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804829/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ccbb5ce5363d4cff8f8af61fdaa0f96e Command failed on smithi085 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804830] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} ----------------------------------------------------------------- time: 00:31:15 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804830/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804830/ sentry: https://sentry.ceph.com/organizations/ceph/?query=be85abdd9ffe4537843712e0aa07874e Timed out waiting for MDS daemons to become healthy [6804831] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:21:44 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804831/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804831/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c6589a420edd443cb8afcb2b2db569f1 reached maximum tries (90) after waiting for 540 seconds [6804832] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:08:30 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804832/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804832/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64c6bcc2097646f8b71f0e99a4e7cb5d Command failed on smithi082 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804835] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:24:40 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804835/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804835/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec63716cab5244cd8b41d851ae493ec4 Timed out waiting for MDS daemons to become healthy [6804836] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} ----------------------------------------------------------------- time: 00:28:56 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804836/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804836/ error during scrub thrashing: Command failed on smithi105 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' [6804837] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:25:45 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804837/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804837/ sentry: https://sentry.ceph.com/organizations/ceph/?query=663e6d830b1346f2b6d3b0ab8bcbfba4 Timed out waiting for MDS daemons to become healthy [6804838] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:13:20 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804838/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804838/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03664c9f3b1a4e348511f8ea667edb18 Test failure: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) [6804840] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:24:14 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804840/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804840/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aaea91d3eefe447aac827395dd60ee1c Extra data: line 2 column 557 (char 557) [6804841] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:08:03 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804841/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804841/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a6a4414a9b414d0996037073bab079ea Command failed on smithi103 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804842] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} ----------------------------------------------------------------- time: 00:32:11 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804842/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804842/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a5a88b30a1c44a99a2ad3cf68250ce8 Timed out waiting for MDS daemons to become healthy [6804843] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:18:49 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804843/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804843/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8128c941388a41f99f2cb7027461cc9c reached maximum tries (90) after waiting for 540 seconds [6804844] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:08:29 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804844/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804844/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f0fe203207c4ef6ad29e5fde111510f Command failed on smithi124 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804845] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/fragment} ----------------------------------------------------------------- time: 00:07:30 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804845/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804845/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b073fe5f8d5246a8aa1ca47b256967fe Command failed on smithi035 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804847] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} ----------------------------------------------------------------- time: 00:32:19 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804847/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804847/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f5d216b71286432398c2fee1d1524a00 Timed out waiting for MDS daemons to become healthy [6804848] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:25:43 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804848/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804848/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7411fff049c6474dbbd65373eac1cfa6 Timed out waiting for MDS daemons to become healthy [6804849] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:22:03 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804849/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804849/ sentry: https://sentry.ceph.com/organizations/ceph/?query=417345f19b9a418f899303dff0c9c559 Timed out waiting for MDS daemons to become healthy [6804851] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:08:47 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804851/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804851/ sentry: https://sentry.ceph.com/organizations/ceph/?query=512ac623c6874b0b85ffb1044030d0c6 Command failed on smithi082 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804854] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:24:23 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804854/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804854/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8117c8b7d0214ba7aaa0bf30399b8873 Timed out waiting for MDS daemons to become healthy [6804856] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/ino_release_cb} ----------------------------------------------------------------- time: 00:13:28 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804856/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804856/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d7661a6c00ba4a9787d245aeeb7af3e2 Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_ino_release_cb' [6804858] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:22:29 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804858/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804858/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df400bb8c02d48fab2ed738e682ce377 Timed out waiting for MDS daemons to become healthy [6804859] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} ----------------------------------------------------------------- time: 00:26:34 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804859/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804859/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e0beb4963cea42069a12aab1f883331b Timed out waiting for MDS daemons to become healthy [6804861] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 06:10:50 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804861/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804861/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0c2368be4e1a4c2ca01ebf193d6c64c8 Command failed (workunit test fs/snaps/untar_snap_rm.sh) on smithi155 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/fs/snaps/untar_snap_rm.sh' [6804863] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:22:04 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804863/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804863/ sentry: https://sentry.ceph.com/organizations/ceph/?query=176e287304ee49298b941a5d1ad0c8eb Timed out waiting for MDS daemons to become healthy [6804864] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{no}} ----------------------------------------------------------------- time: 00:31:22 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804864/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4a9ca6fbff334e39a364b259d3710359 Timed out waiting for MDS daemons to become healthy [6804865] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:08:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804865/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804865/ sentry: https://sentry.ceph.com/organizations/ceph/?query=10d506e6ec2744e5923fbf1974bf5197 Command failed on smithi120 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804869] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} ----------------------------------------------------------------- time: 01:26:58 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804869/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804869/ error during scrub thrashing: Command failed on smithi005 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' [6804870] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:08:43 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804870/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804870/ sentry: https://sentry.ceph.com/organizations/ceph/?query=934ebee0f7564c6085f572c3a07e992e Command failed on smithi003 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804872] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:21:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804872/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804872/ sentry: https://sentry.ceph.com/organizations/ceph/?query=473d545aa6844405af557fb802776961 Timed out waiting for MDS daemons to become healthy [6804873] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:23 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804873/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804873/ sentry: https://sentry.ceph.com/organizations/ceph/?query=648800afeda841f4a6d57c35f0bb769e Timed out waiting for MDS daemons to become healthy [6804874] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:25:21 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804874/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804874/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9b9166ec35ec45d091870236a623a1a8 Timed out waiting for MDS daemons to become healthy [6804876] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:08:38 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804876/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804876/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a060ba11362849b2913fbd225cb30a89 Command failed on smithi134 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804877] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:24:28 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804877/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804877/ sentry: https://sentry.ceph.com/organizations/ceph/?query=92ec3582154f40b58d42da6240c06bb5 Timed out waiting for MDS daemons to become healthy [6804878] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} ----------------------------------------------------------------- time: 00:32:35 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804878/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804878/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00bc9bf2a46647018a9c423666299155 Timed out waiting for MDS daemons to become healthy [6804880] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:21:51 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804880/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804880/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7a71bc942a6f46e98c337ef0baabff07 Timed out waiting for MDS daemons to become healthy [6804882] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/metrics} ----------------------------------------------------------------- time: 00:20:24 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804882/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804882/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da81bf839f8d441ca4f08ecfd543430b Test failure: test_delayed_metrics (tasks.cephfs.test_mds_metrics.TestMDSMetrics) [6804888] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:09:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804888/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804888/ sentry: https://sentry.ceph.com/organizations/ceph/?query=56906f2d237244c48483f1e11875f06f Test failure: test_connection_expiration (tasks.cephfs.test_volumes.TestMisc) [6804890] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:11:15 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804890/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804890/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d85b43a1c66648d1b1b4960562e803ba Command failed on smithi003 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804892] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} ----------------------------------------------------------------- time: 00:26:38 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804892/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804892/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3fdc4c074894bfd8756e5f04c982df4 Timed out waiting for MDS daemons to become healthy [6804893] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:08:30 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804893/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804893/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad652271c498487294bf555bdb42a741 Command failed on smithi123 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804894] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/multimds_misc} ----------------------------------------------------------------- time: 00:15:33 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804894/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70a9d8ecea3e45f59a4bfdbe67c38ffb Test failure: test_apply_tag (tasks.cephfs.test_multimds_misc.TestScrub2) [6804896] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:14:24 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804896/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804896/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f46a7df75ce6418a9c05e476439763c1 Extra data: line 2 column 477 (char 477) [6804900] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} ----------------------------------------------------------------- time: 00:30:50 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804900/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804900/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8a50ac7805bf443fbe443c3df1b98937 Timed out waiting for MDS daemons to become healthy [6804902] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:10:48 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804902/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804902/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b0a3fe887cfb4c1fb59f126700602d08 Command failed on smithi082 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804907] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} ----------------------------------------------------------------- time: 00:36:44 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804907/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804907/ error during scrub thrashing: Command failed on smithi123 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' [6804908] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:18:57 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804908/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804908/ sentry: https://sentry.ceph.com/organizations/ceph/?query=64eb5f49a3cd4eea9cde2a970d0f2b20 reached maximum tries (90) after waiting for 540 seconds [6804910] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:28 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804910/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804910/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ef95e68afaa485daaeb9e46c994a0db Timed out waiting for MDS daemons to become healthy [6804911] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:07:32 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804911/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804911/ sentry: https://sentry.ceph.com/organizations/ceph/?query=56b6899220314e4393bb8c30f75f4ab3 Command failed on smithi003 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804915] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:18:03 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804915/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804915/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c2c0b481d794ea1aacf320ff70be0f0 reached maximum tries (90) after waiting for 540 seconds [6804916] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} ----------------------------------------------------------------- time: 00:07:35 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804916/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804916/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f9fe13193302462986515547107ef87f Command failed on smithi149 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs- mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' [6804917] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:07:41 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804917/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804917/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1d83083a0f548299c4d81c618948ce7 Command failed on smithi038 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs- mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' [6804918] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/quota} ----------------------------------------------------------------- time: 00:07:25 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804918/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804918/ sentry: https://sentry.ceph.com/organizations/ceph/?query=54a5fbb1b48a47708c502c463bb9c531 Command failed on smithi196 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs- mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel' [6804919] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:08:34 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804919/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804919/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e5b1b58dbfc4273a23b53d154ab495d Command failed on smithi008 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804920] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:24:52 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804920/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804920/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe9d1047737444d68c988526ed5b2868 Timed out waiting for MDS daemons to become healthy [6804922] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:08:24 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804922/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804922/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0b1e1c7df378459baddaf038c7952f44 Command failed on smithi082 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804924] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:05:50 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804924/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804924/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e90477305cea4ee089ea42e9c5ec396c No module named 'tasks' [6804925] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/recovery-fs} ----------------------------------------------------------------- time: 00:08:39 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804925/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804925/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a29c08ad8274edebe3bf12b9651c46b Command failed on smithi134 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804927] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{no}} ----------------------------------------------------------------- time: 00:29:23 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804927/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804927/ sentry: https://sentry.ceph.com/organizations/ceph/?query=21622c9bdee84e5dbd76978e3869b9bd Timed out waiting for MDS daemons to become healthy [6804928] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:07:04 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804928/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804928/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fae8729f79e5472cbae1ef87488ea306 Command failed on smithi145 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804929] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:21:32 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804929/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804929/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f6251469b244f9e9de1d029030afcf1 Timed out waiting for MDS daemons to become healthy [6804930] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 00:40:37 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804930/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804930/ sentry: https://sentry.ceph.com/organizations/ceph/?query=47a074a53a334f32a3a436b65ca11b30 Timed out waiting for MDS daemons to become healthy [6804931] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:08:54 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804931/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804931/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e778333e215c41a0886b67be682d91b9 Command failed on smithi003 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804932] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/scrub} ----------------------------------------------------------------- time: 00:09:11 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804932/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804932/ sentry: https://sentry.ceph.com/organizations/ceph/?query=637c163a74f34b85a7a9815326c10c74 Test failure: test_scrub_checks (tasks.cephfs.test_scrub_checks.TestScrubChecks) [6804933] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:24:35 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804933/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804933/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1185cdff89ce4129bc2e8d598a310721 Timed out waiting for MDS daemons to become healthy [6804934] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:08:36 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804934/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804934/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2ef7722c6d754a2684965fa6325db967 Command failed on smithi079 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804935] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} ----------------------------------------------------------------- time: 02:41:36 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804935/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804935/ error during scrub thrashing: Command failed on smithi007 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' [6804937] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:22:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804937/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804937/ sentry: https://sentry.ceph.com/organizations/ceph/?query=68c383c8a196426d98c575d64c547ddf Command failed (workunit test suites/fsstress.sh) on smithi122 with status 126: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [6804938] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:07:39 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804938/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804938/ sentry: https://sentry.ceph.com/organizations/ceph/?query=739830eafd1b44758929ae1bc2cee074 Command failed on smithi005 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804939] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/sessionmap} ----------------------------------------------------------------- time: 00:11:11 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804939/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804939/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ac13421e3264cdbb7d1f74228e6ce9c Test failure: test_session_evict_blocklisted (tasks.cephfs.test_sessionmap.TestSessionMap) [6804941] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} ----------------------------------------------------------------- time: 00:38:40 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804941/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804941/ sentry: https://sentry.ceph.com/organizations/ceph/?query=53ca2fe624714f7786ca587b7df1b14d Timed out waiting for MDS daemons to become healthy [6804942] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:08:29 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804942/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804942/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33dc75ef9e0249e79e122cd9e3f552f3 Command failed on smithi174 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804945] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:15:09 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804945/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804945/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5dd9ccdce92431c873f5300561b8f05 Command failed (workunit test libcephfs/test.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [6804946] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:13:14 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804946/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804946/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f91c69cd100c46e281e33251ea64edff Test failure: test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) [6804947] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} ----------------------------------------------------------------- time: 00:07:24 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804947/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804947/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e7b4613946a40adad02952b3739955e Command failed on smithi019 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804951] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:09:17 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804951/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804951/ sentry: https://sentry.ceph.com/organizations/ceph/?query=308d9f4d4ded41bd848397420c821704 Test failure: test_subvolume_group_snapshot_unsupported_status (tasks.cephfs.test_volumes.TestSubvolumeGroupSnapshots) [6804952] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:23:58 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804952/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804952/ sentry: https://sentry.ceph.com/organizations/ceph/?query=aa07f7fe471f473880f95216743e46de Timed out waiting for MDS daemons to become healthy [6804953] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} ----------------------------------------------------------------- time: 00:35:17 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804953/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804953/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a0799c9d0eea4b968569258d392bd8ac Timed out waiting for MDS daemons to become healthy [6804954] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:18:29 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804954/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804954/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab96611969c44f09be999d3eb0e5cdb7 reached maximum tries (90) after waiting for 540 seconds [6804955] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:23:30 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804955/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804955/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d64700b7d4b74ff4864b6da8037656cd Test failure: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) [6804958] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:22:04 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804958/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804958/ sentry: https://sentry.ceph.com/organizations/ceph/?query=55d2e16e5b654f84a8c1117e0950a9b6 Timed out waiting for MDS daemons to become healthy [6804962] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/snapshots} ----------------------------------------------------------------- time: 00:27:31 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804962/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804962/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8faf430462a3449998c73074e132ba72 Test failure: test_snapclient_cache (tasks.cephfs.test_snapshots.TestSnapshots) [6804963] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:24:35 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804963/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804963/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2351f67b0b8141b3b05f4210523d4afa Timed out waiting for MDS daemons to become healthy [6804964] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} ----------------------------------------------------------------- time: 00:30:32 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804964/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804964/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3737b49f5c84f07839c3ef9a14b9502 Timed out waiting for MDS daemons to become healthy [6804965] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:11:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804965/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804965/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03e26f63d7334ae09cdc19ddd6226d24 Command failed on smithi174 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804969] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/strays} ----------------------------------------------------------------- time: 00:16:16 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804969/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804969/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1441c9a7ce6d4d9ca117938ddc7bdbf2 Test failure: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) [6804971] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} ----------------------------------------------------------------- time: 00:31:37 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804971/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804971/ sentry: https://sentry.ceph.com/organizations/ceph/?query=23ba9393325f40aa8d523ec618c5b980 Timed out waiting for MDS daemons to become healthy [6804973] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:06:59 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804973/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804973/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a105ac8169994decb1d88c0b797e485d Command failed on smithi138 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804975] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} ----------------------------------------------------------------- time: 00:43:42 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804975/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804975/ error during scrub thrashing: Command failed on smithi078 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status' [6804976] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:08:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804976/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804976/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50f223e171a84492be7939aab124f560 Command failed on smithi103 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804977] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:08:47 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804977/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804977/ sentry: https://sentry.ceph.com/organizations/ceph/?query=02e080cb85dd48c9bf8ce456c54c7c31 Command failed on smithi016 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804979] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:11:19 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804979/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804979/ sentry: https://sentry.ceph.com/organizations/ceph/?query=920a5b0986974e3c891072822a183867 Command failed on smithi124 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804980] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:54 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804980/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804980/ sentry: https://sentry.ceph.com/organizations/ceph/?query=919856a437ae426e940763e5793a970e Timed out waiting for MDS daemons to become healthy [6804982] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:10:46 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804982/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804982/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07fdb4ef445441c59af5e8963b15d4b9 Command failed on smithi082 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804985] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:24:58 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804985/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804985/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dfea447d6ef84ab5b5b94fc8be9909d3 Timed out waiting for MDS daemons to become healthy [6804986] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} ----------------------------------------------------------------- time: 00:31:57 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804986/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804986/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75d4ddc52588431cb19887ea515be73f Timed out waiting for MDS daemons to become healthy [6804987] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:08:45 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804987/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804987/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1b7687add13429596e91ab4d58e77c8 Command failed on smithi142 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6804990] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:22:31 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804990/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804990/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f22cbbbf029482a822a386fbcc29fe0 Timed out waiting for MDS daemons to become healthy [6804995] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:25:33 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804995/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804995/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5b7c6467c90445f49f6ad79ee72718ee Timed out waiting for MDS daemons to become healthy [6804996] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} ----------------------------------------------------------------- time: 00:28:49 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804996/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804996/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0143acd96b6848f399547b84fc868a91 Timed out waiting for MDS daemons to become healthy [6804998] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:14:02 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804998/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804998/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6b332419e3434eb9bf069554f8dda8e7 Extra data: line 2 column 478 (char 478) [6804999] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:09:06 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804999/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804999/ sentry: https://sentry.ceph.com/organizations/ceph/?query=584a462111d548898e5bd761e3680aab Command failed on smithi104 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805002] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} ----------------------------------------------------------------- time: 00:33:20 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805002/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805002/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4e817cf36c24428897f3dec34064bf4b Timed out waiting for MDS daemons to become healthy [6805003] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:08:34 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805003/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805003/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2912a9e710e47a19e9e30687ef6e96c Command failed on smithi137 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805005] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/acls} ----------------------------------------------------------------- time: 00:08:50 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805005/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805005/ sentry: https://sentry.ceph.com/organizations/ceph/?query=89dea6455c8b4bff8633ae015f3e463b "2022-04-25T14:34:08.382767+0000 mon.a (mon.0) 223 : cluster [WRN] Health check failed: Degraded data redundancy: 9/48 objects degraded (18.750%), 4 pgs degraded (PG_DEGRADED)" in cluster log [6805011] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:27:59 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805011/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805011/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9db35d843d314beaa3fab45b41852767 Test failure: test_subvolume_group_quota_mds_path_restriction_to_group_path (tasks.cephfs.test_volumes.TestSubvolumeGroups) [6805012] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:02 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805012/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805012/ error during scrub thrashing: Command failed on smithi005 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' [6805014] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:11:10 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805014/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805014/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab02c49a6874494bb3697b8757d9e667 Command failed on smithi063 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805016] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} ----------------------------------------------------------------- time: 00:34:48 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805016/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805016/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19b8412a9d914a4cb3e8134a0837fffd Timed out waiting for MDS daemons to become healthy [6805017] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:21:38 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805017/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805017/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e52d67d1d084a02983e8794d10d4490 Timed out waiting for MDS daemons to become healthy [6805018] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/admin} ----------------------------------------------------------------- time: 00:10:58 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805018/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805018/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6fd9e0a76dfc45b1a574927f2d17a29b Test failure: test_add_already_in_use_metadata_pool (tasks.cephfs.test_admin.TestAddDataPool) [6805019] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:08:50 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805019/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805019/ sentry: https://sentry.ceph.com/organizations/ceph/?query=594473be6f5445f2bb12bafe4f7a5520 Command failed on smithi062 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805021] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{no}} ----------------------------------------------------------------- time: 00:27:31 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805021/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805021/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9200e15a0f8b409392c52fa71855391f Timed out waiting for MDS daemons to become healthy [6805022] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:11:28 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805022/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805022/ sentry: https://sentry.ceph.com/organizations/ceph/?query=afac84415fbd43babecebbce77bef286 Command failed on smithi078 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805024] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:13:54 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805024/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805024/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc43a290553a46258a120b61f38f4bdb Test failure: test_rebuild_simple (tasks.cephfs.test_recovery_pool.TestRecoveryPool) [6805025] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:25:20 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805025/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805025/ sentry: https://sentry.ceph.com/organizations/ceph/?query=19269924617e489da44817d572a78667 Timed out waiting for MDS daemons to become healthy [6805027] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:10:36 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805027/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805027/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a5655a5a680423591a9cc6fcf6392d5 Command failed on smithi080 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0)" [6805028] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} ----------------------------------------------------------------- time: 00:24:55 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805028/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805028/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b10904d29544208afafb5003938cbc0 Extra data: line 2 column 554 (char 554) [6805029] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:09:08 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805029/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805029/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fb5aaad0ff304b56a64f2db6bc954e91 Command failed on smithi017 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805030] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:08:15 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805030/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805030/ sentry: https://sentry.ceph.com/organizations/ceph/?query=87533552760844de8e5f1cdb1896799c "2022-04-25T15:15:12.392865+0000 mon.a (mon.0) 294 : cluster [WRN] Health check failed: Degraded data redundancy: 9/48 objects degraded (18.750%), 4 pgs degraded (PG_DEGRADED)" in cluster log [6805031] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} ----------------------------------------------------------------- time: 00:11:08 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805031/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805031/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a244582fa4c74ecba809811bc81b4bda Command failed on smithi137 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805032] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/dbench} wsync/{yes}} ----------------------------------------------------------------- time: 01:11:26 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805032/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805032/ error during scrub thrashing: Command failed on smithi079 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' [6805033] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:24:12 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805033/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805033/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6cd5b644b051477bae1605090fd68f21 Timed out waiting for MDS daemons to become healthy [6805035] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:22:28 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805035/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805035/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1cf09a74eaeb4d38b012a9bafd8ef194 Timed out waiting for MDS daemons to become healthy [6805037] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:31 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805037/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805037/ sentry: https://sentry.ceph.com/organizations/ceph/?query=902dabecf3dc40f2a7dba5cbf26e35ea Timed out waiting for MDS daemons to become healthy [6805038] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:07:06 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805038/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805038/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fedff95a80dc4a208c38a1071d2e9047 Command failed on smithi062 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805040] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:09:53 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805040/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805040/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a1cb1872d53b4142b74a9ad94db64536 Command failed (workunit test fs/test_python.sh) on smithi063 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [6805042] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:10:55 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805042/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805042/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc09f272eed248ee9c371614170f4120 Command failed on smithi035 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0)" [6805045] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:08:37 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805045/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805045/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b78db72f9cbb48c5b9cbb4ba11427b84 Command failed on smithi160 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805046] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/norstats} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:09 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805046/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805046/ error during scrub thrashing: Command failed on smithi017 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' [6805048] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:11:38 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805048/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805048/ sentry: https://sentry.ceph.com/organizations/ceph/?query=817d39ac02aa45d19fd3e1b2cd771087 Command failed on smithi124 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805050] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{yes}} ----------------------------------------------------------------- time: 00:33:46 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805050/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805050/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0ace5c79d5b141e7905af83b69fc0439 Timed out waiting for MDS daemons to become healthy [6805053] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:08:04 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805053/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805053/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f80eff1b90864679a8059af751a3b53a Command failed on smithi089 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805054] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} ----------------------------------------------------------------- time: 00:26:41 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805054/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805054/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7bc6a8ef5a8745598cd6b13a4f2bbb4e Timed out waiting for MDS daemons to become healthy [6805055] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:21:51 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805055/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805055/ sentry: https://sentry.ceph.com/organizations/ceph/?query=775b015b54254c8c82369c14e457a00a Timed out waiting for MDS daemons to become healthy [6805058] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:08:50 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805058/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805058/ sentry: https://sentry.ceph.com/organizations/ceph/?query=162f247992a543dda0064219afcab24e Command failed on smithi062 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805059] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:24:04 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805059/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805059/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dd4fb64ff54b4f0aaf42d67a703986c3 Timed out waiting for MDS daemons to become healthy [6805061] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:11:38 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805061/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805061/ sentry: https://sentry.ceph.com/organizations/ceph/?query=91af3a860b3040409a45c227e35499f6 Command failed on smithi039 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805063] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits} ----------------------------------------------------------------- time: 00:12:51 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805063/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805063/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3e97353b7804196822a970a1e46d216 Test failure: test_client_min_caps_working_set (tasks.cephfs.test_client_limits.TestClientLimits) [6805064] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:07:50 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805064/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805064/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6173cf0941564eb2be3ebf8d5d86e893 Command failed on smithi035 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805065] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} ----------------------------------------------------------------- time: 00:34:11 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805065/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805065/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0afd42d4d62b4b148e687a57b315e859 Timed out waiting for MDS daemons to become healthy [6805068] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/failover} ----------------------------------------------------------------- time: 00:08:23 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805068/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805068/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd9266f0756941f89827bec0f975c4ad "2022-04-25T15:44:06.267737+0000 mon.a (mon.0) 298 : cluster [WRN] Health check failed: Degraded data redundancy: 5/48 objects degraded (10.417%), 4 pgs degraded (PG_DEGRADED)" in cluster log [6805070] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:11:34 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805070/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805070/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a118c71291804f339581e9c107a9a25f Command failed on smithi037 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805071] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:10:25 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805071/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805071/ sentry: https://sentry.ceph.com/organizations/ceph/?query=902bc81c741b4761981015218182518e Test failure: test_clone_failure_status_failed (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) [6805072] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:08:42 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805072/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805072/ sentry: https://sentry.ceph.com/organizations/ceph/?query=192a726ae7eb4ddab7e019492fe43007 Command failed on smithi106 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805074] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:54 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805074/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805074/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9530c8ae78e64d5495e343aec4f18cc1 Timed out waiting for MDS daemons to become healthy [6805075] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:21:08 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805075/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805075/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49fc8aa173094f49aa889ba5ea11eb0c reached maximum tries (90) after waiting for 540 seconds [6805076] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-readahead} ----------------------------------------------------------------- time: 00:10:21 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805076/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805076/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2bd5a787510e4216aaa948b7a8754519 Test failure: test_flush (tasks.cephfs.test_readahead.TestReadahead) [6805077] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:11:32 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805077/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805077/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9ca629ef33f9423eb4d0328574ba3eab Command failed on smithi062 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805079] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay tasks/{0-check-counter workunit/suites/iozone} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:32 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805079/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805079/ error during scrub thrashing: Command failed on smithi085 with status 110: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive' [6805080] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:07:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805080/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805080/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46c13a7b3d2e4221ae66c844ab4ebc1e Command failed on smithi124 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805081] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:24:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805081/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805081/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58c694a3626245e880af33708f0f8bce Timed out waiting for MDS daemons to become healthy [6805083] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:08:37 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805083/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805083/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0c2c940b36554c7eab6705ae7d6ffd7f Command failed on smithi107 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805084] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} ----------------------------------------------------------------- time: 00:34:10 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805084/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805084/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f9f8b0faa4e465e9786a3cb158b3a55 Timed out waiting for MDS daemons to become healthy [6805087] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:21:14 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805087/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805087/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2899da01d19042a889cdfaafd4b0fc2b Timed out waiting for MDS daemons to become healthy [6805088] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:15:05 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805088/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805088/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33f657e1508c4678af2ed090fed30fd7 Extra data: line 2 column 473 (char 473) [6805089] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:08:48 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805089/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805089/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a5999463c3184380b1ee3695ad45a5a7 Command failed on smithi078 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805091] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay tasks/{0-check-counter workunit/direct_io} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:05 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805091/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805091/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2cbc2595ee194329bfd319234875ce65 Timed out waiting for MDS daemons to become healthy [6805092] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/damage} ----------------------------------------------------------------- time: 00:10:51 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805092/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805092/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d87d4f18d0549058dccc79363fdae47 Test failure: test_damaged_dentry (tasks.cephfs.test_damage.TestDamage) [6805094] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:35:32 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805094/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805094/ SELinux denials found on ubuntu@smithi059.front.sepia.ceph.com: ['type=AVC msg=audit(1650902009.662:192): avc: denied { node_bind } for pid=1478 comm="ping" saddr=172.21.15.59 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [6805096] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:25:23 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805096/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805096/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c140ea589d474a60ad425c068214dec0 Timed out waiting for MDS daemons to become healthy [6805099] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:08:41 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805099/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805099/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e26e4612e65644d3a252c4b1be6f7739 Command failed on smithi138 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805100] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/data-scan} ----------------------------------------------------------------- time: 00:10:41 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805100/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805100/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fddd1a8827bb441c8f2284195faea592 Test failure: test_fragmented_injection (tasks.cephfs.test_data_scan.TestDataScan) [6805101] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:17:47 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805101/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805101/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1984c47b895849d2ad1f0b995cc2127b reached maximum tries (90) after waiting for 540 seconds [6805103] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} ----------------------------------------------------------------- time: 00:33:39 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805103/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805103/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fd1e650ecfa04462ac0a71a94de99e6b Timed out waiting for MDS daemons to become healthy [6805104] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:11:18 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805104/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805104/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9ff71657edf34e9ca3277dbd5babde48 Command failed on smithi120 with status 1: "(cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest- mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs)" [6805105] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:25:50 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805105/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=694fb74762504e7db2ba8510cda6bc46 Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi097 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [6805106] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/exports} ----------------------------------------------------------------- time: 00:16:02 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805106/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805106/ sentry: https://sentry.ceph.com/organizations/ceph/?query=59e7b79ca14e468f84e6889b061eaa12 Test failure: test_ephemeral_pin_dist_failover (tasks.cephfs.test_exports.TestEphemeralPins) Dead ================================================================= [6804759] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804759/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804759/ hit max job timeout [6804821] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804821/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804821/ hit max job timeout [6804879] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804879/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804879/ hit max job timeout [6804883] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804883/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804883/ hit max job timeout [6804913] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 03:32:43 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804913/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804913/ sentry: https://sentry.ceph.com/organizations/ceph/?query=41b6ccfd97384076abd1269b74fcadf2 Command failed (workunit test suites/fsstress.sh) on smithi183 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' [6804926] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804926/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804926/ hit max job timeout [6805015] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805015/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805015/ hit max job timeout [6805073] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805073/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805073/ hit max job timeout [6805097] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805097/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805097/ hit max job timeout [6805102] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 03:32:11 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805102/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805102/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9fb6f044dbc4fe89af7e4a29fde0228 Command failed (workunit test suites/fsstress.sh) on smithi183 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9bc2ff4611c3c4e3862566d39262fbe7a8b33b2c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh' Pass ================================================================= [6804734] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:17:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804734/ [6804737] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} time: 01:06:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804737/ [6804741] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:10:17 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804741/ [6804744] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree} time: 00:19:54 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804744/ [6804745] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:22:21 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804745/ [6804751] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:14:03 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804751/ [6804753] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:45:28 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804753/ [6804761] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:11:56 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804761/ [6804763] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:56:15 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804763/ [6804764] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 00:48:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804764/ [6804765] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/whitelist_health supported-random-distros$/{rhel_8} tasks/fstop} time: 00:13:05 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804765/ [6804766] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:18:42 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804766/ [6804767] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:30:52 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804767/ [6804769] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/dbench validater/valgrind} time: 00:50:05 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804769/ [6804772] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:58:19 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804772/ [6804782] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:36:16 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804782/ [6804784] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:10:56 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804784/ [6804785] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} time: 00:35:31 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804785/ [6804786] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:17:59 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804786/ [6804788] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-limits} time: 00:22:33 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804788/ [6804790] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:15:07 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804790/ [6804792] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:29:13 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804792/ [6804797] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:20:32 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804797/ [6804802] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} time: 00:29:16 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804802/ [6804803] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:26:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804803/ [6804805] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:27:46 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804805/ [6804811] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:20:44 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804811/ [6804815] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} time: 00:29:21 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804815/ [6804816] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:13:46 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804816/ [6804818] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:58:08 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804818/ [6804823] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:10:55 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804823/ [6804825] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:43:01 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804825/ [6804833] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:40:13 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804833/ [6804834] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:18:34 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804834/ [6804839] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:20:24 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804839/ [6804846] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:13:19 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804846/ [6804850] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 01:12:06 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804850/ [6804852] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:34:18 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804852/ [6804853] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/journal-repair} time: 00:16:29 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804853/ [6804855] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:16:56 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804855/ [6804857] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:32 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804857/ [6804860] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:13:08 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804860/ [6804862] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-flush} time: 00:10:17 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804862/ [6804866] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:26:50 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804866/ [6804867] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:11:19 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804867/ [6804868] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds-full} time: 00:21:23 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804868/ [6804871] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:13:24 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804871/ [6804875] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/mds_creation_retry} time: 00:11:06 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804875/ [6804881] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:12:57 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804881/ [6804884] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} time: 00:28:49 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804884/ [6804885] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:42:54 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804885/ [6804886] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:12:46 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804886/ [6804887] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/dbench validater/valgrind} time: 00:52:10 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804887/ [6804889] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} time: 00:30:43 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804889/ [6804891] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 01:01:20 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804891/ [6804895] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:13:02 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804895/ [6804897] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:18:11 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804897/ [6804898] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:44:05 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804898/ [6804899] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:46:31 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804899/ [6804901] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:12:26 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804901/ [6804903] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:33 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804903/ [6804904] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:12:09 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804904/ [6804905] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable} time: 00:13:59 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804905/ [6804906] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 01:08:51 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804906/ [6804909] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:11:12 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804909/ [6804912] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/pool-perm} time: 00:11:57 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804912/ [6804914] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:13:38 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804914/ [6804921] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/pjd} wsync/{no}} time: 00:30:39 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804921/ [6804923] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:12:32 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804923/ [6804936] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:15:30 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804936/ [6804940] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:59:52 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804940/ [6804943] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:47 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804943/ [6804944] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/whitelist_health} time: 00:18:38 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804944/ [6804948] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:27 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804948/ [6804949] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:31:42 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804949/ [6804950] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/fsstress validater/lockdep} time: 00:18:42 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804950/ [6804956] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:13:29 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804956/ [6804957] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/no pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:27:30 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804957/ [6804959] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/ffsb} wsync/{no}} time: 00:55:51 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804959/ [6804960] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:57:18 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804960/ [6804961] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:23:18 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804961/ [6804966] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:48:35 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804966/ [6804967] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:56:07 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804967/ [6804968] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:13:21 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804968/ [6804970] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:14:14 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804970/ [6804972] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-upgrade}} time: 00:08:53 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804972/ [6804974] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:56:14 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804974/ [6804978] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:12:07 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804978/ [6804981] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:12:37 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804981/ [6804983] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/truncate_delay} time: 00:14:03 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804983/ [6804984] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:54 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804984/ [6804988] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:11:16 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804988/ [6804989] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} time: 00:39:11 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804989/ [6804991] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/dir-max-entries} time: 00:12:29 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804991/ [6804992] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:17:56 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804992/ [6804993] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:12:20 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804993/ [6804994] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:14:07 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804994/ [6804997] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:53:41 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6804997/ [6805000] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/workunit/quota} time: 00:11:24 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805000/ [6805001] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:10:57 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805001/ [6805004] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:13:26 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805004/ [6805006] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:11:28 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805006/ [6805007] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{mon-debug whitelist_health whitelist_wrongly_marked_down} tasks/multifs-auth} time: 00:23:06 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805007/ [6805008] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:31:44 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805008/ [6805009] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 00:42:54 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805009/ [6805010] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{mon-debug session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/dbench validater/valgrind} time: 00:59:29 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805010/ [6805013] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:23:00 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805013/ [6805020] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:12:53 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805020/ [6805023] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:52:59 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805023/ [6805026] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} time: 00:38:59 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805026/ [6805034] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/asok_dump_tree} time: 00:21:11 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805034/ [6805036] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:56:21 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805036/ [6805039] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:02 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805039/ [6805041] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:06 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805041/ [6805043] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair} time: 00:15:05 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805043/ [6805044] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{pg-warn syntax whitelist_health whitelist_wrongly_marked_down} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} time: 00:33:26 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805044/ [6805047] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:22:07 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805047/ [6805049] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:22:50 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805049/ [6805051] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/backtrace} time: 00:10:23 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805051/ [6805052] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:25:11 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805052/ [6805056] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{multimds/yes pg-warn whitelist_health whitelist_wrongly_marked_down} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:28:09 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805056/ [6805057] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/cap-flush} time: 00:10:23 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805057/ [6805060] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{no}} time: 00:25:20 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805060/ [6805062] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 01:48:14 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805062/ [6805066] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:22:16 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805066/ [6805067] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/mdtest} time: 00:14:09 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805067/ [6805069] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:51:09 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805069/ [6805078] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:57:53 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805078/ [6805082] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/client-recovery} time: 00:30:57 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805082/ [6805085] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:32 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805085/ [6805086] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:16:33 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805086/ [6805090] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag multifs session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:16:25 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805090/ [6805093] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 01:23:38 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805093/ [6805095] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/no standby-replay tasks/{0-check-counter workunit/fs/misc} wsync/{no}} time: 01:00:05 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805095/ [6805098] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag whitelist_health whitelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:20:12 info: http://pulpito.ceph.com/vshankar-2022-04-25_07:47:28-fs-wip-vshankar-testing1-20220425-093759-testing-default-smithi/6805098/