2022-10-19T17:19:13.414 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-19T17:19:13.415 INFO:teuthology.results:subset: '111/120000' 2022-10-19T17:19:13.416 INFO:teuthology.results:seed: '3526' 2022-10-19T20:54:58.161 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-19T20:54:58.195 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-10-19T23:53:14.309 INFO:teuthology.results:Tests finished! gathering results... 2022-10-19T23:53:31.718 INFO:scrape:Found 181 jobs 2022-10-19T23:54:26.422 INFO:scrape:Found 14 distinct failure reasons 2022-10-19T23:54:26.424 INFO:scrape:Failure: "1666209264.7161775 mon.a (mon.0) 148 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2022-10-19T23:54:26.425 INFO:scrape:66 jobs 2022-10-19T23:54:26.426 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2022-10-19T23:54:26.427 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/32bits/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/shell/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/top/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mon', 'mon-debug', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/ignorelist_health', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'ranks/5', 'rhel_8}', 'session_timeout}', 'supported-random-distros$/{rhel_8}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/admin}', 'tasks/alternate-pool}', 'tasks/asok_dump_tree}', 'tasks/cap-flush}', 'tasks/cephfs-shell}', 'tasks/cephfs_misc_tests}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_blogbench', 'tasks/cfuse_workunit_suites_dbench', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-readahead}', 'tasks/client-recovery}', 'tasks/client}', 'tasks/damage}', 'tasks/data-scan}', 'tasks/exports}', 'tasks/failover}', 'tasks/forward-scrub}', 'tasks/fragment}', 'tasks/fsstress', 'tasks/fstop}', 'tasks/ino_release_cb}', 'tasks/ior-shared-file}', 'tasks/journal-repair}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/libcephfs_python}', 'tasks/mds-flush}', 'tasks/mds-full}', 'tasks/mds_creation_retry}', 'tasks/mdtest}', 'tasks/mirror}', 'tasks/multifs-auth}', 'tasks/openfiletable}', 'tasks/pool-perm}', 'tasks/recovery-fs}', 'tasks/scrub}', 'tasks/sessionmap}', 'tasks/snap-schedule}', 'tasks/strays}', 'tasks/test_journal_migration}', 'tasks/truncate_delay}', 'tasks/volumes/{overrides', 'tasks/workunit/snaps}', 'test/basic}}', 'test/clone}}', 'traceless/50pc}', 'validater/lockdep}'] 2022-10-19T23:54:26.428 INFO:scrape: 2022-10-19T23:54:26.428 INFO:scrape:Failure: reached maximum tries (50) after waiting for 300 seconds 2022-10-19T23:54:26.429 INFO:scrape:2 jobs: ['7073879', '7073962'] 2022-10-19T23:54:26.429 INFO:scrape:suites intersection: ['2-logrotate}'] 2022-10-19T23:54:26.431 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'cephfs-mirror/three-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/mirror-ha/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{whitelist_health}', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'supported-random-distro$/{rhel_8}', 'tasks/{0-subvolume/{with-quota}', 'workloads/cephfs-mirror-ha-workunit}'] 2022-10-19T23:54:26.432 INFO:scrape: 2022-10-19T23:54:26.432 INFO:scrape:Failure: Command failed on smithi052 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' 2022-10-19T23:54:26.433 INFO:scrape:57 jobs 2022-10-19T23:54:26.435 INFO:scrape:suites intersection: [] 2022-10-19T23:54:26.436 INFO:scrape:suites union: ['1-ceph', '1-client', '1-client-upgrade', '1-upgrade}}', '2-client-sanity}}', '2-logrotate}', '2-upgrade', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/fs/snaps}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', '3-client-upgrade', '3-compat_client/no}}', '3-compat_client/quincy}}', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1-mds-2-client-micro', 'clusters/1-node-1-mds-1-osd', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'clusters/small-cluster', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'debug', 'distro/{centos_8}', 'fs/32bits/{begin/{0-install', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/full/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'fs/upgrade/nofs/{bluestore-bitmap', 'fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'multimds/no', 'multimds/yes', 'no-mds-cluster', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides', 'overrides/whitelist_health', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'session_timeout', 'session_timeout}', 'tasks/acls}', 'tasks/auto-repair}', 'tasks/backtrace}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/dbench', 'tasks/fsstress', 'tasks/metrics}', 'tasks/mgr-osd-full}', 'tasks/mirror}}', 'tasks/multifs-auth}', 'tasks/multimds_misc}', 'tasks/quota}', 'tasks/snap_schedule_snapdir}', 'tasks/test_journal_migration}', 'tasks/trim-i24137}', 'tasks/truncate_delay}', 'tasks/volumes/{overrides', 'tasks/workunit/dir-max-entries}', 'tasks/workunit/quota}', 'tasks/workunit/snaps}', 'tasks/{0-nautilus', 'tasks/{0-octopus', 'tasks/{0-pacific', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/basic}}', 'test/clone}}', 'test/misc}}', 'test/snapshot}}', 'thrashosds-health}', 'traceless/50pc}', 'validater/lockdep}', 'validater/valgrind}'] 2022-10-19T23:54:26.437 INFO:scrape: 2022-10-19T23:54:26.437 INFO:scrape:Crash: "1666202463.5406137 mon.a (mon.0) 137 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2022-10-19T23:54:26.438 INFO:scrape:ceph version 18.0.0-547-g5cdb2b2f (5cdb2b2f7c8342868249fd8649ecf4f5ecb19e4c) reef (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fb00d82d420] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x182) [0x7fb00cd1c2bb] 5: /usr/lib/ceph/libceph-common.so.2(+0x31c41d) [0x7fb00cd1c41d] 6: (MDSTableClient::handle_request(boost::intrusive_ptr const&)+0x11d6) [0x5610b2b8a436] 7: (MDSRank::handle_message(boost::intrusive_ptr const&)+0xd45) [0x5610b289b985] 8: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5a6) [0x5610b28b30c6] 9: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5610b28b36bc] 10: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5610b288857f] 11: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x468) [0x7fb00cf877a8] 12: (DispatchQueue::entry()+0x657) [0x7fb00cf84d47] 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fb00d048eb1] 14: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fb00d821609] 15: clone() 2022-10-19T23:54:26.439 INFO:scrape:2 jobs: ['7073826', '7073978'] 2022-10-19T23:54:26.439 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/snapshots}'] 2022-10-19T23:54:26.440 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/snapshots}'] 2022-10-19T23:54:26.440 INFO:scrape: 2022-10-19T23:54:26.441 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi032 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cdb2b2f7c8342868249fd8649ecf4f5ecb19e4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2022-10-19T23:54:26.442 INFO:scrape:1 jobs: ['7073970'] 2022-10-19T23:54:26.443 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2022-10-19T23:54:26.444 INFO:scrape: 2022-10-19T23:54:26.444 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi093 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cdb2b2f7c8342868249fd8649ecf4f5ecb19e4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' 2022-10-19T23:54:26.445 INFO:scrape:3 jobs: ['7073918', '7073810', '7073982'] 2022-10-19T23:54:26.446 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2022-10-19T23:54:26.447 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2022-10-19T23:54:26.448 INFO:scrape: 2022-10-19T23:54:26.449 INFO:scrape:Failure: Command failed on smithi164 with status 1: 'sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap' 2022-10-19T23:54:26.450 INFO:scrape:1 jobs: ['7073968'] 2022-10-19T23:54:26.451 INFO:scrape:suites: ['1-rename', 'distro/single-container-host', 'fs/cephadm/renamevolume/{0-start', 'overrides/ignorelist_health}'] 2022-10-19T23:54:26.452 INFO:scrape: 2022-10-19T23:54:26.453 INFO:scrape:Dead: b'2022-10-19T23:52:25.763 DEBUG:teuthology.exit:Finished running handlers' 2022-10-19T23:54:26.454 INFO:scrape:2 jobs: ['7073874', '7073855'] 2022-10-19T23:54:26.455 INFO:scrape:suites intersection: [] 2022-10-19T23:54:26.456 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-mount', '2-logrotate}', '2-scrub/yes', '2-workload/dbench', '3-snaps/yes', '4-flush/no', '5-workunit/suites/fsstress}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/single-container-host}', 'frag', 'fs/cephadm/multivolume/{0-start', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2022-10-19T23:54:26.457 INFO:scrape: 2022-10-19T23:54:26.458 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi006.front.sepia.ceph.com: ['type=AVC msg=audit(1666212384.724:192): avc: denied { node_bind } for pid=1412 comm="ping" saddr=172.21.15.6 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] 2022-10-19T23:54:26.458 INFO:scrape:1 jobs: ['7073977'] 2022-10-19T23:54:26.459 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/yes', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}'] 2022-10-19T23:54:26.460 INFO:scrape: 2022-10-19T23:54:26.461 INFO:scrape:Failure: Command failed (workunit test suites/pjd.sh) on smithi037 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cdb2b2f7c8342868249fd8649ecf4f5ecb19e4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' 2022-10-19T23:54:26.462 INFO:scrape:1 jobs: ['7073923'] 2022-10-19T23:54:26.463 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/3', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}'] 2022-10-19T23:54:26.466 INFO:scrape: 2022-10-19T23:54:26.467 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2022-10-19T23:54:26.468 INFO:scrape:1 jobs: ['7073940'] 2022-10-19T23:54:26.468 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2022-10-19T23:54:26.469 INFO:scrape: 2022-10-19T23:54:26.470 INFO:scrape:Failure: Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) 2022-10-19T23:54:26.471 INFO:scrape:1 jobs: ['7073876'] 2022-10-19T23:54:26.472 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client-limits}'] 2022-10-19T23:54:26.473 INFO:scrape: 2022-10-19T23:54:26.473 INFO:scrape:Failure: Test failure: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) 2022-10-19T23:54:26.475 INFO:scrape:1 jobs: ['7073828'] 2022-10-19T23:54:26.475 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/strays}'] 2022-10-19T23:54:26.476 INFO:scrape: 2022-10-19T23:54:26.477 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi191 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cdb2b2f7c8342868249fd8649ecf4f5ecb19e4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2022-10-19T23:54:26.478 INFO:scrape:1 jobs: ['7073891'] 2022-10-19T23:54:26.478 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2022-10-19T23:54:26.479 INFO:scrape: 2022-10-19T23:54:26.493 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/ failed: 138 dead: 2 running: 0 waiting: 0 queued: 0 passed: 41 Fail ================================================================= [7073809] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:17:15 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073809/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073809/ "1666201043.3608856 mon.a (mon.0) 142 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073810] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:25:00 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073810/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f26aa7a0480f47de97b04b5a04b43b0b Command failed (workunit test kernel_untar_build.sh) on smithi080 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cdb2b2f7c8342868249fd8649ecf4f5ecb19e4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7073811] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:10:16 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073811/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073811/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03e1144e92384d228c6fc2b9d1af9861 Command failed on smithi006 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073812] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} ----------------------------------------------------------------- time: 00:59:40 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073812/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073812/ "1666201367.5797546 mon.a (mon.0) 119 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073816] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:24:09 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073816/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073816/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d3a49d74b7f4d8cb633e7836d84207d Command failed on smithi063 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073817] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:47:28 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073817/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073817/ "1666201950.0799928 mon.a (mon.0) 524 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [7073819] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:06:21 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073819/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073819/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e0e72666d09c4899b080a3f0cdb47045 Command failed on smithi107 with status 1: 'sudo chmod 777 /var/log/ceph' [7073820] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} ----------------------------------------------------------------- time: 00:11:51 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073820/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073820/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7dc7ba35f1174b1b83066b39c348a44e Command failed on smithi052 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073821] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:12:22 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073821/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073821/ "1666201961.8595107 mon.a (mon.0) 125 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073822] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:26:55 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073822/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073822/ "1666201918.2457733 mon.a (mon.0) 142 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073823] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:11:36 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073823/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073823/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2a2762c43b9b4400aaf0c1019e6e67e9 Command failed on smithi112 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073824] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:11:45 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073824/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073824/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f32f2e207a0e420cadd57f92aadb6fd9 Command failed on smithi017 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073826] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:27:39 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073826/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073826/ "1666202463.5406137 mon.a (mon.0) 137 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073827] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:11:45 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073827/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073827/ sentry: https://sentry.ceph.com/organizations/ceph/?query=449a405fb09e49559b23f374e28e2f98 Command failed on smithi050 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073828] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:15:46 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073828/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073828/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6baaa2dd384841788836c1d2a79bf77f Test failure: test_hardlink_reintegration (tasks.cephfs.test_strays.TestStrays) [7073830] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:06:10 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073830/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073830/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2741314b0d7a412595574b53df3a931f Command failed on smithi120 with status 1: 'sudo chmod 777 /var/log/ceph' [7073831] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:20:32 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073831/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073831/ "1666202618.7917507 mon.a (mon.0) 136 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073833] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:09:44 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073833/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073833/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b686796a2dc24125b4a1206ef495c75a Command failed on smithi008 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073834] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:11:44 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073834/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073834/ "1666202861.5249915 mon.a (mon.0) 188 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073835] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:15:49 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073835/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073835/ "1666202783.234633 mon.a (mon.0) 196 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073837] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:15:07 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073837/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073837/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e1adc03274524ae789877f714bf2fedf Command failed on smithi052 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073840] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} ----------------------------------------------------------------- time: 00:09:58 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073840/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073840/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8c7a5c22913347b9a0fcc2cffd1b0d65 Command failed on smithi112 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073842] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:11:28 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073842/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073842/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a80173b6b103427082f4b42447eb1ac2 Command failed on smithi084 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073843] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 01:49:48 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073843/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073843/ "1666203240.0329285 mon.a (mon.0) 213 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073844] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} ----------------------------------------------------------------- time: 00:16:30 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073844/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073844/ "1666203269.530478 mon.a (mon.0) 141 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073845] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:06:12 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073845/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073845/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e1f2a4b7e834525b214a741e2c38667 Command failed on smithi120 with status 1: 'sudo chmod 777 /var/log/ceph' [7073846] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:09:56 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073846/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073846/ sentry: https://sentry.ceph.com/organizations/ceph/?query=798b66ad70924ef39fa2d661af2b5a15 Command failed on smithi061 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073847] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:21:17 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073847/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073847/ "1666203834.3620348 mon.a (mon.0) 147 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073848] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} ----------------------------------------------------------------- time: 00:15:20 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073848/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073848/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6010186e5494adb951493612c2773de Command failed on smithi063 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073849] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:13:55 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073849/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073849/ "1666203597.943417 mon.a (mon.0) 222 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073850] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:59:12 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073850/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073850/ "1666204185.7500834 mon.a (mon.0) 146 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073851] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:14:57 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073851/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073851/ sentry: https://sentry.ceph.com/organizations/ceph/?query=216f518b17534e3a95c4872ab456ea06 Command failed on smithi044 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073852] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:15:00 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073852/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073852/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4127c982e02a47e3a27a0904742c9a35 Command failed on smithi035 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073854] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 01:09:28 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073854/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073854/ "1666203826.0726478 mon.a (mon.0) 175 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073856] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:16:54 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073856/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073856/ "1666204136.3373234 mon.a (mon.0) 120 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073857] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:09:47 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073857/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073857/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9458c73c77d94334a033d958eeeff4a4 Command failed on smithi164 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073859] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:14:02 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073859/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073859/ "1666204130.558445 mon.a (mon.0) 166 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073860] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:06:11 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073860/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073860/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1186deb71c434cc8b9753f814ed50a08 Command failed on smithi087 with status 1: 'sudo chmod 777 /var/log/ceph' [7073861] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} ----------------------------------------------------------------- time: 00:26:54 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073861/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073861/ "1666204520.6536071 mon.a (mon.0) 106 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073865] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} ----------------------------------------------------------------- time: 00:09:51 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073865/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073865/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8bdfa22548444cae8d8d1464ded27348 Command failed on smithi052 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073867] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} ----------------------------------------------------------------- time: 00:11:14 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073867/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073867/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04304f96d95e4e8cbcb04333d0099526 Command failed on smithi153 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073868] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:06:04 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073868/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073868/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a3d4c096e3cd408a810ea6f91f06e77f Command failed on smithi037 with status 1: 'sudo chmod 777 /var/log/ceph' [7073870] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} ----------------------------------------------------------------- time: 00:16:08 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073870/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073870/ "1666205316.051171 mon.a (mon.0) 144 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073872] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:15:13 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073872/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073872/ "1666205215.9936159 mon.a (mon.0) 208 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073873] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:08:56 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073873/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073873/ sentry: https://sentry.ceph.com/organizations/ceph/?query=944311e31fa0496baa1278729a4ebee0 Command failed on smithi103 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073875] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:08:53 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073875/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073875/ sentry: https://sentry.ceph.com/organizations/ceph/?query=348fd8d35ab74983a96f134578f98db7 Command failed on smithi182 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073876] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:14:54 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073876/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073876/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e907726f9be443fa8a3651888364ef01 Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits) [7073877] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} ----------------------------------------------------------------- time: 00:11:28 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073877/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073877/ "1666205369.056614 mon.a (mon.0) 106 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073878] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} ----------------------------------------------------------------- time: 02:01:20 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073878/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073878/ "1666205227.4650133 mon.a (mon.0) 122 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073879] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{rhel_8} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:26:14 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073879/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073879/ reached maximum tries (50) after waiting for 300 seconds [7073880] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:42:44 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073880/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073880/ "1666205409.2819455 mon.a (mon.0) 146 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073881] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:53:17 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073881/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073881/ "1666207642.8361368 mon.a (mon.0) 1719 : cluster [WRN] Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED)" in cluster log [7073882] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:35:45 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073882/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073882/ "1666205591.4675121 mon.a (mon.0) 208 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073883] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:16:26 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073883/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073883/ "1666205707.35871 mon.a (mon.0) 141 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073884] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:43:02 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073884/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073884/ "1666205762.932481 mon.a (mon.0) 120 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073885] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:57:34 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073885/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073885/ "1666205730.1172788 mon.a (mon.0) 203 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073886] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} ----------------------------------------------------------------- time: 00:10:22 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073886/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073886/ "1666205745.7593133 mon.a (mon.0) 101 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073887] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} ----------------------------------------------------------------- time: 00:23:49 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073887/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073887/ "1666205990.1702707 mon.a (mon.0) 223 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073888] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:06:04 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073888/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073888/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c49fe16e6714a6e998cc5572fb5bb45 Command failed on smithi043 with status 1: 'sudo chmod 777 /var/log/ceph' [7073889] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:17:07 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073889/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073889/ sentry: https://sentry.ceph.com/organizations/ceph/?query=00464ec9648e40dbaf568ec70ebb9115 Command failed on smithi037 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073890] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:09:26 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073890/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073890/ sentry: https://sentry.ceph.com/organizations/ceph/?query=334762527aeb4f73857dfdf1ee13691d Command failed on smithi103 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073891] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:51:38 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073891/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073891/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b74fae4171ea48b2bee110697e6b70aa Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi191 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cdb2b2f7c8342868249fd8649ecf4f5ecb19e4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7073892] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} ----------------------------------------------------------------- time: 00:13:27 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073892/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073892/ "1666206462.8224444 mon.a (mon.0) 192 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073893] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:22:53 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073893/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073893/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ff49c8a3ac674e4eb5205be939c41f71 Command failed on smithi044 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073897] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:36:43 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073897/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073897/ "1666207081.5267897 mon.a (mon.0) 231 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073899] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:24:10 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073899/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073899/ "1666206848.8640635 mon.a (mon.0) 139 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073900] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} ----------------------------------------------------------------- time: 00:06:06 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073900/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073900/ sentry: https://sentry.ceph.com/organizations/ceph/?query=080caff88f164504a8ee5828399635b3 Command failed on smithi043 with status 1: 'sudo chmod 777 /var/log/ceph' [7073901] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:29:38 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073901/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073901/ "1666206986.1664367 mon.a (mon.0) 226 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073904] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} ----------------------------------------------------------------- time: 00:05:57 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073904/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073904/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e3a2b497a2964a44bbef77099986cd81 Command failed on smithi116 with status 1: 'sudo chmod 777 /var/log/ceph' [7073905] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:59:12 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073905/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073905/ "1666207909.1848953 mon.a (mon.0) 228 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log [7073906] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:14:03 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073906/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073906/ "1666207588.7317379 mon.a (mon.0) 110 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073907] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:16:51 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073907/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073907/ "1666207582.765103 mon.a (mon.0) 163 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073908] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:11:33 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073908/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073908/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e386cb60dc724f83854d0765486bbe04 Command failed on smithi112 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073909] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:17:49 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073909/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073909/ "1666207857.2676027 mon.a (mon.0) 135 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073910] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:06:07 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073910/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073910/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b7640f843844406687d14e5b82965bc8 Command failed on smithi063 with status 1: 'sudo chmod 777 /var/log/ceph' [7073913] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:19:37 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073913/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073913/ "1666208053.2507365 mon.a (mon.0) 225 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073914] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:11:53 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073914/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073914/ "1666208322.5871546 mon.a (mon.0) 151 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073915] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:54:28 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073915/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073915/ "1666208254.8701935 mon.a (mon.0) 107 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073916] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 01:01:20 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073916/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073916/ "1666208312.8390818 mon.a (mon.0) 138 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073917] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:12:04 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073917/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073917/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d80c4dad40c4fd09ee6b49494048873 Command failed on smithi006 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073918] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:28:10 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073918/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073918/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ea150f56a0a4a239906a678a71c5f78 Command failed (workunit test kernel_untar_build.sh) on smithi093 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cdb2b2f7c8342868249fd8649ecf4f5ecb19e4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7073919] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} ----------------------------------------------------------------- time: 01:07:36 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073919/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073919/ "1666208342.7439969 mon.a (mon.0) 208 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073920] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:15:00 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073920/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073920/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76fa2c8106d44343a335bc6103f4fa49 Command failed on smithi112 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073921] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:26:44 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073921/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073921/ "1666209264.7161775 mon.a (mon.0) 148 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073922] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:14:06 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073922/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073922/ "1666208861.8138552 mon.a (mon.0) 213 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073923] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:35:37 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073923/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073923/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b96974f7cf9543a9b34967b1d705bf49 Command failed (workunit test suites/pjd.sh) on smithi037 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cdb2b2f7c8342868249fd8649ecf4f5ecb19e4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh' [7073924] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:10:25 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073924/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073924/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b50a0df89264381a975328ac024aff1 Command failed on smithi130 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073925] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:06:09 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073925/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073925/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4cff47011805407e9a744dfe2d8cc42b Command failed on smithi120 with status 1: 'sudo chmod 777 /var/log/ceph' [7073926] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:14:34 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073926/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073926/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad0a513dae9348018965230c61e2a071 Command failed on smithi049 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073927] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:24:27 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073927/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073927/ "1666209217.6923952 mon.a (mon.0) 140 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073928] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:09:28 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073928/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073928/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d159bd05be9549a18ef0e9264f4fd1d0 Command failed on smithi111 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073929] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} ----------------------------------------------------------------- time: 00:13:42 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073929/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073929/ "1666209311.7738454 mon.a (mon.0) 219 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073930] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:16:42 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073930/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073930/ "1666209390.1216223 mon.a (mon.0) 135 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073932] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} ----------------------------------------------------------------- time: 00:19:13 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073932/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073932/ "1666210022.882882 mon.a (mon.0) 160 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073935] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:11:21 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073935/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073935/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ecd64703076443e3ae1bd1949b907897 Command failed on smithi164 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073936] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:09:30 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073936/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073936/ sentry: https://sentry.ceph.com/organizations/ceph/?query=46002c490b504ad0964cdec23d15e146 Command failed on smithi116 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073937] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:06:01 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073937/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073937/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0bb6421e46c04eef8894cae745bd8493 Command failed on smithi120 with status 1: 'sudo chmod 777 /var/log/ceph' [7073938] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} ----------------------------------------------------------------- time: 00:11:41 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073938/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073938/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c07d03c94f84d6b9398bef057388154 Command failed on smithi153 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073940] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:22:59 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073940/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073940/ error during scrub thrashing: rank damage found: {'backtrace'} [7073941] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} ----------------------------------------------------------------- time: 00:16:39 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073941/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073941/ "1666210289.7734206 mon.a (mon.0) 223 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073942] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:19:49 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073942/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073942/ "1666210504.1624255 mon.a (mon.0) 125 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073943] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:14:40 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073943/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073943/ sentry: https://sentry.ceph.com/organizations/ceph/?query=968f26f7945b4ecf93d84b3d2aa5bc16 Command failed on smithi111 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073944] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:51:18 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073944/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073944/ "1666210991.5279732 mon.a (mon.0) 144 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073945] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 02:15:49 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073945/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073945/ "1666211127.6630085 mon.a (mon.0) 241 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073947] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:14:31 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073947/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073947/ "1666210550.7475092 mon.a (mon.0) 221 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073948] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:10:03 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073948/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073948/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58658dcc94ab4f879e845f4ed812501a Command failed on smithi049 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073949] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 01:13:32 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073949/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073949/ "1666210670.5236084 mon.a (mon.0) 205 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073950] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:16:32 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073950/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073950/ "1666210698.9517539 mon.a (mon.0) 208 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073951] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} ----------------------------------------------------------------- time: 00:09:28 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073951/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073951/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e799ac1b753343be84400b6a61f64f71 Command failed on smithi084 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073952] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:06:11 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073952/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073952/ sentry: https://sentry.ceph.com/organizations/ceph/?query=23cd211a1ce04e6aae55a29edc715137 Command failed on smithi050 with status 1: 'sudo chmod 777 /var/log/ceph' [7073953] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:15:09 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073953/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073953/ sentry: https://sentry.ceph.com/organizations/ceph/?query=48c3341185654506b9459a85459499af Command failed on smithi005 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073954] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:11:27 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073954/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073954/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7eec18f8596b416f93e3f4e497eeab93 Command failed on smithi153 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073956] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} ----------------------------------------------------------------- time: 00:15:51 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073956/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073956/ "1666211544.2667916 mon.a (mon.0) 164 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073959] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:23:13 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073959/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073959/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3c04cafd1601413392726e9bd6b936ee Command failed on smithi037 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073960] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:22:05 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073960/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073960/ "1666211652.9745312 mon.a (mon.0) 170 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073962] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 02:07:47 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073962/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073962/ error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds [7073963] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} ----------------------------------------------------------------- time: 00:06:10 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073963/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073963/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4996ac043d4b4c0ea131a277299dd900 Command failed on smithi084 with status 1: 'sudo chmod 777 /var/log/ceph' [7073964] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} ----------------------------------------------------------------- time: 00:17:04 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073964/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073964/ "1666211942.0956247 mon.a (mon.0) 107 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073966] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:56:06 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073966/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073966/ "1666212554.672927 mon.a (mon.0) 242 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073967] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:16:17 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073967/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073967/ "1666212017.7339861 mon.a (mon.0) 202 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073968] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:18:54 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073968/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073968/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fbf3b2f779154cfa8c390b601b9cfdac Command failed on smithi164 with status 1: 'sudo MALLOC_CHECK_=3 adjust- ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap' [7073969] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} ----------------------------------------------------------------- time: 00:10:01 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073969/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073969/ sentry: https://sentry.ceph.com/organizations/ceph/?query=95e91f80b3de4f55a87c55d1580d9df1 Command failed on smithi112 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073970] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:13:38 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073970/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073970/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe0b9291be114350bb5f85d535bba29f Command failed (workunit test libcephfs/test.sh) on smithi032 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cdb2b2f7c8342868249fd8649ecf4f5ecb19e4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7073971] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:52:28 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073971/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073971/ "1666212506.1610057 mon.a (mon.0) 470 : cluster [WRN] Health check failed: Degraded data redundancy: 15/2050 objects degraded (0.732%), 1 pg degraded (PG_DEGRADED)" in cluster log [7073972] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:54:06 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073972/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073972/ "1666212464.3535445 mon.a (mon.0) 148 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073974] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:09:32 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073974/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073974/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4cd1907d9204d00aafb3364d757702a Command failed on smithi111 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073975] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:52:56 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073975/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073975/ "1666212857.2836442 mon.a (mon.0) 428 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073977] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:50:22 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073977/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073977/ SELinux denials found on ubuntu@smithi006.front.sepia.ceph.com: ['type=AVC msg=audit(1666212384.724:192): avc: denied { node_bind } for pid=1412 comm="ping" saddr=172.21.15.6 scontext=system_u:system_r:ping_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=1'] [7073978] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:23:34 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073978/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073978/ "1666212861.0257735 mon.a (mon.0) 170 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073979] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:05:54 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073979/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073979/ sentry: https://sentry.ceph.com/organizations/ceph/?query=96be8f034d2f4656b693e2b4d3945391 Command failed on smithi106 with status 1: 'sudo chmod 777 /var/log/ceph' [7073980] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:14:36 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073980/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073980/ sentry: https://sentry.ceph.com/organizations/ceph/?query=52748c0e4e6b4256b777ec8bcdceb643 Command failed on smithi008 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073981] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:41:07 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073981/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073981/ "1666213751.0252674 mon.a (mon.0) 201 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073982] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:32:15 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073982/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073982/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f632357852b3487cb82b29a041bc0d24 Command failed (workunit test kernel_untar_build.sh) on smithi088 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5cdb2b2f7c8342868249fd8649ecf4f5ecb19e4c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh' [7073983] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} ----------------------------------------------------------------- time: 00:18:44 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073983/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073983/ "1666213395.1768143 mon.a (mon.0) 188 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073984] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} ----------------------------------------------------------------- time: 00:06:09 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073984/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073984/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bdfe14a243ed435a9985878b88de3021 Command failed on smithi061 with status 1: 'sudo chmod 777 /var/log/ceph' [7073987] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:09:32 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073987/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073987/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0bb6ff78d967407b8f734f446eee9922 Command failed on smithi078 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' [7073988] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:14:15 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073988/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073988/ "1666213775.5517275 mon.a (mon.0) 172 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7073989] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:10:14 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073989/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073989/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1536222be22849f0a556c0c2a162ef29 Command failed on smithi043 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 -f /dev/vg_nvme/lv_1' Dead ================================================================= [7073855] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073855/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073855/ [7073874] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073874/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073874/ Pass ================================================================= [7073813] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:10 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073813/ [7073814] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:54:06 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073814/ [7073815] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 01:09:07 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073815/ [7073818] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:29:00 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073818/ [7073825] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:08:42 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073825/ [7073829] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:14:59 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073829/ [7073832] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} time: 01:19:50 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073832/ [7073836] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:23:55 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073836/ [7073838] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:29 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073838/ [7073839] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:21:29 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073839/ [7073841] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:33:27 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073841/ [7073853] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:20:37 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073853/ [7073858] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:26:03 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073858/ [7073862] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 00:51:28 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073862/ [7073863] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:38:48 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073863/ [7073864] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:35:25 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073864/ [7073866] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:24:50 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073866/ [7073869] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:59:56 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073869/ [7073871] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:39:38 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073871/ [7073894] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:20:01 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073894/ [7073895] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:13 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073895/ [7073896] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:41:11 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073896/ [7073898] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:39 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073898/ [7073902] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:15:18 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073902/ [7073903] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 01:24:20 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073903/ [7073911] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:33:30 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073911/ [7073912] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:21:50 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073912/ [7073931] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:48:57 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073931/ [7073933] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:33:59 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073933/ [7073934] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:31:51 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073934/ [7073939] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:16:50 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073939/ [7073946] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:49:45 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073946/ [7073955] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} time: 00:49:23 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073955/ [7073957] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:19:36 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073957/ [7073958] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:46 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073958/ [7073961] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:19:41 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073961/ [7073965] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:15:09 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073965/ [7073973] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:51 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073973/ [7073976] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:17:59 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073976/ [7073985] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:30 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073985/ [7073986] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} time: 01:47:21 info: http://pulpito.ceph.com/yuriw-2022-10-19_14:34:37-fs-wip-yuri11-testing-2022-10-18-1730-distro-default-smithi/7073986/