2023-08-19T09:55:35.280 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-19T09:55:35.301 INFO:teuthology.results:subset: '111/120000' 2023-08-19T09:55:35.301 INFO:teuthology.results:seed: '480' 2023-08-19T11:44:01.359 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-19T11:44:01.388 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-19T23:35:59.210 INFO:teuthology.results:Tests finished! gathering results... 2023-08-19T23:36:14.632 INFO:scrape:Found 191 jobs 2023-08-19T23:36:25.115 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373429/teuthology.log 2023-08-19T23:36:37.306 INFO:scrape:Found 19 distinct failure reasons 2023-08-19T23:36:37.307 INFO:scrape:Failure: "2023-08-19T10:51:57.680471+0000 mon.a (mon.0) 198 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-08-19T23:36:37.308 INFO:scrape:28 jobs: ['7373437', '7373551', '7373451', '7373533', '7373447', '7373532', '7373440', '7373381', '7373485', '7373557', '7373547', '7373456', '7373407', '7373428', '7373502', '7373411', '7373511', '7373375', '7373455', '7373399', '7373530', '7373550', '7373450', '7373539', '7373466', '7373555', '7373418', '7373501'] 2023-08-19T23:36:37.309 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2023-08-19T23:36:37.309 INFO:scrape:suites union: ['1-ceph', '1-tests/fscrypt-dbench}}', '2-logrotate}', '2-three-per-cluster}', 'bluestore-bitmap', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'cephfs-mirror/{1-volume-create-rm', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'clusters/1-mds-1-client', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'clusters/small-cluster', 'conf/{client', 'debug', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/32bits/{begin/{0-install', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/mirror-ha/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/whitelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'pg-warn}', 'supported-random-distro$/{ubuntu_latest}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/asok_dump_tree}', 'tasks/auto-repair}', 'tasks/cap-flush}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/failover}', 'tasks/mds-flush}', 'tasks/mirror}', 'tasks/mirror}}', 'tasks/recovery-fs}', 'tasks/scrub}', 'tasks/sessionmap}', 'tasks/snap_schedule_snapdir}', 'tasks/snapshots}', 'tasks/trim-i24137}', 'tasks/truncate_delay}', 'tasks/volumes/{overrides', 'tasks/workunit/snaps}', 'tasks/{0-client', 'test/clone}}', 'test/snapshot}}', 'traceless/50pc}', 'workloads/cephfs-mirror-ha-workunit}'] 2023-08-19T23:36:37.309 INFO:scrape: 2023-08-19T23:36:37.310 INFO:scrape:Failure: Test failure: test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) 2023-08-19T23:36:37.310 INFO:scrape:1 jobs: ['7373458'] 2023-08-19T23:36:37.310 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}'] 2023-08-19T23:36:37.310 INFO:scrape: 2023-08-19T23:36:37.311 INFO:scrape:Failure: Command failed on smithi069 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-08-19T23:36:37.311 INFO:scrape:4 jobs: ['7373395', '7373492', '7373522', '7373426'] 2023-08-19T23:36:37.311 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health'] 2023-08-19T23:36:37.312 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}'] 2023-08-19T23:36:37.312 INFO:scrape: 2023-08-19T23:36:37.312 INFO:scrape:Failure: No module named 'tasks.fs' 2023-08-19T23:36:37.313 INFO:scrape:3 jobs: ['7373552', '7373560', '7373561'] 2023-08-19T23:36:37.313 INFO:scrape:suites intersection: [] 2023-08-19T23:36:37.313 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-client', '1-rename', '2-logrotate}', '2-scrub/no', '2-upgrade', '3-compat_client/no}}', '3-snaps/no', '4-flush/yes', '5-workunit/suites/blogbench}}', 'begin/{0-install', 'centos_8', 'clusters/1-mds-2-client-micro', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'distro/single-container-host', 'frag', 'fs/cephadm/renamevolume/{0-start', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'multimds/no', 'n/3', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/ignorelist_health}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-octopus', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}'] 2023-08-19T23:36:37.314 INFO:scrape: 2023-08-19T23:36:37.314 INFO:scrape:Failure: Command failed (workunit test suites/iozone.sh) on smithi081 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=749c7706762025163d37481eb7316e2a79035344 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh' 2023-08-19T23:36:37.314 INFO:scrape:2 jobs: ['7373491', '7373376'] 2023-08-19T23:36:37.315 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{centos_latest}', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-08-19T23:36:37.315 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/iozone}}', '2-workunit/suites/iozone}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-08-19T23:36:37.315 INFO:scrape: 2023-08-19T23:36:37.316 INFO:scrape:Failure: Command failed on smithi170 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' 2023-08-19T23:36:37.316 INFO:scrape:2 jobs: ['7373563', '7373562'] 2023-08-19T23:36:37.316 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{centos_8}', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing'] 2023-08-19T23:36:37.316 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/strays}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-08-19T23:36:37.317 INFO:scrape: 2023-08-19T23:36:37.317 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds 2023-08-19T23:36:37.317 INFO:scrape:1 jobs: ['7373544'] 2023-08-19T23:36:37.318 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}'] 2023-08-19T23:36:37.318 INFO:scrape: 2023-08-19T23:36:37.318 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi100 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=749c7706762025163d37481eb7316e2a79035344 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-08-19T23:36:37.319 INFO:scrape:1 jobs: ['7373410'] 2023-08-19T23:36:37.319 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-08-19T23:36:37.319 INFO:scrape: 2023-08-19T23:36:37.319 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=749c7706762025163d37481eb7316e2a79035344 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' 2023-08-19T23:36:37.320 INFO:scrape:1 jobs: ['7373420'] 2023-08-19T23:36:37.320 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/quota}'] 2023-08-19T23:36:37.320 INFO:scrape: 2023-08-19T23:36:37.321 INFO:scrape:Dead: b'2023-08-19T22:05:38.829 DEBUG:teuthology.exit:Finished running handlers' 2023-08-19T23:36:37.321 INFO:scrape:4 jobs: ['7373396', '7373521', '7373459', '7373523'] 2023-08-19T23:36:37.321 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-08-19T23:36:37.322 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'frag', 'fs/multifs/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/multifs-auth}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}'] 2023-08-19T23:36:37.322 INFO:scrape: 2023-08-19T23:36:37.322 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on smithi061 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=749c7706762025163d37481eb7316e2a79035344 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none iozone' 2023-08-19T23:36:37.322 INFO:scrape:1 jobs: ['7373478'] 2023-08-19T23:36:37.323 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-iozone}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2023-08-19T23:36:37.323 INFO:scrape: 2023-08-19T23:36:37.323 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) 2023-08-19T23:36:37.324 INFO:scrape:1 jobs: ['7373417'] 2023-08-19T23:36:37.324 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/xfstests-dev}'] 2023-08-19T23:36:37.324 INFO:scrape: 2023-08-19T23:36:37.325 INFO:scrape:Dead: None 2023-08-19T23:36:37.325 INFO:scrape:3 jobs: ['7373429', '7373430', '7373512'] 2023-08-19T23:36:37.325 INFO:scrape: 2023-08-19T23:36:37.325 INFO:scrape:Failure: Test failure: test_volume_rm (tasks.cephfs.test_volumes.TestVolumes) 2023-08-19T23:36:37.326 INFO:scrape:1 jobs: ['7373398'] 2023-08-19T23:36:37.326 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/volumes/{overrides', 'test/basic}}'] 2023-08-19T23:36:37.326 INFO:scrape: 2023-08-19T23:36:37.327 INFO:scrape:Failure: SSH connection to smithi107 was lost: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd' 2023-08-19T23:36:37.327 INFO:scrape:1 jobs: ['7373465'] 2023-08-19T23:36:37.327 INFO:scrape:suites: ['1-client', '2-upgrade', '3-compat_client/no}}', 'centos_8', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/no', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2023-08-19T23:36:37.328 INFO:scrape: 2023-08-19T23:36:37.328 INFO:scrape:Failure: Host key for server 'smithi107.front.sepia.ceph.com' does not match: got 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC5ahyWELQHKalQUErWKB4OmW8rx8TYQd1rE0V3yrvFRHEGFgdp+yZHXz2wFgT7jfKiJeJJpNXkIyvxpeyS8KYmHU4ldj71ponbudVtXLFoYRdG9g7gw2GcntTcXLrJHWNmCHGCXLzcztnaKD1K+yonAa7z764RzUUqVWUPJ8N/HZu680Il/w+W67uJ0zUMv2GHqa2G0Yfmh2DDEyx/AkXotZQ5kZ5bpm4JiEOZ6wEa6UsEHtT/p42la7QkOY1C5a9Y6yBSRurlE5+eiWxVFJPQpFkrJo8rdJeIfXPsCewNlr/21tMpXDDK+6b3INXhpcLRaNNCRsp/HS30NPNxtsmoLnyvRkv8LOZlXHx3nR+Hb8YKwXI6QTHNzHdlmJ4nR8t5tkfH6ntXPre6feN39P8Uf3JoJXa1kaH/Npu0RVD7C7xCljD9xXt52kIi3pkH4mv/2XWJyo+mSMkbNM2q5xEki/uzNmIK+eCu+unABwTOILF2kACPdqfYc4w6ND3dYdM=' 2023-08-19T23:36:37.328 INFO:scrape:1 jobs: ['7373516'] 2023-08-19T23:36:37.328 INFO:scrape:suites: ['1-client', '2-upgrade', '3-compat_client/no}}', 'centos_8', 'clusters/1-mds-2-client-micro', 'conf/{client', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multimds/yes', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-octopus'] 2023-08-19T23:36:37.329 INFO:scrape: 2023-08-19T23:36:37.329 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-08-19T23:36:37.329 INFO:scrape:1 jobs: ['7373529'] 2023-08-19T23:36:37.330 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/quota}'] 2023-08-19T23:36:37.330 INFO:scrape: 2023-08-19T23:36:37.330 INFO:scrape:Failure: SSH connection to smithi107 was lost: 'sudo yum install -y kernel' 2023-08-19T23:36:37.330 INFO:scrape:1 jobs: ['7373498'] 2023-08-19T23:36:37.331 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_latest}', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{frag', 'tasks/cfuse_workunit_suites_dbench', 'traceless/50pc}'] 2023-08-19T23:36:37.331 INFO:scrape: 2023-08-19T23:36:37.331 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2023-08-19T23:36:37.332 INFO:scrape:1 jobs: ['7373394'] 2023-08-19T23:36:37.332 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-common}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client'] 2023-08-19T23:36:37.332 INFO:scrape: 2023-08-19T23:36:37.343 INFO:teuthology.results:Sending results to lusov@redhat.com: Test Run: leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/ failed: 50 dead: 8 running: 0 waiting: 0 queued: 0 passed: 133 Fail ================================================================= [7373375] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} ----------------------------------------------------------------- time: 00:15:31 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373375/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373375/ "2023-08-19T10:18:56.256201+0000 mon.a (mon.0) 139 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373376] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:22:50 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373376/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373376/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4340f81462249c880f02e36b0206253 Command failed (workunit test suites/iozone.sh) on smithi183 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=749c7706762025163d37481eb7316e2a79035344 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/iozone.sh' [7373381] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} ----------------------------------------------------------------- time: 00:21:33 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373381/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373381/ "2023-08-19T10:22:54.875765+0000 mon.a (mon.0) 118 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373394] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-common}} ----------------------------------------------------------------- time: 00:20:26 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373394/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373394/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0f00c789a22b4264ade17dd96cc16bf0 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7373395] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:22:54 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373395/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373395/ sentry: https://sentry.ceph.com/organizations/ceph/?query=614dcb8f039447ff9c238b722b78ef63 Command failed on smithi069 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7373398] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:36:03 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373398/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373398/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2eca000afc9a4d568b339e3ed9a39110 Test failure: test_volume_rm (tasks.cephfs.test_volumes.TestVolumes) [7373399] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:32:39 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373399/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373399/ "2023-08-19T10:25:45.414574+0000 mon.a (mon.0) 225 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373407] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:19:09 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373407/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373407/ "2023-08-19T10:29:11.909310+0000 mon.a (mon.0) 121 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373410] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:18:38 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373410/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373410/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3107b6c7a9b64915be0f2ce9ca3a5b17 Command failed (workunit test fs/test_python.sh) on smithi100 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=749c7706762025163d37481eb7316e2a79035344 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7373411] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:23:40 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373411/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373411/ "2023-08-19T10:26:26.005121+0000 mon.a (mon.0) 138 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373417] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 08:18:54 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373417/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373417/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a1e18f2f512d488ca685af4cd475183b Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev) [7373418] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:59:02 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373418/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373418/ "2023-08-19T10:23:03.344024+0000 mon.a (mon.0) 107 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373420] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:21:23 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373420/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373420/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ca3d26abd284208baf156b8bbe03f39 Command failed (workunit test fs/quota/quota.sh) on smithi133 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=749c7706762025163d37481eb7316e2a79035344 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh' [7373426] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:22:13 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373426/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373426/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ac1ea398743f4eb0bce0b89a946491d1 Command failed on smithi150 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7373428] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 01:09:33 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373428/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373428/ "2023-08-19T10:24:52.841649+0000 mon.a (mon.0) 133 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373437] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} ----------------------------------------------------------------- time: 00:25:27 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373437/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373437/ "2023-08-19T10:51:57.680471+0000 mon.a (mon.0) 198 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373440] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} ----------------------------------------------------------------- time: 00:15:13 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373440/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373440/ "2023-08-19T10:49:54.371594+0000 mon.a (mon.0) 122 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373447] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} ----------------------------------------------------------------- time: 00:15:18 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373447/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373447/ "2023-08-19T10:57:16.854102+0000 mon.a (mon.0) 140 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373450] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:15:01 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373450/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373450/ "2023-08-19T10:52:42.162850+0000 mon.a (mon.0) 217 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373451] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:12:58 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373451/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373451/ "2023-08-19T10:52:56.242914+0000 mon.a (mon.0) 96 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373455] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror} ----------------------------------------------------------------- time: 02:05:45 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373455/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373455/ "2023-08-19T10:54:53.838519+0000 mon.a (mon.0) 96 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373456] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:24:16 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373456/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373456/ "2023-08-19T10:54:51.869516+0000 mon.a (mon.0) 96 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373458] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:18:08 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373458/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373458/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bf6b5d137e6a437097d7278c51572bf1 Test failure: test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) [7373465] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:22:15 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373465/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4b7c4587bfbe42159e1ebbd71e24e802 SSH connection to smithi107 was lost: 'sudo yum -y upgrade ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph- mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd- mirror rbd-nbd' [7373466] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:05:48 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373466/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373466/ "2023-08-19T11:01:32.535277+0000 mon.a (mon.0) 97 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373478] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-iozone}} ----------------------------------------------------------------- time: 00:17:20 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373478/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373478/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca5275da3cf1483ba6fae5ed20d866d4 Command failed (workunit test fs/fscrypt.sh) on smithi061 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=749c7706762025163d37481eb7316e2a79035344 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none iozone' [7373485] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:17:37 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373485/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373485/ "2023-08-19T11:12:39.830416+0000 mon.a (mon.0) 172 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373491] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:16:22 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373491/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373491/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fcd2bb6087ab4dcbb6767183849b28a5 Command failed (workunit test suites/iozone.sh) on smithi081 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=749c7706762025163d37481eb7316e2a79035344 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh' [7373492] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:15:23 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373492/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373492/ sentry: https://sentry.ceph.com/organizations/ceph/?query=84b3949b59ed4bbbb8f4cb47496a5f23 Command failed on smithi158 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7373501] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:32:14 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373501/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373501/ "2023-08-19T11:22:52.296872+0000 mon.a (mon.0) 113 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373502] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:14:06 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373502/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373502/ "2023-08-19T11:25:32.664624+0000 mon.a (mon.0) 189 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373511] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:16:01 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373511/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373511/ "2023-08-19T11:30:33.216813+0000 mon.a (mon.0) 218 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373516] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373516/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373516/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a817e50db29a45799afaf903cded4b67 Host key for server 'smithi107.front.sepia.ceph.com' does not match: got 'A AAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiU gmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+ 7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs +R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10 DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr /MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8o Z822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJU m9HOQxUlEAvcqa7Ac=', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC5ahyWELQHKalQ UErWKB4OmW8rx8TYQd1rE0V3yrvFRHEGFgdp+yZHXz2wFgT7jfKiJeJJpNXkIyvxpeyS8KYmHU4 ldj71ponbudVtXLFoYRdG9g7gw2GcntTcXLrJHWNmCHGCXLzcztnaKD1K+yonAa7z764RzUUqVW UPJ8N/HZu680Il/w+W67uJ0zUMv2GHqa2G0Yfmh2DDEyx/AkXotZQ5kZ5bpm4JiEOZ6wEa6UsEH tT/p42la7QkOY1C5a9Y6yBSRurlE5+eiWxVFJPQpFkrJo8rdJeIfXPsCewNlr/21tMpXDDK+6b3 INXhpcLRaNNCRsp/HS30NPNxtsmoLnyvRkv8LOZlXHx3nR+Hb8YKwXI6QTHNzHdlmJ4nR8t5tkf H6ntXPre6feN39P8Uf3JoJXa1kaH/Npu0RVD7C7xCljD9xXt52kIi3pkH4mv/2XWJyo+mSMkbNM 2q5xEki/uzNmIK+eCu+unABwTOILF2kACPdqfYc4w6ND3dYdM=' [7373522] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:14:51 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373522/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373522/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3736927b12c417b9b8b15ce4634f801 Command failed on smithi169 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7373529] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:14:25 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373529/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373529/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8d329de26d1d4925ae2abe89ae3b0ffa Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7373530] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:17:16 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373530/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373530/ "2023-08-19T11:42:59.534508+0000 mon.a (mon.0) 144 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373532] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:17:06 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373532/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373532/ "2023-08-19T11:44:51.821929+0000 mon.a (mon.0) 148 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373533] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} ----------------------------------------------------------------- time: 00:47:51 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373533/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373533/ "2023-08-19T11:46:47.151901+0000 mon.a (mon.0) 213 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373539] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:31:18 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373539/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373539/ "2023-08-19T11:48:40.820092+0000 mon.a (mon.0) 122 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373544] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} ----------------------------------------------------------------- time: 01:14:10 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373544/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373544/ error during scrub thrashing: reached maximum tries (31) after waiting for 900 seconds [7373547] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}} ----------------------------------------------------------------- time: 01:17:43 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373547/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373547/ "2023-08-19T11:58:22.122017+0000 mon.a (mon.0) 226 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373550] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} ----------------------------------------------------------------- time: 00:14:11 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373550/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373550/ "2023-08-19T11:57:12.349740+0000 mon.a (mon.0) 222 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373551] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:17:24 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373551/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373551/ "2023-08-19T11:55:49.753898+0000 mon.a (mon.0) 109 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373552] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:20:07 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373552/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373552/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d17f5bcdcef4be9be5b40f37633890d No module named 'tasks.fs' [7373555] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 01:11:15 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373555/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373555/ "2023-08-19T11:56:53.060322+0000 mon.a (mon.0) 169 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373557] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 01:38:06 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373557/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373557/ "2023-08-19T12:00:36.837933+0000 mon.a (mon.0) 108 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7373560] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:23:59 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373560/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373560/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5ee43d8cfbcb4c81aefdef0b8b7b9bb8 No module named 'tasks.fs' [7373561] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:13:38 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373561/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373561/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df65f4b57c78493c9ab7ff727d300a08 No module named 'tasks.ceph_fuse' [7373562] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:10:46 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373562/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373562/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e592c728ccb947b195427e01a6f36878 Command failed on smithi148 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs- mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib' [7373563] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:10:39 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373563/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373563/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c412369022544918afe6309d966e188a Command failed on smithi170 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse python3-cephfs cephfs-top cephfs- mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib' Dead ================================================================= [7373396] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373396/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373396/ hit max job timeout [7373429] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373429/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373429/ Error reimaging machines: Expected smithi107's OS to be ubuntu 20.04 but found centos 8 [7373430] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373430/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373430/ Error reimaging machines: 500 Server Error: Internal Server Error for url: http://fog.front.sepia.ceph.com/fog/host/212/task [7373459] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373459/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373459/ hit max job timeout [7373498] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373498/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373498/ sentry: https://sentry.ceph.com/organizations/ceph/?query=063effe1b04e43038a444f35ef0eec08 SSH connection to smithi107 was lost: 'sudo yum install -y kernel' [7373512] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373512/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373512/ Error reimaging machines: Expected smithi107's OS to be ubuntu 22.04 but found centos 8 [7373521] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373521/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373521/ hit max job timeout [7373523] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373523/ log: http://qa-proxy.ceph.com/teuthology/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373523/ hit max job timeout Pass ================================================================= [7373373] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 00:45:11 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373373/ [7373374] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:16:18 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373374/ [7373377] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/postgres}} time: 00:55:53 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373377/ [7373378] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:31:05 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373378/ [7373379] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:32 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373379/ [7373380] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:59:25 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373380/ [7373382] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:22:38 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373382/ [7373383] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:22:44 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373383/ [7373384] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 01:49:09 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373384/ [7373385] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:39:23 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373385/ [7373386] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:22:43 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373386/ [7373387] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:44:44 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373387/ [7373388] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:35 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373388/ [7373389] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:48:04 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373389/ [7373390] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 00:53:42 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373390/ [7373391] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} time: 01:01:04 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373391/ [7373392] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:19:53 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373392/ [7373393] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:22:35 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373393/ [7373397] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:39:40 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373397/ [7373400] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} time: 01:09:23 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373400/ [7373401] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:23:17 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373401/ [7373402] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:29:17 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373402/ [7373403] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:37:16 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373403/ [7373404] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:25:44 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373404/ [7373405] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:19:00 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373405/ [7373406] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:29:13 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373406/ [7373408] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} time: 01:00:25 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373408/ [7373409] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:19:37 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373409/ [7373412] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:28:12 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373412/ [7373413] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:34:49 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373413/ [7373414] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:20:45 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373414/ [7373415] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:33:02 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373415/ [7373416] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:22:48 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373416/ [7373419] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} time: 00:34:17 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373419/ [7373421] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:47:33 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373421/ [7373422] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:23:35 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373422/ [7373423] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:20:47 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373423/ [7373424] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} time: 00:25:25 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373424/ [7373425] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:22:48 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373425/ [7373427] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:26:20 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373427/ [7373431] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:46:56 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373431/ [7373432] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:26:54 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373432/ [7373433] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:22:20 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373433/ [7373434] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:16:23 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373434/ [7373435] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:24:16 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373435/ [7373436] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:14:34 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373436/ [7373438] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} time: 00:49:45 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373438/ [7373439] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:29:48 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373439/ [7373441] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:14:07 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373441/ [7373442] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:30:00 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373442/ [7373443] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:22:02 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373443/ [7373444] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:14:46 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373444/ [7373445] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:25:20 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373445/ [7373446] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:39:43 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373446/ [7373448] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:36:52 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373448/ [7373449] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:24:51 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373449/ [7373452] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 00:46:55 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373452/ [7373453] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:22:29 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373453/ [7373454] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:14:18 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373454/ [7373457] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:43:54 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373457/ [7373460] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:11 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373460/ [7373461] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:45:02 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373461/ [7373462] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:51:59 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373462/ [7373463] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_20.04} tasks/fstop} time: 00:14:12 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373463/ [7373464] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:23:52 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373464/ [7373467] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:43:07 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373467/ [7373468] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:14:11 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373468/ [7373469] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:14 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373469/ [7373470] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:54:45 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373470/ [7373471] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:17:25 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373471/ [7373472] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:34:54 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373472/ [7373473] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:59:10 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373473/ [7373474] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:13:37 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373474/ [7373475] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:26:18 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373475/ [7373476] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:17:05 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373476/ [7373477] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:14:13 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373477/ [7373479] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:28:25 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373479/ [7373480] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:47:24 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373480/ [7373481] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:16:13 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373481/ [7373482] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:45:37 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373482/ [7373483] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:15 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373483/ [7373484] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 01:01:21 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373484/ [7373486] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:01 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373486/ [7373487] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} time: 00:19:58 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373487/ [7373488] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:17:14 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373488/ [7373489] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:26:51 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373489/ [7373490] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:22:03 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373490/ [7373493] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 01:15:43 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373493/ [7373494] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:50:15 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373494/ [7373495] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:19:37 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373495/ [7373496] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 00:23:45 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373496/ [7373497] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:39:11 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373497/ [7373499] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 00:18:27 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373499/ [7373500] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:21:36 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373500/ [7373503] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} time: 00:35:43 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373503/ [7373504] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:31:19 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373504/ [7373505] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:18:42 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373505/ [7373506] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:20:04 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373506/ [7373507] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:25:37 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373507/ [7373508] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:18:29 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373508/ [7373509] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:15:23 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373509/ [7373510] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:15:14 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373510/ [7373513] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:33:03 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373513/ [7373514] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:25:50 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373514/ [7373515] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 00:53:13 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373515/ [7373517] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:21:13 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373517/ [7373518] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:18:57 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373518/ [7373519] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:18:37 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373519/ [7373520] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 01:15:28 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373520/ [7373524] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:39:58 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373524/ [7373525] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:53:57 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373525/ [7373526] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:14:17 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373526/ [7373527] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:38:38 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373527/ [7373528] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:22:26 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373528/ [7373531] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:43:51 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373531/ [7373534] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:23:20 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373534/ [7373535] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:17:15 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373535/ [7373536] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} time: 00:37:49 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373536/ [7373537] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:32:31 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373537/ [7373538] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:15:45 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373538/ [7373540] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} time: 00:50:56 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373540/ [7373541] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:17:34 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373541/ [7373542] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:26:22 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373542/ [7373543] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:18:43 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373543/ [7373545] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 01:52:49 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373545/ [7373546] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} time: 00:52:24 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373546/ [7373548] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:09:23 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373548/ [7373549] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:16:09 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373549/ [7373553] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 00:24:51 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373553/ [7373554] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:01:07 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373554/ [7373556] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:56 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373556/ [7373558] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:47 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373558/ [7373559] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:28:11 info: http://pulpito.ceph.com/leonidus-2023-08-19_09:54:24-fs-wip-lusov-cap-throttle-event-distro-default-smithi/7373559/