2023-04-21T14:43:45.815 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-21T14:43:45.816 INFO:teuthology.results:subset: '111/12000' 2023-04-21T14:43:45.816 INFO:teuthology.results:seed: '8354' 2023-04-21T21:54:19.540 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-21T21:54:19.593 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-22T08:30:14.343 INFO:teuthology.results:Tests finished! gathering results... 2023-04-22T08:30:31.484 INFO:scrape:Found 195 jobs 2023-04-22T08:30:48.778 WARNING:scrape:Misunderstood line: 2023-04-22T08:30:57.462 INFO:scrape:Found 19 distinct failure reasons 2023-04-22T08:30:57.463 INFO:scrape:Failure: Test failure: test_mount_mon_and_osd_caps_present_mds_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) 2023-04-22T08:30:57.464 INFO:scrape:3 jobs: ['7247834', '7247896', '7247962'] 2023-04-22T08:30:57.464 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'osd}', 'overrides/{ignorelist_health', 'tasks/multifs-auth}'] 2023-04-22T08:30:57.464 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/multifs-auth}'] 2023-04-22T08:30:57.465 INFO:scrape: 2023-04-22T08:30:57.465 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' 2023-04-22T08:30:57.465 INFO:scrape:1 jobs: ['7247957'] 2023-04-22T08:30:57.465 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}'] 2023-04-22T08:30:57.466 INFO:scrape: 2023-04-22T08:30:57.466 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi163 with status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2023-04-22T08:30:57.466 INFO:scrape:1 jobs: ['7247894'] 2023-04-22T08:30:57.467 INFO:scrape:suites: ['1-mount/mount/fuse', '1-tests/kernel_untar_build}}}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/pacific'] 2023-04-22T08:30:57.467 INFO:scrape: 2023-04-22T08:30:57.467 INFO:scrape:Failure: more than one file system available 2023-04-22T08:30:57.468 INFO:scrape:16 jobs: ['7247958', '7248004', '7247983', '7247912', '7247991', '7247845', '7247862', '7247921', '7247810', '7247820', '7247968', '7247934', '7247888', '7247878', '7247854', '7247945'] 2023-04-22T08:30:57.468 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'thrashosds-health}'] 2023-04-22T08:30:57.468 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/iozone}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-04-22T08:30:57.469 INFO:scrape: 2023-04-22T08:30:57.469 INFO:scrape:Failure: Command failed on smithi178 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' 2023-04-22T08:30:57.469 INFO:scrape:6 jobs: ['7247992', '7247927', '7247961', '7247865', '7247895', '7247889'] 2023-04-22T08:30:57.470 INFO:scrape:suites intersection: [] 2023-04-22T08:30:57.470 INFO:scrape:suites union: ['1-ceph', '1-client', '1-rename', '2-logrotate}', '2-upgrade', '3-compat_client/no}}', 'centos_latest', 'clusters/1-mds-2-client', 'clusters/1-mds-2-client-micro', 'clusters/1-mds-3-client', 'conf/{client', 'distro/single-container-host', 'distros/ubuntu_latest', 'fs/cephadm/renamevolume/{0-start', 'fs/multiclient/{begin/{0-install', 'fs/upgrade/featureful_client/old_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'multimds/yes', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/ignorelist_health}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/ior-shared-file}', 'tasks/mdtest}', 'tasks/{0-octopus'] 2023-04-22T08:30:57.470 INFO:scrape: 2023-04-22T08:30:57.471 INFO:scrape:Failure: Command failed on smithi057 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ae71e0d6-e099-11ed-9b00-001a4aab830c -- ceph orch daemon add osd smithi057:vg_nvme/lv_4' 2023-04-22T08:30:57.471 INFO:scrape:1 jobs: ['7247985'] 2023-04-22T08:30:57.471 INFO:scrape:suites: ['1-ranks/1', '1-volume/{0-create', '2-allow_standby_replay/no', '2-client', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/v16.2.4'] 2023-04-22T08:30:57.472 INFO:scrape: 2023-04-22T08:30:57.472 INFO:scrape:Dead: b'2023-04-22T08:20:53.487 DEBUG:teuthology.exit:Finished running handlers' 2023-04-22T08:30:57.472 INFO:scrape:3 jobs: ['7247953', '7247819', '7247827'] 2023-04-22T08:30:57.472 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-04-22T08:30:57.473 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'frag', 'fs/fscrypt/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/v1', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/fscrypt-common}', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}'] 2023-04-22T08:30:57.473 INFO:scrape: 2023-04-22T08:30:57.473 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2023-04-22T08:30:57.474 INFO:scrape:5 jobs: ['7247852', '7247873', '7247892', '7247981', '7248002'] 2023-04-22T08:30:57.474 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'rhel_8}', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2023-04-22T08:30:57.474 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/fs/norstats}}', '5-workunit/kernel_untar_build}}', '5-workunit/suites/blogbench}}', '5-workunit/suites/dbench}}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/secure', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2023-04-22T08:30:57.475 INFO:scrape: 2023-04-22T08:30:57.475 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2023-04-22T08:30:57.475 INFO:scrape:1 jobs: ['7247904'] 2023-04-22T08:30:57.475 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/quota}'] 2023-04-22T08:30:57.476 INFO:scrape: 2023-04-22T08:30:57.476 INFO:scrape:Failure: {'smithi186.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/144bb6d03f4cceafda5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb-comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} 2023-04-22T08:30:57.476 INFO:scrape:4 jobs: ['7247942', '7247965', '7247901', '7247899'] 2023-04-22T08:30:57.477 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}'] 2023-04-22T08:30:57.477 INFO:scrape:suites union: ['1-ceph', '1-client', '2-logrotate}', '2-upgrade', '2-workunit/cfuse_workunit_snaptests}}', '3-client-upgrade', '4-compat_client', '5-client-sanity}}', 'centos_latest', 'clusters/1-mds-2-client-micro', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'multifs', 'multimds/yes', 'no_client_pidfile}', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'overrides/{ignorelist_health', 'pg-warn}', 'session_timeout', 'tasks/admin}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}', 'tasks/{0-octopus', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2023-04-22T08:30:57.477 INFO:scrape: 2023-04-22T08:30:57.478 INFO:scrape:Failure: Command failed (workunit test suites/blogbench.sh) on smithi099 with status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh' 2023-04-22T08:30:57.478 INFO:scrape:1 jobs: ['7247973'] 2023-04-22T08:30:57.478 INFO:scrape:suites: ['1-mount/mount/fuse', '1-tests/blogbench}}}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/nautilus'] 2023-04-22T08:30:57.478 INFO:scrape: 2023-04-22T08:30:57.479 INFO:scrape:Failure: Command failed (workunit test suites/fsstress.sh) on smithi149 with status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' 2023-04-22T08:30:57.479 INFO:scrape:1 jobs: ['7247868'] 2023-04-22T08:30:57.479 INFO:scrape:suites: ['1-mount/mount/fuse', '1-tests/fsstress}}}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/nautilus'] 2023-04-22T08:30:57.480 INFO:scrape: 2023-04-22T08:30:57.480 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi174 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2023-04-22T08:30:57.480 INFO:scrape:1 jobs: ['7247920'] 2023-04-22T08:30:57.480 INFO:scrape:suites: ['1-mount/mount/fuse', '1-tests/dbench}}}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/nautilus'] 2023-04-22T08:30:57.481 INFO:scrape: 2023-04-22T08:30:57.481 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' 2023-04-22T08:30:57.481 INFO:scrape:1 jobs: ['7247976'] 2023-04-22T08:30:57.482 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}'] 2023-04-22T08:30:57.482 INFO:scrape: 2023-04-22T08:30:57.482 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/iozone.sh 2023-04-22T08:30:57.483 INFO:scrape:1 jobs: ['7247951'] 2023-04-22T08:30:57.483 INFO:scrape:suites: ['1-mount/mount/fuse', '1-tests/iozone}}}', '2-workload/stress_tests/{0-client-upgrade', 'centos_latest', 'clusters/{1-mds-1-client-micro}', 'conf/{client', 'fs/upgrade/upgraded_client/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-from/pacific'] 2023-04-22T08:30:57.483 INFO:scrape: 2023-04-22T08:30:57.483 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2023-04-22T08:30:57.484 INFO:scrape:1 jobs: ['7247902'] 2023-04-22T08:30:57.484 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}'] 2023-04-22T08:30:57.484 INFO:scrape: 2023-04-22T08:30:57.485 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2023-04-22T08:30:57.485 INFO:scrape:1 jobs: ['7247841'] 2023-04-22T08:30:57.485 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}'] 2023-04-22T08:30:57.485 INFO:scrape: 2023-04-22T08:30:57.486 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi158 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-04-22T08:30:57.486 INFO:scrape:1 jobs: ['7247997'] 2023-04-22T08:30:57.486 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2023-04-22T08:30:57.487 INFO:scrape: 2023-04-22T08:30:57.487 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh' 2023-04-22T08:30:57.487 INFO:scrape:1 jobs: ['7247932'] 2023-04-22T08:30:57.487 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/snaps}'] 2023-04-22T08:30:57.488 INFO:scrape: 2023-04-22T08:30:57.498 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/ failed: 43 dead: 7 running: 0 waiting: 0 queued: 0 passed: 145 Fail ================================================================= [7247810] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:15:29 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247810/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247810/ sentry: https://sentry.ceph.com/organizations/ceph/?query=820adcbf71034c6796b8bfbd296111bb more than one file system available [7247820] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:14:28 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247820/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247820/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ad957e59b9854221bbbe07914f91e8cd more than one file system available [7247834] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 01:11:22 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247834/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247834/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e2e0f2801cc4d28b628877f9d8bb7be Test failure: test_mount_mon_and_osd_caps_present_mds_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) [7247841] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:18:43 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247841/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247841/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae2c54b67e6543c385af8a1e348f7cc2 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7247845] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:15:53 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247845/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247845/ sentry: https://sentry.ceph.com/organizations/ceph/?query=abec6d3e4e6141ff9439cecf9f7ae497 more than one file system available [7247852] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:15:35 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247852/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247852/ error during scrub thrashing: rank damage found: {'backtrace'} [7247854] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:15:24 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247854/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247854/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c3f39c8d675d46a0ac8f9ab5a99cf22b more than one file system available [7247862] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:17:01 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247862/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247862/ sentry: https://sentry.ceph.com/organizations/ceph/?query=56f826c29b444bc8b0b98957351bd347 more than one file system available [7247865] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:15:01 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247865/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247865/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75383d166246452e949ceb935b80d7c7 Command failed on smithi182 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7247868] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} ----------------------------------------------------------------- time: 01:43:46 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247868/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247868/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b0ac33ccdce43f78e700c3c261f7755 Command failed (workunit test suites/fsstress.sh) on smithi149 with status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh' [7247873] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:30:18 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247873/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247873/ error during scrub thrashing: rank damage found: {'backtrace'} [7247878] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 01:09:11 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247878/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247878/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d605ee0a7b2e498aa28a9e900aaa091a more than one file system available [7247888] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:13:11 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247888/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247888/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4db2370d481f47df95eb998cff51403a more than one file system available [7247889] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247889/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247889/ sentry: https://sentry.ceph.com/organizations/ceph/?query=092e7dac478e423a8ff4f48f72f1c4c4 Command failed on smithi057 with status 1: 'sudo yum install -y kernel' [7247892] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:32:54 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247892/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247892/ error during scrub thrashing: rank damage found: {'backtrace'} [7247894] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} ----------------------------------------------------------------- time: 01:40:39 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247894/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247894/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2109b212499454998021ff1a4eaebd6 Command failed (workunit test kernel_untar_build.sh) on smithi163 with status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' [7247895] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:14:24 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247895/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247895/ sentry: https://sentry.ceph.com/organizations/ceph/?query=112472c79dab4b7682bc4e632a803ba1 Command failed on smithi137 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7247896] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:15:22 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247896/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247896/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a2b222eb9ce84f6eaf167d2b15fc9711 Test failure: test_mount_mon_and_osd_caps_present_mds_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) [7247902] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 01:57:08 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247902/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247902/ sentry: https://sentry.ceph.com/organizations/ceph/?query=845016a854c94f96b9a1aa32d9bce8fb saw valgrind issues [7247904] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:13:06 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247904/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247904/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3adfef1df24d46e69aeec676376cf032 Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) [7247912] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:16:12 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247912/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247912/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50583a73c3ed40e49a8f5c268e7723cd more than one file system available [7247920] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} ----------------------------------------------------------------- time: 01:43:42 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247920/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247920/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5294978ff08f4b3fb1289302de475a94 Command failed (workunit test suites/dbench.sh) on smithi174 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7247921] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:16:13 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247921/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247921/ sentry: https://sentry.ceph.com/organizations/ceph/?query=83dd9bd37b6741918fa25cc7da597d24 more than one file system available [7247927] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247927/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247927/ sentry: https://sentry.ceph.com/organizations/ceph/?query=594ffe2d934840f5b70de0129357be76 Command failed on smithi019 with status 1: 'sudo yum install -y kernel' [7247932] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:44:31 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247932/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247932/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cd5b1679f00c4619b24e7b9419b74a8c Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi117 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest- multiple-capsnaps.sh' [7247934] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:16:55 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247934/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247934/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b427db10aac94af8a012411a193fd2e6 more than one file system available [7247945] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:14:45 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247945/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247945/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f9a6dcada9640828047d0d41f66cc52 more than one file system available [7247951] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} ----------------------------------------------------------------- time: 04:26:59 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247951/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247951/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ce4242e4dbe4e4c9876f63c741aa803 Command failed (workunit test suites/iozone.sh) on smithi049 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iozone.sh' [7247957] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:20:00 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247957/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247957/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2f110658e0241d484ec70655ad84ae0 Command failed (workunit test fs/quota/quota.sh) on smithi183 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh' [7247958] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:20:12 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247958/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247958/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76c89cbe19aa4d54a4bcbddfa2664c2d more than one file system available [7247961] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:14:39 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247961/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247961/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73f828284ce6436294257ec1d0c44166 Command failed on smithi116 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7247962] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:14:26 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247962/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247962/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec147d97832a4fa5aedd1a30d20ff8df Test failure: test_mount_mon_and_osd_caps_present_mds_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) [7247968] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:15:41 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247968/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247968/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c351744408e440ad80c3fcf8281118cf more than one file system available [7247973] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} ----------------------------------------------------------------- time: 01:48:02 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247973/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247973/ sentry: https://sentry.ceph.com/organizations/ceph/?query=803f64c479de4c6f9d68b3a5d8973d50 Command failed (workunit test suites/blogbench.sh) on smithi099 with status 3: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh' [7247976] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:18:12 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247976/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247976/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d957460b3f0348e2938fb0bca68da882 Command failed (workunit test fs/test_python.sh) on smithi023 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh' [7247981] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:44:34 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247981/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247981/ error during scrub thrashing: rank damage found: {'backtrace'} [7247983] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:16:45 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247983/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247983/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b88642374bf5407fb9f25d8c9a5d91e8 more than one file system available [7247985] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:36:39 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247985/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247985/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75187ccd79654d39a90f2248193b9606 Command failed on smithi057 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ae71e0d6-e099-11ed-9b00-001a4aab830c -- ceph orch daemon add osd smithi057:vg_nvme/lv_4' [7247991] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 01:09:14 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247991/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247991/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22357af3a2724c71bee511a2ce6a6248 more than one file system available [7247992] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:14:20 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247992/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247992/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e7857d7993344d79c8606dbe1773c11 Command failed on smithi178 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s' [7247997] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:22:43 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247997/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247997/ sentry: https://sentry.ceph.com/organizations/ceph/?query=dde6801e766147d5a21e1c791b7cdfe1 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi158 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=62b793153702d1553984fbbc66f09b2a8f896145 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7248002] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} ----------------------------------------------------------------- time: 00:37:43 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7248002/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7248002/ error during scrub thrashing: rank damage found: {'backtrace'} [7248004] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:16:09 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7248004/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7248004/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b1f48dfbe0724d36b2c055fdbfcf7b08 more than one file system available Dead ================================================================= [7247819] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/postgres}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247819/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247819/ hit max job timeout [7247827] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247827/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247827/ hit max job timeout [7247899] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:38:50 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247899/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247899/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d725bb184ef242e8926b3fa65674ea22 {'smithi090.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24 680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} [7247901] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} ----------------------------------------------------------------- time: 00:36:18 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247901/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247901/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d0a42b6aeff343adbb0374ffb14c8f6b {'smithi130.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/144bb6d03f4cceaf da5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb- comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} [7247942] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:41:02 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247942/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247942/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4fabb42efa1240beb36a895996c57e87 {'smithi186.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24 680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/144bb6d0 3f4cceafda5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb- comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} [7247953] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247953/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247953/ hit max job timeout [7247965] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:38:48 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247965/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247965/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3fbc8f1dc20949ce964024089067d7f3 {'smithi033.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24 680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/144bb6d0 3f4cceafda5d5248f92ece7a8539e8858d490dfec5ebeb61d487bb20-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/38c2c78c3f89e4d347f07263805d33d8ae17df60ebab2e8a259218c57bcae2fb- comps-PowerTools.x86_64.xml - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} Pass ================================================================= [7247811] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/kernel_untar_build}} time: 02:31:56 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247811/ [7247812] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:14:14 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247812/ [7247813] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:12:59 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247813/ [7247814] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:51:05 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247814/ [7247815] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:18:57 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247815/ [7247816] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:13:35 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247816/ [7247817] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 01:47:40 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247817/ [7247818] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 01:49:19 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247818/ [7247821] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:14:10 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247821/ [7247822] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 01:19:44 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247822/ [7247823] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:20:13 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247823/ [7247824] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:15:56 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247824/ [7247825] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 01:07:05 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247825/ [7247826] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 02:08:26 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247826/ [7247828] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:21:28 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247828/ [7247829] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 01:12:39 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247829/ [7247830] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} time: 02:06:05 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247830/ [7247831] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} time: 00:23:45 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247831/ [7247832] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:53:26 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247832/ [7247833] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:58:44 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247833/ [7247835] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:44 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247835/ [7247836] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:42:47 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247836/ [7247837] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:51:20 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247837/ [7247838] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_8} tasks/fstop} time: 01:14:33 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247838/ [7247839] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:21:21 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247839/ [7247840] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 01:50:28 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247840/ [7247842] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:15:19 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247842/ [7247843] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:52:40 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247843/ [7247844] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} time: 00:50:10 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247844/ [7247846] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:38:15 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247846/ [7247847] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 01:43:11 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247847/ [7247848] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:21:16 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247848/ [7247849] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} time: 00:30:27 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247849/ [7247850] fs/upgrade/upgraded_client/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 01:35:21 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247850/ [7247851] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:23:04 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247851/ [7247853] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} time: 00:45:16 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247853/ [7247855] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 01:57:57 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247855/ [7247856] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} time: 00:49:33 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247856/ [7247857] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 03:51:48 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247857/ [7247858] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 01:09:34 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247858/ [7247859] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:36 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247859/ [7247860] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} time: 00:22:47 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247860/ [7247861] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} time: 00:58:46 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247861/ [7247863] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:19:14 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247863/ [7247864] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} time: 00:26:37 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247864/ [7247866] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:49:39 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247866/ [7247867] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:52:36 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247867/ [7247869] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} time: 01:25:57 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247869/ [7247870] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:25:17 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247870/ [7247871] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 01:34:39 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247871/ [7247872] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:50:40 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247872/ [7247874] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 01:50:32 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247874/ [7247875] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} time: 02:19:36 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247875/ [7247876] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:36:29 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247876/ [7247877] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} time: 01:06:30 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247877/ [7247879] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:53:28 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247879/ [7247880] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} time: 00:28:44 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247880/ [7247881] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:21:23 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247881/ [7247882] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:16:26 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247882/ [7247883] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} time: 00:16:26 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247883/ [7247884] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:19:39 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247884/ [7247885] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:16:03 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247885/ [7247886] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:43:49 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247886/ [7247887] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} time: 00:32:17 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247887/ [7247890] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} time: 00:23:30 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247890/ [7247891] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:13:50 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247891/ [7247893] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} time: 00:25:36 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247893/ [7247897] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:45:09 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247897/ [7247898] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 01:43:20 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247898/ [7247900] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} time: 00:14:43 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247900/ [7247903] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:40:05 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247903/ [7247905] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 01:12:19 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247905/ [7247906] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:14 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247906/ [7247907] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:47:11 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247907/ [7247908] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 01:15:49 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247908/ [7247909] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:49:05 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247909/ [7247910] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} time: 00:14:24 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247910/ [7247911] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:49:09 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247911/ [7247913] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} time: 02:07:29 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247913/ [7247914] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} time: 00:33:04 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247914/ [7247915] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:34:13 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247915/ [7247916] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:18:39 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247916/ [7247917] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} time: 00:20:54 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247917/ [7247918] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 01:45:04 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247918/ [7247919] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} time: 00:52:04 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247919/ [7247922] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:03:08 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247922/ [7247923] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:45:27 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247923/ [7247924] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} time: 00:19:50 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247924/ [7247925] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:01:11 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247925/ [7247926] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:15:10 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247926/ [7247928] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} time: 01:20:07 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247928/ [7247929] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 00:59:26 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247929/ [7247930] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:56:11 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247930/ [7247931] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:20 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247931/ [7247933] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} time: 00:31:00 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247933/ [7247935] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 01:51:34 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247935/ [7247936] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} time: 00:31:31 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247936/ [7247937] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:32:48 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247937/ [7247938] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 01:41:14 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247938/ [7247939] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} time: 02:23:38 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247939/ [7247940] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} time: 00:21:11 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247940/ [7247941] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 01:26:44 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247941/ [7247943] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:19:31 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247943/ [7247944] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} time: 00:13:24 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247944/ [7247946] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} time: 00:34:29 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247946/ [7247947] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:54:26 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247947/ [7247948] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} time: 00:22:32 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247948/ [7247949] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} time: 02:26:27 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247949/ [7247950] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:13:05 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247950/ [7247952] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:18:16 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247952/ [7247954] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:18:44 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247954/ [7247955] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:17:45 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247955/ [7247956] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 01:57:51 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247956/ [7247959] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:32:14 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247959/ [7247960] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} time: 00:18:09 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247960/ [7247963] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:40:34 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247963/ [7247964] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:15:53 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247964/ [7247966] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 01:45:06 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247966/ [7247967] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 02:00:57 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247967/ [7247969] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} time: 00:14:20 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247969/ [7247970] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:26:49 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247970/ [7247971] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} time: 00:24:38 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247971/ [7247972] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:24:07 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247972/ [7247974] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 01:46:34 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247974/ [7247975] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:08 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247975/ [7247977] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:36:12 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247977/ [7247978] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:45 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247978/ [7247979] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:22:32 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247979/ [7247980] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} time: 00:17:13 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247980/ [7247982] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 01:48:15 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247982/ [7247984] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} time: 00:13:54 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247984/ [7247986] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:16:59 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247986/ [7247987] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} time: 00:21:08 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247987/ [7247988] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} time: 00:13:08 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247988/ [7247989] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 01:17:42 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247989/ [7247990] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} time: 00:21:43 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247990/ [7247993] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:56:27 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247993/ [7247994] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:58:51 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247994/ [7247995] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} time: 00:17:19 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247995/ [7247996] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 02:02:00 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247996/ [7247998] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:36:35 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247998/ [7247999] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:20:04 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7247999/ [7248000] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} time: 00:35:36 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7248000/ [7248001] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:15:51 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7248001/ [7248003] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:50:16 info: http://pulpito.ceph.com/yuriw-2023-04-21_14:42:39-fs-wip-yuri7-testing-2023-04-19-1343-distro-default-smithi/7248003/