2023-06-15T15:48:10.161 INFO:root:teuthology version: 0.0.1.dev124+g961f4fb 2023-06-15T15:48:10.162 INFO:teuthology.results:subset: '111/120000' 2023-06-15T15:48:10.162 INFO:teuthology.results:seed: '1223' 2023-06-15T18:04:22.119 INFO:root:teuthology version: 0.0.1.dev124+g961f4fb 2023-06-15T18:04:22.155 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-06-15T21:22:39.842 INFO:teuthology.results:Tests finished! gathering results... 2023-06-15T21:22:55.141 INFO:scrape:Found 185 jobs 2023-06-15T21:23:15.129 INFO:scrape:Found 5 distinct failure reasons 2023-06-15T21:23:15.130 INFO:scrape:Failure: Command failed on smithi073 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' 2023-06-15T21:23:15.130 INFO:scrape:138 jobs 2023-06-15T21:23:15.132 INFO:scrape:suites intersection: ['2-logrotate}'] 2023-06-15T21:23:15.133 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', '2-workunit/cfuse_workunit_suites_fsstress}}', '2-workunit/cfuse_workunit_suites_pjd}}', '2-workunit/cfuse_workunit_trivial_sync}}', '2-workunit/ffsb}}', '2-workunit/fs/snaps}}', '2-workunit/fs/trivial_sync}}', '2-workunit/iozone}}', '2-workunit/suites/ffsb}}', '2-workunit/suites/fsstress}}', '2-workunit/suites/iozone}}', '2-workunit/suites/pjd}}', 'begin/{0-install', 'centos_latest', 'cephfs-mirror/one-per-cluster', 'cephfs-mirror/three-per-cluster', 'clients/mirror', 'clients/{mirror}', 'cluster/1-node', 'cluster/{1-node}', 'clusters/1-mds-1-client-coloc', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'clusters/1-node-1-mds-1-osd', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'clusters/small-cluster', 'conf/{client', 'debug', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/32bits/{begin/{0-install', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/full/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/libcephfs/{begin/{0-install', 'fs/mirror-ha/{begin/{0-install', 'fs/mirror/{begin/{0-install', 'fs/mixed-clients/{begin/{0-install', 'fs/multiclient/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/shell/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/thrash/workloads/{begin/{0-install', 'fs/top/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/valgrind/{begin/{0-install', 'fs/verify/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'kclient-overrides/{distro/stock/{k-stock', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mon-debug', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'msgr-failures/none', 'msgr-failures/osd-mds-delay', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd-asserts}', 'osd}', 'overrides', 'overrides/ignorelist_health', 'overrides/whitelist_health', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'races', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/no', 'scrub/yes', 'session_timeout', 'session_timeout}', 'standby-replay', 'subvolume/{no-subvolume}', 'subvolume/{with-namespace-isolated-and-quota}', 'subvolume/{with-namespace-isolated}', 'subvolume/{with-no-extra-options}', 'subvolume/{with-quota}', 'supported-random-distro$/{centos_8}', 'supported-random-distros$/{rhel_8}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/acls}', 'tasks/admin}', 'tasks/alternate-pool}', 'tasks/auto-repair}', 'tasks/backtrace}', 'tasks/cephfs-shell}', 'tasks/cephfs_misc_tests}', 'tasks/cfuse_workunit_misc}', 'tasks/cfuse_workunit_suites_blogbench', 'tasks/cfuse_workunit_suites_dbench', 'tasks/cfuse_workunit_suites_ffsb', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/client-limits}', 'tasks/client-readahead}', 'tasks/client}', 'tasks/damage}', 'tasks/data-scan}', 'tasks/exports}', 'tasks/failover}', 'tasks/forward-scrub}', 'tasks/fragment}', 'tasks/fsstress', 'tasks/fstop}', 'tasks/ino_release_cb}', 'tasks/ior-shared-file}', 'tasks/kernel_cfuse_workunits_dbench_iozone}', 'tasks/libcephfs/{frag', 'tasks/libcephfs_python}', 'tasks/mds-flush}', 'tasks/mds_creation_retry}', 'tasks/mdtest}', 'tasks/metrics}', 'tasks/mgr-osd-full}', 'tasks/mirror}', 'tasks/mirror}}', 'tasks/multifs-auth}', 'tasks/multimds_misc}', 'tasks/openfiletable}', 'tasks/pool-perm}', 'tasks/recovery-fs}', 'tasks/scrub}', 'tasks/snap-schedule}', 'tasks/snap_schedule_snapdir}', 'tasks/strays}', 'tasks/test_journal_migration}', 'tasks/trim-i24137}', 'tasks/volumes/{overrides', 'tasks/workunit/dir-max-entries}', 'tasks/workunit/quota}', 'tasks/workunit/snaps}', 'tasks/{0-check-counter', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/mon', 'tasks/{1-thrash/osd', 'test/basic}}', 'test/clone}}', 'test/finisher_per_module}}', 'test/misc}}', 'test/snapshot}}', 'test}}', 'thrashosds-health}', 'traceless/50pc}', 'validater/lockdep}', 'workloads/cephfs-mirror-ha-workunit}', 'workunit/direct_io}', 'workunit/fs/misc}', 'workunit/fs/norstats}', 'workunit/fs/test_o_trunc}', 'workunit/kernel_untar_build}', 'workunit/suites/blogbench}', 'workunit/suites/dbench}', 'workunit/suites/ffsb}', 'workunit/suites/fsstress}', 'workunit/suites/fsx}', 'workunit/suites/fsync-tester}', 'workunit/suites/iogen}', 'workunit/suites/iozone}', 'workunit/suites/pjd}', 'wsync/{no}}', 'wsync/{yes}}'] 2023-06-15T21:23:15.133 INFO:scrape: 2023-06-15T21:23:15.134 INFO:scrape:Failure: "2023-06-15T16:24:10.085534+0000 mon.a (mon.0) 314 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs down (PG_AVAILABILITY)" in cluster log 2023-06-15T21:23:15.134 INFO:scrape:14 jobs: ['7305476', '7305535', '7305609', '7305524', '7305502', '7305632', '7305581', '7305513', '7305584', '7305495', '7305550', '7305487', '7305539', '7305559'] 2023-06-15T21:23:15.134 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2023-06-15T21:23:15.135 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-4c-client', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'distro/{ubuntu/{latest', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug', 'mount', 'mount/fuse', 'mount/kclient/{k-testing', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'ranks/3', 'ranks/5', 'session_timeout}', 'tasks/asok_dump_tree}', 'tasks/cap-flush}', 'tasks/client-recovery}', 'tasks/dbench', 'tasks/fsstress', 'tasks/journal-repair}', 'tasks/openfiletable}', 'tasks/quota}', 'tasks/sessionmap}', 'tasks/snapshots}', 'tasks/truncate_delay}', 'tasks/workunit/snaps}', 'validater/lockdep}', 'validater/valgrind}'] 2023-06-15T21:23:15.135 INFO:scrape: 2023-06-15T21:23:15.136 INFO:scrape:Failure: 2023-06-15T21:23:15.136 INFO:scrape:11 jobs: ['7305501', '7305608', '7305525', '7305622', '7305583', '7305457', '7305470', '7305514', '7305551', '7305541', '7305634'] 2023-06-15T21:23:15.137 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2023-06-15T21:23:15.137 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-with-workload', '4-verify}', '4-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-from/v16.2.4'] 2023-06-15T21:23:15.137 INFO:scrape: 2023-06-15T21:23:15.137 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=d1f1c6e0d01e602b94073f3b1b2d8933c2572c92 2023-06-15T21:23:15.138 INFO:scrape:2 jobs: ['7305498', '7305606'] 2023-06-15T21:23:15.138 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-06-15T21:23:15.138 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}'] 2023-06-15T21:23:15.139 INFO:scrape: 2023-06-15T21:23:15.139 INFO:scrape:Failure: "2023-06-15T16:19:00.798890+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 127114.227387 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2023-06-15T21:23:15.139 INFO:scrape:2 jobs: ['7305460', '7305616'] 2023-06-15T21:23:15.140 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/mds-full}'] 2023-06-15T21:23:15.140 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/mds-full}'] 2023-06-15T21:23:15.140 INFO:scrape: 2023-06-15T21:23:15.150 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/ failed: 167 dead: 0 running: 0 waiting: 0 queued: 0 passed: 18 Fail ================================================================= [7305453] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:12:18 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305453/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305453/ sentry: https://sentry.ceph.com/organizations/ceph/?query=18ae8e94e7884ab5af2247235d1fceeb Command failed on smithi086 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305454] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:09:51 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305454/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305454/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c8d3e08d1b984edf85a2aa293d687658 Command failed on smithi018 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305455] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:09:54 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305455/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305455/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe48bc7ffe1047dda56c49a4da9f1a9c Command failed on smithi078 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305456] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:11:09 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305456/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305456/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8bd774ba2da548ed98c6dc315a275141 Command failed on smithi012 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305457] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:18:28 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305457/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305457/ sentry: https://sentry.ceph.com/organizations/ceph/?query=72eb50d3add043dab0633dac54ea43bf [7305458] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:13:17 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305458/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305458/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0e7f667656714abf929dbbd9b7dca500 Command failed on smithi149 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305459] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:37 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305459/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305459/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0cd34c936eeb46c183a8c6ac0fdf9315 Command failed on smithi038 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5' [7305460] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:11:40 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305460/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305460/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5d7f72e0f12545108895f3fb551a78ed "2023-06-15T16:19:00.798890+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 127114.227387 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7305461] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{yes}} ----------------------------------------------------------------- time: 00:21:11 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305461/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305461/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6cefdce658eb48c9931400e06f6d4161 Command failed on smithi084 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true' [7305462] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} ----------------------------------------------------------------- time: 00:13:50 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305462/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305462/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d102b48c6a87459ab37997a4a3cc4ae8 Command failed on smithi144 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305463] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:19:10 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305463/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305463/ sentry: https://sentry.ceph.com/organizations/ceph/?query=edc92fc248aa40a09a4d898d6069973a Command failed on smithi053 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305464] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:09:32 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305464/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305464/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0532cfd708a9427a80822995de86daac Command failed on smithi005 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305465] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:11:52 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305465/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305465/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a37618a70cf544dabb57bc6d6226c612 Command failed on smithi040 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305466] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} ----------------------------------------------------------------- time: 00:13:38 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305466/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305466/ sentry: https://sentry.ceph.com/organizations/ceph/?query=944e5a405b994dc3a16b6c730749c308 Command failed on smithi002 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305468] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:13:39 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305468/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305468/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f349d11dc10949ad85ba12b4049a4d77 Command failed on smithi120 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305469] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:15:58 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305469/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305469/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e28d1f52e33144b0b7c8c08af31555d5 Command failed on smithi073 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305470] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:21:00 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305470/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305470/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07532d9f43fd421e824737e8e0a9bfe9 [7305471] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} ----------------------------------------------------------------- time: 00:17:20 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305471/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305471/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c09f2d1927a94adcb9080c758f0fdf79 Command failed on smithi046 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305472] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/iogen} wsync/{yes}} ----------------------------------------------------------------- time: 00:24:27 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305472/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305472/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b754961636b4d2294ba068460b281b1 Command failed on smithi016 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3' [7305473] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:11:10 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305473/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305473/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c03952c70007451b8035c5674bd9f57a Command failed on smithi101 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305474] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:09:37 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305474/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305474/ sentry: https://sentry.ceph.com/organizations/ceph/?query=98bf60037d6c4a0fbded974d0da8f8ca Command failed on smithi167 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305476] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} ----------------------------------------------------------------- time: 00:10:01 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305476/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305476/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a8899a2cecc942a2aaca96581866acb1 "2023-06-15T16:24:10.085534+0000 mon.a (mon.0) 314 : cluster [WRN] Health check failed: Reduced data availability: 9 pgs down (PG_AVAILABILITY)" in cluster log [7305477] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:15:31 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305477/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305477/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7f56962060494973973c85c507c72638 Command failed on smithi060 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305478] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:10:28 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305478/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305478/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03f2012be13546cbb7c03d53b30405f1 Command failed on smithi019 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305479] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:09:59 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305479/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305479/ sentry: https://sentry.ceph.com/organizations/ceph/?query=285ab76d52d54a0a88d5f4377b36b1b3 Command failed on smithi110 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305480] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:13:23 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305480/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305480/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63ccaa92275c4e65bd7c43212eb443a7 Command failed on smithi006 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305481] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:09:40 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305481/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305481/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8821e22c048546c890f7f7ffce6edb8c Command failed on smithi088 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305483] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} ----------------------------------------------------------------- time: 00:19:49 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305483/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305483/ sentry: https://sentry.ceph.com/organizations/ceph/?query=920ef321552c4f2086338e363e5865dc Command failed on smithi018 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305484] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:15:19 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305484/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305484/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7217201558a4485a9b28a758b2df29ca Command failed on smithi038 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3' [7305485] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 00:11:11 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305485/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305485/ sentry: https://sentry.ceph.com/organizations/ceph/?query=be3e86632ab543bbae2019a0e6b85f5d Command failed on smithi078 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305486] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/iozone} wsync/{no}} ----------------------------------------------------------------- time: 00:25:40 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305486/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305486/ sentry: https://sentry.ceph.com/organizations/ceph/?query=27a1607b0afd4781a97fbc46328078a1 Command failed on smithi057 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5' [7305487] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota} ----------------------------------------------------------------- time: 00:12:41 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305487/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305487/ sentry: https://sentry.ceph.com/organizations/ceph/?query=80f794ec0e7f4f048f0a6172a538eb42 "2023-06-15T16:42:00.848118+0000 mon.a (mon.0) 215 : cluster [WRN] Health check failed: Degraded data redundancy: 9/48 objects degraded (18.750%), 4 pgs degraded (PG_DEGRADED)" in cluster log [7305489] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:14:40 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305489/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305489/ sentry: https://sentry.ceph.com/organizations/ceph/?query=74610cf4751b4f748ecb2d284d6318a4 Command failed on smithi141 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305490] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs} ----------------------------------------------------------------- time: 00:17:26 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305490/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305490/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a59c2a07a02d44c1b90137d06a863549 Command failed on smithi043 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305492] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:16:39 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305492/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305492/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c9fe493bf2144e1babaa7f22608d54c4 Command failed on smithi145 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5' [7305493] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/scrub} ----------------------------------------------------------------- time: 00:11:38 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305493/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305493/ sentry: https://sentry.ceph.com/organizations/ceph/?query=563bc7ad7d8c4dcab17d93ac09c95d44 Command failed on smithi101 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305494] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/suites/pjd} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:02 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305494/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305494/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b4c6c1251c14417ba2f5cbf9bd0fccf6 Command failed on smithi002 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true' [7305495] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/sessionmap} ----------------------------------------------------------------- time: 00:12:22 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305495/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305495/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a862bfb7d5854568b0ee0c9364028dcf "2023-06-15T16:48:56.045246+0000 mon.a (mon.0) 348 : cluster [WRN] Health check failed: Degraded data redundancy: 9/46 objects degraded (19.565%), 4 pgs degraded (PG_DEGRADED)" in cluster log [7305496] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:13:13 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305496/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305496/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a145f2e34fab4ecb952037444a50d587 Command failed on smithi167 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305497] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:09:50 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305497/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305497/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2d306b4dd3114811946b65e6a4f8de95 Command failed on smithi144 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305498] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:08:31 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305498/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305498/ sentry: https://sentry.ceph.com/organizations/ceph/?query=efe345c6458c4aef82d3823ab6489709 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =d1f1c6e0d01e602b94073f3b1b2d8933c2572c92 [7305499] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:13:52 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305499/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305499/ sentry: https://sentry.ceph.com/organizations/ceph/?query=951c070c6443454cad983299333ee8da Command failed on smithi073 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305500] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} ----------------------------------------------------------------- time: 00:12:48 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305500/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305500/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa9922ad198347cea7d2f1f13e32bb6f Command failed on smithi084 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305501] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:20:10 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305501/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305501/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49d422f4d4b64fa887e91ee5e9794428 [7305502] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:24:28 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305502/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305502/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f017ac41139547c68f8963bf68b453be "2023-06-15T16:57:21.315759+0000 mon.a (mon.0) 198 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7305503] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} ----------------------------------------------------------------- time: 00:13:41 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305503/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305503/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e696968949184af9a64046cf71b1487f Command failed on smithi032 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305504] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:15:47 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305504/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305504/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0089b44f111a452781a90504a4d81d07 Command failed on smithi136 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305506] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap_schedule_snapdir} ----------------------------------------------------------------- time: 00:11:56 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305506/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305506/ sentry: https://sentry.ceph.com/organizations/ceph/?query=08b229d7bba046478f411984ba7090eb Command failed on smithi061 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305507] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:10:18 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305507/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305507/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3d3362acf12f4ef2bbb6db38a401aea7 Command failed on smithi053 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305508] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:15:27 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305508/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305508/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9d336dcdbcd244febc207493afc42ac9 Command failed on smithi060 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305509] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:13:35 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305509/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305509/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c335cc6bb6c428faa1faa33047c9df4 Command failed on smithi045 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305510] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:14:58 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305510/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305510/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8e3a6e8c01f24476a3f981a79bfc64b9 Command failed on smithi098 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305511] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/direct_io} wsync/{yes}} ----------------------------------------------------------------- time: 00:25:56 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305511/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305511/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc1a02232f3a4695ac4c862e682e1b5e Command failed on smithi046 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3' [7305512] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:18:23 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305512/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305512/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ea50caac11614d4c97c4b241334040bd Command failed on smithi006 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305513] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots} ----------------------------------------------------------------- time: 00:12:41 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305513/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305513/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b83803ecccf5406594d7ee2ddbddd859 "2023-06-15T16:59:58.467359+0000 mon.a (mon.0) 214 : cluster [WRN] Health check failed: Degraded data redundancy: 10/48 objects degraded (20.833%), 5 pgs degraded (PG_DEGRADED)" in cluster log [7305514] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:23:18 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305514/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305514/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2f41b645bcaa460b9c92e97e0adfe5e0 [7305515] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/strays} ----------------------------------------------------------------- time: 00:10:58 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305515/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305515/ sentry: https://sentry.ceph.com/organizations/ceph/?query=825c40dd63384df8877fc08eb2b5c774 Command failed on smithi078 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305516] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:14:51 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305516/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305516/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4494a964c54a4749935ec518acb48ece Command failed on smithi029 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3' [7305518] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/test_journal_migration} ----------------------------------------------------------------- time: 00:15:27 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305518/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305518/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d866ab2465e04f229b6f0c9dc330250e Command failed on smithi101 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305519] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 00:27:56 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305519/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305519/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5e31ebb965f542f7a9519b4eb4d38d54 Command failed on smithi018 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5' [7305520] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:12:43 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305520/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305520/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2754809306cb4f8d8b3f76116ca98988 Command failed on smithi038 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305521] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:09:33 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305521/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305521/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58a24efc597a4c8e9b9d346f6eca9e6b Command failed on smithi144 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305522] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} ----------------------------------------------------------------- time: 00:11:19 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305522/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305522/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e078e9258984d80b84f02878fa5fb05 Command failed on smithi005 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305523] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:10:03 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305523/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305523/ sentry: https://sentry.ceph.com/organizations/ceph/?query=23797a11849944cd9f497871b014a271 Command failed on smithi141 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305524] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/truncate_delay} ----------------------------------------------------------------- time: 00:12:12 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305524/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305524/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d6c1633a35b7462796aa447c8f495bb7 "2023-06-15T17:09:34.508913+0000 mon.a (mon.0) 315 : cluster [WRN] Health check failed: Degraded data redundancy: 11/46 objects degraded (23.913%), 5 pgs degraded (PG_DEGRADED)" in cluster log [7305525] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:19:03 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305525/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305525/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df56acbcdef149d89d08d5ba7430461a [7305526] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:46 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305526/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305526/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af197a0ad5df43b7af042eb3b8d0c883 Command failed on smithi167 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5' [7305527] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/dir-max-entries} ----------------------------------------------------------------- time: 00:11:28 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305527/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305527/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d7ceb456e064b7eaf6408449add935f Command failed on smithi032 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305528] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:12:42 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305528/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305528/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ec6e9590fdaa4bcca0e82742dab28ee6 Command failed on smithi073 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305529] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:15:34 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305529/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305529/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bed232c09ee4a4588a72cad765c76cf Command failed on smithi120 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305530] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} ----------------------------------------------------------------- time: 00:11:32 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305530/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305530/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43d7e9eb2ce742e5ad4210770f1205ea Command failed on smithi061 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305531] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/kernel_untar_build} wsync/{yes}} ----------------------------------------------------------------- time: 00:23:37 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305531/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305531/ sentry: https://sentry.ceph.com/organizations/ceph/?query=778e412c66f14190b170a4279a58143e Command failed on smithi040 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true' [7305532] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:11:34 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305532/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305532/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6aa8c704796142179c93741ca57f363e Command failed on smithi043 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305533] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} ----------------------------------------------------------------- time: 00:10:56 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305533/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305533/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ae036cc1c71d4945ae18459d82456b73 Command failed on smithi066 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305535] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 00:13:49 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305535/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305535/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0116778728f94b4b9abb0ab32a73b304 "2023-06-15T17:18:01.210274+0000 mon.a (mon.0) 178 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7305536] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls} ----------------------------------------------------------------- time: 00:14:39 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305536/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305536/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8030e0ae3cd14db0908e2a9e7ea99a57 Command failed on smithi053 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305537] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:10:21 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305537/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305537/ sentry: https://sentry.ceph.com/organizations/ceph/?query=03f5889bcd6a4cb1b7a4c58573d54336 Command failed on smithi012 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305538] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:15:48 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305538/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305538/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b3cb0523b91e4e69a5a95bf340818de9 Command failed on smithi060 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305539] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:12:34 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305539/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305539/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4245e39c5f943288868693a81c28f35 "2023-06-15T17:19:37.872813+0000 mon.a (mon.0) 208 : cluster [WRN] Health check failed: Degraded data redundancy: 6/48 objects degraded (12.500%), 4 pgs degraded (PG_DEGRADED)" in cluster log [7305540] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:16:21 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305540/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305540/ sentry: https://sentry.ceph.com/organizations/ceph/?query=796359930e234bbd89c6979f79f0afc8 Command failed on smithi057 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305541] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:18:59 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305541/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305541/ sentry: https://sentry.ceph.com/organizations/ceph/?query=76a65f91f6ee4a3db8202a8731ef72bd [7305542] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin} ----------------------------------------------------------------- time: 00:11:26 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305542/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305542/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7b5ab2350bba4a4aaac8e90ed1b024cd Command failed on smithi033 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305543] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:11:14 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305543/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305543/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f83be6d1c993411a8c0ca3c73e376dc1 Command failed on smithi022 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305544] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:11:42 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305544/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305544/ sentry: https://sentry.ceph.com/organizations/ceph/?query=17a7da51280c4e8dbd3e28f6e78d67ea Command failed on smithi078 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305545] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:15:27 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305545/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305545/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a69b591bfa274128a8cf224dafbf1c4c Command failed on smithi098 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305546] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/blogbench} wsync/{yes}} ----------------------------------------------------------------- time: 00:24:06 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305546/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305546/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3408050eed084548a794665424bdca35 Command failed on smithi016 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3' [7305547] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool} ----------------------------------------------------------------- time: 00:19:57 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305547/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305547/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d1bdca29dcd24228a48d69cc6ce46f37 Command failed on smithi144 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305549] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 00:18:19 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305549/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305549/ sentry: https://sentry.ceph.com/organizations/ceph/?query=120c0d60f19b44238beaf1248e912d42 Command failed on smithi141 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3' [7305550] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/asok_dump_tree} ----------------------------------------------------------------- time: 00:12:23 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305550/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305550/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d44529793b0047caafac8a9617f8c959 "2023-06-15T17:28:03.373723+0000 mon.a (mon.0) 215 : cluster [WRN] Health check failed: Degraded data redundancy: 9/46 objects degraded (19.565%), 4 pgs degraded (PG_DEGRADED)" in cluster log [7305551] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:17:10 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305551/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305551/ sentry: https://sentry.ceph.com/organizations/ceph/?query=332169461612413db2b62f3d2ae28815 [7305552] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/auto-repair} ----------------------------------------------------------------- time: 00:10:57 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305552/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305552/ sentry: https://sentry.ceph.com/organizations/ceph/?query=200fe8816ce2440daaad3d8f9f82cb5f Command failed on smithi167 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305553] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} ----------------------------------------------------------------- time: 00:15:19 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305553/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305553/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a44726daca447148d21a02aaa810793 Command failed on smithi101 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305554] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:12:08 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305554/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305554/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6ff8a91ed46648b49824034376994f79 Command failed on smithi110 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305555] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{no-subvolume} tasks/{0-check-counter workunit/suites/dbench} wsync/{no}} ----------------------------------------------------------------- time: 00:30:27 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305555/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305555/ sentry: https://sentry.ceph.com/organizations/ceph/?query=de6aaaa6202748e08069857480ec3e9f Command failed on smithi019 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5' [7305556] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} ----------------------------------------------------------------- time: 00:11:23 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305556/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305556/ sentry: https://sentry.ceph.com/organizations/ceph/?query=04b2cafaedab4bcf8dec33870d4c3a45 Command failed on smithi046 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305558] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:10:16 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305558/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305558/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4d72dce1cfeb4508ac1b1e0e909d7e97 Command failed on smithi017 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5' [7305559] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cap-flush} ----------------------------------------------------------------- time: 00:12:49 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305559/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305559/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a8fdedc30b8495f942df0ccc0651b4d "2023-06-15T17:34:29.088032+0000 mon.a (mon.0) 249 : cluster [WRN] Health check failed: Degraded data redundancy: 9/48 objects degraded (18.750%), 4 pgs degraded (PG_DEGRADED)" in cluster log [7305560] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} ----------------------------------------------------------------- time: 00:14:14 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305560/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305560/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4346631aab844d7d841a811bb19edeef Command failed on smithi073 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305561] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:09:41 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305561/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305561/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f19cfbbdee644c04bc5fccbd25a22bdf Command failed on smithi043 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305562] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:10:06 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305562/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305562/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2693950ed0e4039ac4e55edfb1080ae Command failed on smithi120 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305564] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} ----------------------------------------------------------------- time: 00:10:22 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305564/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305564/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2492e72599a84df19ed3ee22a6e88c19 Command failed on smithi196 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305565] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} ----------------------------------------------------------------- time: 00:09:43 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305565/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305565/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9217ae655f7445ed9dedf323432aa496 Command failed on smithi012 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305566] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} ----------------------------------------------------------------- time: 00:12:44 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305566/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305566/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f20b89879f9541cd9cc20ed66d25b3aa Command failed on smithi131 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305567] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{rhel_8} tasks/mirror} ----------------------------------------------------------------- time: 00:10:35 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305567/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305567/ sentry: https://sentry.ceph.com/organizations/ceph/?query=78596f473e1c4339bf3dd0c4e5e90101 Command failed on smithi102 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305568] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{centos_8} workloads/cephfs-mirror-ha-workunit} ----------------------------------------------------------------- time: 00:10:47 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305568/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305568/ sentry: https://sentry.ceph.com/organizations/ceph/?query=785ba55de3304f82b4259dedc2c40c5c Command failed on smithi084 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305569] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} ----------------------------------------------------------------- time: 00:15:45 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305569/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305569/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9fb510b6b6d64219b1e9f1afeceb7964 Command failed on smithi066 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305570] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 00:10:40 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305570/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305570/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c80e69cf104c4e62abd7b6c0802f710d Command failed on smithi022 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305571] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:13:39 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305571/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305571/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2789bb4f2ca041a4b0d62f58b3780c05 Command failed on smithi045 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305572] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 00:11:16 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305572/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305572/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e569e8d60fea42c9acd43e6d9452eb75 Command failed on smithi053 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305573] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} ----------------------------------------------------------------- time: 00:09:51 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305573/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305573/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cdf754e46bfa4725a7b0bb052da39ae6 Command failed on smithi018 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305574] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:09:53 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305574/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305574/ sentry: https://sentry.ceph.com/organizations/ceph/?query=92dae15122604efbb8a142922d07e279 Command failed on smithi071 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305575] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/fstop} ----------------------------------------------------------------- time: 00:09:25 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305575/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305575/ sentry: https://sentry.ceph.com/organizations/ceph/?query=63b1ee3e0f934f7789393f2ba72c3dff Command failed on smithi057 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305576] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} ----------------------------------------------------------------- time: 00:09:43 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305576/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305576/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1daf1a7b79274989951489351fccb568 Command failed on smithi153 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305578] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}} ----------------------------------------------------------------- time: 00:17:13 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305578/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305578/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ca0505175dc2451f859859f6d75e42bb Command failed on smithi132 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305579] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/no standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/ffsb} wsync/{yes}} ----------------------------------------------------------------- time: 00:25:43 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305579/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305579/ sentry: https://sentry.ceph.com/organizations/ceph/?query=883db1f64afc46c5affd4247194926e7 Command failed on smithi060 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true' [7305580] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} ----------------------------------------------------------------- time: 00:15:40 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305580/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305580/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50184194d64c457e8c8c521878aebe4a Command failed on smithi178 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305581] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} ----------------------------------------------------------------- time: 00:24:49 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305581/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305581/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e0b71596fdbb4bca8545e7075c2e3127 "2023-06-15T17:57:33.875518+0000 mon.a (mon.0) 297 : cluster [WRN] Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY)" in cluster log [7305582] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} ----------------------------------------------------------------- time: 00:11:14 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305582/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305582/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f0357c6867394c4ea6af49c8a301b834 Command failed on smithi152 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305583] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 00:16:57 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305583/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305583/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e824e0ea26074d37aa66d3128390383c [7305584] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery} ----------------------------------------------------------------- time: 00:12:21 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305584/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305584/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4fa37cb5d29049caa610508d28596fa1 "2023-06-15T17:52:41.998277+0000 mon.a (mon.0) 255 : cluster [WRN] Health check failed: Degraded data redundancy: 9/46 objects degraded (19.565%), 4 pgs degraded (PG_DEGRADED)" in cluster log [7305585] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:09:58 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305585/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305585/ sentry: https://sentry.ceph.com/organizations/ceph/?query=df857dddd49748109ef6e78032b8f137 Command failed on smithi005 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305586] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} ----------------------------------------------------------------- time: 00:16:55 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305586/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305586/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1224c60341a44ec78723e6e944064784 Command failed on smithi017 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305587] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/damage} ----------------------------------------------------------------- time: 00:17:31 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305587/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305587/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a383457d6c9445bcbc911a6f7107683a Command failed on smithi101 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305589] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/yes standby-replay subvolume/{with-quota} tasks/{0-check-counter workunit/fs/norstats} wsync/{no}} ----------------------------------------------------------------- time: 00:22:32 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305589/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305589/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bc3c80a4415e47aeaf648b4885ec5da8 Command failed on smithi002 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3' [7305590] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:18:28 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305590/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305590/ sentry: https://sentry.ceph.com/organizations/ceph/?query=18d4b1b7bee447b2b2350a53dd9529c5 Command failed on smithi120 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3' [7305591] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} ----------------------------------------------------------------- time: 00:13:49 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305591/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305591/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8170195885af42c18639a8b7f472b6f8 Command failed on smithi032 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305593] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/exports} ----------------------------------------------------------------- time: 00:14:36 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305593/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305593/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5be8897fd07c4e23bf44cd00200f946d Command failed on smithi086 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305594] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:11:23 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305594/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305594/ sentry: https://sentry.ceph.com/organizations/ceph/?query=37476e1aec254a13b63bfff40044cbe4 Command failed on smithi110 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305595] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:11:40 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305595/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305595/ sentry: https://sentry.ceph.com/organizations/ceph/?query=58e12267e0294e68adfd8bbcdc5a60a3 Command failed on smithi061 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305596] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} ----------------------------------------------------------------- time: 00:09:46 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305596/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305596/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5185b0c78fb04c1e9c537711ca89d2d3 Command failed on smithi016 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305597] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/no standby-replay subvolume/{with-namespace-isolated-and-quota} tasks/{0-check-counter workunit/suites/fsstress} wsync/{no}} ----------------------------------------------------------------- time: 00:29:20 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305597/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305597/ sentry: https://sentry.ceph.com/organizations/ceph/?query=514bbe16ceb14df581e86ecedbeab0c3 Command failed on smithi111 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5' [7305598] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} ----------------------------------------------------------------- time: 00:15:06 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305598/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305598/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ffb339832e448c5ac7deedf8a1db271 Command failed on smithi073 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305600] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} ----------------------------------------------------------------- time: 00:09:44 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305600/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305600/ sentry: https://sentry.ceph.com/organizations/ceph/?query=51a6431b5f174ccf891c2cbba607e40e Command failed on smithi084 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5' [7305601] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/fragment} ----------------------------------------------------------------- time: 00:13:42 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305601/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305601/ sentry: https://sentry.ceph.com/organizations/ceph/?query=312b35f144054152a787931b4195f15d Command failed on smithi053 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305602] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} ----------------------------------------------------------------- time: 00:10:22 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305602/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305602/ sentry: https://sentry.ceph.com/organizations/ceph/?query=529d7dcaf9224ca88301b2523b535800 Command failed on smithi022 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305603] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} ----------------------------------------------------------------- time: 00:18:11 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305603/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305603/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a16ab458c384413aad7eac9423f35c08 Command failed on smithi018 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305604] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:11:44 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305604/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305604/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4aedbd6b40474b59a6280dbf8e290a25 Command failed on smithi006 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305605] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} ----------------------------------------------------------------- time: 00:12:40 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305605/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305605/ sentry: https://sentry.ceph.com/organizations/ceph/?query=012bc6ac19c04b2ab51c97e9420cc4aa Command failed on smithi029 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305606] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} ----------------------------------------------------------------- time: 00:08:38 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305606/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305606/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1476f92f3f8c434382743e3c2e577f05 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =d1f1c6e0d01e602b94073f3b1b2d8933c2572c92 [7305607] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} ----------------------------------------------------------------- time: 00:09:35 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305607/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305607/ sentry: https://sentry.ceph.com/organizations/ceph/?query=02390cae1a974379bee89a85d74aa7b6 Command failed on smithi071 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305608] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 01:14:49 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305608/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305608/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73cd7319345745b081f09d2fce249ccd [7305609] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/journal-repair} ----------------------------------------------------------------- time: 00:12:30 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305609/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305609/ sentry: https://sentry.ceph.com/organizations/ceph/?query=06d9665f4e004ca19b38082165c3d0a1 "2023-06-15T18:10:05.133644+0000 mon.a (mon.0) 200 : cluster [WRN] Health check failed: Degraded data redundancy: 9/46 objects degraded (19.565%), 4 pgs degraded (PG_DEGRADED)" in cluster log [7305610] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} ----------------------------------------------------------------- time: 01:06:54 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305610/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305610/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5f577ae03d04ebf8abea207bf29b777 Command failed on smithi066 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305611] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 scrub/yes standby-replay subvolume/{with-namespace-isolated} tasks/{0-check-counter workunit/suites/fsx} wsync/{yes}} ----------------------------------------------------------------- time: 00:21:38 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305611/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305611/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9859e58cde9d48908a345811eab9031e Command failed on smithi057 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true' [7305612] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-flush} ----------------------------------------------------------------- time: 00:38:42 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305612/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305612/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ffad9b079c2d46a0ac0f0a9b4327bbc1 Command failed on smithi156 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305613] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:16:09 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305613/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305613/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5f01082fc8c64592a53bc7f83284497c Command failed on smithi088 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305615] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} ----------------------------------------------------------------- time: 01:40:07 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305615/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305615/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d9274a80ab4c4665a52665d214387036 Command failed on smithi038 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3' [7305616] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} ----------------------------------------------------------------- time: 00:10:54 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305616/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305616/ sentry: https://sentry.ceph.com/organizations/ceph/?query=709cb1072b874d2fb1688297a13c259c "2023-06-15T18:09:33.430660+0000 osd.1 (osd.1) 3 : cluster [WRN] OSD bench result of 219867.635286 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.1. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7305617] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} ----------------------------------------------------------------- time: 00:15:32 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305617/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305617/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d5afe2ac148a441183a4703966f79eb1 Command failed on smithi046 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305618] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:15:10 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305618/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305618/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1526d24853c6434484e93715ba34de3d Command failed on smithi110 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305619] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} ----------------------------------------------------------------- time: 00:52:04 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305619/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305619/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c4221e1ee29748108be62f871cd9947b Command failed on smithi016 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305620] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} ----------------------------------------------------------------- time: 01:34:20 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305620/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305620/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bafe35234d941fc8f2573d2a0e208c9 Command failed on smithi121 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305621] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds_creation_retry} ----------------------------------------------------------------- time: 00:54:39 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305621/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305621/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7bb242519fd2498c9c3991a45e5a77c9 Command failed on smithi061 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305622] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 02:48:24 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305622/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305622/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3fcf1d0c0dc54ee082ea86df7ceee8bc [7305623] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-comp omap_limit/10 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/suites/fsync-tester} wsync/{yes}} ----------------------------------------------------------------- time: 00:26:00 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305623/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305623/ sentry: https://sentry.ceph.com/organizations/ceph/?query=39f0d53220db478cb57d6bbd0c404e7d Command failed on smithi136 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3' [7305624] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} ----------------------------------------------------------------- time: 01:39:07 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305624/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305624/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c48f98a9b644e7497444660befb24d1 Command failed on smithi017 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3' [7305625] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/metrics} ----------------------------------------------------------------- time: 00:11:20 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305625/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305625/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b23161480ab45edb5de2c6dac5ed1cb Command failed on smithi101 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305627] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} ----------------------------------------------------------------- time: 00:11:25 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305627/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305627/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a031e83fabf24cbca2a020d8f437e7aa Command failed on smithi060 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set a allow_standby_replay false' [7305628] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc} ----------------------------------------------------------------- time: 00:11:08 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305628/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305628/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70913560cdaf4d0fa7657ea53757c6f3 Command failed on smithi098 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305629] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable} ----------------------------------------------------------------- time: 01:00:45 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305629/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305629/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e84b092ff964ab49097e04897ce8a88 Command failed on smithi078 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305630] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} ----------------------------------------------------------------- time: 00:10:01 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305630/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305630/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f240a89a1f9b4b94a280b4c52b6a35e1 Command failed on smithi097 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305631] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 01:22:51 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305631/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305631/ sentry: https://sentry.ceph.com/organizations/ceph/?query=33988f39426243a4a13c0b1e4de4ef50 Command failed on smithi022 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305632] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 00:11:47 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305632/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305632/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6a931008aa8a4b90a62558dce5c6919e "2023-06-15T18:27:34.576200+0000 mon.a (mon.0) 267 : cluster [WRN] Health check failed: Degraded data redundancy: 9/46 objects degraded (19.565%), 4 pgs degraded (PG_DEGRADED)" in cluster log [7305633] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:12:11 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305633/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305633/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f87676269d6a419c9a45978eb7f88282 Command failed on smithi040 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' [7305634] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} ----------------------------------------------------------------- time: 02:23:50 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305634/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305634/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f74e84f328604eee84153386107692ba [7305635] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:52:48 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305635/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305635/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c93a50c595d04b0ca4b7bdbddc6ffc44 Command failed on smithi002 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5' [7305636] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/5 scrub/yes standby-replay subvolume/{with-no-extra-options} tasks/{0-check-counter workunit/fs/test_o_trunc} wsync/{no}} ----------------------------------------------------------------- time: 00:30:15 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305636/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305636/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b77d250dcc34362a2644935ec41d7a5 Command failed on smithi053 with status 22: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5' Pass ================================================================= [7305452] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:24:48 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305452/ [7305467] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:30:16 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305467/ [7305475] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:27:06 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305475/ [7305482] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:29:31 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305482/ [7305488] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} time: 00:14:16 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305488/ [7305491] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:19:32 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305491/ [7305505] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:33 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305505/ [7305517] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:31:07 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305517/ [7305534] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:08 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305534/ [7305548] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:27:33 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305548/ [7305557] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 00:25:50 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305557/ [7305563] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 01:14:33 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305563/ [7305577] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:30:10 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305577/ [7305588] fs/upgrade/nofs/{bluestore-bitmap centos_latest conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-octopus 1-upgrade}} time: 00:13:01 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305588/ [7305592] fs/upgrade/upgraded_client/from_nautilus/{bluestore-bitmap centos_latest clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-nautilus 1-client-upgrade 2-client-sanity}} time: 00:14:15 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305592/ [7305599] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:49:29 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305599/ [7305614] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/pacific}} time: 02:26:13 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305614/ [7305626] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 01:39:39 info: http://pulpito.ceph.com/yuriw-2023-06-15_15:17:25-fs-wip-yuri4-testing-2023-06-14-0910-quincy-distro-default-smithi/7305626/