2024-02-29T14:28:43.868 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-29T14:28:43.869 INFO:teuthology.results:subset: '743/1024' 2024-02-29T14:28:43.869 INFO:teuthology.results:seed: '8782' 2024-02-29T15:15:59.403 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-02-29T15:15:59.407 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-02-29T17:46:11.509 INFO:teuthology.results:Tests finished! gathering results... 2024-02-29T17:46:15.932 INFO:scrape:Found 52 jobs 2024-02-29T17:46:34.367 INFO:scrape:Found 9 distinct failure reasons 2024-02-29T17:46:34.368 INFO:scrape:Failure: "2024-02-29T15:00:17.215840+0000 mon.a (mon.0) 695 : cluster [WRN] Health check failed: Degraded data redundancy: 4/324 objects degraded (1.235%), 3 pgs degraded (PG_DEGRADED)" in cluster log 2024-02-29T17:46:34.368 INFO:scrape:3 jobs: ['7576609', '7576643', '7576622'] 2024-02-29T17:46:34.368 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2024-02-29T17:46:34.368 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'debug', 'distro/{centos_latest}', 'distro/{ubuntu_latest}', 'fs/valgrind/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mirror/{cephfs-mirror/one-per-cluster', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/ignorelist_health', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mirror}}', 'tasks/volumes/{overrides', 'test/clone}}'] 2024-02-29T17:46:34.369 INFO:scrape: 2024-02-29T17:46:34.369 INFO:scrape:Failure: Command failed on smithi047 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 07c3f51e-d713-11ee-95c3-87774f69a715 -e sha1=7afccd060f911ee22232d7791b04834d5d886a24 -- bash -c 'ceph orch ps'" 2024-02-29T17:46:34.369 INFO:scrape:3 jobs: ['7576614', '7576629', '7576650'] 2024-02-29T17:46:34.369 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}'] 2024-02-29T17:46:34.369 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client', '3-inline/no', '3-inline/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/no', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/reef/{v18.2.0}'] 2024-02-29T17:46:34.369 INFO:scrape: 2024-02-29T17:46:34.369 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-02-29T17:46:34.369 INFO:scrape:5 jobs: ['7576627', '7576639', '7576632', '7576606', '7576618'] 2024-02-29T17:46:34.370 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'session_timeout}', 'standby-replay'] 2024-02-29T17:46:34.370 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/no', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/suites/fsstress}}', '5-workunit/suites/fsx}}', '5-workunit/suites/iogen}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}', 'wsync/yes}'] 2024-02-29T17:46:34.370 INFO:scrape: 2024-02-29T17:46:34.370 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) 2024-02-29T17:46:34.370 INFO:scrape:1 jobs: ['7576656'] 2024-02-29T17:46:34.370 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/acls}'] 2024-02-29T17:46:34.370 INFO:scrape: 2024-02-29T17:46:34.370 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=105e28e68c1101532371a3ec93f10a52312c6311 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-02-29T17:46:34.371 INFO:scrape:1 jobs: ['7576624'] 2024-02-29T17:46:34.371 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}'] 2024-02-29T17:46:34.371 INFO:scrape: 2024-02-29T17:46:34.371 INFO:scrape:Failure: Command failed on smithi055 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" 2024-02-29T17:46:34.371 INFO:scrape:1 jobs: ['7576612'] 2024-02-29T17:46:34.371 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '4-flush/yes', '5-workunit/suites/iozone}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{balancer/automatic', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2024-02-29T17:46:34.371 INFO:scrape: 2024-02-29T17:46:34.371 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2024-02-29T17:46:34.371 INFO:scrape:1 jobs: ['7576621'] 2024-02-29T17:46:34.372 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}'] 2024-02-29T17:46:34.372 INFO:scrape: 2024-02-29T17:46:34.372 INFO:scrape:Failure: Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) 2024-02-29T17:46:34.372 INFO:scrape:1 jobs: ['7576613'] 2024-02-29T17:46:34.372 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/multifs-auth}'] 2024-02-29T17:46:34.372 INFO:scrape: 2024-02-29T17:46:34.372 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi102.front.sepia.ceph.com: ['type=AVC msg=audit(1709217850.305:208): avc: denied { checkpoint_restore } for pid=1201 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-02-29T17:46:34.372 INFO:scrape:1 jobs: ['7576611'] 2024-02-29T17:46:34.372 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_pjd}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{client-shutdown', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}'] 2024-02-29T17:46:34.372 INFO:scrape: 2024-02-29T17:46:34.376 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/ failed: 17 dead: 0 running: 0 waiting: 0 queued: 0 passed: 35 Fail ================================================================= [7576606] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:29:08 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576606/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576606/ error during scrub thrashing: rank damage found: {'backtrace'} [7576609] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 01:06:24 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576609/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576609/ "2024-02-29T15:00:17.215840+0000 mon.a (mon.0) 695 : cluster [WRN] Health check failed: Degraded data redundancy: 4/324 objects degraded (1.235%), 3 pgs degraded (PG_DEGRADED)" in cluster log [7576611] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:19:14 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576611/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576611/ SELinux denials found on ubuntu@smithi102.front.sepia.ceph.com: ['type=AVC msg=audit(1709217850.305:208): avc: denied { checkpoint_restore } for pid=1201 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7576612] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} ----------------------------------------------------------------- time: 00:27:11 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576612/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576612/ Command failed on smithi055 with status 123: "find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip --" [7576613] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:14:56 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576613/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576613/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8f430c634bfc40408979d4dde8f795ea Test failure: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) [7576614] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 02:33:24 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576614/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576614/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bcc560464fd346b784664f3d46662f80 Command failed on smithi047 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 07c3f51e-d713-11ee-95c3-87774f69a715 -e sha1=7afccd060f911ee22232d7791b04834d5d886a24 -- bash -c 'ceph orch ps'" [7576618] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:42:39 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576618/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576618/ error during scrub thrashing: rank damage found: {'backtrace'} [7576621] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 01:29:39 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576621/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576621/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bb39663f81764fa28f27dcbcf4148c6c Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7576622] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 02:24:50 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576622/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576622/ "2024-02-29T15:18:24.633382+0000 mon.a (mon.0) 2140 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY)" in cluster log [7576624] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:56:02 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576624/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576624/ sentry: https://sentry.ceph.com/organizations/ceph/?query=805ca060061b479f9e4bee90c61ffc87 Command failed (workunit test suites/dbench.sh) on smithi049 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=105e28e68c1101532371a3ec93f10a52312c6311 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7576627] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:35:10 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576627/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576627/ error during scrub thrashing: rank damage found: {'backtrace'} [7576629] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 02:32:57 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576629/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576629/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ecd7d7a66264cab8843c7b5073ca608 Command failed on smithi028 with status 1: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c88a2f24-d714-11ee-95c3-87774f69a715 -e sha1=7afccd060f911ee22232d7791b04834d5d886a24 -- bash -c 'ceph orch ps'" [7576632] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} ----------------------------------------------------------------- time: 00:45:40 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576632/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576632/ error during scrub thrashing: rank damage found: {'backtrace'} [7576639] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} ----------------------------------------------------------------- time: 00:53:33 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576639/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576639/ error during scrub thrashing: rank damage found: {'backtrace'} [7576643] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} ----------------------------------------------------------------- time: 00:57:43 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576643/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576643/ "2024-02-29T16:06:18.122693+0000 mon.a (mon.0) 4704 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log [7576650] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:14:50 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576650/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576650/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2e0ad57f7d844c70b83de842c4c078b4 Command failed on smithi090 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v18.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 9d6e11ea-d717-11ee-95c3-87774f69a715 -- ceph mon dump -f json' [7576656] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/acls} ----------------------------------------------------------------- time: 00:20:43 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576656/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576656/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1ac44d7aad1042b3ba9ee49295d1398b Test failure: test_acls (tasks.cephfs.test_acls.TestACLs) Pass ================================================================= [7576605] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} time: 00:40:18 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576605/ [7576607] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:41:23 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576607/ [7576608] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:42:47 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576608/ [7576610] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:52:17 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576610/ [7576615] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} time: 00:37:58 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576615/ [7576616] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:38:24 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576616/ [7576617] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 01:43:39 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576617/ [7576619] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:58:28 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576619/ [7576620] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} time: 00:13:00 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576620/ [7576623] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:41:36 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576623/ [7576625] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 00:37:43 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576625/ [7576626] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:39:54 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576626/ [7576628] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:42:00 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576628/ [7576630] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:27:02 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576630/ [7576631] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:37:38 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576631/ [7576633] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:38:47 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576633/ [7576634] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:27:13 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576634/ [7576635] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:49:40 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576635/ [7576636] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:45:12 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576636/ [7576637] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:30:56 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576637/ [7576638] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} time: 02:34:26 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576638/ [7576640] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} time: 01:53:44 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576640/ [7576641] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 00:42:49 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576641/ [7576642] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:46:22 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576642/ [7576644] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} time: 01:08:16 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576644/ [7576645] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 00:56:21 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576645/ [7576646] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 00:50:09 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576646/ [7576647] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:17:58 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576647/ [7576648] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} time: 00:36:16 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576648/ [7576649] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/strays} time: 00:34:37 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576649/ [7576651] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:18:09 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576651/ [7576652] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:33:07 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576652/ [7576653] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} time: 00:41:34 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576653/ [7576654] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:29:56 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576654/ [7576655] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:35:25 info: https://pulpito.ceph.com/vshankar-2024-02-29_10:46:50-fs-wip-vshankar-testing1-reef-2024-02-28-1133-testing-default-smithi/7576655/