2024-03-05T07:34:36.181 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-05T07:34:36.181 INFO:teuthology.results:subset: '476/1024' 2024-03-05T07:34:36.181 INFO:teuthology.results:seed: '6115' 2024-03-05T11:12:14.541 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-05T11:12:14.561 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-05T19:50:58.101 INFO:teuthology.results:Tests finished! gathering results... 2024-03-05T19:51:15.156 INFO:scrape:Found 215 jobs 2024-03-05T19:51:19.892 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:19.893 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:19.893 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:19.893 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:19.988 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:19.988 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:19.989 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:19.989 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.100 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.100 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.100 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.100 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.196 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.196 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.196 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.196 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.285 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.285 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.286 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.286 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.375 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.375 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.375 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.376 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.467 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.468 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.468 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.468 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.559 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.559 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.559 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:20.559 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.305 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.306 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.306 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.306 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.306 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.306 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.306 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.306 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.604 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.605 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.605 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.605 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.605 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.605 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.605 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.605 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.848 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.849 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.849 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.849 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.849 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.849 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.849 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:23.849 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.094 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.094 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.094 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.094 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.094 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.094 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.094 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.095 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.339 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.340 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.340 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.340 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.340 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.340 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.340 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.340 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.584 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.584 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.584 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.584 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.584 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.584 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.584 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:24.585 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:25.341 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582020/teuthology.log 2024-03-05T19:51:32.332 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.332 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.332 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.332 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.476 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.476 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.477 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.477 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.606 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.606 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.607 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.607 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.751 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.751 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.751 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.752 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.896 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.897 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.897 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:32.897 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:33.029 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:33.029 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:33.029 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:33.029 WARNING:scrape:Saw end of BT but not start 2024-03-05T19:51:33.327 WARNING:scrape:Misunderstood line: 2024-03-05T19:51:43.149 WARNING:scrape:Misunderstood line: 2024-03-05T19:51:44.140 INFO:scrape:Found 16 distinct failure reasons 2024-03-05T19:51:44.140 INFO:scrape:Failure: Command failed on smithi119 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' 2024-03-05T19:51:44.140 INFO:scrape:6 jobs: ['7582115', '7582106', '7582128', '7582129', '7582105', '7582118'] 2024-03-05T19:51:44.141 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{centos_latest}', 'mds', 'mon', 'osd}'] 2024-03-05T19:51:44.141 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_pjd}}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/32bits/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/volumes/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{faked-ino', 'overrides/{frag', 'overrides/{ignorelist_health', 'session_timeout', 'subvol_versions/create_subvol_version_v1', 'tasks/asok_dump_tree}', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/cfuse_workunit_suites_fsstress}', 'tasks/volumes/{overrides', 'tasks/workunit/quota}', 'tasks/{1-thrash/mon', 'test/basic}}', 'thrashosds-health}', 'traceless/50pc}'] 2024-03-05T19:51:44.141 INFO:scrape: 2024-03-05T19:51:44.141 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi120 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=56d2ffd2ee347862a6db7c19ff55a523d3ffb973 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2024-03-05T19:51:44.141 INFO:scrape:2 jobs: ['7582140', '7582083'] 2024-03-05T19:51:44.141 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'thrashosds-health}'] 2024-03-05T19:51:44.141 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_20.04}', 'frag', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{client-shutdown', 'overrides/{distro/stock/{k-stock', 'prefetch_dirfrags/no', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/no', 'races', 'ranks/1', 'rhel_8}', 'session_timeout', 'tasks/{1-thrash/mds', 'tasks/{1-thrash/osd', 'thrashosds-health}'] 2024-03-05T19:51:44.141 INFO:scrape: 2024-03-05T19:51:44.142 INFO:scrape:Failure: SELinux denials found on ubuntu@smithi105.front.sepia.ceph.com: ['type=AVC msg=audit(1709628685.551:188): avc: denied { checkpoint_restore } for pid=1156 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1', 'type=AVC msg=audit(1709628685.551:189): avc: denied { checkpoint_restore } for pid=1159 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-05T19:51:44.142 INFO:scrape:4 jobs: ['7582027', '7581974', '7582063', '7581987'] 2024-03-05T19:51:44.142 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}'] 2024-03-05T19:51:44.142 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_suites_pjd}}', 'centos_latest', 'cluster/{1-node}', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'clusters/small-cluster', 'conf/{client', 'distro/{centos_latest}', 'frag', 'fs/bugs/client_trim_caps/{begin/{0-install', 'fs/functional/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'fs/top/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'msgr-failures/none', 'multifs', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/ignorelist_health', 'overrides/{client-shutdown', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'session_timeout', 'subvol_versions/create_subvol_version_v1', 'supported-random-distros$/{centos_latest}', 'tasks/fstop}', 'tasks/sessionmap}', 'tasks/trim-i24137}', 'tasks/{1-thrash/mds', 'thrashosds-health}'] 2024-03-05T19:51:44.142 INFO:scrape: 2024-03-05T19:51:44.142 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=56d2ffd2ee347862a6db7c19ff55a523d3ffb973 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-03-05T19:51:44.142 INFO:scrape:1 jobs: ['7582017'] 2024-03-05T19:51:44.142 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{balancer/automatic', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/yes}'] 2024-03-05T19:51:44.142 INFO:scrape: 2024-03-05T19:51:44.143 INFO:scrape:Failure: "2024-03-05T08:08:06.305411+0000 mon.a (mon.0) 322 : cluster [WRN] Health check failed: Degraded data redundancy: 29/2050 objects degraded (1.415%), 2 pgs degraded (PG_DEGRADED)" in cluster log 2024-03-05T19:51:44.143 INFO:scrape:2 jobs: ['7581981', '7581944'] 2024-03-05T19:51:44.143 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health'] 2024-03-05T19:51:44.143 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1a3s-mds-1c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'fs/snaps/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/cephfs_misc_tests}', 'tasks/workunit/snaps}'] 2024-03-05T19:51:44.143 INFO:scrape: 2024-03-05T19:51:44.143 INFO:scrape:Dead: b'2024-03-05T19:45:58.315 DEBUG:teuthology.exit:Finished running handlers' 2024-03-05T19:51:44.143 INFO:scrape:1 jobs: ['7581970'] 2024-03-05T19:51:44.143 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/data-scan}'] 2024-03-05T19:51:44.143 INFO:scrape: 2024-03-05T19:51:44.143 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'} 2024-03-05T19:51:44.144 INFO:scrape:3 jobs: ['7582135', '7582084', '7582123'] 2024-03-05T19:51:44.144 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{balancer/random', 'session_timeout}', 'standby-replay'] 2024-03-05T19:51:44.144 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', '5-workunit/fs/norstats}}', '5-workunit/suites/fsx}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-quota}', 'wsync/no}'] 2024-03-05T19:51:44.144 INFO:scrape: 2024-03-05T19:51:44.144 INFO:scrape:Failure: reached maximum tries (51) after waiting for 300 seconds 2024-03-05T19:51:44.144 INFO:scrape:5 jobs: ['7581935', '7582145', '7582107', '7581968', '7582045'] 2024-03-05T19:51:44.144 INFO:scrape:suites intersection: ['1-volume/{0-create', '2-client/fuse', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}'] 2024-03-05T19:51:44.144 INFO:scrape:suites union: ['1-ranks/1', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/no', '2-allow_standby_replay/yes', '2-client/fuse', '3-inline/no', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-verify}', '5-upgrade-with-workload', '6-verify}}', 'centos_8.stream_container_tools', 'conf/{client', 'fail_fs/yes', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'ignorelist_upgrade', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}', 'overrides/{ignorelist_health', 'pg-warn', 'roles', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-from/quincy', 'tasks/{0-from/reef/{reef}', 'tasks/{0-from/reef/{v18.2.0}'] 2024-03-05T19:51:44.145 INFO:scrape: 2024-03-05T19:51:44.145 INFO:scrape:Dead: None 2024-03-05T19:51:44.145 INFO:scrape:3 jobs: ['7582020', '7582081', '7581937'] 2024-03-05T19:51:44.145 INFO:scrape: 2024-03-05T19:51:44.145 INFO:scrape:Failure: "2024-03-05T08:32:30.072001+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log 2024-03-05T19:51:44.145 INFO:scrape:1 jobs: ['7581994'] 2024-03-05T19:51:44.145 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/forward-scrub}'] 2024-03-05T19:51:44.145 INFO:scrape: 2024-03-05T19:51:44.145 INFO:scrape:Failure: Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) 2024-03-05T19:51:44.145 INFO:scrape:1 jobs: ['7581976'] 2024-03-05T19:51:44.145 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/v1', 'osd}', 'overrides/{distro/testing/k-testing}}', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/fscrypt-common}'] 2024-03-05T19:51:44.145 INFO:scrape: 2024-03-05T19:51:44.146 INFO:scrape:Failure: Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) 2024-03-05T19:51:44.146 INFO:scrape:1 jobs: ['7582067'] 2024-03-05T19:51:44.146 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'subvol_versions/create_subvol_version_v2', 'tasks/snap-schedule}'] 2024-03-05T19:51:44.146 INFO:scrape: 2024-03-05T19:51:44.146 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable) 2024-03-05T19:51:44.146 INFO:scrape:1 jobs: ['7581948'] 2024-03-05T19:51:44.146 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/1', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}'] 2024-03-05T19:51:44.146 INFO:scrape: 2024-03-05T19:51:44.146 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi055 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=56d2ffd2ee347862a6db7c19ff55a523d3ffb973 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2024-03-05T19:51:44.147 INFO:scrape:1 jobs: ['7582089'] 2024-03-05T19:51:44.147 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs/{frag', 'test}}'] 2024-03-05T19:51:44.147 INFO:scrape: 2024-03-05T19:51:44.147 INFO:scrape:Failure: "2024-03-05T08:49:18.589847+0000 osd.5 (osd.5) 3 : cluster [WRN] OSD bench result of 123760.533104 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.5. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log 2024-03-05T19:51:44.147 INFO:scrape:1 jobs: ['7582014'] 2024-03-05T19:51:44.148 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'subvol_versions/create_subvol_version_v2', 'tasks/mds-full}'] 2024-03-05T19:51:44.148 INFO:scrape: 2024-03-05T19:51:44.148 INFO:scrape:Failure: Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) 2024-03-05T19:51:44.148 INFO:scrape:1 jobs: ['7581989'] 2024-03-05T19:51:44.148 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/ignorelist_health', 'tasks/mirror}}'] 2024-03-05T19:51:44.149 INFO:scrape: 2024-03-05T19:51:44.159 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi ================================================================= info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/ failed: 30 dead: 4 running: 0 waiting: 0 queued: 0 passed: 181 Fail ================================================================= [7581935] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:45:09 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581935/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581935/ reached maximum tries (51) after waiting for 300 seconds [7581944] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} ----------------------------------------------------------------- time: 01:17:02 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581944/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581944/ "2024-03-05T08:37:12.469918+0000 mon.a (mon.0) 465 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log [7581948] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/fsstress validater/valgrind} ----------------------------------------------------------------- time: 00:29:25 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581948/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581948/ sentry: https://sentry.ceph.com/organizations/ceph/?query=07a38d954afe4d0380946cbe7b0f7ecc saw valgrind issues [7581968] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{v18.2.0} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:34:48 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581968/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581968/ reached maximum tries (51) after waiting for 300 seconds [7581974] fs/bugs/client_trim_caps/{begin/{0-install 1-ceph 2-logrotate} centos_latest clusters/small-cluster conf/{client mds mon osd} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/trim-i24137} ----------------------------------------------------------------- time: 00:12:18 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581974/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581974/ SELinux denials found on ubuntu@smithi181.front.sepia.ceph.com: ['type=AVC msg=audit(1709624923.528:210): avc: denied { checkpoint_restore } for pid=1175 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7581976] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-common} ----------------------------------------------------------------- time: 01:29:04 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581976/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581976/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e172916e332041efa51e89c487a6fa60 Test failure: test_fscrypt_dummy_encryption_with_quick_group (tasks.cephfs.test_fscrypt.TestFscrypt) [7581981] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} ----------------------------------------------------------------- time: 01:00:10 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581981/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581981/ "2024-03-05T08:08:06.305411+0000 mon.a (mon.0) 322 : cluster [WRN] Health check failed: Degraded data redundancy: 29/2050 objects degraded (1.415%), 2 pgs degraded (PG_DEGRADED)" in cluster log [7581987] fs/top/{begin/{0-install 1-ceph 2-logrotate} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/ignorelist_health supported-random-distros$/{centos_latest} tasks/fstop} ----------------------------------------------------------------- time: 00:14:12 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581987/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581987/ SELinux denials found on ubuntu@smithi039.front.sepia.ceph.com: ['type=AVC msg=audit(1709626059.990:209): avc: denied { checkpoint_restore } for pid=1190 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7581989] fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/ignorelist_health tasks/mirror}} ----------------------------------------------------------------- time: 01:27:45 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581989/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581989/ sentry: https://sentry.ceph.com/organizations/ceph/?query=186141cc5f454ac3b1f15f6a68fe25b4 Test failure: test_cephfs_mirror_restart_sync_on_blocklist (tasks.cephfs.test_mirroring.TestMirroring) [7581994] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} ----------------------------------------------------------------- time: 00:20:30 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581994/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581994/ "2024-03-05T08:32:30.072001+0000 mds.c (mds.0) 1 : cluster [ERR] dir 0x10000000000 object missing on disk; some files may be lost (/dir)" in cluster log [7582014] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/mds-full} ----------------------------------------------------------------- time: 00:23:00 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582014/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582014/ "2024-03-05T08:49:18.589847+0000 osd.5 (osd.5) 3 : cluster [WRN] OSD bench result of 123760.533104 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.5. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log [7582017] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:53:24 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582017/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582017/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0c125e6d026e4797ba744ac2d81d8fdd Command failed (workunit test suites/dbench.sh) on smithi055 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=56d2ffd2ee347862a6db7c19ff55a523d3ffb973 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7582027] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:16:25 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582027/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582027/ SELinux denials found on ubuntu@smithi105.front.sepia.ceph.com: ['type=AVC msg=audit(1709628685.551:188): avc: denied { checkpoint_restore } for pid=1156 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1', 'type=AVC msg=audit(1709628685.551:189): avc: denied { checkpoint_restore } for pid=1159 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7582045] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:51:52 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582045/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582045/ reached maximum tries (51) after waiting for 300 seconds [7582063] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/sessionmap} ----------------------------------------------------------------- time: 00:18:40 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582063/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582063/ SELinux denials found on ubuntu@smithi171.front.sepia.ceph.com: ['type=AVC msg=audit(1709631053.111:209): avc: denied { checkpoint_restore } for pid=1205 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] [7582067] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snap-schedule} ----------------------------------------------------------------- time: 01:54:14 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582067/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582067/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c2b6ceb145f841abb5090d72c358a10e Test failure: test_snap_schedule_subvol_and_group_arguments_01 (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSubvolAndGroupArguments) [7582083] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:27:11 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582083/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582083/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1a84cc22cc5d4accade45d92b1232b04 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi100 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=56d2ffd2ee347862a6db7c19ff55a523d3ffb973 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7582084] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}} ----------------------------------------------------------------- time: 01:48:37 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582084/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582084/ error during scrub thrashing: rank damage found: {'backtrace'} [7582089] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_20.04} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} ----------------------------------------------------------------- time: 00:32:43 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582089/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582089/ sentry: https://sentry.ceph.com/organizations/ceph/?query=108f5f75c25545a19bedfb7fa7533f61 Command failed (workunit test libcephfs/test.sh) on smithi055 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=56d2ffd2ee347862a6db7c19ff55a523d3ffb973 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7582105] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/workunit/quota} ----------------------------------------------------------------- time: 00:07:42 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582105/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582105/ sentry: https://sentry.ceph.com/organizations/ceph/?query=94a7a8360433416cb0ee9dbd06b4ad32 Command failed on smithi150 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7582106] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:07:27 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582106/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582106/ sentry: https://sentry.ceph.com/organizations/ceph/?query=641d75ba7e1346a58b1c3d5f046e1f18 Command failed on smithi162 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7582107] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 01:15:39 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582107/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582107/ reached maximum tries (51) after waiting for 300 seconds [7582115] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} ----------------------------------------------------------------- time: 00:06:54 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582115/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582115/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c174b60ab42f435aa943c4160d8f3598 Command failed on smithi119 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7582118] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} ----------------------------------------------------------------- time: 00:07:23 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582118/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582118/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c4511ef4ab544002890f221a63217d2c Command failed on smithi045 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7582123] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/norstats}} ----------------------------------------------------------------- time: 00:40:30 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582123/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582123/ error during scrub thrashing: rank damage found: {'backtrace'} [7582128] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} ----------------------------------------------------------------- time: 00:06:46 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582128/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582128/ sentry: https://sentry.ceph.com/organizations/ceph/?query=50b0d1118ad5400faeef7f5104e7326f Command failed on smithi143 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7582129] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/asok_dump_tree} ----------------------------------------------------------------- time: 00:06:57 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582129/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582129/ sentry: https://sentry.ceph.com/organizations/ceph/?query=43d711dec9bb4a8f859aa43ccfce5536 Command failed on smithi122 with status 1: 'sudo yum -y install ceph- radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph- mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph- mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd- fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf- devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio- devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql- server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql- contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid- devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib' [7582135] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} ----------------------------------------------------------------- time: 00:37:52 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582135/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582135/ error during scrub thrashing: rank damage found: {'backtrace'} [7582140] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} ----------------------------------------------------------------- time: 00:31:36 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582140/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582140/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3ee757c18a2046d6a254301b11cd32a5 Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi120 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=56d2ffd2ee347862a6db7c19ff55a523d3ffb973 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git- ceph.sh' [7582145] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} ----------------------------------------------------------------- time: 00:46:03 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582145/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582145/ reached maximum tries (51) after waiting for 300 seconds Dead ================================================================= [7581937] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581937/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581937/ Error reimaging machines: reached maximum tries (241) after waiting for 3600 seconds [7581970] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/data-scan} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581970/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581970/ hit max job timeout [7582020] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582020/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582020/ Error reimaging machines: reached maximum tries (241) after waiting for 3600 seconds [7582081] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/dbench}}} ----------------------------------------------------------------- time: 00:00:00 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582081/ log: http://qa-proxy.ceph.com/teuthology/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582081/ Error reimaging machines: reached maximum tries (241) after waiting for 3600 seconds Pass ================================================================= [7581936] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/auto-repair} time: 00:15:40 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581936/ [7581938] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-pjd} time: 00:20:06 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581938/ [7581939] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:17:46 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581939/ [7581940] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/backtrace} time: 00:18:28 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581940/ [7581941] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsx}} time: 00:44:26 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581941/ [7581942] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:17:39 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581942/ [7581943] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:51:51 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581943/ [7581945] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}} time: 00:24:48 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581945/ [7581946] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/cap-flush} time: 00:16:44 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581946/ [7581947] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:45:32 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581947/ [7581949] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:38:05 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581949/ [7581950] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}} time: 02:53:24 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581950/ [7581951] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 01:02:48 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581951/ [7581952] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-limits} time: 00:27:27 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581952/ [7581953] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:55:46 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581953/ [7581954] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:21:11 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581954/ [7581955] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:19:11 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581955/ [7581956] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:35:51 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581956/ [7581957] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:51:25 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581957/ [7581958] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc} time: 00:36:08 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581958/ [7581959] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:19:04 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581959/ [7581960] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/client-readahead} time: 00:17:08 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581960/ [7581961] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iogen}} time: 00:39:43 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581961/ [7581962] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/ffsb}} time: 00:23:37 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581962/ [7581963] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/client-recovery} time: 00:54:02 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581963/ [7581964] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:41:49 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581964/ [7581965] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iozone}} time: 00:34:25 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581965/ [7581966] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/damage} time: 00:31:12 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581966/ [7581967] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:29:35 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581967/ [7581969] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} time: 00:38:32 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581969/ [7581971] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/fsstress}} time: 00:24:07 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581971/ [7581972] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/exports} time: 00:49:18 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581972/ [7581973] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:19:38 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581973/ [7581975] fs/cephadm/multivolume/{0-start 1-mount 2-workload/dbench distro/single-container-host} time: 01:08:45 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581975/ [7581977] fs/full/{begin/{0-install 1-ceph 2-logrotate} clusters/1-node-1-mds-1-osd conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mgr-osd-full} time: 00:24:30 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581977/ [7581978] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client} time: 00:30:54 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581978/ [7581979] fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distros$/{ubuntu_20.04} tasks/mirror} time: 02:15:09 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581979/ [7581980] fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/{1-volume-create-rm 2-three-per-cluster} clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{ignorelist_health} supported-random-distro$/{ubuntu_20.04} workloads/cephfs-mirror-ha-workunit} time: 00:34:49 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581980/ [7581982] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:51:18 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581982/ [7581983] fs/nfs/{cluster/{1-node} overrides/ignorelist_health supported-random-distros$/{ubuntu_latest} tasks/nfs} time: 00:40:58 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581983/ [7581984] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:23:24 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581984/ [7581985] fs/shell/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/cephfs-shell} time: 00:48:59 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581985/ [7581986] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:24:41 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581986/ [7581988] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:32 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581988/ [7581990] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} time: 02:06:40 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581990/ [7581991] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/direct_io}} time: 00:27:56 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581991/ [7581992] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}} time: 00:19:50 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581992/ [7581993] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:39 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581993/ [7581995] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} time: 01:11:14 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581995/ [7581996] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:19:53 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581996/ [7581997] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:13:31 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581997/ [7581998] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/dbench validater/lockdep} time: 00:49:02 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581998/ [7581999] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/fragment} time: 00:21:15 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7581999/ [7582000] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/iozone}} time: 00:18:39 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582000/ [7582001] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/kernel_untar_build}} time: 02:05:42 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582001/ [7582002] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}} time: 00:13:42 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582002/ [7582003] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:47:33 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582003/ [7582004] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc} time: 00:22:42 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582004/ [7582005] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/journal-repair} time: 00:25:07 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582005/ [7582006] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}} time: 01:53:57 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582006/ [7582007] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} time: 00:50:39 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582007/ [7582008] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds-flush} time: 00:12:32 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582008/ [7582009] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} time: 00:17:09 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582009/ [7582010] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} time: 00:42:13 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582010/ [7582011] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:38 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582011/ [7582012] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:17:03 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582012/ [7582013] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:02 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582013/ [7582015] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone}} time: 01:02:14 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582015/ [7582016] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:26:09 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582016/ [7582018] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/mds_creation_retry} time: 00:15:33 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582018/ [7582019] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:15:57 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582019/ [7582021] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:02:08 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582021/ [7582022] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}} time: 01:38:00 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582022/ [7582023] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/fsstress}}} time: 00:20:18 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582023/ [7582024] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} time: 01:01:36 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582024/ [7582025] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/metrics} time: 00:38:55 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582025/ [7582026] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:38:48 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582026/ [7582028] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} time: 01:10:23 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582028/ [7582029] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/multimds_misc} time: 00:25:54 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582029/ [7582030] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/norstats}} time: 00:38:38 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582030/ [7582031] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:27:24 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582031/ [7582032] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/ffsb}} time: 00:29:50 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582032/ [7582033] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/openfiletable} time: 00:17:38 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582033/ [7582034] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsstress}} time: 00:30:30 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582034/ [7582035] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} time: 01:31:00 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582035/ [7582036] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress} time: 00:19:31 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582036/ [7582037] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/ino_release_cb} time: 00:12:11 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582037/ [7582038] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:20:06 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582038/ [7582039] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/fsstress}} time: 00:23:27 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582039/ [7582040] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:46:00 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582040/ [7582041] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} time: 00:50:13 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582041/ [7582042] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/pool-perm} time: 00:13:48 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582042/ [7582043] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}} time: 00:13:15 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582043/ [7582044] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} time: 00:39:10 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582044/ [7582046] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}} time: 01:08:01 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582046/ [7582047] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quota} time: 00:21:39 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582047/ [7582048] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 01:26:19 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582048/ [7582049] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/fsync-tester}} time: 00:33:12 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582049/ [7582050] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/trivial_sync}} time: 00:17:38 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582050/ [7582051] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/recovery-fs} time: 00:16:32 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582051/ [7582052] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/kernel_untar_build}}} time: 02:41:40 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582052/ [7582053] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/test_o_trunc}} time: 00:38:51 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582053/ [7582054] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest} time: 00:16:34 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582054/ [7582055] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:50:24 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582055/ [7582056] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:51:52 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582056/ [7582057] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}} time: 00:27:55 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582057/ [7582058] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/scrub} time: 00:32:53 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582058/ [7582059] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 01:19:23 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582059/ [7582060] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} time: 00:57:17 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582060/ [7582061] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:53:02 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582061/ [7582062] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/osd 2-workunit/suites/iozone}} time: 00:25:37 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582062/ [7582064] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:42:31 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582064/ [7582065] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:42:36 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582065/ [7582066] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/iozone}} time: 00:38:13 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582066/ [7582068] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/iozone}} time: 00:16:54 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582068/ [7582069] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-ffsb} time: 01:59:23 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582069/ [7582070] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}} time: 00:44:17 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582070/ [7582071] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/snap_schedule_snapdir} time: 00:14:53 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582071/ [7582072] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-workunit/suites/pjd}} time: 00:32:35 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582072/ [7582073] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/suites/pjd}} time: 00:38:19 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582073/ [7582074] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/snapshots} time: 00:40:49 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582074/ [7582075] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_dbench_iozone} time: 00:49:03 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582075/ [7582076] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_snaptests}} time: 01:46:10 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582076/ [7582077] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_ffsb traceless/50pc} time: 00:40:56 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582077/ [7582078] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:32:45 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582078/ [7582079] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/valgrind} time: 00:57:51 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582079/ [7582080] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/direct_io}} time: 00:30:00 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582080/ [7582082] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/strays} time: 00:34:23 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582082/ [7582085] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/test_journal_migration} time: 00:25:50 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582085/ [7582086] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} time: 00:39:46 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582086/ [7582087] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:21:36 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582087/ [7582088] fs/cephadm/renamevolume/{0-start 1-rename distro/single-container-host overrides/ignorelist_health} time: 00:45:51 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582088/ [7582090] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests} time: 01:00:36 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582090/ [7582091] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth} time: 00:48:11 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582091/ [7582092] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:18 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582092/ [7582093] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 01:31:35 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582093/ [7582094] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/kernel_untar_build}} time: 00:50:37 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582094/ [7582095] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:39:54 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582095/ [7582096] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/truncate_delay} time: 00:23:10 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582096/ [7582097] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_suites_fsstress}} time: 00:29:00 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582097/ [7582098] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:37:44 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582098/ [7582099] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} time: 00:34:59 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582099/ [7582100] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}} time: 00:59:07 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582100/ [7582101] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/workunit/dir-max-entries} time: 00:16:50 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582101/ [7582102] fs/upgrade/nofs/{bluestore-bitmap centos_8 conf/{client mds mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-pacific 1-upgrade}} time: 00:13:42 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582102/ [7582103] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}} time: 00:25:12 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582103/ [7582104] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} time: 00:49:06 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582104/ [7582108] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} time: 01:12:08 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582108/ [7582109] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/acls} time: 00:15:10 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582109/ [7582110] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-iozone} time: 02:22:59 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582110/ [7582111] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-comp-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mds 2-workunit/fs/trivial_sync}} time: 00:17:23 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582111/ [7582112] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/nautilus 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/iozone}}} time: 00:18:48 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582112/ [7582113] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/admin} time: 01:15:45 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582113/ [7582114] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} time: 00:39:55 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582114/ [7582116] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc} time: 00:23:27 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582116/ [7582117] fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench} time: 00:47:09 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582117/ [7582119] fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} time: 00:23:31 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582119/ [7582120] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_trivial_sync}} time: 00:16:28 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582120/ [7582121] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/lockdep} time: 00:21:34 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582121/ [7582122] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/alternate-pool} time: 00:18:04 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582122/ [7582124] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}} time: 00:18:54 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582124/ [7582125] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file} time: 00:14:38 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582125/ [7582126] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover} time: 00:54:05 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582126/ [7582127] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} time: 00:51:31 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582127/ [7582130] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/reef/{reef} 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} time: 00:40:37 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582130/ [7582131] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}} time: 00:32:44 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582131/ [7582132] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/pjd}} time: 00:18:44 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582132/ [7582133] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} time: 00:24:10 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582133/ [7582134] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/auto-repair} time: 00:16:08 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582134/ [7582136] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/ffsb}} time: 00:28:08 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582136/ [7582137] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/backtrace} time: 00:18:42 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582137/ [7582138] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/stress_tests/{0-client-upgrade 1-tests/blogbench}}} time: 00:24:05 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582138/ [7582139] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}} time: 00:29:50 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582139/ [7582141] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:16:45 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582141/ [7582142] fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python} time: 00:19:19 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582142/ [7582143] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} time: 00:18:25 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582143/ [7582144] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/cap-flush} time: 00:18:11 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582144/ [7582146] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/test_o_trunc}} time: 00:41:01 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582146/ [7582147] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}} time: 00:18:44 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582147/ [7582148] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v1 tasks/client-limits} time: 00:25:34 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582148/ [7582149] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} time: 00:32:53 info: https://pulpito.ceph.com/vshankar-2024-03-05_07:34:10-fs-wip-vshankar-testing1-reef-2024-03-05-1017-testing-default-smithi/7582149/