2023-09-07T05:44:53.008 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-07T05:44:53.009 INFO:teuthology.results:subset: '45/1024'
2023-09-07T05:44:53.009 INFO:teuthology.results:seed: '8546'
2023-09-07T06:23:04.768 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc
2023-09-07T06:23:04.772 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-09-07T18:23:57.305 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-09-07T18:24:03.075 INFO:scrape:Found 72 jobs
2023-09-07T18:24:06.088 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390528/teuthology.log
2023-09-07T18:24:15.038 INFO:scrape:Found 18 distinct failure reasons
2023-09-07T18:24:15.039 INFO:scrape:Dead: b'2023-09-07T18:24:02.296 INFO:journalctl@ceph.mon.smithi131.smithi131.stdout:Sep 07 18:24:01 smithi131 ceph-mon[165437]: pgmap v20648: 65 pgs: 65 active+clean; 319 MiB data, 1.2 GiB used, 535 GiB / 536 GiB avail'
2023-09-07T18:24:15.040 INFO:scrape:2 jobs: ['7390578', '7390579']
2023-09-07T18:24:15.040 INFO:scrape:suites intersection: ['conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'osd}']
2023-09-07T18:24:15.040 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '1-ranks/2', '1-volume/{0-create', '2-allow_standby_replay/yes', '2-client', '2-logrotate}', '2-scrub/yes', '3-inline/yes', '3-snaps/yes', '3-upgrade-mgr-staggered', '4-config-upgrade/{fail_fs}', '4-flush/yes', '4-verify}', '5-upgrade-with-workload', '5-workunit/suites/blogbench}}', '6-verify}}', 'begin/{0-install', 'centos_8.stream_container_tools', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fail_fs/yes', 'frag', 'fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/5', 'objectstore-ec/bluestore-comp', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'pg-warn', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'roles', 'session_timeout}', 'standby-replay', 'syntax}', 'tasks/{0-from/pacific', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'wsync/no}']
2023-09-07T18:24:15.041 INFO:scrape:
2023-09-07T18:24:15.041 INFO:scrape:Dead: b'2023-09-07T17:57:49.677 DEBUG:teuthology.exit:Finished running handlers'
2023-09-07T18:24:15.042 INFO:scrape:4 jobs: ['7390519', '7390509', '7390538', '7390516']
2023-09-07T18:24:15.042 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'osd}']
2023-09-07T18:24:15.042 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-2c-client', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'frag', 'fs/functional/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'no_client_pidfile}', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'objectstore/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/always}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/multifs-auth}', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/no}']
2023-09-07T18:24:15.043 INFO:scrape:
2023-09-07T18:24:15.043 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)
2023-09-07T18:24:15.044 INFO:scrape:1 jobs: ['7390573']
2023-09-07T18:24:15.044 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/xfstests-dev}']
2023-09-07T18:24:15.044 INFO:scrape:
2023-09-07T18:24:15.045 INFO:scrape:Failure: "2023-09-07T06:26:24.133921+0000 mon.a (mon.0) 127 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log
2023-09-07T18:24:15.045 INFO:scrape:2 jobs: ['7390520', '7390545']
2023-09-07T18:24:15.046 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}']
2023-09-07T18:24:15.046 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'cephfs-mirror/one-per-cluster', 'clients/{mirror}', 'cluster/{1-node}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/mirror/{begin/{0-install', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'overrides/{whitelist_health}', 'supported-random-distros$/{ubuntu_latest}', 'tasks/failover}', 'tasks/mirror}']
2023-09-07T18:24:15.046 INFO:scrape:
2023-09-07T18:24:15.047 INFO:scrape:Failure: Command failed on smithi144 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-09-07T18:24:15.047 INFO:scrape:4 jobs: ['7390544', '7390508', '7390571', '7390537']
2023-09-07T18:24:15.048 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health']
2023-09-07T18:24:15.048 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}']
2023-09-07T18:24:15.048 INFO:scrape:
2023-09-07T18:24:15.049 INFO:scrape:Failure: [Errno None] Unable to connect to port 22 on 172.21.15.44
2023-09-07T18:24:15.049 INFO:scrape:1 jobs: ['7390561']
2023-09-07T18:24:15.050 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount/fuse', 'objectstore-ec/bluestore-comp-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2023-09-07T18:24:15.050 INFO:scrape:
2023-09-07T18:24:15.050 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-09-07T18:24:15.051 INFO:scrape:1 jobs: ['7390552']
2023-09-07T18:24:15.051 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{centos_8}', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{ignorelist_health', 'ranks/5', 'session_timeout}', 'tasks/fsstress', 'validater/valgrind}']
2023-09-07T18:24:15.052 INFO:scrape:
2023-09-07T18:24:15.052 INFO:scrape:Dead: None
2023-09-07T18:24:15.052 INFO:scrape:7 jobs: ['7390528', '7390562', '7390564', '7390522', '7390523', '7390567', '7390560']
2023-09-07T18:24:15.053 INFO:scrape:
2023-09-07T18:24:15.053 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi161 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2023-09-07T18:24:15.053 INFO:scrape:2 jobs: ['7390556', '7390532']
2023-09-07T18:24:15.054 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-09-07T18:24:15.054 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-09-07T18:24:15.055 INFO:scrape:
2023-09-07T18:24:15.055 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi007 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'
2023-09-07T18:24:15.055 INFO:scrape:1 jobs: ['7390575']
2023-09-07T18:24:15.056 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}']
2023-09-07T18:24:15.056 INFO:scrape:
2023-09-07T18:24:15.057 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi158 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2 CEPH_MNT=/home/ubuntu/cephtest/mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh'
2023-09-07T18:24:15.057 INFO:scrape:2 jobs: ['7390518', '7390576']
2023-09-07T18:24:15.057 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-09-07T18:24:15.058 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-09-07T18:24:15.058 INFO:scrape:
2023-09-07T18:24:15.059 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
2023-09-07T18:24:15.059 INFO:scrape:1 jobs: ['7390558']
2023-09-07T18:24:15.059 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/quota}']
2023-09-07T18:24:15.060 INFO:scrape:
2023-09-07T18:24:15.060 INFO:scrape:Failure: Test failure: test_cephfs_mirror_cancel_mirroring_and_readd (tasks.cephfs.test_mirroring.TestMirroring)
2023-09-07T18:24:15.061 INFO:scrape:1 jobs: ['7390525']
2023-09-07T18:24:15.061 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2023-09-07T18:24:15.061 INFO:scrape:
2023-09-07T18:24:15.062 INFO:scrape:Failure: "2023-09-07T06:26:31.883634+0000 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of 141157.319692 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.0. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log
2023-09-07T18:24:15.062 INFO:scrape:1 jobs: ['7390548']
2023-09-07T18:24:15.062 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-full}']
2023-09-07T18:24:15.063 INFO:scrape:
2023-09-07T18:24:15.063 INFO:scrape:Failure: failed to install new kernel version within timeout
2023-09-07T18:24:15.064 INFO:scrape:1 jobs: ['7390566']
2023-09-07T18:24:15.064 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/no', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}']
2023-09-07T18:24:15.064 INFO:scrape:
2023-09-07T18:24:15.065 INFO:scrape:Failure: error during scrub thrashing: rank damage found: {'backtrace'}
2023-09-07T18:24:15.065 INFO:scrape:1 jobs: ['7390511']
2023-09-07T18:24:15.065 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/yes', '4-flush/yes', '5-workunit/fs/misc}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'objectstore-ec/bluestore-bitmap', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'ranks/multi/{export-check', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/no}']
2023-09-07T18:24:15.068 INFO:scrape:
2023-09-07T18:24:15.069 INFO:scrape:Failure: SSH connection to smithi044 was lost: 'rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.5.0_rc7_g533e5fc8ca06-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/533e5fc8ca061dbf84e56917eecb852edf10a398/centos/8/flavors/default/x86_64/ --input-file=-'
2023-09-07T18:24:15.069 INFO:scrape:1 jobs: ['7390565']
2023-09-07T18:24:15.070 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/suites/iozone}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-09-07T18:24:15.070 INFO:scrape:
2023-09-07T18:24:15.070 INFO:scrape:Failure: machine smithi098.front.sepia.ceph.com is locked by scheduled_nmordech@teuthology, not scheduled_vshankar@teuthology
2023-09-07T18:24:15.071 INFO:scrape:1 jobs: ['7390534']
2023-09-07T18:24:15.071 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_trivial_sync}}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'multifs', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'session_timeout', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-09-07T18:24:15.072 INFO:scrape:
2023-09-07T18:24:15.076 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/
failed:  20
dead:    12
running: 2
waiting: 0
queued:  0
passed:  38


Fail
=================================================================
[7390508]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:17:35
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390508/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390508/
sentry: https://sentry.ceph.com/organizations/ceph/?query=e878d992ccbd4b66ba7b82c3155ee934

    Command failed on smithi137 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7390511]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/fs/misc}}
-----------------------------------------------------------------
time:   00:56:40
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390511/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390511/

    error during scrub thrashing: rank damage found: {'backtrace'}

[7390518]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:19:34
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390518/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390518/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3d62ab9fe1eb4239a81990c503f353ae

    Command failed (workunit test fs/quota/quota.sh) on smithi158 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd --
    /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2
    CEPH_MNT=/home/ubuntu/cephtest/mnt.2 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh'

[7390520]  fs/mirror/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/one-per-cluster clients/{mirror} cluster/{1-node} mount/fuse objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distros$/{ubuntu_latest} tasks/mirror}
-----------------------------------------------------------------
time:   02:10:01
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390520/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390520/

    "2023-09-07T06:26:24.133921+0000 mon.a (mon.0) 127 : cluster [WRN] Health
    check failed: 2 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7390525]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   00:42:21
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390525/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390525/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d22293d1e27549ba84155c77b227ef12

    Test failure: test_cephfs_mirror_cancel_mirroring_and_readd
    (tasks.cephfs.test_mirroring.TestMirroring)

[7390532]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:28:55
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390532/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390532/
sentry: https://sentry.ceph.com/organizations/ceph/?query=791c5bd5853d404ab03b525c9da62d30

    Command failed (workunit test kernel_untar_build.sh) on smithi119 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7390534]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_trivial_sync}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390534/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390534/
sentry: https://sentry.ceph.com/organizations/ceph/?query=b534708cb51c48b09f48c8726ad7e35f

    machine smithi098.front.sepia.ceph.com is locked by
    scheduled_nmordech@teuthology, not scheduled_vshankar@teuthology

[7390537]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:17:01
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390537/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390537/
sentry: https://sentry.ceph.com/organizations/ceph/?query=479c016ad0bf457ba588a7c143d32305

    Command failed on smithi123 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7390544]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:15:33
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390544/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390544/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9c63b7eedb48445eaba5bc6e352b9123

    Command failed on smithi144 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7390545]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   01:15:58
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390545/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390545/

    "2023-09-07T06:34:31.934836+0000 mon.a (mon.0) 4526 : cluster [WRN] Health
    check failed: Reduced data availability: 1 pg inactive, 1 pg peering
    (PG_AVAILABILITY)" in cluster log

[7390548]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
-----------------------------------------------------------------
time:   00:26:26
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390548/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390548/

    "2023-09-07T06:26:31.883634+0000 osd.0 (osd.0) 3 : cluster [WRN] OSD bench
    result of 141157.319692 IOPS exceeded the threshold limit of 500.000000
    IOPS for osd.0. IOPS capacity is unchanged at 315.000000 IOPS. The
    recommendation is to establish the osd's IOPS capacity using other
    benchmark tools (e.g. Fio) and then override
    osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log

[7390552]  fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind}
-----------------------------------------------------------------
time:   06:39:57
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390552/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390552/
sentry: https://sentry.ceph.com/organizations/ceph/?query=31ac5731c3d24148a9377d2193b90e58

    Command failed (workunit test suites/fsstress.sh) on smithi046 with status
    124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

[7390556]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:26:27
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390556/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390556/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9883c437b01343aab85fc4a6a382d794

    Command failed (workunit test kernel_untar_build.sh) on smithi161 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7390558]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
-----------------------------------------------------------------
time:   00:18:33
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390558/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390558/
sentry: https://sentry.ceph.com/organizations/ceph/?query=eddd1b401c004760972e7e87a10db177

    Test failure: test_disable_enable_human_readable_quota_values
    (tasks.cephfs.test_quota.TestQuota)

[7390561]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390561/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390561/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0b3d9bad352749c9825e2ff44572ad4b

    [Errno None] Unable to connect to port 22 on 172.21.15.44

[7390566]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/pjd}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390566/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390566/
sentry: https://sentry.ceph.com/organizations/ceph/?query=6f5cc7495d73437c9255c9574d216660

    failed to install new kernel version within timeout

[7390571]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:15:37
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390571/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390571/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3cf7465927694e3da9b28c7457a5068a

    Command failed on smithi099 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7390573]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   08:12:38
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390573/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390573/
sentry: https://sentry.ceph.com/organizations/ceph/?query=00ec1c800b66441eb759abdc253e2be9

    Test failure: test_generic
    (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)

[7390575]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{centos_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
-----------------------------------------------------------------
time:   00:17:55
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390575/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390575/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9c26380a4a0d41529d368f64c0d9fd03

    Command failed (workunit test fs/test_python.sh) on smithi007 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'

[7390576]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:19:08
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390576/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390576/
sentry: https://sentry.ceph.com/organizations/ceph/?query=8cea3beffe92435f96404719af04bf95

    Command failed (workunit test fs/quota/quota.sh) on smithi116 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd --
    /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=6c8583a87a5de28b8b0ab93d1f128c1a38a1bf71
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2
    CEPH_MNT=/home/ubuntu/cephtest/mnt.2 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh'



Dead
=================================================================
[7390509]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390509/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390509/

    hit max job timeout

[7390516]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390516/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390516/

    hit max job timeout

[7390519]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/postgres}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390519/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390519/

    hit max job timeout

[7390522]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390522/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390522/

    Error reimaging machines: Expected smithi044's OS to be centos 9 but found
    ubuntu 22.04

[7390523]  fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390523/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390523/

    Error reimaging machines: list index out of range

[7390528]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/mon 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390528/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390528/

    Error reimaging machines: reached maximum tries (101) after waiting for 600
    seconds

[7390538]  fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390538/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390538/

    hit max job timeout

[7390560]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390560/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390560/

    Error reimaging machines: Expected smithi044's OS to be ubuntu 22.04 but
    found rhel 8.6

[7390562]  fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390562/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390562/

    Error reimaging machines: Expected smithi044's OS to be centos 9 but found
    centos 8

[7390564]  fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390564/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390564/

    Error reimaging machines: list index out of range

[7390565]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/none objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mon 2-workunit/suites/iozone}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390565/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390565/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c0023f4b45bd459588128c106a1a6118

    SSH connection to smithi044 was lost: 'rm -f /tmp/kernel.x86_64.rpm && echo
    kernel-6.5.0_rc7_g533e5fc8ca06-1.x86_64.rpm | wget -nv -O
    /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/53
    3e5fc8ca061dbf84e56917eecb852edf10a398/centos/8/flavors/default/x86_64/
    --input-file=-'

[7390567]  fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/finisher_per_module}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390567/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390567/

    Error reimaging machines: reached maximum tries (101) after waiting for 600
    seconds



Running
=================================================================
[7390578] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390578/

[7390579] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/blogbench}}
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390579/



Pass
=================================================================
[7390510] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:38:02
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390510/

[7390512] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/iozone}}
time:   00:20:51
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390512/

[7390513] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:16:42
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390513/

[7390514] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_misc}
time:   00:21:37
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390514/

[7390515] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:16:55
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390515/

[7390517] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
time:   01:29:21
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390517/

[7390521] fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cephfs_misc_tests}
time:   01:02:23
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390521/

[7390524] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   01:01:46
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390524/

[7390526] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:33:25
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390526/

[7390527] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/alternate-pool}
time:   00:10:53
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390527/

[7390529] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-iozone}}
time:   02:18:59
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390529/

[7390530] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/misc}}
time:   00:20:17
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390530/

[7390531] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}}
time:   01:07:52
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390531/

[7390533] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
time:   00:45:17
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390533/

[7390535] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
time:   00:10:35
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390535/

[7390536] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-recovery}
time:   00:57:53
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390536/

[7390539] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/ffsb}}
time:   00:26:51
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390539/

[7390540] fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}
time:   00:36:44
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390540/

[7390541] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsstress}}
time:   00:36:03
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390541/

[7390542] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}}
time:   00:38:17
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390542/

[7390543] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:39:27
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390543/

[7390546] fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps}
time:   00:41:43
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390546/

[7390547] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/fsync-tester}}
time:   00:33:26
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390547/

[7390549] fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:18:48
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390549/

[7390550] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:17:51
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390550/

[7390551] fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mon 2-workunit/cfuse_workunit_suites_pjd}}
time:   00:16:01
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390551/

[7390553] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/openfiletable}
time:   00:16:38
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390553/

[7390554] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/osd 2-workunit/suites/fsstress}}
time:   00:19:03
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390554/

[7390555] fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/{0-client 1-tests/fscrypt-dbench}}
time:   01:12:08
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390555/

[7390557] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress traceless/50pc}
time:   00:21:20
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390557/

[7390559] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-bitmap overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:14:24
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390559/

[7390563] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/recovery-fs}
time:   00:10:18
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390563/

[7390568] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snapshots}
time:   00:25:02
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390568/

[7390569] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/snaps}}
time:   00:55:54
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390569/

[7390570] fs/traceless/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_blogbench traceless/50pc}
time:   00:19:20
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390570/

[7390572] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:14:06
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390572/

[7390574] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd}
time:   00:19:05
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390574/

[7390577] fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/1 tasks/{1-thrash/osd 2-workunit/fs/trivial_sync}}
time:   00:21:09
info:   http://pulpito.ceph.com/vshankar-2023-09-07_05:44:31-fs-wip-vshankar-testing-20230830.153114-testing-default-smithi/7390577/