2023-08-24T04:16:51.985 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc
2023-08-24T04:16:51.986 INFO:teuthology.results:subset: '922/1024'
2023-08-24T04:16:51.986 INFO:teuthology.results:seed: '2667'
2023-08-24T04:18:00.100 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc
2023-08-24T04:18:00.104 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-08-24T16:18:51.899 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-08-24T16:18:55.548 INFO:scrape:Found 37 jobs
2023-08-24T16:19:04.345 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378157/teuthology.log
2023-08-24T16:19:09.879 INFO:scrape:Found 13 distinct failure reasons
2023-08-24T16:19:09.880 INFO:scrape:Failure: Command failed (workunit test kernel_untar_build.sh) on smithi066 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'
2023-08-24T16:19:09.880 INFO:scrape:2 jobs: ['7378173', '7378163']
2023-08-24T16:19:09.880 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-08-24T16:19:09.881 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/mixed-clients/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'kclient-overrides/{distro/testing/k-testing', 'mds', 'mon', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd-asserts}', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'tasks/kernel_cfuse_workunits_untarbuild_blogbench}']
2023-08-24T16:19:09.881 INFO:scrape:
2023-08-24T16:19:09.881 INFO:scrape:Failure: Command failed on smithi141 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-08-24T16:19:09.882 INFO:scrape:4 jobs: ['7378154', '7378174', '7378161', '7378169']
2023-08-24T16:19:09.882 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health']
2023-08-24T16:19:09.882 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}']
2023-08-24T16:19:09.883 INFO:scrape:
2023-08-24T16:19:09.883 INFO:scrape:Failure: Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)
2023-08-24T16:19:09.883 INFO:scrape:1 jobs: ['7378177']
2023-08-24T16:19:09.883 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/acls}']
2023-08-24T16:19:09.884 INFO:scrape:
2023-08-24T16:19:09.884 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi012 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2 CEPH_MNT=/home/ubuntu/cephtest/mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh'
2023-08-24T16:19:09.884 INFO:scrape:2 jobs: ['7378176', '7378147']
2023-08-24T16:19:09.885 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-08-24T16:19:09.885 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/workunit/quota}']
2023-08-24T16:19:09.885 INFO:scrape:
2023-08-24T16:19:09.886 INFO:scrape:Failure: Command failed (workunit test suites/ffsb.sh) on smithi043 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'
2023-08-24T16:19:09.886 INFO:scrape:1 jobs: ['7378165']
2023-08-24T16:19:09.886 INFO:scrape:suites: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/ffsb}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'wsync/yes}']
2023-08-24T16:19:09.886 INFO:scrape:
2023-08-24T16:19:09.887 INFO:scrape:Failure: Test failure: test_generic (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)
2023-08-24T16:19:09.887 INFO:scrape:2 jobs: ['7378146', '7378175']
2023-08-24T16:19:09.887 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'osd}', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']
2023-08-24T16:19:09.888 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/xfstests-dev}']
2023-08-24T16:19:09.888 INFO:scrape:
2023-08-24T16:19:09.888 INFO:scrape:Dead: b'2023-08-24T16:18:40.262 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf'
2023-08-24T16:19:09.888 INFO:scrape:3 jobs: ['7378170', '7378141', '7378162']
2023-08-24T16:19:09.889 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'osd}', 'overrides/{ignorelist_health', 'tasks/multifs-auth}']
2023-08-24T16:19:09.889 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{centos_8}', 'distro/{ubuntu_20.04}', 'distro/{ubuntu_latest}', 'fs/multifs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug}', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/multifs-auth}']
2023-08-24T16:19:09.889 INFO:scrape:
2023-08-24T16:19:09.890 INFO:scrape:Failure: "2023-08-24T04:44:00.042095+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 108762.651544 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log
2023-08-24T16:19:09.890 INFO:scrape:1 jobs: ['7378164']
2023-08-24T16:19:09.890 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{centos_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/mds-full}']
2023-08-24T16:19:09.891 INFO:scrape:
2023-08-24T16:19:09.891 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi135 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'
2023-08-24T16:19:09.891 INFO:scrape:1 jobs: ['7378172']
2023-08-24T16:19:09.891 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{rhel_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}']
2023-08-24T16:19:09.892 INFO:scrape:
2023-08-24T16:19:09.892 INFO:scrape:Crash: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi059 with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2023-08-24T16:19:09.892 INFO:scrape:ceph version 18.0.0-5602-g54fd6f84 (54fd6f849d61b151541bf28aca0de4ec83da7fd3) reef (dev)
 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f28caa12ce0]
 2: (CInode::get_dirfrags() const+0x26) [0x55b2c8d50526]
 3: (MDCache::advance_stray()+0x1fa) [0x55b2c8cc106a]
 4: (MDLog::_start_new_segment(SegmentBoundary*)+0x465) [0x55b2c8f03925]
 5: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0xba) [0x55b2c8f03aba]
 6: (MDLog::submit_entry(LogEvent*, MDSLogContextBase*)+0xbf) [0x55b2c8bdc4af]
 7: (MDCache::shutdown_pass()+0xe9f) [0x55b2c8d1ae3f]
 8: (MDSRankDispatcher::tick()+0x300) [0x55b2c8b81300]
 9: (Context::complete(int)+0xd) [0x55b2c8b568fd]
 10: (CommonSafeTimer<ceph::fair_mutex>::timer_thread()+0x181) [0x7f28cc79f641]
 11: (CommonSafeTimerThread<ceph::fair_mutex>::entry()+0x11) [0x7f28cc7a0951]
 12: /lib64/libpthread.so.0(+0x81cf) [0x7f28caa081cf]
 13: clone()
2023-08-24T16:19:09.893 INFO:scrape:1 jobs: ['7378151']
2023-08-24T16:19:09.893 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/no', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/5', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-08-24T16:19:09.893 INFO:scrape:
2023-08-24T16:19:09.893 INFO:scrape:Dead: None
2023-08-24T16:19:09.894 INFO:scrape:1 jobs: ['7378157']
2023-08-24T16:19:09.894 INFO:scrape:
2023-08-24T16:19:09.894 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
2023-08-24T16:19:09.895 INFO:scrape:1 jobs: ['7378168']
2023-08-24T16:19:09.895 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/quota}']
2023-08-24T16:19:09.895 INFO:scrape:
2023-08-24T16:19:09.895 INFO:scrape:Failure: "2023-08-24T04:45:28.469772+0000 mon.a (mon.0) 123 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log
2023-08-24T16:19:09.896 INFO:scrape:1 jobs: ['7378142']
2023-08-24T16:19:09.896 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'centos_latest', 'clients/mirror', 'cluster/1-node', 'debug', 'fs/valgrind/{begin/{0-install', 'mirror/{cephfs-mirror/one-per-cluster', 'mount/fuse', 'overrides/whitelist_health', 'tasks/mirror}}']
2023-08-24T16:19:09.896 INFO:scrape:
2023-08-24T16:19:09.899 INFO:teuthology.results:Sending results to vshankar@redhat.com: Test Run: vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi
=================================================================
info:    http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/
failed:  17
dead:    1
running: 3
waiting: 0
queued:  0
passed:  16


Fail
=================================================================
[7378142]  fs/valgrind/{begin/{0-install 1-ceph 2-logrotate} centos_latest debug mirror/{cephfs-mirror/one-per-cluster clients/mirror cluster/1-node mount/fuse overrides/whitelist_health tasks/mirror}}
-----------------------------------------------------------------
time:   02:30:15
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378142/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378142/

    "2023-08-24T04:45:28.469772+0000 mon.a (mon.0) 123 : cluster [WRN] Health
    check failed: 2 pool(s) do not have an application enabled
    (POOL_APP_NOT_ENABLED)" in cluster log

[7378146]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   00:17:48
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378146/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378146/
sentry: https://sentry.ceph.com/organizations/ceph/?query=0bbd5ed7f5244293b6f17962c850f5f2

    Test failure: test_generic
    (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)

[7378147]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:18:28
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378147/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378147/
sentry: https://sentry.ceph.com/organizations/ceph/?query=912774b3faae4bf89395314cbc3441b2

    Command failed (workunit test fs/quota/quota.sh) on smithi169 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.3/client.3/tmp && cd --
    /home/ubuntu/cephtest/mnt.3/client.3/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="3"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.3
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.3
    CEPH_MNT=/home/ubuntu/cephtest/mnt.3 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.3/qa/workunits/fs/quota/quota.sh'

[7378151]  fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}}
-----------------------------------------------------------------
time:   01:33:53
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378151/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378151/
sentry: https://sentry.ceph.com/organizations/ceph/?query=d23febfa21584a0fab5e6e451cd0c3e7

    Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi059
    with status 135: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp &&
    CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 6h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-
    ceph.sh'

[7378154]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:19:01
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378154/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378154/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4a18ee528ae64c1d92f864dba00e53d3

    Command failed on smithi141 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7378161]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:18:09
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378161/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378161/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5298f650450d4f3e9e754259fc1a70a0

    Command failed on smithi071 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7378163]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:29:58
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378163/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378163/
sentry: https://sentry.ceph.com/organizations/ceph/?query=02a0518021d8436fb60f70f1cf6e3b8e

    Command failed (workunit test kernel_untar_build.sh) on smithi160 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7378164]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}
-----------------------------------------------------------------
time:   00:29:36
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378164/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378164/

    "2023-08-24T04:44:00.042095+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench
    result of 108762.651544 IOPS exceeded the threshold limit of 500.000000
    IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The
    recommendation is to establish the osd's IOPS capacity using other
    benchmark tools (e.g. Fio) and then override
    osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log

[7378165]  fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}}
-----------------------------------------------------------------
time:   00:52:54
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378165/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378165/
sentry: https://sentry.ceph.com/organizations/ceph/?query=ee03d1428c35494883abf98d153e5552

    Command failed (workunit test suites/ffsb.sh) on smithi043 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh'

[7378168]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/quota}
-----------------------------------------------------------------
time:   00:17:09
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378168/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378168/
sentry: https://sentry.ceph.com/organizations/ceph/?query=79081dc8a56a4a649ce8c03c9713913b

    Test failure: test_disable_enable_human_readable_quota_values
    (tasks.cephfs.test_quota.TestQuota)

[7378169]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/ior-shared-file}
-----------------------------------------------------------------
time:   00:18:01
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378169/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378169/
sentry: https://sentry.ceph.com/organizations/ceph/?query=cf5d23abc20246388b25505dec5113f7

    Command failed on smithi083 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7378172]  fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{rhel_8} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs_python}
-----------------------------------------------------------------
time:   00:20:45
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378172/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378172/
sentry: https://sentry.ceph.com/organizations/ceph/?query=5e14ae5a8bf44b54a080bb66cfbd39dc

    Command failed (workunit test fs/test_python.sh) on smithi135 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'

[7378173]  fs/mixed-clients/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} kclient-overrides/{distro/testing/k-testing ms-die-on-skipped} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down osd-asserts} tasks/kernel_cfuse_workunits_untarbuild_blogbench}
-----------------------------------------------------------------
time:   00:29:26
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378173/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378173/
sentry: https://sentry.ceph.com/organizations/ceph/?query=4c6eca76ca864c8da110d4eb0374c9a2

    Command failed (workunit test kernel_untar_build.sh) on smithi066 with
    status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd --
    /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1
    CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.1/qa/workunits/kernel_untar_build.sh'

[7378174]  fs/multiclient/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-2-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/mdtest}
-----------------------------------------------------------------
time:   00:18:49
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378174/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378174/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3645bab07dc04e64afeaad691ecdd001

    Command failed on smithi170 with status 2: 'TESTDIR=/home/ubuntu/cephtest
    bash -s'

[7378175]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/xfstests-dev}
-----------------------------------------------------------------
time:   08:09:44
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378175/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378175/
sentry: https://sentry.ceph.com/organizations/ceph/?query=14577a2d53104c2b9311439dc1c592ba

    Test failure: test_generic
    (tasks.cephfs.tests_from_xfstests_dev.TestXFSTestsDev)

[7378176]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}
-----------------------------------------------------------------
time:   00:20:39
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378176/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378176/
sentry: https://sentry.ceph.com/organizations/ceph/?query=34458f49b9b546238b6e57e1f14a0654

    Command failed (workunit test fs/quota/quota.sh) on smithi012 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.2/client.2/tmp && cd --
    /home/ubuntu/cephtest/mnt.2/client.2/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=54fd6f849d61b151541bf28aca0de4ec83da7fd3
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="2"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.2
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.2
    CEPH_MNT=/home/ubuntu/cephtest/mnt.2 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.2/qa/workunits/fs/quota/quota.sh'

[7378177]  fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
-----------------------------------------------------------------
time:   00:18:55
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378177/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378177/
sentry: https://sentry.ceph.com/organizations/ceph/?query=af72eef9397b43ffa9cb42bf7b4204d0

    Test failure: test_acls (tasks.cephfs.test_acls.TestACLs)



Dead
=================================================================
[7378157]  fs/thrash/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down multifs session_timeout thrashosds-health} tasks/{1-thrash/mds 2-workunit/cfuse_workunit_snaptests}}
-----------------------------------------------------------------
time:   00:00:00
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378157/
log:    http://qa-proxy.ceph.com/teuthology/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378157/

    Error reimaging machines: Expected smithi164's OS to be ubuntu 20.04 but
    found centos 8



Running
=================================================================
[7378141] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378141/

[7378162] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore-ec/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378162/

[7378170] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{ubuntu_20.04} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/multifs-auth}
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378170/



Pass
=================================================================
[7378143] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:32:04
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378143/

[7378144] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/iogen}}
time:   00:37:13
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378144/

[7378145] fs/upgrade/upgraded_client/{bluestore-bitmap centos_8 clusters/{1-mds-1-client-micro} conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} tasks/{0-from/pacific 1-mount/mount/fuse 2-workload/new_ops/{0-clients/fuse-upgrade 1-client-sanity}}}
time:   00:20:07
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378145/

[7378148] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/iozone}}
time:   00:40:37
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378148/

[7378149] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/acls}
time:   00:18:58
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378149/

[7378150] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/kclient/{k-testing mount ms-die-on-skipped} objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/1 tasks/dbench validater/lockdep}
time:   00:53:46
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378150/

[7378152] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/admin}
time:   01:19:29
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378152/

[7378153] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/pjd}}
time:   00:28:14
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378153/

[7378155] fs/multifs/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-2c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug} tasks/failover}
time:   01:16:09
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378155/

[7378156] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/direct_io}}
time:   00:42:44
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378156/

[7378158] fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}}
time:   00:39:57
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378158/

[7378159] fs/32bits/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{faked-ino ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_fsstress}
time:   00:21:32
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378159/

[7378160] fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind}
time:   01:59:36
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378160/

[7378166] fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8 clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}}
time:   00:34:37
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378166/

[7378167] fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/multimds_misc}
time:   00:25:19
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378167/

[7378171] fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/fsync-tester}}
time:   00:41:37
info:   http://pulpito.ceph.com/vshankar-2023-08-24_04:16:25-fs-wip-vshankar-testing-20230822.064807-testing-default-smithi/7378171/