2023-05-18T17:42:21.389 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-18T17:42:21.390 INFO:teuthology.results:subset: '159/256'
2023-05-18T17:42:21.390 INFO:teuthology.results:seed: '8292'
2023-05-18T20:47:03.194 INFO:root:teuthology version: 0.0.1.dev98+g69caab8
2023-05-18T20:47:03.225 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-05-19T08:47:55.514 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds
2023-05-19T08:47:59.389 INFO:scrape:Found 45 jobs
2023-05-19T08:48:06.881 WARNING:scrape:Misunderstood line: 
2023-05-19T08:48:33.942 INFO:scrape:Found 17 distinct failure reasons
2023-05-19T08:48:33.943 INFO:scrape:Failure: Command failed on smithi131 with status 2: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2023-05-19T08:48:33.944 INFO:scrape:4 jobs: ['7277446', '7277474', '7277450', '7277465']
2023-05-19T08:48:33.944 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'osd}', 'overrides/{ignorelist_health']
2023-05-19T08:48:33.944 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1-mds-2-client', 'clusters/1-mds-3-client', 'conf/{client', 'distros/ubuntu_latest', 'fs/multiclient/{begin/{0-install', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/ior-shared-file}', 'tasks/mdtest}']
2023-05-19T08:48:33.945 INFO:scrape:
2023-05-19T08:48:33.945 INFO:scrape:Failure: Command failed (workunit test fs/fscrypt.sh) on smithi156 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27be51263d3d0c721b6d1ca9f42550bf0ab3c97b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/fscrypt.sh none iozone'
2023-05-19T08:48:33.945 INFO:scrape:1 jobs: ['7277438']
2023-05-19T08:48:33.946 INFO:scrape:suites: ['1-ceph', '1-tests/fscrypt-iozone}}', '2-logrotate}', 'bluestore-bitmap', 'clusters/1-mds-1-client', 'conf/{client', 'distro/{centos_latest}', 'fs/fscrypt/{begin/{0-install', 'ignorelist_health_more', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'pg-warn}', 'tasks/{0-client']
2023-05-19T08:48:33.946 INFO:scrape:
2023-05-19T08:48:33.946 INFO:scrape:Failure: error during scrub thrashing: reached maximum tries (30) after waiting for 900 seconds
2023-05-19T08:48:33.946 INFO:scrape:2 jobs: ['7277462', '7277453']
2023-05-19T08:48:33.947 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '4-flush/no', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'session_timeout}', 'standby-replay']
2023-05-19T08:48:33.947 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/no', '5-workunit/fs/misc}}', '5-workunit/kernel_untar_build}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'frag', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}']
2023-05-19T08:48:33.947 INFO:scrape:
2023-05-19T08:48:33.948 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/pjd.sh
2023-05-19T08:48:33.948 INFO:scrape:10 jobs: ['7277443', '7277452', '7277460', '7277467', '7277451', '7277470', '7277442', '7277469', '7277459', '7277434']
2023-05-19T08:48:33.948 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}']
2023-05-19T08:48:33.949 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/pjd}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'frag', 'fs/32bits/{begin/{0-install', 'fs/permission/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{faked-ino', 'overrides/{ignorelist_health', 'ranks/1', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_suites_pjd}', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}']
2023-05-19T08:48:33.949 INFO:scrape:
2023-05-19T08:48:33.949 INFO:scrape:Dead: b'2023-05-19T07:43:43.615 DEBUG:teuthology.exit:Finished running handlers'
2023-05-19T08:48:33.950 INFO:scrape:4 jobs: ['7277449', '7277448', '7277444', '7277436']
2023-05-19T08:48:33.950 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'mds', 'mon', 'osd}']
2023-05-19T08:48:33.950 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/no', '3-snaps/yes', '4-flush/yes', '5-workunit/kernel_untar_build}}', '5-workunit/postgres}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/1a3s-mds-4c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'frag', 'fs/functional/{begin/{0-install', 'fs/traceless/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/secure', 'n/3', 'n/5', 'no_client_pidfile}', 'objectstore-ec/bluestore-bitmap', 'objectstore-ec/bluestore-comp', 'objectstore/bluestore-bitmap', 'omap_limit/10', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{k-stock', 'overrides/{frag', 'overrides/{ignorelist_health', 'ranks/multi/{export-check', 'replication/always}', 'replication/default}', 'rhel_8}', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_suites_blogbench', 'tasks/xfstests-dev}', 'tasks/{0-subvolume/{no-subvolume}', 'tasks/{0-subvolume/{with-no-extra-options}', 'traceless/50pc}', 'wsync/no}']
2023-05-19T08:48:33.951 INFO:scrape:
2023-05-19T08:48:33.951 INFO:scrape:Failure: "1684433340.645037 mon.a (mon.0) 401 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log
2023-05-19T08:48:33.951 INFO:scrape:1 jobs: ['7277433']
2023-05-19T08:48:33.951 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/snap-schedule}']
2023-05-19T08:48:33.952 INFO:scrape:
2023-05-19T08:48:33.952 INFO:scrape:Valgrind: mon (InvalidRead, Leak_PossiblyLost, Leak_StillReachable)
2023-05-19T08:48:33.952 INFO:scrape:1 jobs: ['7277447']
2023-05-19T08:48:33.953 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{ubuntu/{latest', 'fs/verify/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mon-debug', 'mount', 'mount/kclient/{k-testing', 'ms-die-on-skipped}', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'overrides}}', 'ranks/5', 'session_timeout}', 'tasks/dbench', 'validater/valgrind}']
2023-05-19T08:48:33.953 INFO:scrape:
2023-05-19T08:48:33.953 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/suites/ffsb.sh
2023-05-19T08:48:33.953 INFO:scrape:1 jobs: ['7277463']
2023-05-19T08:48:33.954 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/traceless/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'objectstore-ec/bluestore-bitmap', 'osd}', 'overrides/{frag', 'tasks/cfuse_workunit_suites_ffsb', 'traceless/50pc}']
2023-05-19T08:48:33.954 INFO:scrape:
2023-05-19T08:48:33.954 INFO:scrape:Failure: Command failed (workunit test fs/quota/quota.sh) on smithi162 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27be51263d3d0c721b6d1ca9f42550bf0ab3c97b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/fs/quota/quota.sh'
2023-05-19T08:48:33.955 INFO:scrape:1 jobs: ['7277445']
2023-05-19T08:48:33.955 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v1}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{ignorelist_health', 'rhel_8}', 'tasks/workunit/quota}']
2023-05-19T08:48:33.955 INFO:scrape:
2023-05-19T08:48:33.955 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi038 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27be51263d3d0c721b6d1ca9f42550bf0ab3c97b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2023-05-19T08:48:33.956 INFO:scrape:3 jobs: ['7277439', '7277455', '7277476']
2023-05-19T08:48:33.956 INFO:scrape:suites intersection: ['2-logrotate}', 'conf/{client', 'ignorelist_health', 'mds', 'mon', 'objectstore-ec/bluestore-ec-root', 'osd}']
2023-05-19T08:48:33.956 INFO:scrape:suites union: ['1-ceph', '1-cephadm', '1-check-counter', '2-logrotate}', '2-scrub/no', '2-scrub/yes', '3-snaps/yes', '4-flush/no', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{ubuntu_20.04}', 'frag', 'fs/traceless/{begin/{0-install', 'fs/workload/{0-rhel_8', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/legacy', 'n/3', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/multi/{export-check', 'replication/always}', 'session_timeout}', 'standby-replay', 'tasks/cfuse_workunit_suites_dbench', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'traceless/50pc}', 'wsync/yes}']
2023-05-19T08:48:33.957 INFO:scrape:
2023-05-19T08:48:33.957 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi097 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27be51263d3d0c721b6d1ca9f42550bf0ab3c97b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
2023-05-19T08:48:33.957 INFO:scrape:1 jobs: ['7277441']
2023-05-19T08:48:33.958 INFO:scrape:suites: ['1-ceph', '2-logrotate}', '2-workunit/fs/snaps}}', 'clusters/1a5s-mds-1c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/thrash/workloads/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'msgr-failures/osd-mds-delay', 'objectstore-ec/bluestore-comp', 'osd}', 'overrides/{frag', 'prefetch_dirfrags/yes', 'prefetch_entire_dirfrags/yes', 'races', 'ranks/3', 'session_timeout', 'tasks/{1-thrash/mds', 'thrashosds-health}']
2023-05-19T08:48:33.958 INFO:scrape:
2023-05-19T08:48:33.958 INFO:scrape:Timeout 6h running clone.client.0/qa/workunits/suites/fsstress.sh
2023-05-19T08:48:33.958 INFO:scrape:3 jobs: ['7277440', '7277471', '7277473']
2023-05-19T08:48:33.959 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'mds', 'mon', 'mount/fuse', 'osd}']
2023-05-19T08:48:33.959 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', 'clusters/1a5s-mds-1c-client', 'clusters/fixed-2-ucephfs', 'conf/{client', 'distro/{centos_8}', 'distro/{rhel_8}', 'distro/{ubuntu_latest}', 'fs/traceless/{begin/{0-install', 'fs/verify/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mon-debug', 'mount/fuse', 'objectstore-ec/bluestore-comp', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'ranks/5', 'session_timeout}', 'tasks/cfuse_workunit_suites_fsstress', 'tasks/fsstress', 'traceless/50pc}', 'validater/lockdep}']
2023-05-19T08:48:33.959 INFO:scrape:
2023-05-19T08:48:33.960 INFO:scrape:Dead: b'2023-05-19T08:48:01.373 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf'
2023-05-19T08:48:33.960 INFO:scrape:ceph version 18.0.0-3926-g27be5126 (27be51263d3d0c721b6d1ca9f42550bf0ab3c97b) reef (dev)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f73ecbf4420]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x190) [0x7f73ec11e95c]
 5: (MonitorDBStore::apply_transaction(std::shared_ptr<MonitorDBStore::Transaction>)+0x918) [0x559bcb3070a8]
 6: (Paxos::begin(ceph::buffer::v15_2_0::list&)+0x42c) [0x559bcb41266c]
 7: (Paxos::propose_pending()+0x126) [0x559bcb4170c6]
 8: (Paxos::trigger_propose()+0x126) [0x559bcb4174a6]
 9: (PaxosService::propose_pending()+0x235) [0x559bcb421905]
 10: (Context::complete(int)+0xd) [0x559bcb2d9efd]
 11: (CommonSafeTimer<std::mutex>::timer_thread()+0x12c) [0x7f73ec20ffdc]
 12: (CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x7f73ec211071]
 13: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f73ecbe8609]
 14: clone()
2023-05-19T08:48:33.960 INFO:scrape:1 jobs: ['7277477']
2023-05-19T08:48:33.960 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{distro/testing/k-testing', 'overrides/{ignorelist_health', 'tasks/xfstests-dev}']
2023-05-19T08:48:33.961 INFO:scrape:
2023-05-19T08:48:33.961 INFO:scrape:Failure: Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi116 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27be51263d3d0c721b6d1ca9f42550bf0ab3c97b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'
2023-05-19T08:48:33.961 INFO:scrape:2 jobs: ['7277461', '7277464']
2023-05-19T08:48:33.962 INFO:scrape:suites intersection: ['1-ceph', '2-logrotate}', 'conf/{client', 'distro/{ubuntu_latest}', 'mds', 'mon', 'mount/fuse', 'osd}']
2023-05-19T08:48:33.962 INFO:scrape:suites union: ['1-ceph', '2-logrotate}', '2-workunit/cfuse_workunit_snaptests}}', 'clusters/1a3s-mds-1c-client', 'clusters/1a3s-mds-2c-client', 'conf/{client', 'distro/{ubuntu_latest}', 'fs/snaps/{begin/{0-install', 'fs/thrash/multifs/{begin/{0-install', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'ignorelist_wrongly_marked_down}', 'mds', 'mon', 'mount/fuse', 'msgr-failures/none', 'multifs', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{frag', 'overrides/{ignorelist_health', 'session_timeout', 'tasks/workunit/snaps}', 'tasks/{1-thrash/mon', 'thrashosds-health}']
2023-05-19T08:48:33.962 INFO:scrape:
2023-05-19T08:48:33.963 INFO:scrape:Failure: Test failure: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota)
2023-05-19T08:48:33.963 INFO:scrape:1 jobs: ['7277466']
2023-05-19T08:48:33.963 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1a3s-mds-4c-client', 'conf/{client', 'distro/{rhel_8}', 'fs/functional/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'mount/fuse', 'no_client_pidfile}', 'objectstore/bluestore-ec-root', 'osd}', 'overrides/{ignorelist_health', 'tasks/quota}']
2023-05-19T08:48:33.963 INFO:scrape:
2023-05-19T08:48:33.964 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/client/test.sh
2023-05-19T08:48:33.964 INFO:scrape:1 jobs: ['7277435']
2023-05-19T08:48:33.964 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{centos_8}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/client}']
2023-05-19T08:48:33.965 INFO:scrape:
2023-05-19T08:48:33.965 INFO:scrape:Failure: Command failed (workunit test fs/test_python.sh) on smithi081 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=27be51263d3d0c721b6d1ca9f42550bf0ab3c97b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/test_python.sh'
2023-05-19T08:48:33.965 INFO:scrape:1 jobs: ['7277468']
2023-05-19T08:48:33.965 INFO:scrape:suites: ['1-ceph', '2-logrotate}', 'clusters/1-mds-1-client-coloc', 'conf/{client', 'distro/{ubuntu_20.04}', 'fs/libcephfs/{begin/{0-install', 'ignorelist_wrongly_marked_down', 'mds', 'mon', 'no_client_pidfile}', 'objectstore/bluestore-bitmap', 'osd}', 'overrides/{ignorelist_health', 'tasks/libcephfs_python}']
2023-05-19T08:48:33.966 INFO:scrape: