2024-04-16T05:03:47.272 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-16T05:03:47.272 INFO:teuthology.results:seed: '2991'
2024-04-16T05:04:52.231 INFO:root:teuthology version: 0.0.1.dev249+gcd45576
2024-04-16T05:04:52.235 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2024-04-16T05:49:55.290 INFO:teuthology.results:Tests finished! gathering results...
2024-04-16T05:49:55.527 INFO:scrape:Found 5 jobs
2024-04-16T05:49:56.533 INFO:scrape:Found 1 distinct failure reasons
2024-04-16T05:49:56.533 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi067 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a87fcff6340baeedd43af126e793f3241b6400ea TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'
2024-04-16T05:49:56.533 INFO:scrape:5 jobs: ['7657899', '7657902', '7657900', '7657903', '7657901']
2024-04-16T05:49:56.533 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'pg_health', 'session_timeout}', 'standby-replay', 'wsync/yes}']
2024-04-16T05:49:56.533 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'k-stock}', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/stock/{centos_9.stream', 'pg_health', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'wsync/yes}']
2024-04-16T05:49:56.533 INFO:scrape:
2024-04-16T05:49:56.534 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi
=================================================================
info:    https://pulpito.ceph.com/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7657899]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:35:02
info:   https://pulpito.ceph.com/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7657899/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7657899/
sentry: https://sentry.ceph.com/organizations/ceph/?query=828887c547964b0988c83b13e8a1affa

    Command failed (workunit test suites/dbench.sh) on smithi067 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a87fcff6340baeedd43af126e793f3241b6400ea
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7657900]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:34:49
info:   https://pulpito.ceph.com/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7657900/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7657900/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dc6265b9adf940ae882a92366928e7bb

    Command failed (workunit test suites/dbench.sh) on smithi032 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a87fcff6340baeedd43af126e793f3241b6400ea
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7657901]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:35:11
info:   https://pulpito.ceph.com/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7657901/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7657901/
sentry: https://sentry.ceph.com/organizations/ceph/?query=c8dc2f6780524070b550d5c4b0c989e9

    Command failed (workunit test suites/dbench.sh) on smithi086 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a87fcff6340baeedd43af126e793f3241b6400ea
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7657902]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:34:34
info:   https://pulpito.ceph.com/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7657902/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7657902/
sentry: https://sentry.ceph.com/organizations/ceph/?query=226efff1a8024fd18f892b6521791d3d

    Command failed (workunit test suites/dbench.sh) on smithi060 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a87fcff6340baeedd43af126e793f3241b6400ea
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'

[7657903]  fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}}
-----------------------------------------------------------------
time:   00:34:50
info:   https://pulpito.ceph.com/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7657903/
log:    http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_05:02:59-fs-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7657903/
sentry: https://sentry.ceph.com/organizations/ceph/?query=116c54d487a743a195ae7751fe822de7

    Command failed (workunit test suites/dbench.sh) on smithi100 with status 1:
    'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=a87fcff6340baeedd43af126e793f3241b6400ea
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'