2024-04-16T16:33:14.538 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-16T16:33:14.539 INFO:teuthology.results:seed: '2381' 2024-04-16T16:34:04.823 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-04-16T16:34:04.827 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-04-16T17:45:09.949 INFO:teuthology.results:Tests finished! gathering results... 2024-04-16T17:45:10.358 INFO:scrape:Found 5 jobs 2024-04-16T17:45:11.432 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.433 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.433 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.433 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.449 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.449 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.449 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.450 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.877 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.878 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.878 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.878 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.907 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.907 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.907 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:11.907 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.267 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.268 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.268 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.268 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.272 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.272 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.272 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.272 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.752 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.752 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.753 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.753 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.753 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.753 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.753 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.753 WARNING:scrape:Saw end of BT but not start 2024-04-16T17:45:12.855 INFO:scrape:Found 1 distinct failure reasons 2024-04-16T17:45:12.855 INFO:scrape:Failure: Command failed (workunit test suites/dbench.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b137e4a675398e295567f9652210865a4d573fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-04-16T17:45:12.855 INFO:scrape:5 jobs: ['7658439', '7658441', '7658440', '7658442', '7658438'] 2024-04-16T17:45:12.855 INFO:scrape:suites intersection: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'ms-die-on-skipped}}', 'ms_mode/crc', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'session_timeout}', 'standby-replay', 'wsync/yes}'] 2024-04-16T17:45:12.856 INFO:scrape:suites union: ['1-cephadm', '1-check-counter', '2-logrotate', '2-scrub/yes', '3-modules}', '3-snaps/yes', '4-flush/yes', '5-workunit/suites/dbench}}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'export-check', 'frag', 'fs/workload/{0-centos_9.stream', 'ignorelist_health', 'ignorelist_wrongly_marked_down', 'mds', 'mgr', 'mon', 'mount', 'mount/kclient/{base/{mount-syntax/{v1}', 'mount/kclient/{base/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/crc', 'n/3', 'n/5', 'objectstore-ec/bluestore-comp-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{cephsqlite-timeout', 'overrides/{distro/testing/k-testing', 'pg_health', 'ranks/multi/{balancer/automatic', 'ranks/multi/{balancer/random', 'replication/always}', 'replication/default}', 'session_timeout}', 'standby-replay', 'tasks/{0-subvolume/{with-namespace-isolated-and-quota}', 'tasks/{0-subvolume/{with-namespace-isolated}', 'tasks/{0-subvolume/{with-no-extra-options}', 'tasks/{0-subvolume/{with-quota}', 'wsync/yes}'] 2024-04-16T17:45:12.856 INFO:scrape: 2024-04-16T17:45:12.856 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/ failed: 5 dead: 0 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [7658438] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 01:01:13 info: https://pulpito.ceph.com/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7658438/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7658438/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1c3db1d8239f4a198174c124580123a5 Command failed (workunit test suites/dbench.sh) on smithi018 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b137e4a675398e295567f9652210865a4d573fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7658439] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:38:15 info: https://pulpito.ceph.com/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7658439/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7658439/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6df16c87d16b4e09ad8cc292d074c972 Command failed (workunit test suites/dbench.sh) on smithi045 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b137e4a675398e295567f9652210865a4d573fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7658440] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:42:31 info: https://pulpito.ceph.com/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7658440/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7658440/ sentry: https://sentry.ceph.com/organizations/ceph/?query=5249856fb10a4357b41b7252fcaef594 Command failed (workunit test suites/dbench.sh) on smithi044 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b137e4a675398e295567f9652210865a4d573fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7658441] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:38:36 info: https://pulpito.ceph.com/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7658441/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7658441/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f68eed0bcacf4be689c0af0e7634a767 Command failed (workunit test suites/dbench.sh) on smithi042 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b137e4a675398e295567f9652210865a4d573fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' [7658442] fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} ----------------------------------------------------------------- time: 00:46:16 info: https://pulpito.ceph.com/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7658442/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2024-04-16_16:32:10-fs-wip-mchangir-qa-debug-resource-temporarily-unavailable-issue-distro-default-smithi/7658442/ sentry: https://sentry.ceph.com/organizations/ceph/?query=85e81928c3474f87a18a1a383a1d785b Command failed (workunit test suites/dbench.sh) on smithi040 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b137e4a675398e295567f9652210865a4d573fac TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh'