2022-03-30T06:28:35.025 INFO:root:teuthology version: 1.1.0-d877b023 2022-03-30T06:28:35.027 INFO:teuthology.results:seed: '1027' 2022-03-30T06:36:10.578 INFO:root:teuthology version: 1.1.0-d877b023 2022-03-30T06:36:10.589 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-30T07:58:16.829 INFO:teuthology.results:Tests finished! gathering results... 2022-03-30T07:58:17.542 INFO:scrape:Found 10 jobs 2022-03-30T07:58:20.698 INFO:scrape:Found 1 distinct failure reasons 2022-03-30T07:58:20.699 INFO:scrape:Failure: Command failed (workunit test fs/misc/subvolume.sh) on smithi066 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12bfb412379f2916f35214c2398b99cb0a9f60d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/subvolume.sh' 2022-03-30T07:58:20.699 INFO:scrape:8 jobs: ['6768620', '6768618', '6768621', '6768625', '6768623', '6768622', '6768627', '6768626'] 2022-03-30T07:58:20.700 INFO:scrape:suites intersection: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'mds', 'mon', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{frag', 'scrub/yes', 'session_timeout', 'standby-replay', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}'] 2022-03-30T07:58:20.700 INFO:scrape:suites union: ['1-cephadm', '2-logrotate}', 'begin/{0-install', 'clusters/1a11s-mds-1c-client-3node', 'conf/{client', 'fs:workload/{0-rhel_8', 'mds', 'mon', 'mount', 'mount/fuse', 'mount/kclient/{mount-syntax/{v1}', 'mount/kclient/{mount-syntax/{v2}', 'ms-die-on-skipped}}', 'ms_mode/{crc}', 'ms_mode/{legacy}', 'ms_mode/{secure}', 'objectstore-ec/bluestore-comp-ec-root', 'objectstore-ec/bluestore-ec-root', 'omap_limit/10000', 'osd-asserts', 'osd}', 'overrides/{distro/stock/{k-stock', 'overrides/{distro/testing/k-testing', 'overrides/{frag', 'ranks/1', 'ranks/3', 'ranks/5', 'rhel_8}', 'scrub/yes', 'session_timeout', 'standby-replay', 'subvolume/with-namespace-isolated', 'subvolume/with-namespace-isolated-and-quota', 'subvolume/with-no-extra-options', 'subvolume/with-quota', 'tasks/{0-check-counter', 'whitelist_health', 'whitelist_wrongly_marked_down}', 'workunit/fs/misc}', 'wsync/{no}}', 'wsync/{yes}}'] 2022-03-30T07:58:20.701 INFO:scrape: 2022-03-30T07:58:20.703 INFO:teuthology.results:Sending results to mchangir@redhat.com: Test Run: mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/ failed: 8 dead: 0 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [6768618] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay subvolume/with-namespace-isolated-and-quota tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 00:49:56 info: http://pulpito.ceph.com/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768618/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768618/ sentry: https://sentry.ceph.com/organizations/ceph/?query=30dac42e69654450bbfbeadfb19397c9 Command failed (workunit test fs/misc/subvolume.sh) on smithi078 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12bfb412379f2916f35214c2398b99cb0a9f60d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/subvolume.sh' [6768620] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay subvolume/with-quota tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 01:01:00 info: http://pulpito.ceph.com/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768620/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768620/ sentry: https://sentry.ceph.com/organizations/ceph/?query=772efd81640a42e8829480d08ebdddd9 Command failed (workunit test fs/misc/subvolume.sh) on smithi066 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12bfb412379f2916f35214c2398b99cb0a9f60d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/subvolume.sh' [6768621] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay subvolume/with-no-extra-options tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 00:49:52 info: http://pulpito.ceph.com/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768621/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768621/ sentry: https://sentry.ceph.com/organizations/ceph/?query=49e42438d28d465d8f4ad34a3b73c177 Command failed (workunit test fs/misc/subvolume.sh) on smithi071 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12bfb412379f2916f35214c2398b99cb0a9f60d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/subvolume.sh' [6768622] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay subvolume/with-namespace-isolated tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 01:11:34 info: http://pulpito.ceph.com/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768622/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768622/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9c6ef768d2cb44f69efced0315e0779b Command failed (workunit test fs/misc/subvolume.sh) on smithi080 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12bfb412379f2916f35214c2398b99cb0a9f60d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/subvolume.sh' [6768623] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay subvolume/with-namespace-isolated-and-quota tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 01:09:04 info: http://pulpito.ceph.com/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768623/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768623/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b49b64af2b24755871d63f6d893f0f8 Command failed (workunit test fs/misc/subvolume.sh) on smithi050 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12bfb412379f2916f35214c2398b99cb0a9f60d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/subvolume.sh' [6768625] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay subvolume/with-quota tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 01:17:37 info: http://pulpito.ceph.com/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768625/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768625/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f316442b4f614720a6e5a4ef16d5466f Command failed (workunit test fs/misc/subvolume.sh) on smithi002 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12bfb412379f2916f35214c2398b99cb0a9f60d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/subvolume.sh' [6768626] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/5 scrub/yes standby-replay subvolume/with-no-extra-options tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} ----------------------------------------------------------------- time: 00:56:13 info: http://pulpito.ceph.com/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768626/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768626/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a1e9a105f0004d53a5dd5301826676e5 Command failed (workunit test fs/misc/subvolume.sh) on smithi005 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12bfb412379f2916f35214c2398b99cb0a9f60d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/subvolume.sh' [6768627] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{legacy} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay subvolume/with-namespace-isolated tasks/{0-check-counter workunit/fs/misc} wsync/{no}} ----------------------------------------------------------------- time: 00:51:57 info: http://pulpito.ceph.com/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768627/ log: http://qa-proxy.ceph.com/teuthology/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768627/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c63f74d19d7142759424bb877160cefd Command failed (workunit test fs/misc/subvolume.sh) on smithi145 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12bfb412379f2916f35214c2398b99cb0a9f60d1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/subvolume.sh' Pass ================================================================= [6768619] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse ms_mode/{secure} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/1 scrub/yes standby-replay subvolume/no-subvolume tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 01:14:09 info: http://pulpito.ceph.com/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768619/ [6768624] fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{frag osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/yes standby-replay subvolume/no-subvolume tasks/{0-check-counter workunit/fs/misc} wsync/{yes}} time: 00:58:44 info: http://pulpito.ceph.com/mchangir-2022-03-30_06:26:55-fs:workload-wip-mchangir-qa-add-subvolume-option-flavors-distro-default-smithi/6768624/