2023-04-03T08:56:09.750 INFO:root:teuthology version: 0.0.1.dev91+g5338cb9
2023-04-03T08:56:09.751 INFO:teuthology.results:seed: '9277'
2023-04-03T08:56:57.700 INFO:root:teuthology version: 0.0.1.dev91+g5338cb9
2023-04-03T08:56:57.704 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-04-03T09:27:59.754 INFO:teuthology.results:Tests finished! gathering results...
2023-04-03T09:27:59.935 INFO:scrape:Found 5 jobs
2023-04-03T09:28:00.441 INFO:scrape:Found 2 distinct failure reasons
2023-04-03T09:28:00.443 INFO:scrape:Failure: Command failed (workunit test rgw/test_librgw_file.sh) on smithi060 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=932086279cc10e7dbf3aa5c59db0acf68e9c00cb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh'
2023-04-03T09:28:00.443 INFO:scrape:3 jobs: ['7230262', '7230263', '7230264']
2023-04-03T09:28:00.444 INFO:scrape:suites intersection: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'tasks/{cls', 'ubuntu_20.04', 'validater/valgrind}']
2023-04-03T09:28:00.445 INFO:scrape:suites union: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/{cls', 'ubuntu_20.04', 'validater/valgrind}']
2023-04-03T09:28:00.445 INFO:scrape:
2023-04-03T09:28:00.446 INFO:scrape:Failure: Command failed (workunit test rgw/run-datacache.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=932086279cc10e7dbf3aa5c59db0acf68e9c00cb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RGW_DATACACHE_PATH=/tmp/rgw_datacache adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-datacache.sh'
2023-04-03T09:28:00.446 INFO:scrape:2 jobs: ['7230266', '7230265']
2023-04-03T09:28:00.447 INFO:scrape:suites intersection: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{single}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'ubuntu_20.04', 'validater/valgrind}']
2023-04-03T09:28:00.448 INFO:scrape:suites union: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'ubuntu_20.04', 'validater/valgrind}']
2023-04-03T09:28:00.448 INFO:scrape:
2023-04-03T09:28:00.449 INFO:teuthology.results:Sending results to mkogan@redhat.com: Test Run: mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/
failed:  5
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7230262]  rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_20.04 validater/valgrind}
-----------------------------------------------------------------
time:   00:21:02
info:   http://pulpito.ceph.com/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/7230262/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/7230262/
sentry: https://sentry.ceph.com/organizations/ceph/?query=2363b5dda92a4cd786914543e03db210

    Command failed (workunit test rgw/test_librgw_file.sh) on smithi060 with
    status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=932086279cc10e7dbf3aa5c59db0acf68e9c00cb
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh'

[7230263]  rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_20.04 validater/valgrind}
-----------------------------------------------------------------
time:   00:21:04
info:   http://pulpito.ceph.com/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/7230263/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/7230263/
sentry: https://sentry.ceph.com/organizations/ceph/?query=45a6905da0b44aa69310dc241ee27449

    Command failed (workunit test rgw/test_librgw_file.sh) on smithi139 with
    status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=932086279cc10e7dbf3aa5c59db0acf68e9c00cb
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh'

[7230264]  rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_20.04 validater/valgrind}
-----------------------------------------------------------------
time:   00:20:48
info:   http://pulpito.ceph.com/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/7230264/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/7230264/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9e4d24c31136467ba5e722318ad17fec

    Command failed (workunit test rgw/test_librgw_file.sh) on smithi151 with
    status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=932086279cc10e7dbf3aa5c59db0acf68e9c00cb
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh'

[7230265]  rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_20.04 validater/valgrind}
-----------------------------------------------------------------
time:   00:13:10
info:   http://pulpito.ceph.com/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/7230265/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/7230265/
sentry: https://sentry.ceph.com/organizations/ceph/?query=fd7e663799074d4ea213c88a282a6f2c

    Command failed (workunit test rgw/run-datacache.sh) on smithi141 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=932086279cc10e7dbf3aa5c59db0acf68e9c00cb
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RGW_DATACACHE_PATH=/tmp/rgw_datacache
    adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout
    3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-datacache.sh'

[7230266]  rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} ubuntu_20.04 validater/valgrind}
-----------------------------------------------------------------
time:   00:13:00
info:   http://pulpito.ceph.com/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/7230266/
log:    http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_08:56:00-rgw:verify-wip-ubu-SigDoesNotMatch-cont--i01-distro-default-smithi/7230266/
sentry: https://sentry.ceph.com/organizations/ceph/?query=776ea830d4454569bff4d762f81f4b20

    Command failed (workunit test rgw/run-datacache.sh) on smithi027 with
    status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=932086279cc10e7dbf3aa5c59db0acf68e9c00cb
    TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0"
    PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RGW_DATACACHE_PATH=/tmp/rgw_datacache
    adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout
    3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/run-datacache.sh'