2023-08-27T07:41:48.000 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc
2023-08-27T07:41:48.001 INFO:teuthology.results:seed: '7510'
2023-08-27T07:42:50.335 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc
2023-08-27T07:42:50.338 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish...
2023-08-27T11:25:06.928 INFO:teuthology.results:Tests finished! gathering results...
2023-08-27T11:25:07.504 INFO:scrape:Found 6 jobs
2023-08-27T11:25:09.636 INFO:scrape:Found 3 distinct failure reasons
2023-08-27T11:25:09.637 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi159 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'
2023-08-27T11:25:09.637 INFO:scrape:3 jobs: ['7381979', '7381982', '7381980']
2023-08-27T11:25:09.638 INFO:scrape:suites intersection: ['1-ceph-install/quincy', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack']
2023-08-27T11:25:09.638 INFO:scrape:suites union: ['1-ceph-install/quincy', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp', 'objectstore/bluestore-stupid', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack']
2023-08-27T11:25:09.639 INFO:scrape:
2023-08-27T11:25:09.639 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi101 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3'"
2023-08-27T11:25:09.639 INFO:scrape:2 jobs: ['7381978', '7381981']
2023-08-27T11:25:09.640 INFO:scrape:suites intersection: ['upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade']
2023-08-27T11:25:09.640 INFO:scrape:suites union: ['supported-all-distro/centos_8}', 'supported-all-distro/ubuntu_latest}', 'upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade']
2023-08-27T11:25:09.640 INFO:scrape:
2023-08-27T11:25:09.641 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json
2023-08-27T11:25:09.641 INFO:scrape:1 jobs: ['7381983']
2023-08-27T11:25:09.642 INFO:scrape:suites: ['1-ceph-install/quincy', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack']
2023-08-27T11:25:09.642 INFO:scrape:
2023-08-27T11:25:09.643 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi
=================================================================
info:    http://pulpito.ceph.com/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/
logs:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/
failed:  6
dead:    0
running: 0
waiting: 0
queued:  0
passed:  0


Fail
=================================================================
[7381978]  upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_8}
-----------------------------------------------------------------
time:   00:53:36
info:   http://pulpito.ceph.com/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381978/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381978/
sentry: https://sentry.ceph.com/organizations/ceph/?query=51c997e0339841b0a1e18f199d90eedd

    Command failed (s3 tests against rgw) on smithi101 with status 1:
    "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
    /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w
    /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_exp
    iration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_su
    bdomain,!sse-s3'"

[7381979]  upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   03:32:09
info:   http://pulpito.ceph.com/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381979/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381979/
sentry: https://sentry.ceph.com/organizations/ceph/?query=dd9a81734cdb4e7b82bc14379776b58b

    Command failed (workunit test cls/test_cls_rbd.sh) on smithi159 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'

[7381980]  upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-comp supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   02:13:27
info:   http://pulpito.ceph.com/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381980/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381980/
sentry: https://sentry.ceph.com/organizations/ceph/?query=3810e3685a90466cb4d2de446e48f86e

    Command failed (workunit test cls/test_cls_rbd.sh) on smithi122 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'

[7381981]  upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest}
-----------------------------------------------------------------
time:   00:50:30
info:   http://pulpito.ceph.com/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381981/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381981/
sentry: https://sentry.ceph.com/organizations/ceph/?query=14dc90ce63a240139386f8ef3268779d

    Command failed (s3 tests against rgw) on smithi158 with status 1:
    "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
    BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg
    REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
    /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w
    /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_exp
    iration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_su
    bdomain,!sse-s3'"

[7381982]  upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-stupid supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   03:28:28
info:   http://pulpito.ceph.com/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381982/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381982/
sentry: https://sentry.ceph.com/organizations/ceph/?query=9a5131d9cbb84e52bcd1dee450ee8173

    Command failed (workunit test cls/test_cls_rbd.sh) on smithi072 with status
    1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd --
    /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
    CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph"
    CEPH_ID="0" PATH=$PATH:/usr/sbin
    CEPH_BASE=/home/ubuntu/cephtest/clone.client.0
    CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0
    CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage
    /home/ubuntu/cephtest/archive/coverage timeout 3h
    /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh'

[7381983]  upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health}
-----------------------------------------------------------------
time:   01:21:30
info:   http://pulpito.ceph.com/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381983/
log:    http://qa-proxy.ceph.com/teuthology/teuthology-2023-08-27_01:35:02-upgrade:quincy-p2p-quincy-distro-default-smithi/7381983/
sentry: https://sentry.ceph.com/organizations/ceph/?query=77011d165dbe492ebd3f860bbce352cb

    Command failed on smithi118 with status 124: 'sudo adjust-ulimits ceph-
    coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster
    ceph osd dump --format=json'