2023-09-08T11:03:49.132 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-08T11:03:49.133 INFO:teuthology.results:seed: '4309' 2023-09-08T11:04:41.828 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-08T11:04:41.831 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-09-08T15:54:01.227 INFO:teuthology.results:Tests finished! gathering results... 2023-09-08T15:54:01.749 INFO:scrape:Found 6 jobs 2023-09-08T15:54:04.027 INFO:scrape:Found 3 distinct failure reasons 2023-09-08T15:54:04.028 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi155 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2023-09-08T15:54:04.029 INFO:scrape:1 jobs: ['7391057'] 2023-09-08T15:54:04.029 INFO:scrape:suites: ['1-ceph-install/quincy', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/bluestore-stupid', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack'] 2023-09-08T15:54:04.030 INFO:scrape: 2023-09-08T15:54:04.030 INFO:scrape:Failure: Command failed (s3 tests against rgw) on smithi125 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_subdomain,!sse-s3'" 2023-09-08T15:54:04.030 INFO:scrape:2 jobs: ['7391056', '7391053'] 2023-09-08T15:54:04.031 INFO:scrape:suites intersection: ['upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade'] 2023-09-08T15:54:04.031 INFO:scrape:suites union: ['supported-all-distro/centos_8}', 'supported-all-distro/ubuntu_latest}', 'upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade'] 2023-09-08T15:54:04.032 INFO:scrape: 2023-09-08T15:54:04.032 INFO:scrape:Timeout 120 running ceph --cluster ceph osd dump --format=json 2023-09-08T15:54:04.033 INFO:scrape:1 jobs: ['7391058'] 2023-09-08T15:54:04.033 INFO:scrape:suites: ['1-ceph-install/quincy', '1.1.short_pg_log', '2-partial-upgrade/firsthalf', '3-thrash/default', '4-workload/{fsx', '5-finish-upgrade', '6-final-workload/{rbd-python', 'objectstore/filestore-xfs', 'radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'snaps-many-objects}', 'start}', 'supported-all-distro/ubuntu_latest', 'thrashosds-health}', 'upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack'] 2023-09-08T15:54:04.033 INFO:scrape: 2023-09-08T15:54:04.034 INFO:teuthology.results:Sending results to idryomov@gmail.com: Test Run: dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 2 Fail ================================================================= [7391053] upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/centos_8} ----------------------------------------------------------------- time: 00:51:45 info: http://pulpito.ceph.com/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/7391053/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/7391053/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9941414923d04df7bcf21f92d4e5ab87 Command failed (s3 tests against rgw) on smithi090 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_exp iration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_su bdomain,!sse-s3'" [7391056] upgrade:quincy-p2p/quincy-p2p-parallel/{point-to-point-upgrade supported-all-distro/ubuntu_latest} ----------------------------------------------------------------- time: 00:50:53 info: http://pulpito.ceph.com/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/7391056/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/7391056/ sentry: https://sentry.ceph.com/organizations/ceph/?query=98913553756343bfb688520f68e009bd Command failed (s3 tests against rgw) on smithi125 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt /home/ubuntu/cephtest/s3-tests-client.0/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/s3-tests-client.0 -v -a '!fails_on_rgw,!lifecycle_exp iration,!fails_strict_rfc2616,!test_of_sts,!webidentity_test,!fails_with_su bdomain,!sse-s3'" [7391057] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-stupid supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 03:30:34 info: http://pulpito.ceph.com/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/7391057/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/7391057/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0d3c59aeed234b8897ea297e331eaf9a Command failed (workunit test cls/test_cls_rbd.sh) on smithi155 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=quincy TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7391058] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/filestore-xfs supported-all-distro/ubuntu_latest thrashosds-health} ----------------------------------------------------------------- time: 01:35:57 info: http://pulpito.ceph.com/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/7391058/ log: http://qa-proxy.ceph.com/teuthology/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/7391058/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6c8a0dd09c164a00954c2510d4d28ec8 Command failed on smithi007 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' Pass ================================================================= [7391054] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-bitmap supported-all-distro/ubuntu_latest thrashosds-health} time: 04:39:01 info: http://pulpito.ceph.com/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/7391054/ [7391055] upgrade:quincy-p2p/quincy-p2p-stress-split/{0-cluster/{openstack start} 1-ceph-install/quincy 1.1.short_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{fsx radosbench rbd-cls rbd-import-export rbd_api readwrite snaps-few-objects} 5-finish-upgrade 6-final-workload/{rbd-python snaps-many-objects} objectstore/bluestore-comp supported-all-distro/ubuntu_latest thrashosds-health} time: 03:18:44 info: http://pulpito.ceph.com/dis-2023-09-08_11:03:42-upgrade:quincy-p2p-quincy-distro-default-smithi/7391055/