2023-03-14T14:11:56.806 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-14T14:11:56.807 INFO:teuthology.results:subset: '111/120000' 2023-03-14T14:11:56.807 INFO:teuthology.results:seed: '8904' 2023-03-14T14:39:27.178 INFO:root:teuthology version: 0.0.1.dev57+g5c9057f 2023-03-14T14:39:27.182 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-03-14T17:12:39.935 INFO:teuthology.results:Tests finished! gathering results... 2023-03-14T17:12:43.690 INFO:scrape:Found 41 jobs 2023-03-14T17:12:45.595 INFO:scrape:Found 2 distinct failure reasons 2023-03-14T17:12:45.596 INFO:scrape:Failure: Command failed on smithi088 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' 2023-03-14T17:12:45.596 INFO:scrape:5 jobs: ['7206455', '7206466', '7206472', '7206445', '7206448'] 2023-03-14T17:12:45.597 INFO:scrape:suites intersection: ['frontend/beast', 'overrides'] 2023-03-14T17:12:45.597 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/tempest/{clusters/fixed-1', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/rgw_tempest', 'tasks/{cls', 'ubuntu_latest}', 'validater/lockdep}', 'validater/valgrind}'] 2023-03-14T17:12:45.597 INFO:scrape: 2023-03-14T17:12:45.598 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=2255065110167b1355215bbbded1cf4ec621a330 2023-03-14T17:12:45.598 INFO:scrape:2 jobs: ['7206461', '7206438'] 2023-03-14T17:12:45.598 INFO:scrape:suites intersection: ['overrides', 'rgw/hadoop-s3a/{clusters/fixed-2', 's3a-hadoop}'] 2023-03-14T17:12:45.599 INFO:scrape:suites union: ['hadoop/default', 'hadoop/v32', 'overrides', 'rgw/hadoop-s3a/{clusters/fixed-2', 's3a-hadoop}'] 2023-03-14T17:12:45.599 INFO:scrape: 2023-03-14T17:12:45.601 INFO:teuthology.results:Sending results to yweinstein@radhat.com: Test Run: yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 34 Fail ================================================================= [7206438] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} ----------------------------------------------------------------- time: 00:07:43 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206438/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206438/ sentry: https://sentry.ceph.com/organizations/ceph/?query=51e12eb9f41b4326883a49bb3ed52190 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =2255065110167b1355215bbbded1cf4ec621a330 [7206445] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:23:08 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206445/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206445/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2b1eae50773c425a9fb6500367ff3a69 Command failed on smithi138 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7206448] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:39:25 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206448/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206448/ sentry: https://sentry.ceph.com/organizations/ceph/?query=23eeccd8ba174d2a87bb57735aa8453e Command failed on smithi044 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' [7206455] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:46:20 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206455/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206455/ sentry: https://sentry.ceph.com/organizations/ceph/?query=39211ae036494569acfbd155cadb3ef2 Command failed on smithi088 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' [7206461] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} ----------------------------------------------------------------- time: 00:07:33 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206461/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206461/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f3d209d15672482e9b83c5c3e64be2e6 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1 =2255065110167b1355215bbbded1cf4ec621a330 [7206466] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:49:25 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206466/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206466/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b023de388ccf4a948a15bcbf5543712d Command failed on smithi144 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' [7206472] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:50:19 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206472/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206472/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d96c5d33894c4cfd8d7676c2bcd34d8b Command failed on smithi017 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' Pass ================================================================= [7206436] rgw/cloud-transition/{cluster overrides supported-random-distro$/{centos_8} tasks/cloud_transition_s3tests} time: 00:32:07 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206436/ [7206437] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:23:00 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206437/ [7206439] rgw/lifecycle/{cluster overrides supported-random-distro$/{ubuntu_latest} tasks/rgw_s3tests} time: 00:24:35 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206439/ [7206440] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:45 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206440/ [7206441] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi} time: 00:42:23 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206441/ [7206442] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:34:04 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206442/ [7206443] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:15:11 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206443/ [7206444] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{rhel_8} tasks/{0-install first ststests}} time: 00:32:02 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206444/ [7206446] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:16:02 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206446/ [7206447] rgw/tools/{centos_latest cluster tasks} time: 02:48:14 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206447/ [7206449] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:19:17 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206449/ [7206450] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:13:37 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206450/ [7206451] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{centos_8}} time: 00:17:14 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206451/ [7206452] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:09:56 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206452/ [7206453] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:14:29 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206453/ [7206454] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:19:13 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206454/ [7206456] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:13:54 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206456/ [7206457] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:15:23 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206457/ [7206458] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:15:06 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206458/ [7206459] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:12:58 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206459/ [7206460] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:14:06 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206460/ [7206462] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:10:07 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206462/ [7206463] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} time: 00:32:44 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206463/ [7206464] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:14:51 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206464/ [7206465] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:15 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206465/ [7206467] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:14:08 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206467/ [7206468] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{rhel_8}} time: 00:16:59 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206468/ [7206469] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:11:21 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206469/ [7206470] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:11:54 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206470/ [7206471] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:18:23 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206471/ [7206473] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability supported-random-distro$/{ubuntu_latest}} time: 00:14:41 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206473/ [7206474] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:42 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206474/ [7206475] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} time: 00:14:10 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206475/ [7206476] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:51 info: http://pulpito.ceph.com/yuriw-2023-03-14_14:09:30-rgw-wip-yuri10-testing-2023-03-13-1318-quincy-distro-default-smithi/7206476/