2023-11-14T15:37:01.539 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-14T15:37:01.541 INFO:teuthology.results:subset: '111/12000' 2023-11-14T15:37:01.541 INFO:teuthology.results:seed: '6710' 2023-11-14T16:08:03.659 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-14T16:08:03.679 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-14T20:25:24.442 INFO:teuthology.results:Tests finished! gathering results... 2023-11-14T20:25:27.819 INFO:scrape:Found 43 jobs 2023-11-14T20:25:29.803 INFO:scrape:Found 2 distinct failure reasons 2023-11-14T20:25:29.804 INFO:scrape:Failure: rgw multisite, test failures 2023-11-14T20:25:29.804 INFO:scrape:4 jobs: ['7457418', '7457407', '7457385', '7457397'] 2023-11-14T20:25:29.804 INFO:scrape:suites intersection: ['ignore-pg-availability', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2023-11-14T20:25:29.805 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2023-11-14T20:25:29.805 INFO:scrape: 2023-11-14T20:25:29.806 INFO:scrape:Failure: Command failed on smithi002 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' 2023-11-14T20:25:29.806 INFO:scrape:2 jobs: ['7457388', '7457410'] 2023-11-14T20:25:29.806 INFO:scrape:suites intersection: ['overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2023-11-14T20:25:29.807 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2023-11-14T20:25:29.807 INFO:scrape: 2023-11-14T20:25:29.809 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 37 Fail ================================================================= [7457385] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 00:12:45 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457385/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457385/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b13c92b7c42c4bfda371d5ba10826674 rgw multisite, test failures [7457388] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:22:01 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457388/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457388/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f1928636e64a4c5fb3d9809a4ebcc24c Command failed on smithi002 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7457397] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 00:32:10 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457397/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee7a98eda09d4213ad64060360cb7e7a rgw multisite, test failures [7457407] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:15:42 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457407/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457407/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9565b97ebabb4ca0a154103beea446bc rgw multisite, test failures [7457410] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:21:39 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457410/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457410/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f4975f37ba064e15a68abf20d38f699c Command failed on smithi146 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7457418] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 01:27:29 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457418/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457418/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b814bb05c9e940398eda8e40e3c84f01 rgw multisite, test failures Pass ================================================================= [7457382] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:23:45 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457382/ [7457383] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:19:32 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457383/ [7457384] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:53 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457384/ [7457386] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:16:41 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457386/ [7457387] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}} time: 00:29:40 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457387/ [7457389] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:46 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457389/ [7457390] rgw/tools/{centos_latest cluster tasks} time: 04:14:27 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457390/ [7457391] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 01:04:00 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457391/ [7457392] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:18:39 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457392/ [7457393] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:14:02 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457393/ [7457394] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:15:00 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457394/ [7457395] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:10:04 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457395/ [7457396] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:43:39 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457396/ [7457398] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:19:40 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457398/ [7457399] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 01:04:54 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457399/ [7457400] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:20 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457400/ [7457401] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:15:08 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457401/ [7457402] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:12:06 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457402/ [7457403] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:15:43 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457403/ [7457404] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:20 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457404/ [7457405] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:18:43 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457405/ [7457406] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:10:43 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457406/ [7457408] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:11:47 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457408/ [7457409] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/civetweb supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:32:20 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457409/ [7457411] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:33 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457411/ [7457412] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:37:56 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457412/ [7457413] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} time: 00:19:17 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457413/ [7457414] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:14:44 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457414/ [7457415] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:45 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457415/ [7457416] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:11:37 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457416/ [7457417] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:16:39 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457417/ [7457419] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:18:09 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457419/ [7457420] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 00:35:52 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457420/ [7457421] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:26 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457421/ [7457422] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:23 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457422/ [7457423] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:16:13 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457423/ [7457424] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:46 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:35:15-rgw-wip-yuri6-testing-2023-11-13-1419-pacific-distro-default-smithi/7457424/