2023-11-14T17:16:19.103 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-14T17:16:19.104 INFO:teuthology.results:subset: '111/12000' 2023-11-14T17:16:19.104 INFO:teuthology.results:seed: '1019' 2023-11-14T18:03:38.067 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-14T18:03:38.069 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-14T20:57:52.171 INFO:teuthology.results:Tests finished! gathering results... 2023-11-14T20:57:55.340 INFO:scrape:Found 43 jobs 2023-11-14T20:57:57.348 INFO:scrape:Found 2 distinct failure reasons 2023-11-14T20:57:57.349 INFO:scrape:Failure: rgw multisite, test failures 2023-11-14T20:57:57.349 INFO:scrape:4 jobs: ['7457508', '7457497', '7457475', '7457487'] 2023-11-14T20:57:57.349 INFO:scrape:suites intersection: ['ignore-pg-availability', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2023-11-14T20:57:57.350 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2023-11-14T20:57:57.350 INFO:scrape: 2023-11-14T20:57:57.351 INFO:scrape:Failure: Command failed on smithi195 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' 2023-11-14T20:57:57.351 INFO:scrape:2 jobs: ['7457500', '7457478'] 2023-11-14T20:57:57.351 INFO:scrape:suites intersection: ['overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2023-11-14T20:57:57.352 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2023-11-14T20:57:57.352 INFO:scrape: 2023-11-14T20:57:57.354 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 37 Fail ================================================================= [7457475] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:16:46 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457475/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457475/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4c5a8597ca4a411c853c35202cd60b48 rgw multisite, test failures [7457478] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:21:19 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457478/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457478/ sentry: https://sentry.ceph.com/organizations/ceph/?query=72d0c03f180449e6bb66c8a3e019b180 Command failed on smithi044 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7457487] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 02:34:38 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457487/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457487/ sentry: https://sentry.ceph.com/organizations/ceph/?query=097440aa6b9444fe91b257e03ab65c68 rgw multisite, test failures [7457497] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zones supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 00:17:45 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457497/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457497/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd585c0dc118463d828016559069d145 rgw multisite, test failures [7457500] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:21:03 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457500/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457500/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ab416c14a08248b789ee12271a8b4a1a Command failed on smithi195 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7457508] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 02:39:45 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457508/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457508/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ee9a8444d8da46e1bd1f846f7f46432e rgw multisite, test failures Pass ================================================================= [7457472] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:23:08 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457472/ [7457473] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:18:55 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457473/ [7457474] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload ubuntu_latest} time: 00:11:03 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457474/ [7457476] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:16:57 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457476/ [7457477] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:31:25 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457477/ [7457479] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:14:38 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457479/ [7457480] rgw/tools/{centos_latest cluster tasks} time: 02:46:45 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457480/ [7457481] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:34:22 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457481/ [7457482] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:19:18 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457482/ [7457483] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:13:30 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457483/ [7457484] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:44 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457484/ [7457485] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed ubuntu_latest} time: 00:09:45 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457485/ [7457486] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:16:29 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457486/ [7457488] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:18:15 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457488/ [7457489] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 00:37:58 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457489/ [7457490] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:00 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457490/ [7457491] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests ubuntu_latest} time: 00:15:02 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457491/ [7457492] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:16:54 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457492/ [7457493] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:41 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457493/ [7457494] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:03 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457494/ [7457495] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:18:03 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457495/ [7457496] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota ubuntu_latest} time: 00:10:27 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457496/ [7457498] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:18:11 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457498/ [7457499] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/civetweb supported-random-distro$/{rhel_8} tasks/{0-install first ststests}} time: 00:33:15 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457499/ [7457501] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:15:11 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457501/ [7457502] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:31:53 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457502/ [7457503] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} time: 00:18:41 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457503/ [7457504] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:14:09 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457504/ [7457505] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:17 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457505/ [7457506] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota ubuntu_latest} time: 00:11:34 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457506/ [7457507] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:15:17 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457507/ [7457509] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:17:27 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457509/ [7457510] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install bucket-check cls ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 00:39:17 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457510/ [7457511] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:12 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457511/ [7457512] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload ubuntu_latest} time: 00:10:25 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457512/ [7457513] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} time: 00:15:39 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457513/ [7457514] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:13:56 info: https://pulpito.ceph.com/yuriw-2023-11-14_15:38:06-rgw-wip-yuri2-testing-2023-11-13-0820-pacific-distro-default-smithi/7457514/