2022-12-22T21:13:11.532 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-22T21:13:11.533 INFO:teuthology.results:subset: '111/120000' 2022-12-22T21:13:11.533 INFO:teuthology.results:seed: '1651' 2022-12-22T22:11:35.590 INFO:root:teuthology version: 0.0.1.dev18+g19d18db 2022-12-22T22:11:35.612 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-12-23T02:19:56.578 INFO:teuthology.results:Tests finished! gathering results... 2022-12-23T02:20:00.378 INFO:scrape:Found 43 jobs 2022-12-23T02:20:04.272 INFO:scrape:Found 4 distinct failure reasons 2022-12-23T02:20:04.272 INFO:scrape:Failure: Command failed on smithi060 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' 2022-12-23T02:20:04.273 INFO:scrape:2 jobs: ['7125349', '7125401'] 2022-12-23T02:20:04.273 INFO:scrape:suites intersection: ['overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-12-23T02:20:04.273 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'overrides', 'rgw/tempest/{clusters/fixed-1', 'tasks/rgw_tempest', 'ubuntu_latest}'] 2022-12-23T02:20:04.273 INFO:scrape: 2022-12-23T02:20:04.273 INFO:scrape:Failure: rgw multisite pubsub, test failures 2022-12-23T02:20:04.273 INFO:scrape:3 jobs: ['7125397', '7125342', '7125378'] 2022-12-23T02:20:04.273 INFO:scrape:suites intersection: ['ignore-pg-availability', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-12-23T02:20:04.273 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-12-23T02:20:04.274 INFO:scrape: 2022-12-23T02:20:04.274 INFO:scrape:Failure: "2022-12-22T22:37:37.345981+0000 mon.a (mon.0) 512 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log 2022-12-23T02:20:04.274 INFO:scrape:2 jobs: ['7125422', '7125368'] 2022-12-23T02:20:04.274 INFO:scrape:suites intersection: [] 2022-12-23T02:20:04.274 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/kmip', '3-rgw/rgw', '4-tests/{s3tests}', 'cls', 'clusters/fixed-2', 'frontend/civetweb', 'ignore-pg-availability', 'ignore-pg-availability}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/verify/{centos_latest', 'rgw_pool_type/ec-profile', 's3tests-java', 's3tests}', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{0-install', 'validater/valgrind}'] 2022-12-23T02:20:04.274 INFO:scrape: 2022-12-23T02:20:04.274 INFO:scrape:Failure: Cannot create secret 2022-12-23T02:20:04.274 INFO:scrape:1 jobs: ['7125336'] 2022-12-23T02:20:04.274 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability}', 'rgw/crypt/{0-cluster/fixed-1'] 2022-12-23T02:20:04.275 INFO:scrape: 2022-12-23T02:20:04.276 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/ failed: 8 dead: 0 running: 0 waiting: 0 queued: 0 passed: 35 Fail ================================================================= [7125336] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:16:14 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125336/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125336/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2ec4e28f3b5498abed785270f21f35a Cannot create secret [7125342] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 04:49:18 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125342/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125342/ sentry: https://sentry.ceph.com/organizations/ceph/?query=02e40605197d473da321f632db16e46e rgw multisite, rgw multisite pubsub, test failures [7125349] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:20:57 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125349/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125349/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2c6a4900f1d049aeb1d3f64a121058ee Command failed on smithi060 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7125368] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:13:05 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125368/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125368/ "2022-12-22T21:50:38.050675+0000 mon.a (mon.0) 158 : cluster [WRN] Health check failed: Degraded data redundancy: 8/1564 objects degraded (0.512%), 1 pg degraded (PG_DEGRADED)" in cluster log [7125378] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} ----------------------------------------------------------------- time: 02:28:13 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125378/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125378/ sentry: https://sentry.ceph.com/organizations/ceph/?query=eec2267251cc4d169f1c8e614d9508a5 rgw multisite, rgw multisite pubsub, test failures [7125397] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 00:40:25 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125397/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125397/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f61e130a54746229f78624aaa925d9f rgw multisite pubsub, test failures [7125401] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} ----------------------------------------------------------------- time: 00:20:30 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125401/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125401/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0aa8b9a353f348cca24e87e5ae7595c2 Command failed on smithi060 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest' [7125422] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:36:50 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125422/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125422/ "2022-12-22T22:37:37.345981+0000 mon.a (mon.0) 512 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log Pass ================================================================= [7125338] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:16:36 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125338/ [7125340] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload} time: 00:10:10 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125340/ [7125343] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:14:11 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125343/ [7125345] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{ubuntu_latest} tasks/{0-install first ststests}} time: 00:28:10 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125345/ [7125352] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:14:13 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125352/ [7125355] rgw/tools/{centos_latest cluster tasks} time: 02:52:30 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125355/ [7125358] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:28:27 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125358/ [7125361] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:17:26 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125361/ [7125365] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:12:47 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125365/ [7125372] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed} time: 00:09:17 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125372/ [7125375] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:12:11 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125375/ [7125379] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:17:21 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125379/ [7125382] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:32:34 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125382/ [7125383] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:12:17 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125383/ [7125386] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests} time: 00:13:34 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125386/ [7125387] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:11:52 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125387/ [7125389] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:12:40 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125389/ [7125391] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:13:18 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125391/ [7125393] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:14:52 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125393/ [7125395] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota} time: 00:09:51 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125395/ [7125398] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:11:11 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125398/ [7125399] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/civetweb supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:29:43 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125399/ [7125404] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:13:38 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125404/ [7125406] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:29:24 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125406/ [7125407] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} time: 00:17:12 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125407/ [7125409] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:13:45 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125409/ [7125412] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:12:34 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125412/ [7125414] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota} time: 00:10:12 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125414/ [7125416] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:14:14 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125416/ [7125418] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} time: 00:31:33 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125418/ [7125420] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:16:28 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125420/ [7125425] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:12:16 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125425/ [7125428] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload} time: 00:09:10 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125428/ [7125429] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:10:59 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125429/ [7125432] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:15:42 info: http://pulpito.ceph.com/yuriw-2022-12-22_21:11:51-rgw-pacific_16.2.11_RC3-distro-default-smithi/7125432/