2022-08-05T19:04:24.624 INFO:root:teuthology version: 1.1.1.dev407+g9e4bf63f 2022-08-05T19:04:24.628 INFO:teuthology.results:subset: '111/120000' 2022-08-05T19:04:24.628 INFO:teuthology.results:seed: '1173' 2022-08-05T19:20:51.823 INFO:root:teuthology version: 1.1.1.dev407+g9e4bf63f 2022-08-05T19:20:51.826 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-08-05T23:49:11.377 INFO:teuthology.results:Tests finished! gathering results... 2022-08-05T23:49:15.032 INFO:scrape:Found 43 jobs 2022-08-05T23:49:16.778 INFO:scrape:Found 3 distinct failure reasons 2022-08-05T23:49:16.779 INFO:scrape:Failure: rgw multisite pubsub, test failures 2022-08-05T23:49:16.779 INFO:scrape:4 jobs: ['6959889', '6959900', '6959879', '6959867'] 2022-08-05T23:49:16.780 INFO:scrape:suites intersection: ['ignore-pg-availability', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-08-05T23:49:16.780 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-08-05T23:49:16.781 INFO:scrape: 2022-08-05T23:49:16.781 INFO:scrape:Failure: Cannot create secret 2022-08-05T23:49:16.782 INFO:scrape:1 jobs: ['6959864'] 2022-08-05T23:49:16.782 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability}', 'rgw/crypt/{0-cluster/fixed-1'] 2022-08-05T23:49:16.782 INFO:scrape: 2022-08-05T23:49:16.783 INFO:scrape:Failure: "2022-08-05T19:35:12.622011+0000 mon.a (mon.0) 184 : cluster [WRN] Health check failed: Degraded data redundancy: 11/1940 objects degraded (0.567%), 1 pg degraded (PG_DEGRADED)" in cluster log 2022-08-05T23:49:16.783 INFO:scrape:1 jobs: ['6959898'] 2022-08-05T23:49:16.784 INFO:scrape:suites: ['frontend/civetweb', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/ec', 'tasks/rgw_bucket_quota}'] 2022-08-05T23:49:16.784 INFO:scrape: 2022-08-05T23:49:16.786 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 37 Fail ================================================================= [6959864] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:18:40 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959864/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=88f967bd134643158c95f995c73d6edb Cannot create secret [6959867] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 04:35:05 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959867/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959867/ sentry: https://sentry.ceph.com/organizations/ceph/?query=125d76377f01423c8be837cbe087da2c rgw multisite, rgw multisite pubsub, test failures [6959879] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} ----------------------------------------------------------------- time: 00:47:48 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959879/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959879/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8b88aaceb2c248a3bcfab8df36565f88 rgw multisite, test failures [6959889] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 00:40:10 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959889/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959889/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1310531dc9004312a57e10bbaf1434c8 rgw multisite pubsub, test failures [6959898] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota} ----------------------------------------------------------------- time: 00:11:59 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959898/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959898/ "2022-08-05T19:35:12.622011+0000 mon.a (mon.0) 184 : cluster [WRN] Health check failed: Degraded data redundancy: 11/1940 objects degraded (0.567%), 1 pg degraded (PG_DEGRADED)" in cluster log [6959900] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} ----------------------------------------------------------------- time: 02:23:34 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959900/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959900/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d050db3741e9420280a29ec24d8863ee rgw multisite, rgw multisite pubsub, test failures Pass ================================================================= [6959865] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:17:11 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959865/ [6959866] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload} time: 00:10:08 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959866/ [6959868] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:16:38 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959868/ [6959869] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{centos_8} tasks/{0-install ststests webidentity}} time: 00:30:40 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959869/ [6959870] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} time: 00:23:16 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959870/ [6959871] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:16:56 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959871/ [6959872] rgw/tools/{centos_latest cluster tasks} time: 02:45:59 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959872/ [6959873] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:28:53 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959873/ [6959874] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:18:11 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959874/ [6959875] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:12:49 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959875/ [6959876] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:27 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959876/ [6959877] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed} time: 00:09:41 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959877/ [6959878] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{rhel_8}} time: 00:16:27 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959878/ [6959880] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:18:08 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959880/ [6959881] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:33:59 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959881/ [6959882] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:13:24 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959882/ [6959883] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests} time: 00:14:33 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959883/ [6959884] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} time: 00:15:19 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959884/ [6959885] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:16:29 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959885/ [6959886] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:13:42 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959886/ [6959887] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:16:42 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959887/ [6959888] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota} time: 00:10:52 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959888/ [6959890] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{rhel_8}} time: 00:17:32 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959890/ [6959891] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/civetweb supported-random-distro$/{centos_8} tasks/{0-install ststests webidentity}} time: 00:31:32 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959891/ [6959892] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} time: 00:26:23 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959892/ [6959893] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:15:19 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959893/ [6959894] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:30:20 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959894/ [6959895] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} time: 00:19:05 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959895/ [6959896] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:14:33 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959896/ [6959897] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:14:01 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959897/ [6959899] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:13:01 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959899/ [6959901] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:17:16 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959901/ [6959902] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:34:22 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959902/ [6959903] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:13:09 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959903/ [6959904] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload} time: 00:10:36 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959904/ [6959905] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:12:08 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959905/ [6959906] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:14:37 info: http://pulpito.ceph.com/yuriw-2022-08-05_18:54:06-rgw-wip-yuri5-testing-2022-08-05-0827-pacific-distro-default-smithi/6959906/