2022-10-07T18:42:04.333 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-07T18:42:04.335 INFO:teuthology.results:subset: '111/120000' 2022-10-07T18:42:04.335 INFO:teuthology.results:seed: '2511' 2022-10-07T19:33:04.559 INFO:root:teuthology version: 1.1.1.dev427+g35ea38a9 2022-10-07T19:33:04.587 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-10-08T00:37:29.701 INFO:teuthology.results:Tests finished! gathering results... 2022-10-08T00:37:33.482 INFO:scrape:Found 43 jobs 2022-10-08T00:37:36.187 INFO:scrape:Found 4 distinct failure reasons 2022-10-08T00:37:36.188 INFO:scrape:Failure: "2022-10-07T19:25:03.800175+0000 mon.a (mon.0) 178 : cluster [WRN] Health check failed: Degraded data redundancy: 67/11688 objects degraded (0.573%), 3 pgs degraded (PG_DEGRADED)" in cluster log 2022-10-08T00:37:36.189 INFO:scrape:1 jobs: ['7058110'] 2022-10-08T00:37:36.189 INFO:scrape:suites: ['frontend/civetweb', 'ignore-pg-availability', 'objectstore/bluestore-bitmap', 'overrides', 'rgw/multifs/{clusters/fixed-2', 'rgw_pool_type/replicated', 'tasks/rgw_s3tests}'] 2022-10-08T00:37:36.190 INFO:scrape: 2022-10-08T00:37:36.191 INFO:scrape:Failure: Cannot create secret 2022-10-08T00:37:36.191 INFO:scrape:1 jobs: ['7058091'] 2022-10-08T00:37:36.192 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability}', 'rgw/crypt/{0-cluster/fixed-1'] 2022-10-08T00:37:36.192 INFO:scrape: 2022-10-08T00:37:36.193 INFO:scrape:Failure: rgw multisite, test failures 2022-10-08T00:37:36.194 INFO:scrape:3 jobs: ['7058127', '7058094', '7058116'] 2022-10-08T00:37:36.194 INFO:scrape:suites intersection: ['ignore-pg-availability', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-10-08T00:37:36.195 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-10-08T00:37:36.196 INFO:scrape: 2022-10-08T00:37:36.197 INFO:scrape:Failure: Command failed on smithi158 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' 2022-10-08T00:37:36.197 INFO:scrape:1 jobs: ['7058129'] 2022-10-08T00:37:36.198 INFO:scrape:suites: ['cls', 'clusters/fixed-2', 'frontend/civetweb', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{centos_latest', 'rgw_pool_type/ec-profile', 's3tests-java', 's3tests}', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{0-install', 'validater/valgrind}'] 2022-10-08T00:37:36.199 INFO:scrape: 2022-10-08T00:37:36.202 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 37 Fail ================================================================= [7058091] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:13:13 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058091/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058091/ sentry: https://sentry.ceph.com/organizations/ceph/?query=57716a1272e741e4aefaece59e555912 Cannot create secret [7058094] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 05:46:07 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058094/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058094/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af2500a7986e4e41b71709c5aea98762 rgw multisite, rgw multisite pubsub, test failures [7058110] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests} ----------------------------------------------------------------- time: 00:10:47 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058110/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058110/ "2022-10-07T19:25:03.800175+0000 mon.a (mon.0) 178 : cluster [WRN] Health check failed: Degraded data redundancy: 67/11688 objects degraded (0.573%), 3 pgs degraded (PG_DEGRADED)" in cluster log [7058116] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 00:34:31 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058116/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058116/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4d2a84793b248c29d95d563798af58c rgw multisite, rgw multisite pubsub, test failures [7058127] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} ----------------------------------------------------------------- time: 01:07:56 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058127/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058127/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fc5f64778b2a408592d4cd2b64343bf3 rgw multisite, test failures [7058129] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:23:11 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058129/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058129/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bea8b99d049340b994b8c7fab845eb6a Command failed on smithi158 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' Pass ================================================================= [7058092] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:14:07 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058092/ [7058093] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload} time: 00:07:39 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058093/ [7058095] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:08:21 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058095/ [7058096] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{rhel_8} tasks/{0-install first ststests}} time: 00:27:28 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058096/ [7058097] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} time: 00:19:40 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058097/ [7058098] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:11:29 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058098/ [7058099] rgw/tools/{centos_latest cluster tasks} time: 02:44:10 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058099/ [7058100] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:27:11 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058100/ [7058101] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:13:58 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058101/ [7058102] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:10:17 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058102/ [7058103] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:10:24 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058103/ [7058104] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed} time: 00:06:22 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058104/ [7058105] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:10:08 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058105/ [7058106] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} time: 00:31:49 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058106/ [7058107] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:14:24 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058107/ [7058108] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:29:27 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058108/ [7058109] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:10:00 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058109/ [7058111] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:11:26 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058111/ [7058112] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:10:38 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058112/ [7058113] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:09:57 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058113/ [7058114] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:12:49 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058114/ [7058115] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota} time: 00:07:04 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058115/ [7058117] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:10:22 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058117/ [7058118] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/civetweb supported-random-distro$/{centos_8} tasks/{0-install first ststests}} time: 00:26:25 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058118/ [7058119] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} time: 00:19:34 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058119/ [7058120] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:11:53 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058120/ [7058121] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:23:25 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058121/ [7058122] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} time: 00:14:12 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058122/ [7058123] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:10:42 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058123/ [7058124] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:10:07 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058124/ [7058125] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota} time: 00:07:58 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058125/ [7058126] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:08:27 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058126/ [7058128] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:13:50 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058128/ [7058130] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:10:05 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058130/ [7058131] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload} time: 00:06:39 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058131/ [7058132] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} time: 00:10:04 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058132/ [7058133] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:10:37 info: http://pulpito.ceph.com/yuriw-2022-10-07_16:44:11-rgw-wip-yuri2-testing-2022-10-05-1340-pacific-distro-default-smithi/7058133/