2022-06-09T13:42:22.501 INFO:root:teuthology version: 1.1.1.dev370+g2290146e 2022-06-09T13:42:22.502 INFO:teuthology.results:subset: '111/120000' 2022-06-09T13:42:22.502 INFO:teuthology.results:seed: '228' 2022-06-09T21:30:17.590 INFO:root:teuthology version: 1.1.1.dev370+g2290146e 2022-06-09T21:30:17.607 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-06-10T09:23:18.056 INFO:teuthology.results:Tests finished! gathering results... 2022-06-10T09:23:21.340 INFO:scrape:Found 43 jobs 2022-06-10T09:23:21.537 WARNING:scrape:Not-understood crash indication 2022-06-09T21:25:07.810 INFO:tasks.rgw.c1.client.0.smithi174.stderr:daemon-helper: command crashed with signal 11 2022-06-10T09:23:21.538 WARNING:scrape:Not-understood crash indication 2022-06-10T09:23:21.705 WARNING:scrape:Not-understood crash indication 2022-06-09T21:37:27.756 INFO:tasks.rgw.c1.client.0.smithi046.stderr:daemon-helper: command crashed with signal 11 2022-06-10T09:23:21.707 WARNING:scrape:Not-understood crash indication 2022-06-09T21:37:31.749 INFO:tasks.rgw.c1.client.1.smithi046.stderr:daemon-helper: command crashed with signal 11 2022-06-10T09:23:21.707 WARNING:scrape:Not-understood crash indication 2022-06-10T09:23:21.726 WARNING:scrape:Not-understood crash indication 2022-06-09T21:37:27.756 INFO:tasks.rgw.c1.client.0.smithi046.stderr:daemon-helper: command crashed with signal 11 2022-06-10T09:23:21.727 WARNING:scrape:Not-understood crash indication 2022-06-09T21:37:31.749 INFO:tasks.rgw.c1.client.1.smithi046.stderr:daemon-helper: command crashed with signal 11 2022-06-10T09:23:21.728 WARNING:scrape:Not-understood crash indication 2022-06-10T09:23:21.747 WARNING:scrape:Not-understood crash indication 2022-06-09T21:37:27.756 INFO:tasks.rgw.c1.client.0.smithi046.stderr:daemon-helper: command crashed with signal 11 2022-06-10T09:23:21.749 WARNING:scrape:Not-understood crash indication 2022-06-09T21:37:31.749 INFO:tasks.rgw.c1.client.1.smithi046.stderr:daemon-helper: command crashed with signal 11 2022-06-10T09:23:21.749 WARNING:scrape:Not-understood crash indication 2022-06-10T09:23:22.015 WARNING:scrape:Not-understood crash indication 2022-06-09T14:01:44.681 INFO:tasks.rgw.c1.client.0.smithi060.stderr:daemon-helper: command crashed with signal 11 2022-06-10T09:23:22.016 WARNING:scrape:Not-understood crash indication 2022-06-10T09:23:22.017 INFO:scrape:Found 3 distinct failure reasons 2022-06-10T09:23:22.018 INFO:scrape:Dead: b'2022-06-10T09:16:25.772 DEBUG:teuthology.exit:Finished running handlers' 2022-06-10T09:23:22.018 INFO:scrape:2 jobs: ['6869889', '6869867'] 2022-06-10T09:23:22.018 INFO:scrape:suites intersection: ['ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-06-10T09:23:22.019 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/three-zone-plus-pubsub', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-06-10T09:23:22.019 INFO:scrape: 2022-06-10T09:23:22.020 INFO:scrape:Failure: rgw multisite, rgw multisite pubsub, test failures 2022-06-10T09:23:22.020 INFO:scrape:2 jobs: ['6869900', '6869879'] 2022-06-10T09:23:22.021 INFO:scrape:suites intersection: ['ignore-pg-availability', 'omap_limits', 'overrides', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-06-10T09:23:22.021 INFO:scrape:suites union: ['frontend/beast', 'frontend/civetweb', 'ignore-pg-availability', 'omap_limits', 'overrides', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'tasks/test_multi', 'valgrind}'] 2022-06-10T09:23:22.021 INFO:scrape: 2022-06-10T09:23:22.022 INFO:scrape:Failure: Cannot create secret 2022-06-10T09:23:22.022 INFO:scrape:1 jobs: ['6869864'] 2022-06-10T09:23:22.023 INFO:scrape:suites: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'ignore-pg-availability}', 'rgw/crypt/{0-cluster/fixed-1'] 2022-06-10T09:23:22.023 INFO:scrape: 2022-06-10T09:23:22.026 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/ failed: 3 dead: 2 running: 0 waiting: 0 queued: 0 passed: 38 Fail ================================================================= [6869864] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} ----------------------------------------------------------------- time: 00:14:52 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869864/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4bc572fd2d8842c19a742bbbb21c9f4e Cannot create secret [6869879] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} ----------------------------------------------------------------- time: 06:27:29 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869879/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869879/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2fa8e8fe7993441c9b52673b04ca6089 rgw multisite, rgw multisite pubsub, test failures [6869900] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/two-zonegroup tasks/test_multi valgrind} ----------------------------------------------------------------- time: 07:22:02 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869900/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869900/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ba0d53fb8c74db68165fd42a33bd500 rgw multisite, rgw multisite pubsub, test failures Dead ================================================================= [6869867] rgw/multisite/{clusters frontend/beast ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869867/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869867/ hit max job timeout [6869889] rgw/multisite/{clusters frontend/civetweb ignore-pg-availability omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869889/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869889/ hit max job timeout Pass ================================================================= [6869865] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:15:38 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869865/ [6869866] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile tasks/rgw_multipart_upload} time: 00:08:22 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869866/ [6869868] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:09:07 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869868/ [6869869] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast supported-random-distro$/{ubuntu_latest} tasks/{0-install ststests webidentity}} time: 00:25:21 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869869/ [6869870] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} time: 00:20:26 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869870/ [6869871] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:12:07 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869871/ [6869872] rgw/tools/{centos_latest cluster tasks} time: 02:48:12 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869872/ [6869873] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:24:06 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869873/ [6869874] rgw/website/{clusters/fixed-2 frontend/beast http overrides tasks/s3tests-website ubuntu_latest} time: 00:14:39 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869874/ [6869875] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:09:39 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869875/ [6869876] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:10:33 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869876/ [6869877] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec tasks/rgw_ragweed} time: 00:06:49 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869877/ [6869878] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec supported-random-distro$/{centos_8}} time: 00:10:16 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869878/ [6869880] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:14:38 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869880/ [6869881] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:29:43 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869881/ [6869882] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:09:30 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869882/ [6869883] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated tasks/rgw_s3tests} time: 00:10:35 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869883/ [6869884] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} time: 00:09:54 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869884/ [6869885] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:09:20 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869885/ [6869886] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:10:06 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869886/ [6869887] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} time: 00:11:34 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869887/ [6869888] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/ec-profile tasks/rgw_user_quota} time: 00:06:43 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869888/ [6869890] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/ec-profile supported-random-distro$/{centos_8}} time: 00:09:53 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869890/ [6869891] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/civetweb supported-random-distro$/{ubuntu_latest} tasks/{0-install ststests webidentity}} time: 00:24:10 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869891/ [6869892] rgw/tempest/{clusters/fixed-1 frontend/civetweb overrides tasks/rgw_tempest ubuntu_latest} time: 00:19:10 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869892/ [6869893] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_bucket_quota} time: 00:11:33 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869893/ [6869894] rgw/verify/{centos_latest clusters/fixed-2 frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:24:21 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869894/ [6869895] rgw/website/{clusters/fixed-2 frontend/civetweb http overrides tasks/s3tests-website ubuntu_latest} time: 00:14:24 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869895/ [6869896] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_multipart_upload} time: 00:12:52 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869896/ [6869897] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:09:57 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869897/ [6869898] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec tasks/rgw_bucket_quota} time: 00:07:52 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869898/ [6869899] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:08:28 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869899/ [6869901] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/filestore-xfs thrasher/default thrashosds-health workload/rgw_s3tests} time: 00:16:11 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869901/ [6869902] rgw/verify/{centos_latest clusters/fixed-2 frontend/civetweb ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{0-install cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:29:07 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869902/ [6869903] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability} time: 00:09:59 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869903/ [6869904] rgw/multifs/{clusters/fixed-2 frontend/civetweb ignore-pg-availability objectstore/filestore-xfs overrides rgw_pool_type/replicated tasks/rgw_multipart_upload} time: 00:06:51 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869904/ [6869905] rgw/singleton/{all/radosgw-admin frontend/civetweb objectstore/filestore-xfs overrides rgw_pool_type/replicated supported-random-distro$/{rhel_8}} time: 00:22:15 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869905/ [6869906] rgw/thrash/{civetweb clusters/fixed-2 install objectstore/bluestore-bitmap thrasher/default thrashosds-health workload/rgw_user_quota} time: 00:10:07 info: http://pulpito.ceph.com/yuriw-2022-06-09_13:25:04-rgw-wip-yuri-testing-2022-06-08-1015-pacific-distro-default-smithi/6869906/