2024-03-06T17:25:01.471 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-06T17:25:01.471 INFO:teuthology.results:subset: '1/50' 2024-03-06T17:25:01.471 INFO:teuthology.results:seed: '831' 2024-03-06T17:26:25.597 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-06T17:26:25.601 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-06T21:16:42.208 INFO:teuthology.results:Tests finished! gathering results... 2024-03-06T21:16:45.673 INFO:scrape:Found 41 jobs 2024-03-06T21:16:46.595 WARNING:scrape:Not-understood crash indication 2024-03-06T18:46:51.666 INFO:tasks.rgw.c2.client.0.smithi172.stderr:daemon-helper: command crashed with signal 11 2024-03-06T21:16:46.595 WARNING:scrape:Not-understood crash indication 2024-03-06T20:44:13.655 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi172.stderr: INFO: daemon-helper: command crashed with signal 11 2024-03-06T21:16:46.595 WARNING:scrape:Not-understood crash indication 2024-03-06T21:16:46.843 WARNING:scrape:Not-understood crash indication 2024-03-06T18:46:51.666 INFO:tasks.rgw.c2.client.0.smithi172.stderr:daemon-helper: command crashed with signal 11 2024-03-06T21:16:46.843 WARNING:scrape:Not-understood crash indication 2024-03-06T20:44:13.655 INFO:tasks.rgw_multisite_tests:tasks.rgw.c2.client.0.smithi172.stderr: INFO: daemon-helper: command crashed with signal 11 2024-03-06T21:16:46.844 WARNING:scrape:Not-understood crash indication 2024-03-06T21:16:46.844 INFO:scrape:Found 2 distinct failure reasons 2024-03-06T21:16:46.844 INFO:scrape:Failure: Command failed (bucket notification tests against different endpoints) on smithi078 with status 1: "BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a '!kafka_test,!amqp_test,!amqp_ssl_test,!kafka_security_test,!modification_required,!manual_test'" 2024-03-06T21:16:46.844 INFO:scrape:1 jobs: ['7583567'] 2024-03-06T21:16:46.844 INFO:scrape:suites: ['bluestore-bitmap', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'supported-distros/{ubuntu_latest}', 'tasks/others/{0-install', 'test_others}}'] 2024-03-06T21:16:46.844 INFO:scrape: 2024-03-06T21:16:46.845 INFO:scrape:Failure: rgw multisite test failures 2024-03-06T21:16:46.845 INFO:scrape:2 jobs: ['7583561', '7583540'] 2024-03-06T21:16:46.845 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2024-03-06T21:16:46.845 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2024-03-06T21:16:46.845 INFO:scrape: 2024-03-06T21:16:46.846 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 38 Fail ================================================================= [7583540] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 03:38:41 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583540/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583540/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b206bd437c7941f28cd75557e68890db rgw multisite test failures [7583561] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 03:19:12 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583561/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583561/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a9492c2263034aceb566a0db03cc5786 rgw multisite test failures [7583567] rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/others/{0-install supported-distros/{ubuntu_latest} test_others}} ----------------------------------------------------------------- time: 00:24:07 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583567/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583567/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e4f4c7c5e3194aef9267ec73fe926e24 Command failed (bucket notification tests against different endpoints) on smithi078 with status 1: "BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/ rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/s rc/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a '!kafka_test,!amqp_test,!amqp_ssl_test,!kafka_security_test,!modificatio n_required,!manual_test'" Pass ================================================================= [7583532] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:16:45 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583532/ [7583533] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/cloud_transition_s3tests} time: 00:28:32 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583533/ [7583534] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:30:45 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583534/ [7583535] rgw/d4n/{cluster ignore-pg-availability overrides supported-random-distro$/{centos_latest} tasks/rgw_d4ntests} time: 00:12:05 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583535/ [7583536] rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests} time: 00:15:12 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583536/ [7583537] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default ignore-pg-availability overrides s3a-hadoop supported-random-distro$/{centos_latest}} time: 00:19:54 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583537/ [7583538] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests} time: 00:33:56 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583538/ [7583539] rgw/lua/{beast bluestore-bitmap fixed-2 overrides supported-distros/{centos_latest} tasks/{0-install test_lua}} time: 00:13:39 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583539/ [7583541] rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/amqp/{0-install centos_latest test_amqp}} time: 00:18:19 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583541/ [7583542] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest} time: 00:13:55 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583542/ [7583543] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:16:32 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583543/ [7583544] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{centos_latest} tasks/{0-install 1-keycloak 2-s3tests}} time: 00:34:34 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583544/ [7583545] rgw/tempest/{0-install clusters/fixed-1 frontend/beast ignore-pg-availability overrides s3tests-branch tasks/s3tests ubuntu_latest} time: 00:32:34 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583545/ [7583546] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:19:02 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583546/ [7583547] rgw/tools/{centos_latest cluster ignore-pg-availability tasks} time: 02:42:55 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583547/ [7583548] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:17:57 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583548/ [7583549] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch tasks/s3tests-website ubuntu_latest} time: 00:22:39 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583549/ [7583550] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:22:31 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583550/ [7583551] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:22:14 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583551/ [7583552] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:14:55 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583552/ [7583553] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:15:24 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583553/ [7583554] rgw/upgrade/{1-install/reef/{distro$/{ubuntu_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:17:38 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583554/ [7583555] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:16:33 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583555/ [7583556] rgw/notifications/{beast bluestore-bitmap fixed-2 ignore-pg-availability overrides tasks/kafka/{0-install supported-distros/{ubuntu_latest} test_kafka}} time: 00:25:06 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583556/ [7583557] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{centos_latest}} time: 00:14:37 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583557/ [7583558] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:22:10 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583558/ [7583559] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:15:39 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583559/ [7583560] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 ignore-pg-availability overrides s3a-hadoop supported-random-distro$/{ubuntu_latest}} time: 00:19:43 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583560/ [7583562] rgw/tempest/{0-install clusters/fixed-1 frontend/beast ignore-pg-availability overrides s3tests-branch tasks/tempest ubuntu_latest} time: 00:23:11 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583562/ [7583563] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:22:47 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583563/ [7583564] rgw/upgrade/{1-install/reef/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:15:46 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583564/ [7583565] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:22:37 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583565/ [7583566] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:17:04 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583566/ [7583568] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_latest}} time: 00:14:07 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583568/ [7583569] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:23:02 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583569/ [7583570] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:16:52 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583570/ [7583571] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:16:52 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583571/ [7583572] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:22:40 info: https://pulpito.ceph.com/ivancich-2024-03-06_17:24:06-rgw-wip-eric-testing-1-distro-default-smithi/7583572/