2024-03-02T08:23:30.357 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-02T08:23:30.357 INFO:teuthology.results:subset: '111/120000' 2024-03-02T08:23:30.357 INFO:teuthology.results:seed: '8797' 2024-03-02T08:58:58.199 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-03-02T08:58:58.223 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-03-02T13:41:19.997 INFO:teuthology.results:Tests finished! gathering results... 2024-03-02T13:41:23.755 INFO:scrape:Found 43 jobs 2024-03-02T13:41:25.828 WARNING:scrape:Not-understood crash indication 2024-03-02T09:14:22.395 INFO:tasks.rgw.client.0.smithi057.stderr:daemon-helper: command crashed with signal 6 2024-03-02T13:41:25.828 WARNING:scrape:Not-understood crash indication 2024-03-02T13:41:25.871 WARNING:scrape:Not-understood crash indication 2024-03-02T09:14:22.395 INFO:tasks.rgw.client.0.smithi057.stderr:daemon-helper: command crashed with signal 6 2024-03-02T13:41:25.872 WARNING:scrape:Not-understood crash indication 2024-03-02T13:41:25.918 WARNING:scrape:Not-understood crash indication 2024-03-02T09:14:22.395 INFO:tasks.rgw.client.0.smithi057.stderr:daemon-helper: command crashed with signal 6 2024-03-02T13:41:25.918 WARNING:scrape:Not-understood crash indication 2024-03-02T13:41:25.960 WARNING:scrape:Not-understood crash indication 2024-03-02T09:14:22.395 INFO:tasks.rgw.client.0.smithi057.stderr:daemon-helper: command crashed with signal 6 2024-03-02T13:41:25.961 WARNING:scrape:Not-understood crash indication 2024-03-02T13:41:25.961 INFO:scrape:Found 2 distinct failure reasons 2024-03-02T13:41:25.961 INFO:scrape:Failure: rgw multisite test failures 2024-03-02T13:41:25.961 INFO:scrape:2 jobs: ['7578997', '7579018'] 2024-03-02T13:41:25.961 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2024-03-02T13:41:25.962 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_latest}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/test_multi}'] 2024-03-02T13:41:25.962 INFO:scrape: 2024-03-02T13:41:25.962 INFO:scrape:Failure: Command failed (bucket notification tests against different endpoints) on smithi057 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test' 2024-03-02T13:41:25.962 INFO:scrape:1 jobs: ['7578998'] 2024-03-02T13:41:25.962 INFO:scrape:suites: ['bluestore-bitmap', 'centos_latest', 'fixed-2', 'ignore-pg-availability', 'overrides', 'rgw/notifications/{beast', 'tasks/{0-install', 'test_amqp', 'test_kafka', 'test_others}}'] 2024-03-02T13:41:25.962 INFO:scrape: 2024-03-02T13:41:25.964 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 40 Fail ================================================================= [7578997] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{ubuntu_latest} tasks/test_multi} ----------------------------------------------------------------- time: 04:47:54 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578997/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578997/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e5d17697ccfb4522b8ee25a0f37aaca3 rgw multisite test failures [7578998] rgw/notifications/{beast bluestore-bitmap centos_latest fixed-2 ignore-pg-availability overrides tasks/{0-install test_amqp test_kafka test_others}} ----------------------------------------------------------------- time: 00:26:55 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578998/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578998/ sentry: https://sentry.ceph.com/organizations/ceph/?query=963d0d3f45ec41acbb4d69f3433966f0 Command failed (bucket notification tests against different endpoints) on smithi057 with status 1: 'BNTESTS_CONF=/home/ubuntu/cephtest/ceph/src/test/ rgw/bucket_notification/bn-tests.client.0.conf /home/ubuntu/cephtest/ceph/s rc/test/rgw/bucket_notification/virtualenv/bin/python -m nose -s /home/ubuntu/cephtest/ceph/src/test/rgw/bucket_notification/test_bn.py -v -a kafka_test' [7579018] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{centos_latest} tasks/test_multi} ----------------------------------------------------------------- time: 03:16:58 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579018/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579018/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4204fb79d3ff484c8e55b3d99fb65793 rgw multisite test failures Pass ================================================================= [7578977] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:21:25 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578977/ [7578978] rgw/upgrade/{1-install/reef/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:14:04 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578978/ [7578979] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:44:03 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578979/ [7578980] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:21:03 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578980/ [7578981] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:14:55 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578981/ [7578982] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{ubuntu_latest}} time: 00:14:38 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578982/ [7578983] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:21:44 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578983/ [7578984] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} time: 00:14:15 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578984/ [7578985] rgw/upgrade/{1-install/quincy/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:14:46 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578985/ [7578986] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 01:20:39 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578986/ [7578987] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:21:47 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578987/ [7578988] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:13:03 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578988/ [7578989] rgw/cloud-transition/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/cloud_transition_s3tests} time: 00:26:01 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578989/ [7578990] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:30:47 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578990/ [7578991] rgw/d4n/{cluster ignore-pg-availability overrides supported-random-distro$/{centos_latest} tasks/rgw_d4ntests} time: 00:09:47 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578991/ [7578992] rgw/dbstore/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests} time: 00:13:07 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578992/ [7578993] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default ignore-pg-availability overrides s3a-hadoop supported-random-distro$/{centos_latest}} time: 00:16:22 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578993/ [7578994] rgw/lifecycle/{cluster ignore-pg-availability overrides s3tests-branch supported-random-distro$/{centos_latest} tasks/rgw_s3tests} time: 00:32:47 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578994/ [7578995] rgw/lua/{beast bluestore-bitmap fixed-2 overrides supported-distros/{ubuntu_latest} tasks/{0-install test_lua}} time: 00:13:36 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578995/ [7578996] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:14:31 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578996/ [7578999] rgw/service-token/{clusters/fixed-1 frontend/beast ignore-pg-availability overrides tasks/service-token ubuntu_latest} time: 00:12:53 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7578999/ [7579000] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_latest}} time: 00:14:48 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579000/ [7579001] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/{0-install 1-keycloak 2-s3tests}} time: 00:39:40 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579001/ [7579002] rgw/tempest/{0-install clusters/fixed-1 frontend/beast ignore-pg-availability overrides s3tests-branch tasks/s3tests ubuntu_latest} time: 00:32:21 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579002/ [7579003] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:15:12 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579003/ [7579004] rgw/tools/{centos_latest cluster ignore-pg-availability tasks} time: 02:37:01 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579004/ [7579005] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:15:20 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579005/ [7579006] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/lockdep} time: 00:43:10 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579006/ [7579007] rgw/website/{clusters/fixed-2 frontend/beast http ignore-pg-availability overrides s3tests-branch tasks/s3tests-website ubuntu_latest} time: 00:21:15 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579007/ [7579008] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:13:22 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579008/ [7579009] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:21:02 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579009/ [7579010] rgw/thrash/{clusters/fixed-2 frontend/beast ignore-pg-availability install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:14:09 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579010/ [7579011] rgw/upgrade/{1-install/reef/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides} time: 00:14:45 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579011/ [7579012] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{bucket-check cls mp_reupload ragweed reshard s3tests-java s3tests versioning} validater/valgrind} time: 01:17:22 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579012/ [7579013] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:12:39 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579013/ [7579014] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_latest}} time: 00:15:20 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579014/ [7579015] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch ubuntu_latest} time: 00:20:53 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579015/ [7579016] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:21:35 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579016/ [7579017] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 ignore-pg-availability overrides s3a-hadoop supported-random-distro$/{centos_latest}} time: 00:15:54 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579017/ [7579019] rgw/tempest/{0-install clusters/fixed-1 frontend/beast ignore-pg-availability overrides s3tests-branch tasks/tempest ubuntu_latest} time: 00:21:37 info: https://pulpito.ceph.com/yuriw-2024-03-01_20:58:15-rgw-squid-distro-default-smithi/7579019/