2023-06-28T21:28:01.532 INFO:root:teuthology version: 0.0.1.dev134+ge25bf39 2023-06-28T21:28:01.533 INFO:teuthology.results:seed: '9461' 2023-06-28T22:05:17.439 INFO:root:teuthology version: 0.0.1.dev134+ge25bf39 2023-06-28T22:05:17.459 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-06-29T05:02:45.891 INFO:teuthology.results:Tests finished! gathering results... 2023-06-29T05:02:51.093 INFO:scrape:Found 70 jobs 2023-06-29T05:02:51.094 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320893/teuthology.log 2023-06-29T05:02:51.862 WARNING:scrape:Not-understood crash indication 2023-06-28T22:36:19.958 INFO:tasks.rgw.c1.client.0.smithi125.stderr:daemon-helper: command crashed with signal 6 2023-06-29T05:02:51.862 WARNING:scrape:Not-understood crash indication 2023-06-29T04:44:19.394 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi125.stderr: INFO: daemon-helper: command crashed with signal 6 2023-06-29T05:02:51.863 WARNING:scrape:Not-understood crash indication 2023-06-29T05:02:52.093 WARNING:scrape:Not-understood crash indication 2023-06-28T22:36:19.958 INFO:tasks.rgw.c1.client.0.smithi125.stderr:daemon-helper: command crashed with signal 6 2023-06-29T05:02:52.094 WARNING:scrape:Not-understood crash indication 2023-06-29T04:44:19.394 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi125.stderr: INFO: daemon-helper: command crashed with signal 6 2023-06-29T05:02:52.094 WARNING:scrape:Not-understood crash indication 2023-06-29T05:02:52.331 WARNING:scrape:Not-understood crash indication 2023-06-28T22:36:19.958 INFO:tasks.rgw.c1.client.0.smithi125.stderr:daemon-helper: command crashed with signal 6 2023-06-29T05:02:52.332 WARNING:scrape:Not-understood crash indication 2023-06-29T04:44:19.394 INFO:tasks.rgw_multisite_tests:tasks.rgw.c1.client.0.smithi125.stderr: INFO: daemon-helper: command crashed with signal 6 2023-06-29T05:02:52.332 WARNING:scrape:Not-understood crash indication 2023-06-29T05:02:52.757 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320892/teuthology.log matches 2023-06-29T05:02:52.758 WARNING:scrape:Misunderstood line: 2023-06-29T05:02:53.207 INFO:scrape:Found 3 distinct failure reasons 2023-06-29T05:02:53.208 INFO:scrape:Dead: None 2023-06-29T05:02:53.208 INFO:scrape:14 jobs: ['7320893', '7320915', '7320887', '7320898', '7320912', '7320913', '7320871', '7320881', '7320877', '7320901', '7320865', '7320911', '7320891', '7320867'] 2023-06-29T05:02:53.208 INFO:scrape: 2023-06-29T05:02:53.209 INFO:scrape:Failure: rgw multisite test failures 2023-06-29T05:02:53.209 INFO:scrape:2 jobs: ['7320899', '7320863'] 2023-06-29T05:02:53.209 INFO:scrape:suites intersection: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'rgw/multisite/{clusters', 'tasks/test_multi}'] 2023-06-29T05:02:53.210 INFO:scrape:suites union: ['frontend/beast', 'ignore-pg-availability', 'notify', 'omap_limits', 'overrides', 'realms/three-zones', 'realms/two-zonegroup', 'rgw/multisite/{clusters', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'tasks/test_multi}'] 2023-06-29T05:02:53.210 INFO:scrape: 2023-06-29T05:02:53.210 INFO:scrape:Valgrind: 2023-06-29T05:02:53.210 INFO:scrape:3 jobs: ['7320892', '7320857', '7320859'] 2023-06-29T05:02:53.211 INFO:scrape:suites intersection: ['s3tests-branch'] 2023-06-29T05:02:53.211 INFO:scrape:suites union: ['1-ceph-install/install', '2-kms/barbican', '3-rgw/rgw', '4-tests/{s3tests}', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw/crypt/{0-cluster/fixed-1', 'rgw/dbstore/{cluster', 'rgw/verify/{0-install', 'rgw_pool_type/ec-profile', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/rgw_s3tests}', 'tasks/{cls', 'validater/valgrind}'] 2023-06-29T05:02:53.211 INFO:scrape: 2023-06-29T05:02:53.214 INFO:teuthology.results:Sending results to mkogan@redhat.com: Test Run: mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/ failed: 5 dead: 14 running: 0 waiting: 0 queued: 0 passed: 51 Fail ================================================================= [7320857] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/barbican 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}} ----------------------------------------------------------------- time: 00:20:31 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320857/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320857/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9dc665ed027144709205f33cbfd8083a Command failed on smithi060 with status 1: "cd /home/ubuntu/cephtest/keystone && source .tox/venv/bin/activate && openstack project create --or-show --os-username admin --os-password ADMIN --os-user-domain-id default --os-project-name admin --os-project-domain-id default --os-identity-api-version 3 --os-auth-url http://smithi060.front.sepia.ceph.com:5000/v3 --description 'Encryption Tenant' --domain default rgwcrypt --debug" [7320859] rgw/dbstore/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/rgw_s3tests} ----------------------------------------------------------------- time: 00:20:16 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320859/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320859/ sentry: https://sentry.ceph.com/organizations/ceph/?query=73e8017fd87c4e8b8824098b01a3eafd Command failed (s3 tests against rgw) on smithi090 with status 1: "source /home/ubuntu/cephtest/tox-venv/bin/activate && cd /home/ubuntu/cephtest/s3-tests-client.0 && S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto-client.0.cfg REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt tox -- -v -m 'not fails_on_rgw and not lifecycle_expiration and not test_of_sts and not webidentity_test and not fails_with_subdomain and not sse_s3 and not fails_on_dbstore'" [7320863] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/three-zones supported-random-distro$/{centos_8} tasks/test_multi} ----------------------------------------------------------------- time: 03:17:35 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320863/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320863/ sentry: https://sentry.ceph.com/organizations/ceph/?query=da484bfd84bf448f9e12341269043df8 rgw multisite test failures [7320892] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:21:32 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320892/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320892/ saw valgrind issues [7320899] rgw/multisite/{clusters frontend/beast ignore-pg-availability notify omap_limits overrides realms/two-zonegroup supported-random-distro$/{rhel_8} tasks/test_multi} ----------------------------------------------------------------- time: 07:16:49 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320899/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320899/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f8b6eb8a04342f6a3d21d6dd7b47e1b rgw multisite test failures Dead ================================================================= [7320865] rgw/service-token/{clusters/fixed-1 frontend/beast overrides tasks/service-token ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320865/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320865/ Error reimaging machines: [Errno 104] Connection reset by peer [7320867] rgw/sts/{cluster ignore-pg-availability objectstore overrides pool-type rgw_frontend/beast s3tests-branch supported-random-distro$/{ubuntu_20.04} tasks/{0-install first ststests}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320867/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320867/ Error reimaging machines: [Errno 104] Connection reset by peer [7320871] rgw/upgrade/{1-install/pacific/{distro$/{centos_latest} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320871/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320871/ Error reimaging machines: [Errno 104] Connection reset by peer [7320877] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320877/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320877/ Error reimaging machines: [Errno 104] Connection reset by peer [7320881] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_s3tests ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320881/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320881/ Error reimaging machines: [Errno 104] Connection reset by peer [7320887] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320887/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320887/ Error reimaging machines: [Errno 104] Connection reset by peer [7320891] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320891/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320891/ Error reimaging machines: [Errno 104] Connection reset by peer [7320893] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320893/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320893/ Error reimaging machines: [Errno 104] Connection reset by peer [7320898] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/v32 overrides s3a-hadoop} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320898/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320898/ Error reimaging machines: [Errno 104] Connection reset by peer [7320901] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/osds-then-rgws cluster frontend/beast objectstore/bluestore-bitmap overrides} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320901/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320901/ Error reimaging machines: [Errno 104] Connection reset by peer [7320911] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated supported-random-distro$/{centos_8}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320911/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320911/ Error reimaging machines: [Errno 104] Connection reset by peer [7320912] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320912/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320912/ Error reimaging machines: [Errno 104] Connection reset by peer [7320913] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320913/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320913/ Error reimaging machines: [Errno 104] Connection reset by peer [7320915] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_user_quota} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320915/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320915/ Error reimaging machines: [Errno 104] Connection reset by peer Pass ================================================================= [7320856] rgw/cloud-transition/{cluster overrides s3tests-branch supported-random-distro$/{ubuntu_latest} tasks/cloud_transition_s3tests} time: 00:29:50 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320856/ [7320858] rgw/d4n/{cluster overrides supported-random-distro$/{ubuntu_20.04} tasks/rgw_d4ntests} time: 00:15:47 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320858/ [7320860] rgw/hadoop-s3a/{clusters/fixed-2 hadoop/default overrides s3a-hadoop} time: 00:20:58 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320860/ [7320861] rgw/lifecycle/{cluster overrides s3tests-branch supported-random-distro$/{centos_8} tasks/rgw_s3tests} time: 00:22:06 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320861/ [7320862] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:19:19 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320862/ [7320864] rgw/notifications/{beast bluestore-bitmap fixed-2 overrides supported-all-distro$/{centos_8} tasks/{0-install test_amqp test_kafka test_others}} time: 00:41:19 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320864/ [7320866] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile supported-random-distro$/{ubuntu_20.04}} time: 00:21:06 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320866/ [7320868] rgw/tempest/{clusters/fixed-1 frontend/beast overrides tasks/rgw_tempest ubuntu_latest} time: 00:31:38 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320868/ [7320869] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} time: 00:17:54 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320869/ [7320870] rgw/tools/{centos_latest cluster tasks} time: 02:50:18 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320870/ [7320872] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:31:36 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320872/ [7320873] rgw/website/{clusters/fixed-2 frontend/beast http overrides s3tests-branch tasks/s3tests-website ubuntu_latest} time: 00:25:30 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320873/ [7320874] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:28:17 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320874/ [7320875] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:16:50 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320875/ [7320876] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:42:42 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320876/ [7320878] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:20:30 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320878/ [7320879] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/kmip 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}} time: 00:25:27 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320879/ [7320880] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:31:47 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320880/ [7320882] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:21:07 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320882/ [7320883] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_multipart_upload} time: 00:15:58 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320883/ [7320884] rgw/upgrade/{1-install/quincy/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides} time: 00:20:32 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320884/ [7320885] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:47:49 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320885/ [7320886] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:20:22 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320886/ [7320888] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/testing 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}} time: 00:28:25 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320888/ [7320889] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:16:40 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320889/ [7320890] rgw/singleton/{all/radosgw-admin frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec supported-random-distro$/{ubuntu_20.04}} time: 00:20:28 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320890/ [7320894] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:36:36 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320894/ [7320895] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:17:01 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320895/ [7320896] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:25:09 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320896/ [7320897] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_kv 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{centos_8}} time: 00:23:32 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320897/ [7320900] rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/bluestore-bitmap s3tests-branch thrasher/default thrashosds-health ubuntu_latest workload/rgw_s3tests} time: 00:27:37 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320900/ [7320902] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:31:41 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320902/ [7320904] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:27:57 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320904/ [7320905] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:59:34 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320905/ [7320906] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:33:13 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320906/ [7320907] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:17:14 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320907/ [7320908] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:07:35 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320908/ [7320909] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_old 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{ubuntu_latest}} time: 00:28:05 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320909/ [7320910] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_bucket_quota ubuntu_latest} time: 00:23:05 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320910/ [7320914] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_multipart_upload ubuntu_latest} time: 00:21:42 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320914/ [7320916] rgw/upgrade/{1-install/pacific/{distro$/{ubuntu_20.04} install overrides} 2-setup 3-upgrade-sequence/rgws-then-osds cluster frontend/beast objectstore/bluestore-bitmap overrides} time: 00:21:47 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320916/ [7320917] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:35:09 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320917/ [7320918] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:17:11 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320918/ [7320919] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec-profile s3tests-branch tasks/rgw_ragweed ubuntu_latest} time: 00:20:34 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320919/ [7320920] rgw/crypt/{0-cluster/fixed-1 1-ceph-install/install 2-kms/vault_transit 3-rgw/rgw 4-tests/{s3tests} ignore-pg-availability s3tests-branch supported-random-distro$/{rhel_8}} time: 00:25:16 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320920/ [7320921] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:34:07 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320921/ [7320922] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/ec s3tests-branch tasks/rgw_s3tests ubuntu_latest} time: 00:27:02 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320922/ [7320923] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:56:16 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320923/ [7320924] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:44:11 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320924/ [7320925] rgw/multifs/{clusters/fixed-2 frontend/beast ignore-pg-availability objectstore/bluestore-bitmap overrides rgw_pool_type/replicated s3tests-branch tasks/rgw_user_quota ubuntu_latest} time: 00:16:50 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320925/ [7320926] rgw/verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:15:28 info: http://pulpito.ceph.com/mkogan-2023-06-28_16:38:09-rgw-wip-t61661-d3n-valg_i01-distro-default-smithi/7320926/