2021-05-20T13:47:11.042 INFO:root:teuthology version: 1.1.0-471a2f1e 2021-05-20T13:47:11.044 INFO:teuthology.results:seed: '6255' 2021-05-20T18:46:26.547 INFO:root:teuthology version: 1.1.0-471a2f1e 2021-05-20T18:46:26.577 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-05-20T18:46:26.593 INFO:teuthology.results:Tests finished! gathering results... 2021-05-20T18:46:28.222 INFO:scrape:Found 19 jobs 2021-05-20T18:46:28.647 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124864/teuthology.log matches 2021-05-20T18:46:28.648 WARNING:scrape:Not-understood crash indication 2021-05-20T18:46:28.836 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124864/teuthology.log matches 2021-05-20T18:46:28.837 WARNING:scrape:Not-understood crash indication 2021-05-20T18:46:29.019 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124864/teuthology.log matches 2021-05-20T18:46:29.020 WARNING:scrape:Not-understood crash indication 2021-05-20T18:46:29.207 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124864/teuthology.log matches 2021-05-20T18:46:29.208 WARNING:scrape:Not-understood crash indication 2021-05-20T18:46:29.441 WARNING:scrape:Not-understood crash indication 2021-05-20T14:18:19.409 INFO:tasks.rgw.client.0.smithi079.stderr:daemon-helper: command crashed with signal 6 2021-05-20T18:46:29.442 WARNING:scrape:Not-understood crash indication 2021-05-20T18:46:29.509 WARNING:scrape:Not-understood crash indication 2021-05-20T14:18:19.409 INFO:tasks.rgw.client.0.smithi079.stderr:daemon-helper: command crashed with signal 6 2021-05-20T18:46:29.510 WARNING:scrape:Not-understood crash indication 2021-05-20T18:46:29.578 WARNING:scrape:Not-understood crash indication 2021-05-20T14:18:19.409 INFO:tasks.rgw.client.0.smithi079.stderr:daemon-helper: command crashed with signal 6 2021-05-20T18:46:29.579 WARNING:scrape:Not-understood crash indication 2021-05-20T18:46:29.645 WARNING:scrape:Not-understood crash indication 2021-05-20T14:18:19.409 INFO:tasks.rgw.client.0.smithi079.stderr:daemon-helper: command crashed with signal 6 2021-05-20T18:46:29.646 WARNING:scrape:Not-understood crash indication 2021-05-20T18:46:29.711 WARNING:scrape:Not-understood crash indication 2021-05-20T14:18:19.409 INFO:tasks.rgw.client.0.smithi079.stderr:daemon-helper: command crashed with signal 6 2021-05-20T18:46:29.712 WARNING:scrape:Not-understood crash indication 2021-05-20T18:46:29.838 INFO:scrape:Found 3 distinct failure reasons 2021-05-20T18:46:29.839 INFO:scrape:Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-rgw-d3n-cache-upstreaming_i93 2021-05-20T18:46:29.839 INFO:scrape:4 jobs: ['6124865', '6124862', '6124863', '6124861'] 2021-05-20T18:46:29.840 INFO:scrape: 2021-05-20T18:46:29.840 INFO:scrape:Failure: Command failed on smithi081 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph' 2021-05-20T18:46:29.841 INFO:scrape:1 jobs: ['6124864'] 2021-05-20T18:46:29.841 INFO:scrape:suites: ['centos_latest', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 's3tests-java', 's3tests}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'tasks/{cls', 'validater/valgrind}'] 2021-05-20T18:46:29.842 INFO:scrape: 2021-05-20T18:46:29.842 INFO:scrape:Failure: Command failed (ragweed tests against rgw) on smithi079 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" 2021-05-20T18:46:29.843 INFO:scrape:1 jobs: ['6124858'] 2021-05-20T18:46:29.843 INFO:scrape:suites: ['centos_latest', 'clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/filestore-xfs', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 's3tests-java', 's3tests}', 'sharding$/{default}', 'striping$/{stripe-equals-chunk}', 'tasks/{cls', 'validater/valgrind}'] 2021-05-20T18:46:29.843 INFO:scrape: 2021-05-20T18:46:29.845 INFO:teuthology.results:Sending results to mkogan@redhat.com: Test Run: mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi ================================================================= info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 13 Fail ================================================================= [6124858] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:24:27 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124858/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124858/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4ba4943ec2014811a9cfe9c009e47654 Command failed (ragweed tests against rgw) on smithi079 with status 1: "RAGWEED_CONF=/home/ubuntu/cephtest/archive/ragweed.client.0.conf RAGWEED_STAGES=prepare,check BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/ragweed/virtualenv/bin/python -m nose -w /home/ubuntu/cephtest/ragweed -v -a '!fails_on_rgw'" [6124861] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124861/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124861/ sentry: https://sentry.ceph.com/organizations/ceph/?query=96655e27c756416b88130af8c74ffc70 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip- rgw-d3n-cache-upstreaming_i93 [6124862] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124862/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124862/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6f1c8039da9549ea81e9ca0fcefe17e8 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip- rgw-d3n-cache-upstreaming_i93 [6124863] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124863/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124863/ sentry: https://sentry.ceph.com/organizations/ceph/?query=af423630d4d0407fbe749c1fd09bffcf Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip- rgw-d3n-cache-upstreaming_i93 [6124864] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 02:42:24 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124864/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124864/ sentry: https://sentry.ceph.com/organizations/ceph/?query=97cbdf39543c4ed1936d6534901038be Command failed on smithi081 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph' [6124865] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124865/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124865/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a137173118e94a90a7624549fc9d50f6 Failed to fetch package version from https://shaman.ceph.com/api/search/?st atus=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip- rgw-d3n-cache-upstreaming_i93 Pass ================================================================= [6124857] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:29:26 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124857/ [6124859] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:28:22 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124859/ [6124860] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:15:43 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124860/ [6124866] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:05:58 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124866/ [6124867] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:29:32 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124867/ [6124868] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:07:18 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124868/ [6124869] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:28:37 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124869/ [6124870] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:18:13 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124870/ [6124871] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:27:33 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124871/ [6124872] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:22:53 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124872/ [6124873] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:28:33 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124873/ [6124874] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:21:37 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124874/ [6124875] rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:29:41 info: http://pulpito.ceph.com/mkogan-2021-05-20_13:46:55-rgw:verify-wip-rgw-d3n-cache-upstreaming_i93-distro-basic-smithi/6124875/