2021-09-11T05:05:20.629 INFO:root:teuthology version: 1.1.0-b135a4f8 2021-09-11T05:05:20.630 INFO:teuthology.results:subset: '1/3' 2021-09-11T05:05:20.631 INFO:teuthology.results:seed: '831' 2021-09-11T07:40:39.605 INFO:root:teuthology version: 1.1.0-b135a4f8 2021-09-11T07:40:39.641 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2021-09-11T10:15:49.739 INFO:teuthology.results:Tests finished! gathering results... 2021-09-11T10:15:50.133 INFO:scrape:Found 16 jobs 2021-09-11T10:15:50.538 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384906/teuthology.log matches 2021-09-11T10:15:50.539 WARNING:scrape:Not-understood crash indication 2021-09-11T10:15:50.574 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384906/teuthology.log matches 2021-09-11T10:15:50.575 WARNING:scrape:Not-understood crash indication 2021-09-11T10:15:50.609 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384906/teuthology.log matches 2021-09-11T10:15:50.610 WARNING:scrape:Not-understood crash indication 2021-09-11T10:15:50.613 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384904/teuthology.log 2021-09-11T10:15:50.786 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384898/teuthology.log matches 2021-09-11T10:15:50.787 WARNING:scrape:Not-understood crash indication 2021-09-11T10:15:50.940 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384910/teuthology.log matches 2021-09-11T10:15:50.941 WARNING:scrape:Not-understood crash indication 2021-09-11T10:15:51.147 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384899/teuthology.log matches 2021-09-11T10:15:51.147 WARNING:scrape:Not-understood crash indication 2021-09-11T10:15:51.359 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384907/teuthology.log matches 2021-09-11T10:15:51.360 WARNING:scrape:Not-understood crash indication 2021-09-11T10:15:51.532 WARNING:scrape:Not-understood crash indication Binary file /home/teuthworker/archive/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384895/teuthology.log matches 2021-09-11T10:15:51.533 WARNING:scrape:Not-understood crash indication 2021-09-11T10:15:51.534 INFO:scrape:Found 2 distinct failure reasons 2021-09-11T10:15:51.535 INFO:scrape:Failure: Command failed on gibba032 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' 2021-09-11T10:15:51.535 INFO:scrape:6 jobs: ['6384906', '6384898', '6384910', '6384899', '6384907', '6384895'] 2021-09-11T10:15:51.536 INFO:scrape:suites intersection: ['centos_latest', 'clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests-java', 's3tests}', 'tasks/{cls'] 2021-09-11T10:15:51.536 INFO:scrape:suites union: ['centos_latest', 'clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/filestore-xfs', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}'] 2021-09-11T10:15:51.537 INFO:scrape: 2021-09-11T10:15:51.537 INFO:scrape:Dead: None 2021-09-11T10:15:51.537 INFO:scrape:10 jobs: ['6384904', '6384896', '6384901', '6384897', '6384909', '6384902', '6384908', '6384905', '6384900', '6384903'] 2021-09-11T10:15:51.538 INFO:scrape: 2021-09-11T10:15:51.539 INFO:teuthology.results:Sending results to ivancich@redhat.com: Test Run: ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba ================================================================= info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/ logs: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/ failed: 6 dead: 10 running: 0 waiting: 0 queued: 0 passed: 0 Fail ================================================================= [6384895] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 02:20:01 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384895/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384895/ sentry: https://sentry.ceph.com/organizations/ceph/?query=70d62a9ed2a84ccdafbc6caaa35cd5eb Command failed on gibba001 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' [6384898] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 02:27:08 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384898/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384898/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d8e851459dd34063aacf701273115d51 Command failed on gibba015 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' [6384899] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 02:20:49 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384899/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384899/ sentry: https://sentry.ceph.com/organizations/ceph/?query=cc1671d4bb6f4f4bbce59324a5cb8bfb Command failed on gibba004 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' [6384906] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 02:25:53 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384906/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384906/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9ee102e944c1490699e254aa22ab42f5 Command failed on gibba032 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' [6384907] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 02:20:53 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384907/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384907/ sentry: https://sentry.ceph.com/organizations/ceph/?query=becbfb3273f14614abfce0fff61a3d7c Command failed on gibba026 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' [6384910] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/ec sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 02:25:38 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384910/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384910/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a265b597548f4a8891fc701e43ebdc60 Command failed on gibba001 with status 1: 'cd /home/ubuntu/cephtest/s3-tests-java && /opt/gradle/gradle/bin/gradle clean test --rerun-tasks --no-build-cache --tests ObjectTest' Dead ================================================================= [6384896] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384896/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384896/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384897] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384897/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384897/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384900] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384900/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384900/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384901] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384901/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384901/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384902] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384902/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384902/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384903] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384903/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384903/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384904] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384904/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384904/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384905] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384905/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384905/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384908] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/https rgw_pool_type/replicated sharding$/{default} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384908/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384908/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds [6384909] rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile sharding$/{single} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384909/ log: http://qa-proxy.ceph.com/teuthology/ivancich-2021-09-11_04:56:24-rgw-wip-none-stats-fix-distro-basic-gibba/6384909/ Error reimaging machines: reached maximum tries (60) after waiting for 900 seconds