2023-05-06T04:11:21.641 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-06T04:11:21.649 INFO:teuthology.results:seed: '4813' 2023-05-06T12:37:24.259 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-06T12:37:24.280 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-06T14:04:30.993 INFO:teuthology.results:Tests finished! gathering results... 2023-05-06T14:04:32.942 INFO:scrape:Found 24 jobs 2023-05-06T14:04:33.270 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262651/teuthology.log matches 2023-05-06T14:04:33.270 WARNING:scrape:Misunderstood line: 2023-05-06T14:04:33.292 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262647/teuthology.log matches 2023-05-06T14:04:33.293 WARNING:scrape:Misunderstood line: 2023-05-06T14:04:33.350 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262633/teuthology.log matches 2023-05-06T14:04:33.351 WARNING:scrape:Misunderstood line: 2023-05-06T14:04:33.660 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262645/teuthology.log matches 2023-05-06T14:04:33.661 WARNING:scrape:Misunderstood line: 2023-05-06T14:04:33.965 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262649/teuthology.log matches 2023-05-06T14:04:33.966 WARNING:scrape:Misunderstood line: 2023-05-06T14:04:33.983 WARNING:scrape:Misunderstood line: Binary file /home/teuthworker/archive/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262629/teuthology.log matches 2023-05-06T14:04:33.984 WARNING:scrape:Misunderstood line: 2023-05-06T14:04:33.984 INFO:scrape:Found 1 distinct failure reasons 2023-05-06T14:04:33.984 INFO:scrape:Valgrind: 2023-05-06T14:04:33.985 INFO:scrape:9 jobs: ['7262651', '7262647', '7262632', '7262636', '7262633', '7262645', '7262628', '7262649', '7262629'] 2023-05-06T14:04:33.985 INFO:scrape:suites intersection: ['clusters/fixed-2', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw:verify/{0-install', 's3tests-branch', 's3tests-java', 's3tests}', 'tasks/{cls'] 2023-05-06T14:04:33.986 INFO:scrape:suites union: ['clusters/fixed-2', 'datacache/no_datacache', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw:verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_20.04}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}'] 2023-05-06T14:04:33.986 INFO:scrape: 2023-05-06T14:04:33.988 INFO:teuthology.results:Sending results to gsalomon@redhat.com: Test Run: gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/ failed: 9 dead: 0 running: 0 waiting: 0 queued: 0 passed: 15 Fail ================================================================= [7262628] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:37:55 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262628/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262628/ "1683347678.3557594 mon.a (mon.0) 369 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7262629] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:39:12 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262629/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262629/ "1683350593.363475 mon.a (mon.0) 358 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7262632] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:38:25 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262632/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262632/ "1683357109.848198 mon.a (mon.0) 361 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7262633] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:40:44 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262633/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262633/ "1683357881.2187524 mon.a (mon.0) 337 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7262636] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:40:56 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262636/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262636/ "1683364472.080163 mon.a (mon.0) 344 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7262645] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:20:12 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262645/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262645/ saw valgrind issues [7262647] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:02:37 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262647/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262647/ saw valgrind issues [7262649] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:14:10 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262649/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262649/ "1683378040.5858362 mon.a (mon.0) 356 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7262651] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:05:18 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262651/ log: http://qa-proxy.ceph.com/teuthology/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262651/ saw valgrind issues Pass ================================================================= [7262630] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:31:20 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262630/ [7262631] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:54:57 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262631/ [7262634] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:38:54 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262634/ [7262635] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:47:09 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262635/ [7262637] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:50:11 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262637/ [7262638] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:34:34 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262638/ [7262639] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:52:55 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262639/ [7262640] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:32:38 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262640/ [7262641] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:53:54 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262641/ [7262642] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:34:10 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262642/ [7262643] rgw:verify/{0-install clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:54:12 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262643/ [7262644] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:28:02 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262644/ [7262646] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:31:05 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262646/ [7262648] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:36:09 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262648/ [7262650] rgw:verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 00:37:20 info: http://pulpito.ceph.com/gsalomon-2023-05-06_04:06:24-rgw:verify-json_s3tests_and_more-distro-default-smithi/7262650/