2023-04-03T17:48:00.698 INFO:root:teuthology version: 0.0.1.dev91+g5338cb9 2023-04-03T17:48:00.699 INFO:teuthology.results:seed: '75' 2023-04-03T17:48:51.470 INFO:root:teuthology version: 0.0.1.dev91+g5338cb9 2023-04-03T17:48:51.476 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-03T19:55:00.494 INFO:teuthology.results:Tests finished! gathering results... 2023-04-03T19:55:00.865 INFO:scrape:Found 12 jobs 2023-04-03T19:55:01.335 INFO:scrape:Found 2 distinct failure reasons 2023-04-03T19:55:01.336 INFO:scrape:Crash: "1680545626.2082732 mon.a (mon.0) 356 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-04-03T19:55:01.336 INFO:scrape:ceph version 18.0.0-3281-gaa7b5e61 (aa7b5e6108e7061a3de70b290dfd084dc21e2ff4) reef (dev) 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f0a8d3bd520] 2: pthread_kill() 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7f0a8e7e473b] 6: /usr/lib/ceph/libceph-common.so.2(+0x161895) [0x7f0a8e7e4895] 7: (std::__detail::__variant::__gen_vtable_impl (*)(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)>, std::integer_sequence >::__visit_invoke(FaultInjector > >::check(std::basic_string_view > const&) const::visitor&&, std::variant const&)+0x41) [0x55f016e22f51] 8: radosgw-admin(+0x7f7a6a) [0x55f016e29a6a] 9: (RGWBucketReshard::execute(int, FaultInjector > >&, int, DoutPrefixProvider const*, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x239) [0x55f016e2e989] 10: main() 11: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f0a8d3a4d90] 12: __libc_start_main() 13: _start() 2023-04-03T19:55:01.337 INFO:scrape:6 jobs: ['7230390', '7230397', '7230392', '7230393', '7230387', '7230389'] 2023-04-03T19:55:01.337 INFO:scrape:suites intersection: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'ragweed', 'reshard', 'rgw/verify/{0-install', 's3tests-branch', 's3tests-java', 's3tests}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls'] 2023-04-03T19:55:01.337 INFO:scrape:suites union: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'inline-data$/{on}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/http', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/ec', 'rgw_pool_type/ec-profile', 'rgw_pool_type/replicated', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'sharding$/{single}', 'striping$/{stripe-equals-chunk}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{cls', 'validater/lockdep}', 'validater/valgrind}'] 2023-04-03T19:55:01.338 INFO:scrape: 2023-04-03T19:55:01.338 INFO:scrape:Failure: Command failed (workunit test rgw/test_librgw_file.sh) on smithi027 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4735011c744a19323c162860f03b03c4def71636 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh' 2023-04-03T19:55:01.338 INFO:scrape:1 jobs: ['7230388'] 2023-04-03T19:55:01.339 INFO:scrape:suites: ['clusters/fixed-2', 'datacache/rgw-datacache', 'frontend/beast', 'ignore-pg-availability', 'inline-data$/{off}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'overrides', 'proto/https', 'ragweed', 'reshard', 'rgw/verify/{0-install', 'rgw_pool_type/replicated', 's3tests-branch', 's3tests-java', 's3tests}', 'sharding$/{default}', 'striping$/{stripe-greater-than-chunk}', 'supported-random-distro$/{ubuntu_20.04}', 'tasks/{cls', 'validater/valgrind}'] 2023-04-03T19:55:01.339 INFO:scrape: 2023-04-03T19:55:01.340 INFO:teuthology.results:Sending results to mkogan@redhat.com: Test Run: mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/ failed: 7 dead: 0 running: 0 waiting: 0 queued: 0 passed: 5 Fail ================================================================= [7230387] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:41:53 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230387/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230387/ "1680545740.9934664 mon.a (mon.0) 349 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7230388] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{default} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_20.04} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 00:21:58 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230388/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230388/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0c35d9c76986423388c287c6bb9ac0fb Command failed (workunit test rgw/test_librgw_file.sh) on smithi027 with status 134: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4735011c744a19323c162860f03b03c4def71636 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rgw/test_librgw_file.sh' [7230389] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:11:59 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230389/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230389/ "1680545768.3732183 mon.a (mon.0) 362 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7230390] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} ----------------------------------------------------------------- time: 01:13:28 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230390/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230390/ "1680545626.2082732 mon.a (mon.0) 356 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7230392] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec-profile s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:41:24 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230392/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230392/ "1680545768.7946773 mon.a (mon.0) 350 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7230393] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:41:04 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230393/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230393/ "1680545656.7558053 mon.a (mon.0) 354 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7230397] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{ubuntu_latest} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} ----------------------------------------------------------------- time: 00:44:46 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230397/ log: http://qa-proxy.ceph.com/teuthology/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230397/ "1680545565.6767938 mon.a (mon.0) 345 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log Pass ================================================================= [7230386] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{on} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 01:34:42 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230386/ [7230391] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{rhel_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} time: 00:51:27 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230391/ [7230394] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/ec s3tests-branch sharding$/{single} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 01:08:54 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230394/ [7230395] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec-profile s3tests-branch sharding$/{default} striping$/{stripe-equals-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 01:19:24 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230395/ [7230396] rgw/verify/{0-install clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability inline-data$/{off} msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/replicated s3tests-branch sharding$/{single} striping$/{stripe-greater-than-chunk} supported-random-distro$/{centos_8} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} time: 01:12:07 info: http://pulpito.ceph.com/mkogan-2023-04-03_17:47:49-rgw-wip-ubu-SigDoesNotMatch-cont--i02-distro-default-smithi/7230396/