2023-10-14T12:21:25.644 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-14T12:21:25.645 INFO:teuthology.results:subset: '111/120000' 2023-10-14T12:21:25.645 INFO:teuthology.results:seed: '7103' 2023-10-14T12:53:23.116 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-14T12:53:23.139 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-10-14T13:41:26.972 INFO:teuthology.results:Tests finished! gathering results... 2023-10-14T13:41:28.269 INFO:scrape:Found 24 jobs 2023-10-14T13:41:32.221 WARNING:scrape:Missing teuthology log /home/teuthworker/archive/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425349/teuthology.log 2023-10-14T13:41:32.396 INFO:scrape:Found 3 distinct failure reasons 2023-10-14T13:41:32.396 INFO:scrape:Failure: "1697287842.4689066 mon.a (mon.0) 97 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log 2023-10-14T13:41:32.397 INFO:scrape:14 jobs: ['7425337', '7425338', '7425339', '7425350', '7425335', '7425352', '7425334', '7425355', '7425333', '7425336', '7425354', '7425332', '7425345', '7425353'] 2023-10-14T13:41:32.397 INFO:scrape:suites intersection: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'tasks/{0-install'] 2023-10-14T13:41:32.398 INFO:scrape:suites union: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'tasks/{0-install', 'test/cfuse_workunit_suites_blogbench}}', 'test/cfuse_workunit_suites_fsstress}}', 'test/cfuse_workunit_suites_iozone}}', 'test/cfuse_workunit_suites_pjd}}', 'test/kclient_workunit_direct_io}}', 'test/kclient_workunit_suites_dbench}}', 'test/kclient_workunit_suites_fsstress}}', 'test/kclient_workunit_suites_pjd}}', 'test/rados_cls_all}}', 'test/rbd_cli_import_export}}', 'test/rbd_python_api_tests}}', 'test/rbd_workunit_suites_iozone}}', 'test/rgw_ec_s3tests}}', 'test/rgw_s3tests}}'] 2023-10-14T13:41:32.398 INFO:scrape: 2023-10-14T13:41:32.399 INFO:scrape:Failure: Command failed (workunit test libcephfs/test.sh) on smithi102 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' 2023-10-14T13:41:32.399 INFO:scrape:1 jobs: ['7425340'] 2023-10-14T13:41:32.399 INFO:scrape:suites: ['objectstore/bluestore-bitmap', 'openstack}', 'smoke/basic/{clusters/{fixed-3-cephfs', 'supported-random-distro$/{rhel_8}', 'tasks/{0-install', 'test/libcephfs_interface_tests}}'] 2023-10-14T13:41:32.400 INFO:scrape: 2023-10-14T13:41:32.400 INFO:scrape:Dead: None 2023-10-14T13:41:32.401 INFO:scrape:1 jobs: ['7425349'] 2023-10-14T13:41:32.401 INFO:scrape: 2023-10-14T13:41:32.402 INFO:teuthology.results:Sending results to yweinste@redhat.com: Test Run: yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/ failed: 15 dead: 1 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [7425332] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/cfuse_workunit_suites_blogbench}} ----------------------------------------------------------------- time: 00:26:57 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425332/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425332/ "1697287550.801321 mon.a (mon.0) 98 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425333] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/cfuse_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:15:25 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425333/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425333/ "1697287397.9885457 mon.a (mon.0) 127 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425334] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/cfuse_workunit_suites_iozone}} ----------------------------------------------------------------- time: 00:22:38 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425334/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425334/ "1697287622.53472 mon.a (mon.0) 73 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425335] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/cfuse_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:24:01 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425335/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425335/ "1697287742.786819 mon.a (mon.0) 102 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425336] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/kclient_workunit_direct_io}} ----------------------------------------------------------------- time: 00:21:22 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425336/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425336/ "1697287811.916963 mon.a (mon.0) 98 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425337] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/kclient_workunit_suites_dbench}} ----------------------------------------------------------------- time: 00:48:47 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425337/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425337/ "1697287842.4689066 mon.a (mon.0) 97 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425338] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/kclient_workunit_suites_fsstress}} ----------------------------------------------------------------- time: 00:22:04 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425338/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425338/ "1697287891.4324358 mon.a (mon.0) 96 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425339] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/kclient_workunit_suites_pjd}} ----------------------------------------------------------------- time: 00:24:27 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425339/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425339/ "1697288034.9500475 mon.a (mon.0) 148 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425340] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/libcephfs_interface_tests}} ----------------------------------------------------------------- time: 00:26:54 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425340/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425340/ sentry: https://sentry.ceph.com/organizations/ceph/?query=431a9b0e57fb4aed9b8fc489995a7f2a Command failed (workunit test libcephfs/test.sh) on smithi102 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9d6d32bb3452d5179bde2ee1cfa05df7a65f4586 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh' [7425345] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_cls_all}} ----------------------------------------------------------------- time: 00:22:47 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425345/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425345/ "1697288194.3077664 mon.a (mon.0) 85 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425350] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rbd_cli_import_export}} ----------------------------------------------------------------- time: 00:18:15 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425350/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425350/ "1697289006.346009 mon.a (mon.0) 98 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425352] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rbd_python_api_tests}} ----------------------------------------------------------------- time: 00:32:31 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425352/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425352/ "1697289195.165406 mon.a (mon.0) 83 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425353] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rbd_workunit_suites_iozone}} ----------------------------------------------------------------- time: 00:18:39 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425353/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425353/ "1697288933.6730297 mon.a (mon.0) 81 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425354] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rgw_ec_s3tests}} ----------------------------------------------------------------- time: 00:23:58 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425354/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425354/ "1697288927.9284656 mon.a (mon.0) 159 : cluster [WRN] Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log [7425355] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rgw_s3tests}} ----------------------------------------------------------------- time: 00:24:05 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425355/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425355/ "1697289113.6353207 mon.a (mon.0) 87 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log Dead ================================================================= [7425349] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rbd_api_tests}} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425349/ log: http://qa-proxy.ceph.com/teuthology/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425349/ Error reimaging machines: Failed to power on smithi055 Pass ================================================================= [7425341] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/mon_thrash}} time: 00:40:45 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425341/ [7425342] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rados_api_tests}} time: 00:41:43 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425342/ [7425343] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rados_bench}} time: 01:00:52 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425343/ [7425344] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/rados_cache_snaps}} time: 00:28:39 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425344/ [7425346] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{rhel_8} tasks/{0-install test/rados_ec_snaps}} time: 00:31:50 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425346/ [7425347] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_python}} time: 00:20:38 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425347/ [7425348] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rados_workunit_loadgen_mix}} time: 00:25:38 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425348/ [7425351] smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} tasks/{0-install test/rbd_fsx}} time: 00:14:48 info: http://pulpito.ceph.com/yuriw-2023-10-13_19:39:30-smoke-quincy-release-distro-default-smithi/7425351/