2023-06-08T03:27:14.078 INFO:root:teuthology version: 0.0.1.dev120+g2ef0dcd 2023-06-08T03:27:14.079 INFO:teuthology.results:seed: '6388' 2023-06-08T03:28:15.132 INFO:root:teuthology version: 0.0.1.dev120+g2ef0dcd 2023-06-08T03:28:15.135 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-06-08T06:59:29.823 INFO:teuthology.results:Tests finished! gathering results... 2023-06-08T06:59:31.573 INFO:scrape:Found 20 jobs 2023-06-08T06:59:31.911 INFO:scrape:Found 2 distinct failure reasons 2023-06-08T06:59:31.912 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2023-06-08T06:59:31.912 INFO:scrape:2 jobs: ['7298075', '7298072'] 2023-06-08T06:59:31.913 INFO:scrape:suites intersection: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2023-06-08T06:59:31.913 INFO:scrape:suites union: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}'] 2023-06-08T06:59:31.913 INFO:scrape: 2023-06-08T06:59:31.914 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2023-06-08T06:59:31.915 INFO:scrape:1 jobs: ['7298071'] 2023-06-08T06:59:31.915 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2023-06-08T06:59:31.915 INFO:scrape: 2023-06-08T06:59:31.917 INFO:teuthology.results:Sending results to chunmei.liu@intel.com: Test Run: yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 17 Fail ================================================================= [7298071] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:19:54 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298071/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298071/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9a039e781e9a42238b71c1ec3d3021b1 Command failed (workunit test rados/test.sh) on smithi018 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6fc43019e0026c4a1ddd83a3fab6fb8c23187b49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7298072] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} ----------------------------------------------------------------- time: 03:16:09 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298072/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298072/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1fb12087a19943ee99c0b1e7b0f382fb Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi111 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6fc43019e0026c4a1ddd83a3fab6fb8c23187b49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crims on_librbd.sh' [7298075] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 03:16:08 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298075/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298075/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0508202607e04434a4da481d01ba18f9 Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi033 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=6fc43019e0026c4a1ddd83a3fab6fb8c23187b49 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' Pass ================================================================= [7298073] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:23:05 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298073/ [7298074] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:36:07 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298074/ [7298076] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:30:31 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298076/ [7298077] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:35:58 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298077/ [7298078] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:21:56 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298078/ [7298079] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 00:20:52 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298079/ [7298080] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:29:24 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298080/ [7298081] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:29:17 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298081/ [7298082] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} time: 00:15:03 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298082/ [7298083] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:24:04 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298083/ [7298084] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:38:14 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298084/ [7298085] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:21:10 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298085/ [7298087] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} time: 00:40:47 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298087/ [7298088] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:37:49 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298088/ [7298090] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:34:08 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298090/ [7298092] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} time: 00:39:16 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298092/ [7298094] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:21:02 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:27:09-crimson-rados-wip-chunmei-ceph-distro-default-smithi/7298094/