2023-06-08T03:02:47.312 INFO:root:teuthology version: 0.0.1.dev120+g2ef0dcd 2023-06-08T03:02:47.313 INFO:teuthology.results:seed: '5641' 2023-06-08T03:04:28.627 INFO:root:teuthology version: 0.0.1.dev120+g2ef0dcd 2023-06-08T03:04:28.631 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-06-08T06:33:43.056 INFO:teuthology.results:Tests finished! gathering results... 2023-06-08T06:33:44.730 INFO:scrape:Found 20 jobs 2023-06-08T06:33:45.147 INFO:scrape:Found 3 distinct failure reasons 2023-06-08T06:33:45.148 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2023-06-08T06:33:45.148 INFO:scrape:1 jobs: ['7298040'] 2023-06-08T06:33:45.148 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests}'] 2023-06-08T06:33:45.149 INFO:scrape: 2023-06-08T06:33:45.149 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh --eval-attr \ 2023-06-08T06:33:45.149 INFO:scrape:1 jobs: ['7298054'] 2023-06-08T06:33:45.150 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests}'] 2023-06-08T06:33:45.150 INFO:scrape: 2023-06-08T06:33:45.150 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json 2023-06-08T06:33:45.150 INFO:scrape:1 jobs: ['7298057'] 2023-06-08T06:33:45.151 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2023-06-08T06:33:45.151 INFO:scrape: 2023-06-08T06:33:45.152 INFO:teuthology.results:Sending results to chunmei.liu@intel.com: Test Run: yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 17 Fail ================================================================= [7298040] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} ----------------------------------------------------------------- time: 03:15:35 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298040/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298040/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3a6b55743c1f41ea80b9e09c913308ca Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi160 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3ee726d7821fc66dc0554305838831cc13d320f9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crims on_librbd.sh' [7298054] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} ----------------------------------------------------------------- time: 03:16:08 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298054/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298054/ sentry: https://sentry.ceph.com/organizations/ceph/?query=61e8e9f4580347f9b77af4d3a3a31f61 Command failed (workunit test rbd/test_librbd_python.sh) on smithi072 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3ee726d7821fc66dc0554305838831cc13d320f9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh --eval-attr \'not (SKIP_IF_CRIMSON)\'' [7298057] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 00:41:22 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298057/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298057/ Command failed on smithi125 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' Pass ================================================================= [7298039] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} time: 00:41:20 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298039/ [7298041] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:20:15 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298041/ [7298042] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:36:00 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298042/ [7298043] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} time: 00:30:40 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298043/ [7298044] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:32:59 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298044/ [7298045] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:37:03 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298045/ [7298046] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:22:33 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298046/ [7298047] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 00:21:06 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298047/ [7298048] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:23:38 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298048/ [7298049] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:24:28 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298049/ [7298050] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} time: 00:15:44 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298050/ [7298051] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:28:53 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298051/ [7298052] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:33:42 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298052/ [7298053] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:25:05 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298053/ [7298055] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:37:41 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298055/ [7298056] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:33:22 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298056/ [7298058] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:19:58 info: http://pulpito.ceph.com/yingxin-2023-06-08_03:02:36-crimson-rados-wip-chunmei-ceph-pr-distro-default-smithi/7298058/