2023-05-09T04:05:48.700 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-09T04:05:48.701 INFO:teuthology.results:seed: '3867' 2023-05-09T04:15:58.466 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-09T04:15:58.513 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-09T07:33:14.377 INFO:teuthology.results:Tests finished! gathering results... 2023-05-09T07:33:15.618 INFO:scrape:Found 14 jobs 2023-05-09T07:33:16.021 INFO:scrape:Found 3 distinct failure reasons 2023-05-09T07:33:16.022 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2023-05-09T07:33:16.022 INFO:scrape:1 jobs: ['7267712'] 2023-05-09T07:33:16.023 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2023-05-09T07:33:16.023 INFO:scrape: 2023-05-09T07:33:16.023 INFO:scrape:Failure: Command failed (workunit test rbd/test_lock_fence.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=154c3118b41f1a851d4faba0dd149a8d4ccaa67d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' 2023-05-09T07:33:16.024 INFO:scrape:1 jobs: ['7267708'] 2023-05-09T07:33:16.024 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2023-05-09T07:33:16.024 INFO:scrape: 2023-05-09T07:33:16.025 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi101 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=154c3118b41f1a851d4faba0dd149a8d4ccaa67d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2023-05-09T07:33:16.025 INFO:scrape:1 jobs: ['7267704'] 2023-05-09T07:33:16.025 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_cls_tests}'] 2023-05-09T07:33:16.025 INFO:scrape: 2023-05-09T07:33:16.027 INFO:teuthology.results:Sending results to chunmei.liu@intel.com: Test Run: yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/ failed: 3 dead: 0 running: 0 waiting: 0 queued: 0 passed: 11 Fail ================================================================= [7267704] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} ----------------------------------------------------------------- time: 00:14:44 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267704/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267704/ sentry: https://sentry.ceph.com/organizations/ceph/?query=6755a7e11bf6407594ddff15fd6d566c Command failed (workunit test cls/test_cls_rbd.sh) on smithi101 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=154c3118b41f1a851d4faba0dd149a8d4ccaa67d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7267708] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 00:12:05 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267708/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267708/ sentry: https://sentry.ceph.com/organizations/ceph/?query=462231853eaf48f98ff10a2b97d5f5e8 Command failed (workunit test rbd/test_lock_fence.sh) on smithi157 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=154c3118b41f1a851d4faba0dd149a8d4ccaa67d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' [7267712] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 03:10:05 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267712/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267712/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f2271eae152e462595800bdcec7fa315 Command failed (workunit test rbd/test_librbd_python.sh) on smithi165 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=154c3118b41f1a851d4faba0dd149a8d4ccaa67d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' Pass ================================================================= [7267700] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} time: 00:29:16 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267700/ [7267701] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} time: 00:22:50 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267701/ [7267702] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:15:26 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267702/ [7267703] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:25:40 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267703/ [7267705] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:26:30 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267705/ [7267706] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:16:39 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267706/ [7267707] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:21:42 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267707/ [7267709] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:20:02 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267709/ [7267710] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:15:40 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267710/ [7267711] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:20:00 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267711/ [7267713] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:14:40 info: http://pulpito.ceph.com/yingxin-2023-05-09_01:26:38-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7267713/