2023-05-05T01:15:17.628 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-05T01:15:17.629 INFO:teuthology.results:seed: '790' 2023-05-05T01:16:12.300 INFO:root:teuthology version: 0.0.1.dev98+g69caab8 2023-05-05T01:16:12.302 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-05-05T04:38:26.348 INFO:teuthology.results:Tests finished! gathering results... 2023-05-05T04:38:27.508 INFO:scrape:Found 14 jobs 2023-05-05T04:38:27.938 INFO:scrape:Found 4 distinct failure reasons 2023-05-05T04:38:27.939 INFO:scrape:Failure: Command failed (workunit test rbd/test_lock_fence.sh) on smithi135 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf092b40c93e5884dca0dc5f559f2b07d125fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' 2023-05-05T04:38:27.939 INFO:scrape:1 jobs: ['7262066'] 2023-05-05T04:38:27.939 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2023-05-05T04:38:27.940 INFO:scrape: 2023-05-05T04:38:27.940 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2023-05-05T04:38:27.940 INFO:scrape:1 jobs: ['7262070'] 2023-05-05T04:38:27.940 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2023-05-05T04:38:27.941 INFO:scrape: 2023-05-05T04:38:27.941 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi063 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf092b40c93e5884dca0dc5f559f2b07d125fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2023-05-05T04:38:27.941 INFO:scrape:1 jobs: ['7262062'] 2023-05-05T04:38:27.942 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_cls_tests}'] 2023-05-05T04:38:27.942 INFO:scrape: 2023-05-05T04:38:27.942 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf092b40c93e5884dca0dc5f559f2b07d125fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2023-05-05T04:38:27.943 INFO:scrape:1 jobs: ['7262058'] 2023-05-05T04:38:27.943 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2023-05-05T04:38:27.943 INFO:scrape: 2023-05-05T04:38:27.944 INFO:teuthology.results:Sending results to chunmei.liu@intel.com: Test Run: yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 10 Fail ================================================================= [7262058] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 01:06:59 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262058/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262058/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4e5ef4240184c8e9e0ac3edbc8b0aaa Command failed (workunit test rados/test.sh) on smithi062 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf092b40c93e5884dca0dc5f559f2b07d125fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7262062] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} ----------------------------------------------------------------- time: 00:14:16 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262062/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262062/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3e70a83b65e04b898e8c083eba1f80b6 Command failed (workunit test cls/test_cls_rbd.sh) on smithi063 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf092b40c93e5884dca0dc5f559f2b07d125fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7262066] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 00:19:17 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262066/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262066/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9be0e39dcf1a45f0902e63ba2386801c Command failed (workunit test rbd/test_lock_fence.sh) on smithi135 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf092b40c93e5884dca0dc5f559f2b07d125fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' [7262070] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 03:15:25 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262070/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262070/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3804240df3ec4d94a946aecf56df4133 Command failed (workunit test rbd/test_librbd_python.sh) on smithi007 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbf092b40c93e5884dca0dc5f559f2b07d125fd6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' Pass ================================================================= [7262059] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} time: 00:32:48 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262059/ [7262060] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:21:47 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262060/ [7262061] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:30:08 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262061/ [7262063] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:36:41 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262063/ [7262064] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:25:17 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262064/ [7262065] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:29:21 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262065/ [7262067] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:27:48 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262067/ [7262068] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:17:18 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262068/ [7262069] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:27:50 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262069/ [7262071] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:22:46 info: http://pulpito.ceph.com/yingxin-2023-05-05_01:15:02-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7262071/