2023-04-04T07:55:21.723 INFO:root:teuthology version: 0.0.1.dev93+gd7c4062 2023-04-04T07:55:21.724 INFO:teuthology.results:seed: '5162' 2023-04-04T07:56:16.509 INFO:root:teuthology version: 0.0.1.dev93+gd7c4062 2023-04-04T07:56:16.512 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-04T13:02:36.932 INFO:teuthology.results:Tests finished! gathering results... 2023-04-04T13:02:37.770 INFO:scrape:Found 14 jobs 2023-04-04T13:02:39.155 INFO:scrape:Found 5 distinct failure reasons 2023-04-04T13:02:39.156 INFO:scrape:Failure: Command failed on smithi044 with status 1: 'sudo yum install -y kernel' 2023-04-04T13:02:39.156 INFO:scrape:2 jobs: ['7230641', '7230639'] 2023-04-04T13:02:39.156 INFO:scrape:suites intersection: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health'] 2023-04-04T13:02:39.157 INFO:scrape:suites union: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}', 'workloads/write_fadvise_dontneed}'] 2023-04-04T13:02:39.157 INFO:scrape: 2023-04-04T13:02:39.157 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=327705dfa13c5a336b298805184b52afc4c4a9f1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2023-04-04T13:02:39.158 INFO:scrape:1 jobs: ['7230628'] 2023-04-04T13:02:39.158 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2023-04-04T13:02:39.158 INFO:scrape: 2023-04-04T13:02:39.159 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2023-04-04T13:02:39.159 INFO:scrape:1 jobs: ['7230629'] 2023-04-04T13:02:39.159 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}'] 2023-04-04T13:02:39.159 INFO:scrape: 2023-04-04T13:02:39.160 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2023-04-04T13:02:39.160 INFO:scrape:1 jobs: ['7230640'] 2023-04-04T13:02:39.160 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2023-04-04T13:02:39.161 INFO:scrape: 2023-04-04T13:02:39.161 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2023-04-04T13:02:39.161 INFO:scrape:1 jobs: ['7230636'] 2023-04-04T13:02:39.162 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2023-04-04T13:02:39.162 INFO:scrape: 2023-04-04T13:02:39.163 INFO:teuthology.results:Sending results to yingxincheng@gmail.com: Test Run: yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/ failed: 6 dead: 0 running: 0 waiting: 0 queued: 0 passed: 8 Fail ================================================================= [7230628] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 02:10:32 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230628/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230628/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d7eff1f0f07e400cb9b6518625718528 Command failed (workunit test rados/test.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=327705dfa13c5a336b298805184b52afc4c4a9f1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7230629] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 04:36:42 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230629/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230629/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fa744cfc45fb4933ac4b6fd1d91743f8 Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi083 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=327705dfa13c5a336b298805184b52afc4c4a9f1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' [7230636] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 04:17:25 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230636/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230636/ sentry: https://sentry.ceph.com/organizations/ceph/?query=653a7b6d82574788bd49db720a393992 Command failed (workunit test rbd/test_lock_fence.sh) on smithi139 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=327705dfa13c5a336b298805184b52afc4c4a9f1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' [7230639] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230639/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230639/ sentry: https://sentry.ceph.com/organizations/ceph/?query=87ab3f1a73e14cd5b14d4bf4abed95f8 Command failed on smithi146 with status 1: 'sudo yum install -y kernel' [7230640] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 04:17:28 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230640/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230640/ sentry: https://sentry.ceph.com/organizations/ceph/?query=52b3957082534b01b629294f2b59857c Command failed (workunit test rbd/test_librbd_python.sh) on smithi137 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=327705dfa13c5a336b298805184b52afc4c4a9f1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' [7230641] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230641/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230641/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9449ff578f3a48e7b55e22ad406b5e28 Command failed on smithi044 with status 1: 'sudo yum install -y kernel' Pass ================================================================= [7230630] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 01:22:29 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230630/ [7230631] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 01:36:13 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230631/ [7230632] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 01:23:04 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230632/ [7230633] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 01:33:36 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230633/ [7230634] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python} time: 01:32:55 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230634/ [7230635] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 01:31:06 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230635/ [7230637] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 01:26:58 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230637/ [7230638] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/readwrite} time: 01:20:51 info: http://pulpito.ceph.com/yingxin-2023-04-04_07:55:01-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-base-1-distro-default-smithi/7230638/