2023-04-04T03:13:28.484 INFO:root:teuthology version: 0.0.1.dev93+gd7c4062 2023-04-04T03:13:28.486 INFO:teuthology.results:seed: '7711' 2023-04-04T03:17:15.557 INFO:root:teuthology version: 0.0.1.dev93+gd7c4062 2023-04-04T03:17:15.571 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-04T08:01:36.035 INFO:teuthology.results:Tests finished! gathering results... 2023-04-04T08:01:37.091 INFO:scrape:Found 14 jobs 2023-04-04T08:01:37.480 INFO:scrape:Found 4 distinct failure reasons 2023-04-04T08:01:37.481 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2023-04-04T08:01:37.481 INFO:scrape:1 jobs: ['7230535'] 2023-04-04T08:01:37.482 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2023-04-04T08:01:37.482 INFO:scrape: 2023-04-04T08:01:37.482 INFO:scrape:Failure: Command failed (workunit test rbd/test_lock_fence.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00bb462ba6c10f353aac93ffcf07294760fcaab1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' 2023-04-04T08:01:37.482 INFO:scrape:1 jobs: ['7230531'] 2023-04-04T08:01:37.483 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2023-04-04T08:01:37.483 INFO:scrape: 2023-04-04T08:01:37.483 INFO:scrape:Failure: Command failed on smithi043 with status 1: 'sudo yum install -y kernel' 2023-04-04T08:01:37.484 INFO:scrape:1 jobs: ['7230523'] 2023-04-04T08:01:37.484 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2023-04-04T08:01:37.484 INFO:scrape: 2023-04-04T08:01:37.484 INFO:scrape:Failure: Command failed (workunit test cls/test_cls_rbd.sh) on smithi154 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00bb462ba6c10f353aac93ffcf07294760fcaab1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' 2023-04-04T08:01:37.485 INFO:scrape:1 jobs: ['7230527'] 2023-04-04T08:01:37.485 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_cls_tests}'] 2023-04-04T08:01:37.485 INFO:scrape: 2023-04-04T08:01:37.486 INFO:teuthology.results:Sending results to yingxincheng@gmail.com: Test Run: yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 10 Fail ================================================================= [7230523] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230523/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230523/ sentry: https://sentry.ceph.com/organizations/ceph/?query=75f50b695c684dbabbc5e7bf362a47b5 Command failed on smithi043 with status 1: 'sudo yum install -y kernel' [7230527] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} ----------------------------------------------------------------- time: 01:17:31 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230527/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230527/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f5b4a7ac34c340b3a68accd17557a1f2 Command failed (workunit test cls/test_cls_rbd.sh) on smithi154 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00bb462ba6c10f353aac93ffcf07294760fcaab1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [7230531] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 01:19:25 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230531/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230531/ sentry: https://sentry.ceph.com/organizations/ceph/?query=112873ca1ebf43cf8ed965a3f6269bda Command failed (workunit test rbd/test_lock_fence.sh) on smithi027 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00bb462ba6c10f353aac93ffcf07294760fcaab1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' [7230535] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 04:17:20 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230535/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230535/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1890c29277bd490fa69250a7f438b462 Command failed (workunit test rbd/test_librbd_python.sh) on smithi144 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00bb462ba6c10f353aac93ffcf07294760fcaab1 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' Pass ================================================================= [7230524] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} time: 01:31:24 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230524/ [7230525] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 01:22:30 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230525/ [7230526] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 01:34:55 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230526/ [7230528] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 01:33:23 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230528/ [7230529] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python} time: 01:30:11 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230529/ [7230530] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 01:27:02 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230530/ [7230532] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 01:26:47 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230532/ [7230533] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/readwrite} time: 01:20:50 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230533/ [7230534] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 01:26:56 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230534/ [7230536] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 01:21:56 info: http://pulpito.ceph.com/yingxin-2023-04-04_03:13:23-crimson-rados-wip-yingxin-adjust-conn-ref-crimson-only-3-distro-default-smithi/7230536/