2023-11-27T09:03:52.435 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-27T09:03:52.436 INFO:teuthology.results:seed: '2012' 2023-11-27T09:04:09.480 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-27T09:04:09.480 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-11-27T12:32:23.321 INFO:teuthology.results:Tests finished! gathering results... 2023-11-27T12:32:25.949 INFO:scrape:Found 34 jobs 2023-11-27T12:32:26.510 INFO:scrape:Found 3 distinct failure reasons 2023-11-27T12:32:26.511 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh -m \ 2023-11-27T12:32:26.511 INFO:scrape:1 jobs: ['7467545'] 2023-11-27T12:32:26.511 INFO:scrape:suites: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests}'] 2023-11-27T12:32:26.511 INFO:scrape: 2023-11-27T12:32:26.512 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh 2023-11-27T12:32:26.512 INFO:scrape:2 jobs: ['7467522', '7467528'] 2023-11-27T12:32:26.512 INFO:scrape:suites intersection: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph'] 2023-11-27T12:32:26.512 INFO:scrape:suites union: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}', 'tasks/rbd_api_tests}'] 2023-11-27T12:32:26.512 INFO:scrape: 2023-11-27T12:32:26.512 INFO:scrape:Failure: Command failed on smithi114 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' 2023-11-27T12:32:26.512 INFO:scrape:1 jobs: ['7467535'] 2023-11-27T12:32:26.512 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}'] 2023-11-27T12:32:26.512 INFO:scrape: 2023-11-27T12:32:26.514 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/ failed: 4 dead: 0 running: 0 waiting: 0 queued: 0 passed: 30 Fail ================================================================= [7467522] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} ----------------------------------------------------------------- time: 03:17:50 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467522/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467522/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c72d88b1e5624d8eafe8f2081d08f1cb Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi172 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crims on_librbd.sh' [7467528] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 03:17:41 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467528/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467528/ sentry: https://sentry.ceph.com/organizations/ceph/?query=ed51df953ce6461ca5faf5265e4ed343 Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi022 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' [7467535] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:26:05 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467535/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467535/ Command failed on smithi114 with status 6: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats' [7467545] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} ----------------------------------------------------------------- time: 03:13:41 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467545/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467545/ sentry: https://sentry.ceph.com/organizations/ceph/?query=22832f2e307f44f092253e29766f08b4 Command failed (workunit test rbd/test_librbd_python.sh) on smithi122 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=705aae7f46f3b33a89617c3a1f2fcac8a51ce7b3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh -m \'not skip_if_crimson\'' Pass ================================================================= [7467520] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} time: 00:36:10 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467520/ [7467521] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:21:13 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467521/ [7467523] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:16:24 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467523/ [7467524] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:22:33 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467524/ [7467525] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:20:39 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467525/ [7467526] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:32:57 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467526/ [7467527] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:19:41 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467527/ [7467529] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:30:17 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467529/ [7467530] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:22:06 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467530/ [7467531] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:32:17 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467531/ [7467532] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} time: 00:18:08 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467532/ [7467533] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:21:44 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467533/ [7467534] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 00:17:08 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467534/ [7467536] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} time: 00:17:46 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467536/ [7467537] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:25:45 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467537/ [7467538] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} time: 00:17:10 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467538/ [7467539] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} time: 00:16:28 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467539/ [7467540] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} time: 00:18:46 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467540/ [7467541] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:26:17 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467541/ [7467542] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} time: 00:18:28 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467542/ [7467543] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:33:34 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467543/ [7467544] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:19:11 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467544/ [7467546] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:17:31 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467546/ [7467547] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:33:50 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467547/ [7467548] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:28:01 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467548/ [7467549] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:33:42 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467549/ [7467550] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} time: 00:23:26 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467550/ [7467551] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:21:34 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467551/ [7467552] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:20:40 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467552/ [7467553] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:17:27 info: https://pulpito.ceph.com/yingxin-2023-11-27_09:03:31-crimson-rados-wip-yingxin-crimson-make-crosscore-send-ordered-distro-default-smithi/7467553/