2024-01-23T07:03:16.898 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-23T07:03:16.898 INFO:teuthology.results:seed: '8265' 2024-01-23T07:29:34.140 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-23T07:29:34.144 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2024-01-23T10:47:49.484 INFO:teuthology.results:Tests finished! gathering results... 2024-01-23T10:47:52.033 INFO:scrape:Found 34 jobs 2024-01-23T10:47:52.427 INFO:scrape:Found 2 distinct failure reasons 2024-01-23T10:47:52.427 INFO:scrape:Failure: Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12e4686aee4cae72453c09f13ec877e9b23a5a65 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' 2024-01-23T10:47:52.428 INFO:scrape:1 jobs: ['7527511'] 2024-01-23T10:47:52.428 INFO:scrape:suites: ['crimson-rados/rbd/{clusters/fixed-1', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}'] 2024-01-23T10:47:52.428 INFO:scrape: 2024-01-23T10:47:52.428 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh --crimson 2024-01-23T10:47:52.428 INFO:scrape:1 jobs: ['7527503'] 2024-01-23T10:47:52.428 INFO:scrape:suites: ['crimson-rados/basic/{clusters/fixed-2', 'crimson-supported-all-distro/centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2024-01-23T10:47:52.428 INFO:scrape: 2024-01-23T10:47:52.430 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi ================================================================= info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/ failed: 2 dead: 0 running: 0 waiting: 0 queued: 0 passed: 32 Fail ================================================================= [7527503] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:22:56 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527503/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527503/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1eb2c8e4f5874dd2ac9002bbdfe021ba Command failed (workunit test rados/test.sh) on smithi113 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12e4686aee4cae72453c09f13ec877e9b23a5a65 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh --crimson' [7527511] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 00:16:45 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527511/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527511/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3bd7f1ad0d524d8f815a483e77359923 Command failed (workunit test rbd/crimson/test_crimson_librbd.sh) on smithi059 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=12e4686aee4cae72453c09f13ec877e9b23a5a65 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/crimson/test_crimson_librbd.sh' Pass ================================================================= [7527504] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:15:27 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527504/ [7527505] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} time: 00:58:11 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527505/ [7527506] crimson-rados/singleton/{all/osd-backfill crimson-supported-all-distro/centos_latest crimson_qa_overrides objectstore/bluestore rados} time: 00:32:32 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527506/ [7527507] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:41:19 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527507/ [7527508] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:34:43 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527508/ [7527509] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:28:40 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527509/ [7527510] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:40:20 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527510/ [7527512] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:50:52 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527512/ [7527513] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:34:32 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527513/ [7527514] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:56:48 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527514/ [7527515] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} time: 00:33:12 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527515/ [7527516] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:26:31 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527516/ [7527517] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 00:18:38 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527517/ [7527518] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:39:44 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527518/ [7527519] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} time: 00:13:25 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527519/ [7527520] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:30:42 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527520/ [7527521] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} time: 00:37:34 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527521/ [7527522] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} time: 00:12:57 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527522/ [7527523] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} time: 00:12:03 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527523/ [7527524] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 00:21:39 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527524/ [7527525] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} time: 00:11:14 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527525/ [7527526] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:41:13 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527526/ [7527527] crimson-rados/basic/{clusters/fixed-2 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:32:42 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527527/ [7527528] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} time: 00:22:12 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527528/ [7527529] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:10:31 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527529/ [7527530] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:51:02 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527530/ [7527531] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:22:04 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527531/ [7527532] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:29:21 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527532/ [7527533] crimson-rados/rbd/{clusters/fixed-1 crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} time: 00:19:05 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527533/ [7527534] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:25:43 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527534/ [7527535] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:21:06 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527535/ [7527536] crimson-rados/perf/{clusters/{fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:11:32 info: https://pulpito.ceph.com/yingxin-2024-01-23_01:28:42-crimson-rados-wip-yingxin-crimson-save-conn-foreign-copy-9-distro-default-smithi/7527536/