2023-08-08T02:00:54.872 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-08T02:00:54.873 INFO:teuthology.results:seed: '7393' 2023-08-08T02:39:57.888 INFO:root:teuthology version: 0.0.1.dev181+g7fda959 2023-08-08T02:39:57.908 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-08-08T03:30:01.898 INFO:teuthology.results:Tests finished! gathering results... 2023-08-08T03:30:02.939 INFO:scrape:Found 33 jobs 2023-08-08T03:30:02.996 INFO:scrape:Found 1 distinct failure reasons 2023-08-08T03:30:02.997 INFO:scrape:Timeout 120 running ceph --cluster ceph pg dump --format=json 2023-08-08T03:30:02.997 INFO:scrape:1 jobs: ['7362410'] 2023-08-08T03:30:02.997 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_8.stream', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}'] 2023-08-08T03:30:02.998 INFO:scrape: 2023-08-08T03:30:03.000 INFO:teuthology.results:Sending results to yingxin.cheng@intel.com: Test Run: yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/ failed: 1 dead: 0 running: 0 waiting: 0 queued: 0 passed: 32 Fail ================================================================= [7362410] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:31:42 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362410/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362410/ sentry: https://sentry.ceph.com/organizations/ceph/?query=99260dfa1f0742858733c9a3cfb1f975 Command failed on smithi005 with status 124: 'sudo adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' Pass ================================================================= [7362390] crimson-rados/basic/{centos_8.stream clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} time: 00:39:00 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362390/ [7362391] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_read} time: 00:22:07 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362391/ [7362392] crimson-rados/rbd/{centos_8.stream clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests} time: 00:53:20 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362392/ [7362393] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} time: 00:20:11 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362393/ [7362394] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4K_rand_rw} time: 00:27:54 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362394/ [7362395] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/pool-snaps-few-objects} time: 00:33:45 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362395/ [7362396] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_read} time: 00:18:56 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362396/ [7362397] crimson-rados/rbd/{centos_8.stream clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} time: 00:24:59 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362397/ [7362398] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 00:31:32 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362398/ [7362399] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_rw} time: 00:22:23 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362399/ [7362400] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 00:34:36 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362400/ [7362401] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/fio_4M_rand_write} time: 00:24:09 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362401/ [7362402] crimson-rados/basic/{centos_8.stream clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python} time: 00:24:43 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362402/ [7362403] crimson-rados/rbd/{centos_8.stream clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 00:25:44 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362403/ [7362404] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} time: 00:31:29 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362404/ [7362405] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_rand_read} time: 00:16:35 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362405/ [7362406] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 00:28:31 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362406/ [7362407] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4K_seq_read} time: 00:18:29 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362407/ [7362408] crimson-rados/rbd/{centos_8.stream clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} time: 00:15:32 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362408/ [7362409] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_rand_read} time: 00:19:02 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362409/ [7362411] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_seq_read} time: 00:15:54 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362411/ [7362412] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-balanced} time: 00:36:48 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362412/ [7362413] crimson-rados/basic/{centos_8.stream clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/readwrite} time: 00:22:18 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362413/ [7362414] crimson-rados/rbd/{centos_8.stream clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests} time: 00:29:51 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362414/ [7362415] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_4M_write} time: 00:17:56 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362415/ [7362416] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects-localized} time: 00:43:54 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362416/ [7362417] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/radosbench_omap_write} time: 00:28:23 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362417/ [7362418] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/snaps-few-objects} time: 00:40:09 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362418/ [7362419] crimson-rados/rbd/{centos_8.stream clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} time: 00:23:10 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362419/ [7362420] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_fio} time: 00:19:56 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362420/ [7362421] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 00:27:30 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362421/ [7362422] crimson-rados/perf/{centos_8.stream clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore settings/optimized workloads/sample_radosbench} time: 00:15:47 info: http://pulpito.ceph.com/yingxin-2023-08-08_02:00:15-crimson-rados-wip-yingxin-crimson-cleanup-osd-sharddispatcher-distro-default-smithi/7362422/