2023-04-20T03:37:59.048 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-20T03:37:59.049 INFO:teuthology.results:seed: '9873' 2023-04-20T03:38:53.933 INFO:root:teuthology version: 0.0.1.dev96+g8d156ae 2023-04-20T03:38:53.937 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2023-04-20T08:43:16.849 INFO:teuthology.results:Tests finished! gathering results... 2023-04-20T08:43:17.859 INFO:scrape:Found 14 jobs 2023-04-20T08:43:18.524 INFO:scrape:Found 5 distinct failure reasons 2023-04-20T08:43:18.525 INFO:scrape:Failure: Command failed on smithi027 with status 1: 'sudo yum install -y kernel' 2023-04-20T08:43:18.525 INFO:scrape:1 jobs: ['7245756'] 2023-04-20T08:43:18.525 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/admin_socket_objecter_requests}'] 2023-04-20T08:43:18.526 INFO:scrape: 2023-04-20T08:43:18.526 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2023-04-20T08:43:18.526 INFO:scrape:1 jobs: ['7245766'] 2023-04-20T08:43:18.527 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2023-04-20T08:43:18.527 INFO:scrape: 2023-04-20T08:43:18.527 INFO:scrape:Failure: {'smithi123.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} 2023-04-20T08:43:18.528 INFO:scrape:1 jobs: ['7245761'] 2023-04-20T08:43:18.528 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects-balanced}'] 2023-04-20T08:43:18.528 INFO:scrape: 2023-04-20T08:43:18.528 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2023-04-20T08:43:18.529 INFO:scrape:1 jobs: ['7245762'] 2023-04-20T08:43:18.529 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2023-04-20T08:43:18.529 INFO:scrape: 2023-04-20T08:43:18.530 INFO:scrape:Failure: Command failed (workunit test rados/test.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35ec3f3976e2d906d6ae515fa30c7848480555a4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2023-04-20T08:43:18.530 INFO:scrape:1 jobs: ['7245754'] 2023-04-20T08:43:18.530 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rados_api_tests}'] 2023-04-20T08:43:18.530 INFO:scrape: 2023-04-20T08:43:18.531 INFO:teuthology.results:Sending results to yingxincheng@gmail.com: Test Run: yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/ failed: 4 dead: 1 running: 0 waiting: 0 queued: 0 passed: 9 Fail ================================================================= [7245754] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 02:27:23 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245754/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245754/ sentry: https://sentry.ceph.com/organizations/ceph/?query=7683d84541684d0283e420f2eca1ac90 Command failed (workunit test rados/test.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35ec3f3976e2d906d6ae515fa30c7848480555a4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [7245756] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245756/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245756/ sentry: https://sentry.ceph.com/organizations/ceph/?query=2209cb0975a144069926c8519be2d6b7 Command failed on smithi027 with status 1: 'sudo yum install -y kernel' [7245762] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 04:35:39 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245762/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245762/ sentry: https://sentry.ceph.com/organizations/ceph/?query=bd6f7291bf6e47edb60334d7e4042da4 Command failed (workunit test rbd/test_lock_fence.sh) on smithi143 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35ec3f3976e2d906d6ae515fa30c7848480555a4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' [7245766] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 04:33:40 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245766/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245766/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a4f486014ebb4d799d086cb1c9ce236a Command failed (workunit test rbd/test_librbd_python.sh) on smithi063 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=35ec3f3976e2d906d6ae515fa30c7848480555a4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph- coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cep htest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' Dead ================================================================= [7245761] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-balanced} ----------------------------------------------------------------- time: 00:36:13 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245761/ log: http://qa-proxy.ceph.com/teuthology/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245761/ sentry: https://sentry.ceph.com/organizations/ceph/?query=0a2682c9bd3b45e8a7b0f0138d67debc {'smithi123.front.sepia.ceph.com': {'_ansible_no_log': False, 'changed': False, 'invocation': {'module_args': {'allow_downgrade': False, 'allowerasing': False, 'autoremove': False, 'bugfix': False, 'conf_file': None, 'disable_excludes': None, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_dir': None, 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'install_repoquery': True, 'install_weak_deps': True, 'installroot': '/', 'list': None, 'lock_timeout': 30, 'name': ['krb5-workstation'], 'releasever': None, 'security': False, 'skip_broken': False, 'state': 'present', 'update_cache': False, 'update_only': False, 'validate_certs': True}}, 'msg': "Failed to download metadata for repo 'CentOS-PowerTools': Yum repo downloading error: Downloading error(s): repodata/e830a7a4e881ef24 680d161802ae07874dd447031dd12d47f2d3d4a911245522-primary.xml.gz - Cannot download, all mirrors were already tried without success", 'rc': 1, 'results': []}} Pass ================================================================= [7245755] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format} time: 01:50:02 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245755/ [7245757] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency} time: 01:48:12 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245757/ [7245758] crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests} time: 01:40:52 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245758/ [7245759] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench} time: 01:53:45 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245759/ [7245760] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python} time: 01:48:10 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245760/ [7245763] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects-localized} time: 01:42:41 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245763/ [7245764] crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/readwrite} time: 01:36:10 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245764/ [7245765] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} time: 01:42:51 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245765/ [7245767] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed} time: 01:46:43 info: http://pulpito.ceph.com/yingxin-2023-04-20_03:37:46-crimson-rados-wip-chunmei-multicore-os-crimson-only-distro-default-smithi/7245767/