2022-05-29T01:01:15.800 INFO:root:teuthology version: 1.1.1.dev348+g8df7ff2b 2022-05-29T01:01:15.801 INFO:teuthology.results:subset: '24197/100000' 2022-05-29T01:01:15.801 INFO:teuthology.results:seed: '9' 2022-05-29T01:01:23.096 INFO:root:teuthology version: 1.1.1.dev348+g8df7ff2b 2022-05-29T01:01:23.097 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-05-29T07:39:53.678 INFO:teuthology.results:Tests finished! gathering results... 2022-05-29T07:39:54.395 INFO:scrape:Found 8 jobs 2022-05-29T07:39:56.450 INFO:scrape:Found 7 distinct failure reasons 2022-05-29T07:39:56.450 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2022-05-29T07:39:56.451 INFO:scrape:1 jobs: ['6851698'] 2022-05-29T07:39:56.451 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2022-05-29T07:39:56.451 INFO:scrape: 2022-05-29T07:39:56.452 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh 2022-05-29T07:39:56.453 INFO:scrape:1 jobs: ['6851692'] 2022-05-29T07:39:56.453 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}'] 2022-05-29T07:39:56.453 INFO:scrape: 2022-05-29T07:39:56.453 INFO:scrape:Dead: b'2022-05-29T07:37:04.867 DEBUG:teuthology.exit:Finished running handlers' 2022-05-29T07:39:56.453 INFO:scrape:1 jobs: ['6851693'] 2022-05-29T07:39:56.453 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}'] 2022-05-29T07:39:56.454 INFO:scrape: 2022-05-29T07:39:56.454 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2022-05-29T07:39:56.456 INFO:scrape:1 jobs: ['6851691'] 2022-05-29T07:39:56.456 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'deploy/ceph', 'tasks/rados_api_tests}'] 2022-05-29T07:39:56.456 INFO:scrape: 2022-05-29T07:39:56.456 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test_python.sh --eval-attr \ 2022-05-29T07:39:56.456 INFO:scrape:1 jobs: ['6851695'] 2022-05-29T07:39:56.457 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'deploy/ceph', 'tasks/rados_python}'] 2022-05-29T07:39:56.457 INFO:scrape: 2022-05-29T07:39:56.457 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2022-05-29T07:39:56.457 INFO:scrape:1 jobs: ['6851694'] 2022-05-29T07:39:56.457 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_cls_tests}'] 2022-05-29T07:39:56.457 INFO:scrape: 2022-05-29T07:39:56.458 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2022-05-29T07:39:56.458 INFO:scrape:1 jobs: ['6851696'] 2022-05-29T07:39:56.458 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2022-05-29T07:39:56.458 INFO:scrape: 2022-05-29T07:39:56.459 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/ failed: 6 dead: 1 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [6851691] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:14:28 info: http://pulpito.ceph.com/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851691/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851691/ sentry: https://sentry.ceph.com/organizations/ceph/?query=35e875e0d16c407581cae23728a3d903 Command failed (workunit test rados/test.sh) on smithi142 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d5fe31a530fe6cfdad42a875e11a1ed21a1c61c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [6851692] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 03:13:45 info: http://pulpito.ceph.com/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851692/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851692/ sentry: https://sentry.ceph.com/organizations/ceph/?query=1b0fb1ad1ec74640a449f48e2da76457 Command failed (workunit test rbd/test_librbd.sh) on smithi158 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d5fe31a530fe6cfdad42a875e11a1ed21a1c61c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' [6851694] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_cls_tests} ----------------------------------------------------------------- time: 03:14:59 info: http://pulpito.ceph.com/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851694/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851694/ sentry: https://sentry.ceph.com/organizations/ceph/?query=3b52a58e1f864c9581a30347d48955a4 Command failed (workunit test cls/test_cls_rbd.sh) on smithi029 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d5fe31a530fe6cfdad42a875e11a1ed21a1c61c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [6851695] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/rados_python} ----------------------------------------------------------------- time: 03:32:28 info: http://pulpito.ceph.com/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851695/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851695/ sentry: https://sentry.ceph.com/organizations/ceph/?query=8ad4f7ac13954e79a89995187c4082f0 Command failed (workunit test rados/test_python.sh) on smithi145 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d5fe31a530fe6cfdad42a875e11a1ed21a1c61c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh --eval-attr \'not (tier or snap or ec or bench or stats)\'' [6851696] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 03:16:56 info: http://pulpito.ceph.com/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851696/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851696/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f98586f90f1742bd8f1a2b4821ede23d Command failed (workunit test rbd/test_lock_fence.sh) on smithi139 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d5fe31a530fe6cfdad42a875e11a1ed21a1c61c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' [6851698] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 03:13:45 info: http://pulpito.ceph.com/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851698/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851698/ sentry: https://sentry.ceph.com/organizations/ceph/?query=69dcaa1d8b924a98843a5dda36784977 Command failed (workunit test rbd/test_librbd_python.sh) on smithi086 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d5fe31a530fe6cfdad42a875e11a1ed21a1c61c TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/test_librbd_python.sh' Dead ================================================================= [6851693] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851693/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851693/ hit max job timeout Pass ================================================================= [6851697] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/readwrite} time: 00:16:28 info: http://pulpito.ceph.com/teuthology-2022-05-29_01:01:03-crimson-rados-quincy-distro-default-smithi/6851697/