2022-01-30T10:59:36.598 INFO:root:teuthology version: 1.1.0-7c0cb867 2022-01-30T10:59:36.599 INFO:teuthology.results:subset: '32056/100000' 2022-01-30T10:59:36.600 INFO:teuthology.results:seed: '4270' 2022-01-30T11:07:30.732 INFO:root:teuthology version: 1.1.0-7c0cb867 2022-01-30T11:07:30.738 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-01-30T17:38:59.756 INFO:teuthology.results:Tests finished! gathering results... 2022-01-30T17:39:00.483 INFO:scrape:Found 8 jobs 2022-01-30T17:39:02.541 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648307/remote/smithi040/log/ceph-osd.3.log.gz 2022-01-30T17:39:02.544 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648307/remote/smithi019/log/ceph-osd.1.log.gz 2022-01-30T17:39:02.545 WARNING:scrape:Not-understood crash indication 2022-01-30T17:39:03.007 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648307/remote/smithi040/log/ceph-osd.3.log.gz 2022-01-30T17:39:03.008 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648307/remote/smithi019/log/ceph-osd.1.log.gz 2022-01-30T17:39:03.009 WARNING:scrape:Not-understood crash indication 2022-01-30T17:39:03.634 INFO:scrape:Found 7 distinct failure reasons 2022-01-30T17:39:03.635 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh 2022-01-30T17:39:03.636 INFO:scrape:1 jobs: ['6648308'] 2022-01-30T17:39:03.636 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}'] 2022-01-30T17:39:03.637 INFO:scrape: 2022-01-30T17:39:03.637 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test_python.sh --eval-attr \ 2022-01-30T17:39:03.637 INFO:scrape:1 jobs: ['6648311'] 2022-01-30T17:39:03.638 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'deploy/ceph', 'tasks/rados_python}'] 2022-01-30T17:39:03.638 INFO:scrape: 2022-01-30T17:39:03.639 INFO:scrape:Dead: b'2022-01-30T17:35:50.923 DEBUG:teuthology.exit:Finished running handlers' 2022-01-30T17:39:03.639 INFO:scrape:1 jobs: ['6648309'] 2022-01-30T17:39:03.640 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}'] 2022-01-30T17:39:03.640 INFO:scrape: 2022-01-30T17:39:03.641 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2022-01-30T17:39:03.641 INFO:scrape:1 jobs: ['6648307'] 2022-01-30T17:39:03.642 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'deploy/ceph', 'tasks/rados_api_tests}'] 2022-01-30T17:39:03.642 INFO:scrape: 2022-01-30T17:39:03.642 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2022-01-30T17:39:03.643 INFO:scrape:1 jobs: ['6648310'] 2022-01-30T17:39:03.643 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_cls_tests}'] 2022-01-30T17:39:03.644 INFO:scrape: 2022-01-30T17:39:03.644 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2022-01-30T17:39:03.645 INFO:scrape:1 jobs: ['6648312'] 2022-01-30T17:39:03.645 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2022-01-30T17:39:03.645 INFO:scrape: 2022-01-30T17:39:03.646 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2022-01-30T17:39:03.646 INFO:scrape:1 jobs: ['6648314'] 2022-01-30T17:39:03.647 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2022-01-30T17:39:03.647 INFO:scrape: 2022-01-30T17:39:03.648 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/ failed: 6 dead: 1 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [6648307] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:13:26 info: http://pulpito.ceph.com/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648307/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648307/ sentry: https://sentry.ceph.com/organizations/ceph/?query=9e2e5bd930f547da86a7a4a4a84b6dfa Command failed (workunit test rados/test.sh) on smithi019 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b5a77eb1a142d339394614264c9540faf281cca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [6648308] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 03:12:36 info: http://pulpito.ceph.com/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648308/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648308/ sentry: https://sentry.ceph.com/organizations/ceph/?query=d84888e375a24227801b3d3b2b6ed45b Command failed (workunit test rbd/test_librbd.sh) on smithi149 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b5a77eb1a142d339394614264c9540faf281cca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' [6648310] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_cls_tests} ----------------------------------------------------------------- time: 03:12:43 info: http://pulpito.ceph.com/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648310/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648310/ sentry: https://sentry.ceph.com/organizations/ceph/?query=817670e51fc048509dc4730321860a8e Command failed (workunit test cls/test_cls_rbd.sh) on smithi107 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b5a77eb1a142d339394614264c9540faf281cca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [6648311] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/rados_python} ----------------------------------------------------------------- time: 03:31:32 info: http://pulpito.ceph.com/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648311/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648311/ sentry: https://sentry.ceph.com/organizations/ceph/?query=16724a9aa64945b0a4fbcbd385ccde6a Command failed (workunit test rados/test_python.sh) on smithi155 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b5a77eb1a142d339394614264c9540faf281cca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh --eval-attr \'not (tier or snap or ec or bench or stats)\'' [6648312] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 03:16:05 info: http://pulpito.ceph.com/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648312/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648312/ sentry: https://sentry.ceph.com/organizations/ceph/?query=48f65a34fd674e4eb0b5aec2388b50cc Command failed (workunit test rbd/test_lock_fence.sh) on smithi132 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b5a77eb1a142d339394614264c9540faf281cca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' [6648314] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 03:13:40 info: http://pulpito.ceph.com/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648314/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648314/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c7a4a990c48940aca6716e121a4111ee Command failed (workunit test rbd/test_librbd_python.sh) on smithi072 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b5a77eb1a142d339394614264c9540faf281cca TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/test_librbd_python.sh' Dead ================================================================= [6648309] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648309/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648309/ hit max job timeout Pass ================================================================= [6648313] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/readwrite} time: 00:15:20 info: http://pulpito.ceph.com/teuthology-2022-01-30_01:01:03-crimson-rados-master-distro-default-smithi/6648313/