2022-03-13T08:00:34.302 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-13T08:00:34.305 INFO:teuthology.results:subset: '2372/100000' 2022-03-13T08:00:34.305 INFO:teuthology.results:seed: '9125' 2022-03-13T08:05:58.312 INFO:root:teuthology version: 1.1.0-fbfc7b38 2022-03-13T08:05:58.316 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-03-13T14:39:27.568 INFO:teuthology.results:Tests finished! gathering results... 2022-03-13T14:39:28.181 INFO:scrape:Found 8 jobs 2022-03-13T14:39:28.426 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733975/remote/smithi122/log/ceph-osd.1.log.gz 2022-03-13T14:39:28.427 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733975/remote/smithi122/log/ceph-osd.2.log.gz 2022-03-13T14:39:28.428 WARNING:scrape:Not-understood crash indication 2022-03-13T14:39:28.495 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733975/remote/smithi122/log/ceph-osd.1.log.gz 2022-03-13T14:39:28.496 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733975/remote/smithi122/log/ceph-osd.2.log.gz 2022-03-13T14:39:28.497 WARNING:scrape:Not-understood crash indication 2022-03-13T14:39:29.149 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733970/remote/smithi007/log/ceph-osd.0.log.gz 2022-03-13T14:39:29.150 WARNING:scrape:Not-understood crash indication 2022-03-13T14:39:29.350 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733971/remote/smithi169/log/ceph-osd.0.log.gz 2022-03-13T14:39:29.351 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733971/remote/smithi169/log/ceph-osd.2.log.gz 2022-03-13T14:39:29.352 WARNING:scrape:Not-understood crash indication 2022-03-13T14:39:29.409 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733971/remote/smithi169/log/ceph-osd.0.log.gz 2022-03-13T14:39:29.410 WARNING:scrape:Missing log /home/teuthworker/archive/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733971/remote/smithi169/log/ceph-osd.2.log.gz 2022-03-13T14:39:29.411 WARNING:scrape:Not-understood crash indication 2022-03-13T14:39:29.413 INFO:scrape:Found 7 distinct failure reasons 2022-03-13T14:39:29.414 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2022-03-13T14:39:29.414 INFO:scrape:1 jobs: ['6733975'] 2022-03-13T14:39:29.415 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2022-03-13T14:39:29.415 INFO:scrape: 2022-03-13T14:39:29.415 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test_python.sh --eval-attr \ 2022-03-13T14:39:29.416 INFO:scrape:1 jobs: ['6733972'] 2022-03-13T14:39:29.416 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'deploy/ceph', 'tasks/rados_python}'] 2022-03-13T14:39:29.417 INFO:scrape: 2022-03-13T14:39:29.417 INFO:scrape:Dead: b'2022-03-13T14:36:20.310 DEBUG:teuthology.exit:Finished running handlers' 2022-03-13T14:39:29.417 INFO:scrape:1 jobs: ['6733970'] 2022-03-13T14:39:29.418 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}'] 2022-03-13T14:39:29.418 INFO:scrape: 2022-03-13T14:39:29.419 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2022-03-13T14:39:29.419 INFO:scrape:1 jobs: ['6733971'] 2022-03-13T14:39:29.419 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_cls_tests}'] 2022-03-13T14:39:29.420 INFO:scrape: 2022-03-13T14:39:29.420 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2022-03-13T14:39:29.421 INFO:scrape:1 jobs: ['6733973'] 2022-03-13T14:39:29.421 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2022-03-13T14:39:29.421 INFO:scrape: 2022-03-13T14:39:29.422 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2022-03-13T14:39:29.422 INFO:scrape:1 jobs: ['6733968'] 2022-03-13T14:39:29.423 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'deploy/ceph', 'tasks/rados_api_tests}'] 2022-03-13T14:39:29.423 INFO:scrape: 2022-03-13T14:39:29.423 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh 2022-03-13T14:39:29.424 INFO:scrape:1 jobs: ['6733969'] 2022-03-13T14:39:29.424 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}'] 2022-03-13T14:39:29.425 INFO:scrape: 2022-03-13T14:39:29.425 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/ failed: 6 dead: 1 running: 0 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [6733968] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:12:50 info: http://pulpito.ceph.com/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733968/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733968/ sentry: https://sentry.ceph.com/organizations/ceph/?query=fe375847ee7b4fb89e83cbfc092889df Command failed (workunit test rados/test.sh) on smithi149 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e39304dcd6158065e31e85efe4fcca6b6a43e7f2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [6733969] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 03:13:19 info: http://pulpito.ceph.com/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733969/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733969/ sentry: https://sentry.ceph.com/organizations/ceph/?query=f22823ff62254fac8af5bdbb1eca0066 Command failed (workunit test rbd/test_librbd.sh) on smithi026 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e39304dcd6158065e31e85efe4fcca6b6a43e7f2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' [6733971] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_cls_tests} ----------------------------------------------------------------- time: 03:12:33 info: http://pulpito.ceph.com/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733971/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733971/ sentry: https://sentry.ceph.com/organizations/ceph/?query=4f074a0806ba423bba8897d82203c539 Command failed (workunit test cls/test_cls_rbd.sh) on smithi169 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e39304dcd6158065e31e85efe4fcca6b6a43e7f2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [6733972] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/rados_python} ----------------------------------------------------------------- time: 03:30:40 info: http://pulpito.ceph.com/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733972/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733972/ sentry: https://sentry.ceph.com/organizations/ceph/?query=b15eb617cecc4619b3fc380a6540032a Command failed (workunit test rados/test_python.sh) on smithi171 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e39304dcd6158065e31e85efe4fcca6b6a43e7f2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh --eval-attr \'not (tier or snap or ec or bench or stats)\'' [6733973] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 03:15:34 info: http://pulpito.ceph.com/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733973/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733973/ sentry: https://sentry.ceph.com/organizations/ceph/?query=41c677de227843a0a10d953825b4d693 Command failed (workunit test rbd/test_lock_fence.sh) on smithi093 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e39304dcd6158065e31e85efe4fcca6b6a43e7f2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' [6733975] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 03:12:24 info: http://pulpito.ceph.com/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733975/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733975/ sentry: https://sentry.ceph.com/organizations/ceph/?query=c42067aea6614d8196a4fa7506a89c18 Command failed (workunit test rbd/test_librbd_python.sh) on smithi122 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e39304dcd6158065e31e85efe4fcca6b6a43e7f2 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/test_librbd_python.sh' Dead ================================================================= [6733970] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} ----------------------------------------------------------------- time: 00:00:00 info: http://pulpito.ceph.com/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733970/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733970/ hit max job timeout Pass ================================================================= [6733974] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/readwrite} time: 00:14:42 info: http://pulpito.ceph.com/teuthology-2022-03-13_01:01:03-crimson-rados-master-distro-default-smithi/6733974/