2022-07-17T12:54:12.120 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-17T12:54:12.121 INFO:teuthology.results:subset: '31440/100000' 2022-07-17T12:54:12.122 INFO:teuthology.results:seed: '7042' 2022-07-17T13:01:05.509 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-17T13:01:05.514 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-07-18T01:02:04.006 WARNING:teuthology.results:test(s) did not finish before timeout of 43200 seconds 2022-07-18T01:02:04.632 INFO:scrape:Found 8 jobs 2022-07-18T01:02:05.295 INFO:scrape:Found 7 distinct failure reasons 2022-07-18T01:02:05.296 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test_python.sh --eval-attr \ 2022-07-18T01:02:05.296 INFO:scrape:1 jobs: ['6933125'] 2022-07-18T01:02:05.297 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'deploy/ceph', 'tasks/rados_python}'] 2022-07-18T01:02:05.297 INFO:scrape: 2022-07-18T01:02:05.297 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd_python.sh 2022-07-18T01:02:05.298 INFO:scrape:1 jobs: ['6933128'] 2022-07-18T01:02:05.299 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2022-07-18T01:02:05.299 INFO:scrape: 2022-07-18T01:02:05.300 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rados/test.sh 2022-07-18T01:02:05.300 INFO:scrape:1 jobs: ['6933121'] 2022-07-18T01:02:05.301 INFO:scrape:suites: ['clusters/fixed-2', 'crimson-rados/basic/{centos_latest', 'deploy/ceph', 'tasks/rados_api_tests}'] 2022-07-18T01:02:05.301 INFO:scrape: 2022-07-18T01:02:05.301 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2022-07-18T01:02:05.302 INFO:scrape:1 jobs: ['6933126'] 2022-07-18T01:02:05.302 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2022-07-18T01:02:05.303 INFO:scrape: 2022-07-18T01:02:05.303 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh 2022-07-18T01:02:05.303 INFO:scrape:1 jobs: ['6933122'] 2022-07-18T01:02:05.304 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}'] 2022-07-18T01:02:05.304 INFO:scrape: 2022-07-18T01:02:05.305 INFO:scrape:Dead: b"2022-07-18T01:01:03.758 DEBUG:teuthology.orchestra.run.smithi043:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false" 2022-07-18T01:02:05.305 INFO:scrape:1 jobs: ['6933123'] 2022-07-18T01:02:05.306 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}'] 2022-07-18T01:02:05.306 INFO:scrape: 2022-07-18T01:02:05.307 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/cls/test_cls_rbd.sh 2022-07-18T01:02:05.307 INFO:scrape:1 jobs: ['6933124'] 2022-07-18T01:02:05.307 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'deploy/ceph', 'tasks/rbd_cls_tests}'] 2022-07-18T01:02:05.308 INFO:scrape: 2022-07-18T01:02:05.309 INFO:teuthology.results:Sending results to ceph-qa@ceph.io: Test Run: teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi ================================================================= info: http://pulpito.ceph.com/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/ logs: http://qa-proxy.ceph.com/teuthology/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/ failed: 6 dead: 0 running: 1 waiting: 0 queued: 0 passed: 1 Fail ================================================================= [6933121] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/rados_api_tests} ----------------------------------------------------------------- time: 03:10:55 info: http://pulpito.ceph.com/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933121/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933121/ sentry: https://sentry.ceph.com/organizations/ceph/?query=258edda681bb4611b74847d89096a693 Command failed (workunit test rados/test.sh) on smithi119 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bc7284a3d1e4cb88ce099a8c15575d65dbfc5de0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' [6933122] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_api_tests_old_format} ----------------------------------------------------------------- time: 03:11:01 info: http://pulpito.ceph.com/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933122/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933122/ sentry: https://sentry.ceph.com/organizations/ceph/?query=41bf4068775345eb9e7b9da82a51e441 Command failed (workunit test rbd/test_librbd.sh) on smithi019 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bc7284a3d1e4cb88ce099a8c15575d65dbfc5de0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' [6933124] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_cls_tests} ----------------------------------------------------------------- time: 03:10:39 info: http://pulpito.ceph.com/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933124/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933124/ sentry: https://sentry.ceph.com/organizations/ceph/?query=261a6ff30af44fb088ad76a165ef26c3 Command failed (workunit test cls/test_cls_rbd.sh) on smithi130 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bc7284a3d1e4cb88ce099a8c15575d65dbfc5de0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' [6933125] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/rados_python} ----------------------------------------------------------------- time: 03:29:06 info: http://pulpito.ceph.com/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933125/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933125/ sentry: https://sentry.ceph.com/organizations/ceph/?query=a1fa6e8c5b5c43c596b3da03be79e31f Command failed (workunit test rados/test_python.sh) on smithi136 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bc7284a3d1e4cb88ce099a8c15575d65dbfc5de0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_python.sh --eval-attr \'not (tier or snap or ec or bench or stats)\'' [6933126] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_lock_and_fence} ----------------------------------------------------------------- time: 03:13:57 info: http://pulpito.ceph.com/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933126/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933126/ sentry: https://sentry.ceph.com/organizations/ceph/?query=e740472f5f354b6395a4288bc15d2445 Command failed (workunit test rbd/test_lock_fence.sh) on smithi084 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bc7284a3d1e4cb88ce099a8c15575d65dbfc5de0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh' [6933128] crimson-rados/rbd/{centos_latest clusters/fixed-1 deploy/ceph tasks/rbd_python_api_tests_old_format} ----------------------------------------------------------------- time: 03:10:25 info: http://pulpito.ceph.com/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933128/ log: http://qa-proxy.ceph.com/teuthology/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933128/ sentry: https://sentry.ceph.com/organizations/ceph/?query=02553f8b80c94b6787e222233a388dd7 Command failed (workunit test rbd/test_librbd_python.sh) on smithi103 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bc7284a3d1e4cb88ce099a8c15575d65dbfc5de0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clo ne.client.0/qa/workunits/rbd/test_librbd_python.sh' Running ================================================================= [6933123] crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects} info: http://pulpito.ceph.com/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933123/ Pass ================================================================= [6933127] crimson-rados/basic/{centos_latest clusters/fixed-2 deploy/ceph tasks/readwrite} time: 00:13:00 info: http://pulpito.ceph.com/teuthology-2022-07-17_01:01:04-crimson-rados-quincy-distro-default-smithi/6933127/