2022-11-24T09:59:01.119 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf 2022-11-24T09:59:01.119 INFO:teuthology.results:seed: '5097' 2022-11-24T10:16:14.213 INFO:root:teuthology version: 0.0.1.dev3+g4da97cf 2022-11-24T10:16:14.218 INFO:teuthology.results:Waiting up to 43200 seconds for tests to finish... 2022-11-24T14:16:33.043 INFO:teuthology.results:Tests finished! gathering results... 2022-11-24T14:16:34.067 INFO:scrape:Found 12 jobs 2022-11-24T14:16:34.534 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fcef38e9bee] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x55d3cf0bf0ca] 2022-11-24T14:16:34.535 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:09:16.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740201 got 34359740199 for osd.1 , 2022-11-24T11:09:17.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 , 2022-11-24T11:09:17.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-11-24T14:16:34.536 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:09:39.200 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.m1hGY6ydqh , 2022-11-24T11:09:39.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.m1hGY6ydqh , 2022-11-24T11:09:39.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.m1hGY6ydqh 2022-11-24T14:16:34.536 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:10:56.892 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... , 2022-11-24T11:10:56.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2022-11-24T11:10:56.895 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-11-24T14:16:34.663 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fcef38e9bee] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x55d3cf0bf0ca] 2022-11-24T14:16:34.664 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:09:16.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740201 got 34359740199 for osd.1 , 2022-11-24T11:09:17.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 , 2022-11-24T11:09:17.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-11-24T14:16:34.664 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:09:39.200 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.m1hGY6ydqh , 2022-11-24T11:09:39.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.m1hGY6ydqh , 2022-11-24T11:09:39.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.m1hGY6ydqh 2022-11-24T14:16:34.664 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:10:56.892 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... , 2022-11-24T11:10:56.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2022-11-24T11:10:56.895 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-11-24T14:16:34.793 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fcef38e9bee] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x55d3cf0bf0ca] 2022-11-24T14:16:34.794 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:09:16.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740201 got 34359740199 for osd.1 , 2022-11-24T11:09:17.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 , 2022-11-24T11:09:17.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-11-24T14:16:34.795 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:09:39.200 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.m1hGY6ydqh , 2022-11-24T11:09:39.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.m1hGY6ydqh , 2022-11-24T11:09:39.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.m1hGY6ydqh 2022-11-24T14:16:34.795 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:10:56.892 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... , 2022-11-24T11:10:56.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2022-11-24T11:10:56.895 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-11-24T14:16:34.920 WARNING:scrape:Ignoring malparsed backtrace: ceph version 18.0.0-941-ga1b7f9ac (a1b7f9ac6434451a681e5e6e4b41516a163d8b8a) reef (dev) , 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xcc) [0x7fcef38e9bee] , 2: (WriteOp::_finish(TestOp::CallbackInfo*)+0x1b0) [0x55d3cf0bf0ca] 2022-11-24T14:16:34.921 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:09:16.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359740201 got 34359740199 for osd.1 , 2022-11-24T11:09:17.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 , 2022-11-24T11:09:17.818 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-11-24T14:16:34.921 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:09:39.200 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.m1hGY6ydqh , 2022-11-24T11:09:39.201 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.m1hGY6ydqh , 2022-11-24T11:09:39.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.m1hGY6ydqh 2022-11-24T14:16:34.922 WARNING:scrape:Ignoring malparsed backtrace: 2022-11-24T11:10:56.892 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... , 2022-11-24T11:10:56.893 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz , 2022-11-24T11:10:56.895 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-11-24T14:16:35.068 INFO:scrape:Found 4 distinct failure reasons 2022-11-24T14:16:35.069 INFO:scrape:Failure: Command failed (workunit test rbd/test_librbd_python.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3d15501d1ba1667ece7942aaccbed888deca959a TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh' 2022-11-24T14:16:35.069 INFO:scrape:1 jobs: ['7091047'] 2022-11-24T14:16:35.069 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_python_api_tests_old_format}'] 2022-11-24T14:16:35.069 INFO:scrape: 2022-11-24T14:16:35.070 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh 2022-11-24T14:16:35.070 INFO:scrape:1 jobs: ['7091038'] 2022-11-24T14:16:35.070 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_api_tests_old_format}'] 2022-11-24T14:16:35.070 INFO:scrape: 2022-11-24T14:16:35.071 INFO:scrape:Failure: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0' 2022-11-24T14:16:35.071 INFO:scrape:1 jobs: ['7091045'] 2022-11-24T14:16:35.071 INFO:scrape:suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', 'centos_latest', 'clusters/{fixed-2}', 'crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size', 'crimson_qa_overrides', 'deploy/ceph', 'objectstore/bluestore', 'thrashers/default', 'thrashosds-health', 'workloads/small-objects}'] 2022-11-24T14:16:35.072 INFO:scrape: 2022-11-24T14:16:35.072 INFO:scrape:Timeout 3h running clone.client.0/qa/workunits/rbd/test_lock_fence.sh 2022-11-24T14:16:35.072 INFO:scrape:1 jobs: ['7091044'] 2022-11-24T14:16:35.073 INFO:scrape:suites: ['clusters/fixed-1', 'crimson-rados/rbd/{centos_latest', 'crimson_qa_overrides', 'deploy/ceph', 'tasks/rbd_lock_and_fence}'] 2022-11-24T14:16:35.073 INFO:scrape: