Found 179 jobs Ignoring malparsed backtrace: ceph version 17.1.0-26-g0722784c (0722784cdb161090ed35fecef7cbbff70ecf9deb) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f500b6d97cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f500b6d99de] Ignoring malparsed backtrace: Removing image: 73% complete... , Removing image: 74% complete... , Removing image: 75% complete... Ignoring malparsed backtrace: Removing image: 67% complete... , Removing image: 68% complete... , Removing image: 69% complete... Ignoring malparsed backtrace: Removing image: 61% complete... , Removing image: 62% complete... , Removing image: 63% complete... Ignoring malparsed backtrace: (Reading database ... 40% , (Reading database ... 45% , (Reading database ... 50% Ignoring malparsed backtrace: Removing image: 81% complete... , Removing image: 82% complete... , Removing image: 83% complete... Ignoring malparsed backtrace: Removing image: 78% complete... , Removing image: 79% complete... , Removing image: 80% complete... Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph_77b782875d733496cac8473a67f611f0795b892b/qa/tasks/qemu.py", line 694, in task , yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Ignoring malparsed backtrace: instructed osd(s) 3 to deep-scrub , 2022-03-14T03:35:45.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-03-14T03:35:45.882 INFO:teuthology.orchestra.run.smithi059.stdout: Ignoring malparsed backtrace: 2022-03-14T03:36:08.312 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.cuq0KCqIhg , 2022-03-14T03:36:08.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.cuq0KCqIhg , 2022-03-14T03:36:08.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.cuq0KCqIhg Ignoring malparsed backtrace: 2022-03-14T03:36:10.393 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2022-03-14T03:36:10.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2022-03-14T03:36:10.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers Not-understood crash indication 2022-03-14T03:30:31.830 INFO:tasks.qemu.client.0.smithi196.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 17.1.0-26-g0722784c (0722784cdb161090ed35fecef7cbbff70ecf9deb) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f500b6d97cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f500b6d99de] Ignoring malparsed backtrace: Removing image: 73% complete... , Removing image: 74% complete... , Removing image: 75% complete... Ignoring malparsed backtrace: Removing image: 67% complete... , Removing image: 68% complete... , Removing image: 69% complete... Ignoring malparsed backtrace: Removing image: 61% complete... , Removing image: 62% complete... , Removing image: 63% complete... Ignoring malparsed backtrace: (Reading database ... 40% , (Reading database ... 45% , (Reading database ... 50% Ignoring malparsed backtrace: Removing image: 81% complete... , Removing image: 82% complete... , Removing image: 83% complete... Ignoring malparsed backtrace: Removing image: 78% complete... , Removing image: 79% complete... , Removing image: 80% complete... Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph_77b782875d733496cac8473a67f611f0795b892b/qa/tasks/qemu.py", line 694, in task , yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Ignoring malparsed backtrace: instructed osd(s) 3 to deep-scrub , 2022-03-14T03:35:45.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-03-14T03:35:45.882 INFO:teuthology.orchestra.run.smithi059.stdout: Ignoring malparsed backtrace: 2022-03-14T03:36:08.312 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.cuq0KCqIhg , 2022-03-14T03:36:08.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.cuq0KCqIhg , 2022-03-14T03:36:08.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.cuq0KCqIhg Ignoring malparsed backtrace: 2022-03-14T03:36:10.393 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2022-03-14T03:36:10.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2022-03-14T03:36:10.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers Not-understood crash indication 2022-03-14T03:30:31.830 INFO:tasks.qemu.client.0.smithi196.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 17.1.0-26-g0722784c (0722784cdb161090ed35fecef7cbbff70ecf9deb) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f500b6d97cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f500b6d99de] Ignoring malparsed backtrace: Removing image: 73% complete... , Removing image: 74% complete... , Removing image: 75% complete... Ignoring malparsed backtrace: Removing image: 67% complete... , Removing image: 68% complete... , Removing image: 69% complete... Ignoring malparsed backtrace: Removing image: 61% complete... , Removing image: 62% complete... , Removing image: 63% complete... Ignoring malparsed backtrace: (Reading database ... 40% , (Reading database ... 45% , (Reading database ... 50% Ignoring malparsed backtrace: Removing image: 81% complete... , Removing image: 82% complete... , Removing image: 83% complete... Ignoring malparsed backtrace: Removing image: 78% complete... , Removing image: 79% complete... , Removing image: 80% complete... Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph_77b782875d733496cac8473a67f611f0795b892b/qa/tasks/qemu.py", line 694, in task , yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Ignoring malparsed backtrace: instructed osd(s) 3 to deep-scrub , 2022-03-14T03:35:45.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-03-14T03:35:45.882 INFO:teuthology.orchestra.run.smithi059.stdout: Ignoring malparsed backtrace: 2022-03-14T03:36:08.312 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.cuq0KCqIhg , 2022-03-14T03:36:08.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.cuq0KCqIhg , 2022-03-14T03:36:08.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.cuq0KCqIhg Ignoring malparsed backtrace: 2022-03-14T03:36:10.393 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2022-03-14T03:36:10.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2022-03-14T03:36:10.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers Not-understood crash indication 2022-03-14T03:30:31.830 INFO:tasks.qemu.client.0.smithi196.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 17.1.0-26-g0722784c (0722784cdb161090ed35fecef7cbbff70ecf9deb) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f500b6d97cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f500b6d99de] Ignoring malparsed backtrace: Removing image: 73% complete... , Removing image: 74% complete... , Removing image: 75% complete... Ignoring malparsed backtrace: Removing image: 67% complete... , Removing image: 68% complete... , Removing image: 69% complete... Ignoring malparsed backtrace: Removing image: 61% complete... , Removing image: 62% complete... , Removing image: 63% complete... Ignoring malparsed backtrace: (Reading database ... 40% , (Reading database ... 45% , (Reading database ... 50% Ignoring malparsed backtrace: Removing image: 81% complete... , Removing image: 82% complete... , Removing image: 83% complete... Ignoring malparsed backtrace: Removing image: 78% complete... , Removing image: 79% complete... , Removing image: 80% complete... Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph_77b782875d733496cac8473a67f611f0795b892b/qa/tasks/qemu.py", line 694, in task , yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Ignoring malparsed backtrace: instructed osd(s) 3 to deep-scrub , 2022-03-14T03:35:45.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-03-14T03:35:45.882 INFO:teuthology.orchestra.run.smithi059.stdout: Ignoring malparsed backtrace: 2022-03-14T03:36:08.312 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.cuq0KCqIhg , 2022-03-14T03:36:08.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.cuq0KCqIhg , 2022-03-14T03:36:08.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.cuq0KCqIhg Ignoring malparsed backtrace: 2022-03-14T03:36:10.393 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2022-03-14T03:36:10.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2022-03-14T03:36:10.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers Not-understood crash indication 2022-03-14T03:30:31.830 INFO:tasks.qemu.client.0.smithi196.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Ignoring malparsed backtrace: ceph version 17.1.0-26-g0722784c (0722784cdb161090ed35fecef7cbbff70ecf9deb) quincy (dev) , 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x7f500b6d97cc] , 2: /usr/lib/ceph/libceph-common.so.2(+0x27b9de) [0x7f500b6d99de] Ignoring malparsed backtrace: Removing image: 73% complete... , Removing image: 74% complete... , Removing image: 75% complete... Ignoring malparsed backtrace: Removing image: 67% complete... , Removing image: 68% complete... , Removing image: 69% complete... Ignoring malparsed backtrace: Removing image: 61% complete... , Removing image: 62% complete... , Removing image: 63% complete... Ignoring malparsed backtrace: (Reading database ... 40% , (Reading database ... 45% , (Reading database ... 50% Ignoring malparsed backtrace: Removing image: 81% complete... , Removing image: 82% complete... , Removing image: 83% complete... Ignoring malparsed backtrace: Removing image: 78% complete... , Removing image: 79% complete... , Removing image: 80% complete... Ignoring malparsed backtrace: File "/home/teuthworker/src/github.com_ceph_ceph_77b782875d733496cac8473a67f611f0795b892b/qa/tasks/qemu.py", line 694, in task , yield , File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ Ignoring malparsed backtrace: instructed osd(s) 3 to deep-scrub , 2022-03-14T03:35:45.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json , 2022-03-14T03:35:45.882 INFO:teuthology.orchestra.run.smithi059.stdout: Ignoring malparsed backtrace: 2022-03-14T03:36:08.312 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.cuq0KCqIhg , 2022-03-14T03:36:08.312 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.cuq0KCqIhg , 2022-03-14T03:36:08.391 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.cuq0KCqIhg Ignoring malparsed backtrace: 2022-03-14T03:36:10.393 INFO:teuthology.task.internal:Restoring /etc/sudoers... , 2022-03-14T03:36:10.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2022-03-14T03:36:10.410 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers Not-understood crash indication 2022-03-14T03:30:31.830 INFO:tasks.qemu.client.0.smithi196.stderr:daemon-helper: command crashed with signal 6 Not-understood crash indication Found 7 distinct failure reasons Timeout 3h running clone.client.0/qa/workunits/rbd/qemu_dynamic_features.sh 3 jobs: ['6735215', '6735162', '6735082'] suites intersection: ['clusters/{fixed-3', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/dynamic_features_no_cache}'] suites union: ['clusters/{fixed-3', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-stupid', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'workloads/dynamic_features_no_cache}'] Failure: reached maximum tries (90) after waiting for 540 seconds 1 jobs: ['6735118'] suites: ['cluster/{fixed-3', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{rhel_8}', 'thrashers/default', 'thrashosds-health', 'workloads/rbd_fsx_nbd}'] Failure: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi115 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=77b782875d733496cac8473a67f611f0795b892b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=journal MIRROR_POOL_MODE=pool RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 1 jobs: ['6735186'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-stupid', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rbd-mirror-journal-stress-workunit}'] Failure: Command failed on smithi196 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term qemu-system-x86_64 -enable-kvm -nographic -cpu host -smp 4 -m 4096 -cdrom /home/ubuntu/cephtest/qemu/client.0.iso -drive file=rbd:rbd/client.0.0-clone:id=0,format=raw,if=virtio,cache=writeback -drive file=rbd:rbd/client.0.1-clone:id=0,format=raw,if=virtio,cache=writeback -drive file=rbd:rbd/client.0.2-clone:id=0,format=raw,if=virtio,cache=writeback' 1 jobs: ['6735188'] suites: ['openstack}', 'pool/ceph_and_immutable_object_cache', 'rbd/immutable-object-cache/{clusters/{fix-2', 'supported-random-distro$/{ubuntu_latest}', 'workloads/qemu_on_immutable_object_cache_and_thrash}'] Timeout 3h running clone.client.0/qa/workunits/rbd/cli_generic.sh 5 jobs: ['6735180', '6735117', '6735178', '6735148', '6735091'] suites intersection: ['clusters/{fixed-1', 'msgr-failures/few', 'openstack}', 'pool/small-cache-pool', 'workloads/rbd_cli_generic}'] suites union: ['clusters/{fixed-1', 'features/format-1', 'features/journaling', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-stupid', 'openstack}', 'pool/small-cache-pool', 'rbd/cli/{base/install', 'rbd/cli_v1/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_cli_generic}'] Failure: Command failed on smithi105 with status 1: 'test -f /home/ubuntu/cephtest/archive/qemu/client.0/success' 1 jobs: ['6735141'] suites: ['2-cluster/{fix-2', '3-supported-random-distro$/{ubuntu_latest}', '4-pool/cache', '5-cache-mode/ssd', '6-workloads/qemu_xfstests}', 'openstack}', 'rbd/persistent-writeback-cache/{1-base/install'] Failure: "2022-03-14T03:15:09.310735+0000 mon.a (mon.0) 214 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4620 objects degraded (0.043%), 1 pg degraded (PG_DEGRADED)" in cluster log 2 jobs: ['6735169', '6735119'] suites intersection: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'openstack}', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install'] suites union: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-comp-zlib', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-fsx-workunit}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}']