Found 385 jobs Missing teuthology log /home/teuthworker/archive/teuthology-2021-10-17_07:01:03-rados-master-distro-default-gibba/6447029/teuthology.log Ignoring malparsed backtrace: ceph version 17.0.0-8337-g5db11d39 (5db11d398856f1e5a93048865af3a8da8e128e8d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f7dd17033c0] , 2: gsignal() Ignoring malparsed backtrace: 2021-10-18T20:19:57.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-10-18T20:19:57.942 INFO:teuthology.task.internal:Transferring archived files... , 2021-10-18T20:19:57.944 DEBUG:teuthology.misc:Transferring archived files from gibba019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2021-10-17_07:01:03-rados-master-distro-default-gibba/6447057/remote/gibba019 Ignoring malparsed backtrace: ceph version 17.0.0-8337-g5db11d39 (5db11d398856f1e5a93048865af3a8da8e128e8d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f7dd17033c0] , 2: gsignal() Ignoring malparsed backtrace: 2021-10-18T20:19:57.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-10-18T20:19:57.942 INFO:teuthology.task.internal:Transferring archived files... , 2021-10-18T20:19:57.944 DEBUG:teuthology.misc:Transferring archived files from gibba019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2021-10-17_07:01:03-rados-master-distro-default-gibba/6447057/remote/gibba019 Ignoring malparsed backtrace: ceph version 17.0.0-8337-g5db11d39 (5db11d398856f1e5a93048865af3a8da8e128e8d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f7dd17033c0] , 2: gsignal() Ignoring malparsed backtrace: 2021-10-18T20:19:57.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-10-18T20:19:57.942 INFO:teuthology.task.internal:Transferring archived files... , 2021-10-18T20:19:57.944 DEBUG:teuthology.misc:Transferring archived files from gibba019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2021-10-17_07:01:03-rados-master-distro-default-gibba/6447057/remote/gibba019 Ignoring malparsed backtrace: ceph version 17.0.0-8337-g5db11d39 (5db11d398856f1e5a93048865af3a8da8e128e8d) quincy (dev) , 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f7dd17033c0] , 2: gsignal() Ignoring malparsed backtrace: 2021-10-18T20:19:57.929 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive , 2021-10-18T20:19:57.942 INFO:teuthology.task.internal:Transferring archived files... , 2021-10-18T20:19:57.944 DEBUG:teuthology.misc:Transferring archived files from gibba019:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2021-10-17_07:01:03-rados-master-distro-default-gibba/6447057/remote/gibba019 Ignoring malparsed backtrace: ceph version 17.0.0-8337-g5db11d39 (5db11d398856f1e5a93048865af3a8da8e128e8d) quincy (dev) , 1: /lib64/libpthread.so.0(+0x12b20) [0x7fe79d5ecb20] , 2: gsignal() Ignoring malparsed backtrace: 2021-10-18T17:55:35.746 DEBUG:teuthology.orchestra.run.gibba005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2021-10-18T17:55:35.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump , 2021-10-18T17:55:35.806 DEBUG:teuthology.orchestra.run.gibba005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump Ignoring malparsed backtrace: ceph version 17.0.0-8337-g5db11d39 (5db11d398856f1e5a93048865af3a8da8e128e8d) quincy (dev) , 1: /lib64/libpthread.so.0(+0x12b20) [0x7fe79d5ecb20] , 2: gsignal() Ignoring malparsed backtrace: 2021-10-18T17:55:35.746 DEBUG:teuthology.orchestra.run.gibba005:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers , 2021-10-18T17:55:35.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump , 2021-10-18T17:55:35.806 DEBUG:teuthology.orchestra.run.gibba005:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump Found 8 distinct failure reasons Failure: Command failed (workunit test scrub/osd-scrub-repair.sh) on gibba008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5db11d398856f1e5a93048865af3a8da8e128e8d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh' 1 jobs: ['6447081'] suites: ['rados/standalone/{supported-random-distro$/{centos_8.stream}', 'workloads/scrub}'] Dead: None 8 jobs: ['6447029', '6447024', '6446946', '6447229', '6446950', '6446944', '6446956', '6447025'] Failure: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c \'mkdir $TESTDIR/archive/ostest && cd $TESTDIR/archive/ostest && ulimit -Sn 16384 && CEPH_ARGS="--no-log-to-stderr --log-file $TESTDIR/archive/ceph_test_objectstore.log --debug-bluestore 20" ceph_test_objectstore --gtest_filter=*SyntheticMatrixC*/2 --gtest_catch_exceptions=0\'' 2 jobs: ['6447057', '6447020'] suites intersection: [] suites union: ['rados/objectstore/{backends/objectstore-bluestore-a', 'rados/objectstore/{backends/objectstore-bluestore-b', 'supported-random-distro$/{rhel_8}}', 'supported-random-distro$/{ubuntu_latest}}'] Failure: Command failed on gibba015 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:5db11d398856f1e5a93048865af3a8da8e128e8d shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5effc88e-3077-11ec-8c28-001a4aab830c -- ceph mon dump -f json' 3 jobs: ['6447216', '6447224', '6446995'] suites intersection: ['0-nvme-loop', '1-start', '3-final}'] suites union: ['0-nvme-loop', '1-start', '2-services/client-keyring', '2-services/nfs-ingress', '2-services/nfs-ingress2', '3-final}', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.2_container_tools_3.0', 'rados/cephadm/smoke-roleless/{0-distro/centos_8.3_container_tools_3.0'] Failure: reached maximum tries (60) after waiting for 900 seconds 1 jobs: ['6447036'] Failure: Failed to power on gibba027 3 jobs: ['6447237', '6446951', '6446989'] Failure: Command failed (workunit test cephadm/test_dashboard_e2e.sh) on gibba008 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5db11d398856f1e5a93048865af3a8da8e128e8d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_dashboard_e2e.sh' 1 jobs: ['6447222'] suites: ['debug/mgr', 'mon_election/connectivity', 'rados/dashboard/{centos_8.2_container_tools_3.0', 'random-objectstore$/{bluestore-comp-snappy}', 'tasks/e2e}'] Failure: Could not reconnect to ubuntu@gibba008.front.sepia.ceph.com 1 jobs: ['6447190'] suites: ['1-pg-log-overrides/normal_pg_log', '2-recovery-overrides/{default}', '3-scrub-overrides/{max-simultaneous-scrubs-2}', 'backoff/normal', 'ceph', 'clusters/{fixed-2', 'crc-failures/bad_map_crc_failure', 'd-balancer/on', 'mon_election/connectivity', 'msgr-failures/osd-dispatch-delay', 'msgr/async-v2only', 'objectstore/bluestore-stupid', 'openstack}', 'rados', 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size', 'supported-random-distro$/{centos_8.stream}', 'thrashers/mapgap', 'thrashosds-health', 'workloads/cache-agent-big}']