Found 181 jobs Found 12 distinct failure reasons Failure: Could not reconnect to ubuntu@gibba008.front.sepia.ceph.com 1 jobs: ['6471820'] suites: ['clusters/{fixed-3', 'objectstore/filestore-xfs', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{centos_8.stream}', 'workloads/dynamic_features_no_cache}'] Failure: Command failed on gibba032 with status 151: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 512 -w 512 -t 512 -h 512 -l 250000000 -S 0 -N 6000 -M pool_client.0 image_client.0' 1 jobs: ['6471839'] suites: ['cluster/{fixed-3', 'msgr-failures/few', 'objectstore/bluestore-comp-zstd', 'openstack}', 'rbd/nbd/{base/install', 'supported-random-distro$/{ubuntu_latest}', 'thrashers/default', 'thrashosds-health', 'workloads/rbd_fsx_nbd}'] Timeout 3h running clone.client.0/qa/workunits/rbd/cli_generic.sh 6 jobs: ['6471868', '6471873', '6471777', '6471807', '6471897', '6471838'] suites intersection: ['clusters/{fixed-1', 'msgr-failures/few', 'openstack}', 'pool/small-cache-pool', 'workloads/rbd_cli_generic}'] suites union: ['clusters/{fixed-1', 'features/defaults', 'features/format-1', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-snappy', 'objectstore/bluestore-comp-zstd', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/filestore-xfs', 'openstack}', 'pool/small-cache-pool', 'rbd/cli/{base/install', 'rbd/cli_v1/{base/install', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd_cli_generic}'] Failure: Failure object was: {'gibba037.front.sepia.ceph.com': {'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" install \'docker.io\'\' failed: E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 8092 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'stdout': '', 'stderr': 'E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 8092 (apt-get)\nE: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?\n', 'rc': 100, 'cache_updated': False, 'cache_update_time': 1636284116, 'invocation': {'module_args': {'name': ['docker.io'], 'state': 'latest', 'package': ['docker.io'], 'cache_valid_time': 0, 'purge': False, 'force': False, 'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'autoclean': False, 'only_upgrade': False, 'force_apt_get': False, 'allow_unauthenticated': False, 'update_cache': None, 'deb': None, 'default_release': None, 'install_recommends': None, 'upgrade': None, 'policy_rc_d': None}}, 'stdout_lines': [], 'stderr_lines': ['E: Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 8092 (apt-get)', 'E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?'], '_ansible_no_log': False, 'changed': False}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_c56135d151713269e811ede3163c9743c2e269de/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)yaml.representer.RepresenterError: ('cannot represent an object', 'cache_update_time') 1 jobs: ['6471893'] suites: ['clusters/{fixed-3', 'features/defaults', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'openstack}', 'pool/ec-data-pool', 'rbd/encryption/{cache/none', 'supported-random-distro$/{ubuntu_latest}', 'workloads/qemu_xfstests_luks2}'] Failure: Command failed on gibba015 with status 1: 'test -n "$(ls /tmp/ceph-immutable-object-cache )"' 1 jobs: ['6471896'] suites: ['openstack}', 'pool/ceph_and_immutable_object_cache', 'rbd/immutable-object-cache/{clusters/{fix-2', 'supported-random-distro$/{centos_8.stream}', 'workloads/qemu_on_immutable_object_cache_and_thrash}'] Failure: "2021-11-07T10:28:23.488031+0000 mon.a (mon.0) 314 : cluster [WRN] Health check failed: Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded (PG_DEGRADED)" in cluster log 5 jobs: ['6471874', '6471872', '6471902', '6471835', '6471806'] suites intersection: ['msgr-failures/few', 'openstack}'] suites union: ['clients/mirror', 'clients/{mirror-extra', 'cluster/{2-node', 'clusters/{fixed-3', 'features/defaults', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-zstd', 'objectstore/filestore-xfs', 'openstack}', 'policy/none', 'pool/ec-cache-pool', 'rbd-mirror/four-per-cluster', 'rbd/encryption/{cache/writethrough', 'rbd/mirror-thrash/{base/install', 'rbd/mirror/{base/install', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{rhel_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/qemu_xfstests_luks1}', 'workloads/rbd-mirror-journal-stress-workunit}', 'workloads/rbd-mirror-snapshot-bootstrap-workunit}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'workloads/rbd-mirror-workunit-min-compat-client-octopus}'] Dead: b'2021-11-07T12:27:58.964 DEBUG:teuthology.exit:Finished running handlers' ceph version 17.0.0-8645-g9b3fdce0 (9b3fdce0ab1c02dde23b7912d1b0cc7a087dcde8) quincy (dev) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fc02caff3c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x1b0) [0x55d2d147cf25] 5: (KernelDevice::_aio_thread()+0xe19) [0x55d2d2140929] 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x55d2d21467e1] 7: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fc02caf3609] 8: clone() 3 jobs: ['6471756', '6471771', '6471737'] suites intersection: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'openstack}', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install'] suites union: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-hybrid', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'policy/none', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_8.stream}', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{ubuntu_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'workloads/rbd-mirror-snapshot-stress-workunit-fast-diff}', 'workloads/rbd-mirror-snapshot-stress-workunit-minimum}'] Failure: Command failed (workunit test rbd/test_librbd.sh) on gibba032 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9b3fdce0ab1c02dde23b7912d1b0cc7a087dcde8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 RBD_FEATURES=61 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh' 2 jobs: ['6471864', '6471774'] suites intersection: ['2-cluster/{fix-2', '4-pool/cache', '6-workloads/c_api_tests_with_defaults}', 'openstack}', 'rbd/persistent-writeback-cache/{1-base/install'] suites union: ['2-cluster/{fix-2', '3-supported-random-distro$/{centos_8.stream}', '3-supported-random-distro$/{ubuntu_latest}', '4-pool/cache', '5-cache-mode/rwl', '5-cache-mode/ssd', '6-workloads/c_api_tests_with_defaults}', 'openstack}', 'rbd/persistent-writeback-cache/{1-base/install'] Timeout 3h running clone.client.0/qa/workunits/rbd/qemu_dynamic_features.sh 2 jobs: ['6471834', '6471729'] suites intersection: ['clusters/{fixed-3', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{centos_8}'] suites union: ['clusters/{fixed-3', 'objectstore/bluestore-bitmap', 'objectstore/bluestore-comp-lz4', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{centos_8}', 'workloads/dynamic_features_no_cache}', 'workloads/dynamic_features}'] Failure: Command failed on gibba001 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:9b3fdce0ab1c02dde23b7912d1b0cc7a087dcde8 -v bootstrap --fsid 4b596302-3f6e-11ec-8c28-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.2.101 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring' 1 jobs: ['6471773'] suites: ['cluster/{fixed-3', 'openstack}', 'rbd/iscsi/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/cephadm_iscsi}'] Timeout 3h running clone.client.0/qa/workunits/rbd/qemu_rebuild_object_map.sh 1 jobs: ['6471889'] suites: ['clusters/{fixed-3', 'objectstore/bluestore-comp-zstd', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/rebuild_object_map}'] Failure: reached maximum tries (90) after waiting for 540 seconds 1 jobs: ['6471803'] suites: ['clusters/{fixed-2', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'rbd/thrash/{base/install', 'supported-random-distro$/{centos_8}', 'thrashers/cache', 'thrashosds-health', 'workloads/rbd_fsx_cache_writethrough}']