Found 221 jobs Found 7 distinct failure reasons Failure: Command failed on smithi189 with status 1: "/usr/bin/sudo TESTDIR=/home/ubuntu/cephtest adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/bash /home/ubuntu/cephtest/run_xfstests.sh -c 1 -f ext4 -t /dev/rbd0 -s /dev/rbd1 -x /tmp/excludeszi8y1ui -r -- '-g auto -g blockdev -x clone'" 2 jobs: ['6719567', '6719456'] suites intersection: ['conf', 'krbd/singleton/{bluestore-bitmap', 'tasks/rbd_xfstests}'] suites union: ['conf', 'krbd/singleton/{bluestore-bitmap', 'ms_mode$/{crc}', 'ms_mode$/{secure}', 'msgr-failures/few', 'msgr-failures/many', 'tasks/rbd_xfstests}'] Failure: Command failed on smithi093 with status 100: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=16.2.7-526-g2fbeafc6-1focal cephadm=16.2.7-526-g2fbeafc6-1focal ceph-mds=16.2.7-526-g2fbeafc6-1focal ceph-mgr=16.2.7-526-g2fbeafc6-1focal ceph-common=16.2.7-526-g2fbeafc6-1focal ceph-fuse=16.2.7-526-g2fbeafc6-1focal ceph-test=16.2.7-526-g2fbeafc6-1focal radosgw=16.2.7-526-g2fbeafc6-1focal python3-rados=16.2.7-526-g2fbeafc6-1focal python3-rgw=16.2.7-526-g2fbeafc6-1focal python3-cephfs=16.2.7-526-g2fbeafc6-1focal python3-rbd=16.2.7-526-g2fbeafc6-1focal libcephfs2=16.2.7-526-g2fbeafc6-1focal libcephfs-dev=16.2.7-526-g2fbeafc6-1focal librados2=16.2.7-526-g2fbeafc6-1focal librbd1=16.2.7-526-g2fbeafc6-1focal rbd-fuse=16.2.7-526-g2fbeafc6-1focal' 2 jobs: ['6719533', '6719510'] suites intersection: ['conf'] suites union: ['clusters/3-node', 'clusters/fixed-3', 'conf', 'features/no-object-map', 'krbd/fsx/{ceph/ceph', 'krbd/rbd/{bluestore-bitmap', 'ms_mode$/{legacy}', 'ms_mode/secure', 'msgr-failures/many', 'objectstore/filestore-xfs', 'randomized-striping-off}', 'striping/default/{msgr-failures/few', 'tasks/fsx-1-client}', 'tasks/rbd_workunit_kernel_untar_build}'] Failure: "2022-03-04T08:20:04.106616+0000 osd.1 (osd.1) 3 : cluster [WRN] slow request osd_op(client.4399.0:332639 2.4 2:3298c2fb:::rbd_data.10e8e74a8160.00000000000003e9:head [set-alloc-hint object_size 4194304 write_size 4194304,write 4169728~16384 in=16384b] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected e20) initiated 2022-03-04T08:19:34.099015+0000 currently waiting for sub ops" in cluster log 2 jobs: ['6719615', '6719500'] suites intersection: ['clusters/fixed-3', 'conf', 'msgr-failures/many'] suites union: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'ms_mode/crc$/{crc}', 'ms_mode/secure', 'msgr-failures/many', 'tasks/krbd_latest_osdmap_on_map}', 'tasks/rbd_workunit_suites_ffsb}'] Failure: {'smithi105.front.sepia.ceph.com': {'_ansible_no_log': False, 'cache_update_time': 1646376565, 'cache_updated': False, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'name': ['nagios-nrpe-server', 'nagios-plugins-basic'], 'only_upgrade': False, 'package': ['nagios-nrpe-server', 'nagios-plugins-basic'], 'policy_rc_d': None, 'purge': False, 'state': 'latest', 'update_cache': None, 'upgrade': None}}, 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" install \'nagios-plugins-basic\'\' failed: E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/m/monitoring-plugins/monitoring-plugins-basic_2.2-6ubuntu1.2_amd64.deb Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::23), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::24), connection timed out\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n', 'rc': 100, 'stderr': 'E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/m/monitoring-plugins/monitoring-plugins-basic_2.2-6ubuntu1.2_amd64.deb Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::23), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::24), connection timed out\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n', 'stderr_lines': ['E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/m/monitoring-plugins/monitoring-plugins-basic_2.2-6ubuntu1.2_amd64.deb Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::23), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::24), connection timed out', 'E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?'], 'stdout': 'Reading package lists...\nBuilding dependency tree...\nReading state information...\nSuggested packages:\n icinga | icinga2\nThe following packages will be upgraded:\n monitoring-plugins-basic\n1 upgraded, 0 newly installed, 0 to remove and 72 not upgraded.\nNeed to get 272 kB of archives.\nAfter this operation, 0 B of additional disk space will be used.\nErr:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 monitoring-plugins-basic amd64 2.2-6ubuntu1.2\n Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::23), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::24), connection timed out\n', 'stdout_lines': ['Reading package lists...', 'Building dependency tree...', 'Reading state information...', 'Suggested packages:', ' icinga | icinga2', 'The following packages will be upgraded:', ' monitoring-plugins-basic', '1 upgraded, 0 newly installed, 0 to remove and 72 not upgraded.', 'Need to get 272 kB of archives.', 'After this operation, 0 B of additional disk space will be used.', 'Err:1 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 monitoring-plugins-basic amd64 2.2-6ubuntu1.2', ' Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::23), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::24), connection timed out']}} 2 jobs: ['6719536', '6719537'] suites intersection: ['clusters/fixed-3', 'conf', 'msgr-failures/few'] suites union: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'krbd/rbd/{bluestore-bitmap', 'ms_mode/crc$/{crc}', 'ms_mode/secure', 'msgr-failures/few', 'tasks/krbd_latest_osdmap_on_map}', 'tasks/rbd_workunit_suites_dbench}'] Failure: Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=kernel&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nightly_pre-single-major 1 jobs: ['6719458'] suites: ['clusters/separate-client', 'conf', 'filestore-xfs', 'kernels/pre-single-major', 'krbd/unmap/{ceph/ceph', 'tasks/unmap}'] Failure: {'smithi120.front.sepia.ceph.com': {'_ansible_no_log': False, 'cache_update_time': 1646376519, 'cache_updated': False, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': True, 'force_apt_get': False, 'install_recommends': None, 'name': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'libfcgi0ldbl', 'python-dev', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'only_upgrade': False, 'package': ['mpich', 'qemu-system-x86', 'lttng-tools', 'libtool-bin', 'docker.io', 'python3-nose', 'python3-venv', 'python3-virtualenv', 'python3-configobj', 'python3-gevent', 'python3-numpy', 'python3-matplotlib', 'python3-setuptools', 'libfcgi0ldbl', 'python-dev', 'libev-dev', 'perl', 'libwww-perl', 'lsb-release', 'build-essential', 'sysstat', 'gdb', 'libedit2', 'cryptsetup-bin', 'xfsprogs', 'gdisk', 'parted', 'libuuid1', 'libatomic-ops-dev', 'git-core', 'attr', 'dbench', 'bonnie++', 'valgrind', 'ant', 'libtool', 'automake', 'gettext', 'uuid-dev', 'libacl1-dev', 'bc', 'xfsdump', 'xfslibs-dev', 'libattr1-dev', 'quota', 'libcap2-bin', 'libncurses5-dev', 'lvm2', 'vim', 'pdsh', 'blktrace', 'genisoimage', 'libjson-xs-perl', 'xml-twig-tools', 'default-jdk', 'junit4', 'tgt', 'open-iscsi', 'cifs-utils', 'ipcalc', 'nfs-common', 'nfs-kernel-server', 'software-properties-common', 'libgnutls30'], 'policy_rc_d': None, 'purge': False, 'state': 'present', 'update_cache': None, 'upgrade': None}}, 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" --force-yes install \'qemu-system-x86\' \'git-core\'\' failed: W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librados2_15.2.14-0ubuntu0.20.04.2_amd64.deb Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::24), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::23), connection timed out\nE: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librbd1_15.2.14-0ubuntu0.20.04.2_amd64.deb Unable to connect to archive.ubuntu.com:http:\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n', 'rc': 100, 'stderr': 'W: --force-yes is deprecated, use one of the options starting with --allow instead.\nE: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librados2_15.2.14-0ubuntu0.20.04.2_amd64.deb Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::24), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::23), connection timed out\nE: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librbd1_15.2.14-0ubuntu0.20.04.2_amd64.deb Unable to connect to archive.ubuntu.com:http:\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n', 'stderr_lines': ['W: --force-yes is deprecated, use one of the options starting with --allow instead.', 'E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librados2_15.2.14-0ubuntu0.20.04.2_amd64.deb Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::24), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::23), connection timed out', 'E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/librbd1_15.2.14-0ubuntu0.20.04.2_amd64.deb Unable to connect to archive.ubuntu.com:http:', 'E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?'], 'stdout': 'Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.25.1-1ubuntu3.2).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-data\n qemu-system-gui qemu-utils\nSuggested packages:\n samba vde2 debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\nThe following packages will be upgraded:\n qemu-system-data qemu-system-gui\n2 upgraded, 6 newly installed, 0 to remove and 66 not upgraded.\nNeed to get 14.3 MB of archives.\nAfter this operation, 62.7 MB of additional disk space will be used.\nErr:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.14-0ubuntu0.20.04.2\n Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::24), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::23), connection timed out\nErr:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.14-0ubuntu0.20.04.2\n Unable to connect to archive.ubuntu.com:http:\nIgn:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.21\nIgn:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.21\nIgn:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-data all 1:4.2-3ubuntu6.21\nIgn:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-gui amd64 1:4.2-3ubuntu6.21\nIgn:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.21\nIgn:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.21\nIgn:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-data all 1:4.2-3ubuntu6.21\nGet:3 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.21 [54.8 kB]\nGet:4 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.21 [1056 kB]\nGet:6 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-gui amd64 1:4.2-3ubuntu6.21 [40.7 kB]\nGet:7 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.21 [6726 kB]\nGet:8 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.21 [976 kB]\nGet:5 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-data all 1:4.2-3ubuntu6.21 [563 kB]\nFetched 9416 kB in 31s (300 kB/s)\n', 'stdout_lines': ['Reading package lists...', 'Building dependency tree...', 'Reading state information...', 'git is already the newest version (1:2.25.1-1ubuntu3.2).', 'The following additional packages will be installed:', ' librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-data', ' qemu-system-gui qemu-utils', 'Suggested packages:', ' samba vde2 debootstrap', 'The following NEW packages will be installed:', ' librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86', ' qemu-utils', 'The following packages will be upgraded:', ' qemu-system-data qemu-system-gui', '2 upgraded, 6 newly installed, 0 to remove and 66 not upgraded.', 'Need to get 14.3 MB of archives.', 'After this operation, 62.7 MB of additional disk space will be used.', 'Err:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.14-0ubuntu0.20.04.2', ' Could not connect to archive.ubuntu.com:80 (91.189.88.142), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::24), connection timed out Could not connect to archive.ubuntu.com:80 (91.189.88.152), connection timed out Could not connect to archive.ubuntu.com:80 (2001:67c:1360:8001::23), connection timed out', 'Err:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.14-0ubuntu0.20.04.2', ' Unable to connect to archive.ubuntu.com:http:', 'Ign:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.21', 'Ign:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.21', 'Ign:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-data all 1:4.2-3ubuntu6.21', 'Ign:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-gui amd64 1:4.2-3ubuntu6.21', 'Ign:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.21', 'Ign:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.21', 'Ign:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-data all 1:4.2-3ubuntu6.21', 'Get:3 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.21 [54.8 kB]', 'Get:4 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-common amd64 1:4.2-3ubuntu6.21 [1056 kB]', 'Get:6 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-gui amd64 1:4.2-3ubuntu6.21 [40.7 kB]', 'Get:7 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-x86 amd64 1:4.2-3ubuntu6.21 [6726 kB]', 'Get:8 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.21 [976 kB]', 'Get:5 http://security.ubuntu.com/ubuntu focal-updates/main amd64 qemu-system-data all 1:4.2-3ubuntu6.21 [563 kB]', 'Fetched 9416 kB in 31s (300 kB/s)']}} 1 jobs: ['6719532'] suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/secure', 'msgr-failures/many', 'tasks/krbd_exclusive_option}'] Failure: Found coredumps on ubuntu@smithi162.front.sepia.ceph.com 1 jobs: ['6719514'] suites: ['clusters/fixed-3', 'conf', 'install/ceph', 'krbd/rbd-nomount/{bluestore-bitmap', 'ms_mode/crc$/{crc-rxbounce}', 'msgr-failures/many', 'tasks/rbd_concurrent}']