Found 245 jobs Missing teuthology log /home/teuthworker/archive/dis-2023-09-02_07:44:41-rbd-wip-dis-testing-distro-default-smithi/7386812/teuthology.log Found 14 distinct failure reasons Crash: "2023-09-02T08:37:12.537193+0000 mon.a (mon.0) 1550 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log ceph version 18.0.0-5777-g846f5fdc (846f5fdc524dcf6398cd9846bf4d90915c248ebd) reef (dev) 1: /lib64/libc.so.6(+0x54db0) [0x7f926f054db0] 2: /lib64/libc.so.6(+0xa154c) [0x7f926f0a154c] 3: raise() 4: abort() 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x187) [0x556f4f6052de] 6: (KernelDevice::_aio_thread()+0x10f6) [0x556f4ff3af76] 7: ceph-osd(+0xd970e1) [0x556f4ff3b0e1] 8: /lib64/libc.so.6(+0x9f802) [0x7f926f09f802] 9: /lib64/libc.so.6(+0x3f450) [0x7f926f03f450] 1 jobs: ['7386851'] suites: ['clients/mirror', 'cluster/{2-node', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'policy/simple', 'rbd-mirror/four-per-cluster', 'rbd/mirror-thrash/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}'] Dead: None 7 jobs: ['7386812', '7386913', '7387030', '7386976', '7386839', '7386889', '7386866'] Failure: "2023-09-02T08:20:16.497429+0000 mon.a (mon.0) 1041 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY)" in cluster log 1 jobs: ['7386819'] suites: ['clusters/{fixed-3', 'config/copy-on-read', 'min-compat-client/default', 'msgr-failures/few', 'objectstore/bluestore-comp-snappy', 'openstack}', 'pool/small-cache-pool', 'rbd/librbd/{cache/writearound', 'supported-random-distro$/{centos_latest}', 'workloads/c_api_tests_with_defaults}'] Failure: Host key for server 'smithi183.front.sepia.ceph.com' does not match: got 'AAAAB3NzaC1yc2EAAAADAQABAAABgQCoF6FTDslzGw43Je/RXgVV9hhEs/TjglRkGYcDX7ozaMWdv075xqNWn0Cbt5QqvRJpXjd2dUzD8Dgf0/FD38UPB6YSuvpMkXpmD749ZcKraAPi10duE0uAFpKpaPmSYnLzlOCrt4PkIlEunwBh+NUp/27u0Sg8+B+ZQ04O1NypW1gYleJysarsvlQ4fE+7P8/Usw2BcpXyege+0KYoSNEC6PYStUQzO4eG9oVKoJWA4Cttx31uLHPurecGbmwbvk/dWH6vayGQ7K7Jo74iHh5MfDL21NFvAoy5Mke/j+JEqTZypBI6rgNqUx6wEJKDOP/IGNygQoia0ZGKWN5Z/TeXAkrtV1106D+jcQU8l5Tx2OGly6rWol7Hq6ocMaXTlTopeTsiPp6mpEahjqyC2YwL9Oo9FmmNIduSDWdWsEDxNGAWnFeTeoCXB85btgk0sH1YiKC0gOx7MSKD0FA/Zshdj2GouiyhyqRA0y/Pho8H9YEaxTYDWCaHYQEbS6ryKg8=', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDERYkXm/4GuQNylHwlMJUEX5BLeF2TvVKwiWkrnovRsJw1gMTqpUvtMRrgORzHmsOMJ3sSmP58+zzxLIySthSrnxv5K62JMd+yIwVKT0qzzdWF5+c+VS2/FIXqa+ysTKD9AJS4GpgOkCSOGcI/dajxpFFZ3QFVFFhzw6RLETlWDHtWZP7aJCDL/hjsjc5vhSy54VSH0qTUOdZZEEInvYcqc80aWcl8B1OpYowmwP55UjMEJ+OCHQXxm4OEJRanodpXBtXd4jMt25FkDFjputTNpObijX7ofOgopQaRk1n2RA7mV0VzXdq+HbmojL3jzAFMSehBYJlJlXYcOXCVkIZxUMCU8KyJBCflT6FvHOaVdhq+xmXRxpccWbgPVSJ4xYz6TFKgRsvzc2bgxCg/OJGMyW6Fn/MrS51SSs3hJaNdH8JM1TWfOP0YIFZh6uDSVKt2LV/WyucMbdg+bbWo67WMHmdNTv5NxiwxrVV8pKoSzjb1jOUvSL1dj5VwvKq4Rbc=' 1 jobs: ['7386893'] suites: ['2-clusters/{fixed-3', '3-objectstore/bluestore-stupid', '4-supported-random-distro$/{centos_latest}', '5-pool/none', '6-prepare/qcow2-http', '7-io-workloads/qemu_xfstests', '8-migrate-workloads/execute', '9-cleanup/cleanup}', 'openstack}', 'rbd/migration/{1-base/install'] Dead: b'2023-09-02T22:48:49.281 DEBUG:teuthology.exit:Finished running handlers' 1 jobs: ['7386990'] suites: ['openstack}', 'pool/ceph_and_immutable_object_cache', 'rbd/immutable-object-cache/{clusters/{fix-2', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_on_immutable_object_cache_and_thrash}'] Failure: smithi141.front.sepia.ceph.com: _ansible_no_log: false changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - krb5-workstation releasever: null security: false skip_broken: false state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''epel'': Yum repo downloading error: Downloading error(s): repodata/c0b8484273cba936c1dacae7746e9afa56070a666f7b3a5f307e965704e00262-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/c1777098a6912dfab4142aad585aa9286c19caf7eb5a92bac389271050767178-filelists.xml.gz - Cannot download, all mirrors were already tried without success; repodata/8ae517934765ff2c3ddf5c51a62cdc8d3a0cb70a0461f379febc6c36011dd51b-prestodelta.xml.xz - Cannot download, all mirrors were already tried without success; repodata/60bec892c18a25cb1c1adca6a42070589402f2d0504389451970e26dbc5917bb-updateinfo.xml.bz2 - Cannot download, all mirrors were already tried without success' rc: 1 results: [] 1 jobs: ['7386909'] suites: ['openstack}', 'pool/ceph_and_immutable_object_cache', 'rbd/immutable-object-cache/{clusters/{fix-2', 'supported-random-distro$/{rhel_8}', 'workloads/fio_on_immutable_object_cache}'] Failure: 1 jobs: ['7387021'] suites: ['clusters/{fixed-3', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{rhel_8}', 'workloads/dynamic_features}'] Failure: ['ubuntu@smithi183.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).'] 1 jobs: ['7386884'] suites: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{ubuntu_20.04}', 'workloads/rbd-mirror-workunit-policy-simple}'] Failure: Command failed on smithi055 with status 1: 'sudo yum install -y kernel' 1 jobs: ['7386995'] suites: ['clusters/{fixed-3', 'config/permit-partial-discard', 'min-compat-client/default', 'msgr-failures/few', 'objectstore/bluestore-comp-lz4', 'openstack}', 'pool/replicated-data-pool', 'rbd/librbd/{cache/writearound', 'supported-random-distro$/{centos_8}', 'workloads/python_api_tests_with_defaults}'] Timeout 3h running clone.client.0/qa/workunits/rbd/test_librbd.sh 2 jobs: ['7386813', '7386946'] suites intersection: ['msgr-failures/few', 'openstack}'] suites union: ['clusters/{fixed-2', 'clusters/{fixed-3', 'config/permit-partial-discard', 'min-compat-client/default', 'msgr-failures/few', 'objectstore/bluestore-low-osd-mem-target', 'objectstore/bluestore-stupid', 'openstack}', 'pool/replicated-data-pool', 'rbd/librbd/{cache/writearound', 'rbd/thrash/{base/install', 'supported-random-distro$/{centos_8}', 'supported-random-distro$/{centos_latest}', 'thrashers/cache', 'thrashosds-health', 'workloads/c_api_tests_with_defaults}', 'workloads/rbd_api_tests_copy_on_read}'] Timeout 3h running clone.client.0/qa/workunits/rbd/qemu_dynamic_features.sh 1 jobs: ['7386840'] suites: ['clusters/{fixed-3', 'objectstore/bluestore-hybrid', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{centos_latest}', 'workloads/dynamic_features}'] Failure: Command failed on smithi183 with status 2: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.0' 1 jobs: ['7386897'] suites: ['clusters/{fixed-3', 'features/defaults', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'openstack}', 'pool/ec-data-pool', 'rbd/encryption/{cache/writeback', 'supported-random-distro$/{centos_latest}', 'workloads/qemu_xfstests_luks1}'] Failure: Stale jobs detected, aborting. 1 jobs: ['7386818'] suites: ['clusters/{fixed-3', 'objectstore/bluestore-comp-zstd', 'openstack}', 'qemu/xfstests', 'rbd/maintenance/{base/install', 'supported-random-distro$/{ubuntu_20.04}', 'workloads/rebuild_object_map}'] Failure: ['Failed to update apt cache: unknown reason'] 1 jobs: ['7386935'] suites: ['clients/{mirror-extra', 'cluster/{2-node', 'mirror}', 'msgr-failures/few', 'objectstore/bluestore-bitmap', 'openstack}', 'rbd/mirror/{base/install', 'supported-random-distro$/{ubuntu_20.04}', 'workloads/rbd-mirror-snapshot-bootstrap-workunit}']