Found 123 jobs Missing teuthology log /home/teuthworker/archive/teuthology-2023-07-08_14:23:11-upgrade:quincy-x-reef-distro-default-smithi/7329236/teuthology.log Found 5 distinct failure reasons Failure: {'smithi137.front.sepia.ceph.com': {'_ansible_no_log': False, 'cache_update_time': 1688966214, 'cache_updated': False, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'name': ['docker.io', 'python3-setuptools', 'python3-pip'], 'only_upgrade': False, 'package': ['docker.io', 'python3-setuptools', 'python3-pip'], 'policy_rc_d': None, 'purge': False, 'state': 'latest', 'update_cache': None, 'update_cache_retries': 5, 'update_cache_retry_max_delay': 12, 'upgrade': None}}, 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" install \'docker.io\' \'python3-setuptools\' \'python3-pip\'\' failed: E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/universe/d/docker.io/docker.io_20.10.21-0ubuntu1~20.04.2_amd64.deb Temporary failure resolving \'archive.ubuntu.com\'\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n', 'rc': 100, 'stderr': "E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/universe/d/docker.io/docker.io_20.10.21-0ubuntu1~20.04.2_amd64.deb Temporary failure resolving 'archive.ubuntu.com'\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n", 'stderr_lines': ["E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/universe/d/docker.io/docker.io_20.10.21-0ubuntu1~20.04.2_amd64.deb Temporary failure resolving 'archive.ubuntu.com'", 'E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?'], 'stdout': "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n python-pip-whl python3-pkg-resources\nSuggested packages:\n aufs-tools btrfs-progs cgroupfs-mount | cgroup-lite debootstrap docker-doc\n rinse zfs-fuse | zfsutils python-setuptools-doc\nThe following packages will be upgraded:\n docker.io python-pip-whl python3-pip python3-pkg-resources\n python3-setuptools\n5 upgraded, 0 newly installed, 0 to remove and 210 not upgraded.\nNeed to get 32.8 MB of archives.\nAfter this operation, 23.1 MB disk space will be freed.\nIgn:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1\nIgn:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1\nIgn:3 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.21-0ubuntu1~20.04.2\nErr:3 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.21-0ubuntu1~20.04.2\n Temporary failure resolving 'archive.ubuntu.com'\nIgn:4 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9\nIgn:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9\nIgn:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1\nIgn:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1\nIgn:4 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9\nIgn:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9\nGet:1 http://security.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1 [330 kB]\nGet:2 http://security.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1 [130 kB]\nGet:4 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9 [231 kB]\nGet:5 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9 [1805 kB]\nFetched 2495 kB in 1min 25s (29.5 kB/s)\n", 'stdout_lines': ['Reading package lists...', 'Building dependency tree...', 'Reading state information...', 'The following additional packages will be installed:', ' python-pip-whl python3-pkg-resources', 'Suggested packages:', ' aufs-tools btrfs-progs cgroupfs-mount | cgroup-lite debootstrap docker-doc', ' rinse zfs-fuse | zfsutils python-setuptools-doc', 'The following packages will be upgraded:', ' docker.io python-pip-whl python3-pip python3-pkg-resources', ' python3-setuptools', '5 upgraded, 0 newly installed, 0 to remove and 210 not upgraded.', 'Need to get 32.8 MB of archives.', 'After this operation, 23.1 MB disk space will be freed.', 'Ign:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1', 'Ign:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1', 'Ign:3 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.21-0ubuntu1~20.04.2', 'Err:3 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.21-0ubuntu1~20.04.2', " Temporary failure resolving 'archive.ubuntu.com'", 'Ign:4 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9', 'Ign:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9', 'Ign:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1', 'Ign:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1', 'Ign:4 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9', 'Ign:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9', 'Get:1 http://security.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1 [330 kB]', 'Get:2 http://security.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1 [130 kB]', 'Get:4 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9 [231 kB]', 'Get:5 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9 [1805 kB]', 'Fetched 2495 kB in 1min 25s (29.5 kB/s)']}, 'smithi145.front.sepia.ceph.com': {'_ansible_no_log': False, 'cache_update_time': 1688966214, 'cache_updated': False, 'changed': False, 'invocation': {'module_args': {'allow_unauthenticated': False, 'autoclean': False, 'autoremove': False, 'cache_valid_time': 0, 'deb': None, 'default_release': None, 'dpkg_options': 'force-confdef,force-confold', 'force': False, 'force_apt_get': False, 'install_recommends': None, 'name': ['docker.io', 'python3-setuptools', 'python3-pip'], 'only_upgrade': False, 'package': ['docker.io', 'python3-setuptools', 'python3-pip'], 'policy_rc_d': None, 'purge': False, 'state': 'latest', 'update_cache': None, 'update_cache_retries': 5, 'update_cache_retry_max_delay': 12, 'upgrade': None}}, 'msg': '\'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" install \'docker.io\' \'python3-setuptools\' \'python3-pip\'\' failed: E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/universe/d/docker.io/docker.io_20.10.21-0ubuntu1~20.04.2_amd64.deb Temporary failure resolving \'archive.ubuntu.com\'\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n', 'rc': 100, 'stderr': "E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/universe/d/docker.io/docker.io_20.10.21-0ubuntu1~20.04.2_amd64.deb Temporary failure resolving 'archive.ubuntu.com'\nE: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n", 'stderr_lines': ["E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/universe/d/docker.io/docker.io_20.10.21-0ubuntu1~20.04.2_amd64.deb Temporary failure resolving 'archive.ubuntu.com'", 'E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?'], 'stdout': "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n python-pip-whl python3-pkg-resources\nSuggested packages:\n aufs-tools btrfs-progs cgroupfs-mount | cgroup-lite debootstrap docker-doc\n rinse zfs-fuse | zfsutils python-setuptools-doc\nThe following packages will be upgraded:\n docker.io python-pip-whl python3-pip python3-pkg-resources\n python3-setuptools\n5 upgraded, 0 newly installed, 0 to remove and 210 not upgraded.\nNeed to get 32.8 MB of archives.\nAfter this operation, 23.1 MB disk space will be freed.\nIgn:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1\nIgn:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1\nIgn:3 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.21-0ubuntu1~20.04.2\nErr:3 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.21-0ubuntu1~20.04.2\n Temporary failure resolving 'archive.ubuntu.com'\nIgn:4 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9\nIgn:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9\nIgn:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1\nIgn:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1\nIgn:4 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9\nIgn:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9\nGet:1 http://security.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1 [330 kB]\nGet:2 http://security.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1 [130 kB]\nGet:4 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9 [231 kB]\nGet:5 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9 [1805 kB]\nFetched 2495 kB in 1min 25s (29.5 kB/s)\n", 'stdout_lines': ['Reading package lists...', 'Building dependency tree...', 'Reading state information...', 'The following additional packages will be installed:', ' python-pip-whl python3-pkg-resources', 'Suggested packages:', ' aufs-tools btrfs-progs cgroupfs-mount | cgroup-lite debootstrap docker-doc', ' rinse zfs-fuse | zfsutils python-setuptools-doc', 'The following packages will be upgraded:', ' docker.io python-pip-whl python3-pip python3-pkg-resources', ' python3-setuptools', '5 upgraded, 0 newly installed, 0 to remove and 210 not upgraded.', 'Need to get 32.8 MB of archives.', 'After this operation, 23.1 MB disk space will be freed.', 'Ign:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1', 'Ign:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1', 'Ign:3 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.21-0ubuntu1~20.04.2', 'Err:3 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 docker.io amd64 20.10.21-0ubuntu1~20.04.2', " Temporary failure resolving 'archive.ubuntu.com'", 'Ign:4 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9', 'Ign:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9', 'Ign:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1', 'Ign:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1', 'Ign:4 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9', 'Ign:5 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9', 'Get:1 http://security.ubuntu.com/ubuntu focal-updates/main amd64 python3-setuptools all 45.2.0-1ubuntu0.1 [330 kB]', 'Get:2 http://security.ubuntu.com/ubuntu focal-updates/main amd64 python3-pkg-resources all 45.2.0-1ubuntu0.1 [130 kB]', 'Get:4 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 python3-pip all 20.0.2-5ubuntu1.9 [231 kB]', 'Get:5 http://security.ubuntu.com/ubuntu focal-updates/universe amd64 python-pip-whl all 20.0.2-5ubuntu1.9 [1805 kB]', 'Fetched 2495 kB in 1min 25s (29.5 kB/s)']}} 1 jobs: ['7329260'] suites: ['0-roles', '1-start', '2-first-half-tasks/readwrite', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:quincy-x/stress-split/{0-distro/ubuntu_20.04'] Failure: Command failed on smithi131 with status 127: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 79432922-1d9f-11ee-9b31-001a4aab830c -- ceph mon dump -f json' 3 jobs: ['7329164', '7329305', '7329303'] suites intersection: ['0-roles', '1-start', '3-stress-tasks/{radosbench', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}'] suites union: ['0-roles', '1-start', '2-first-half-tasks/rbd-cls', '2-first-half-tasks/rbd-import-export', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'mon_election/connectivity}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools', 'upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun', 'upgrade:quincy-x/stress-split/{0-distro/rhel_8.6_container_tools_3.0'] Failure: Found coredumps on ubuntu@smithi115.front.sepia.ceph.com 1 jobs: ['7329212'] suites: ['0-roles', '1-start', '2-first-half-tasks/snaps-few-objects', '3-stress-tasks/{radosbench', '4-second-half-tasks/rbd-import-export', 'mon_election/classic}', 'rbd-cls', 'rbd-import-export', 'rbd_api', 'readwrite', 'snaps-few-objects}', 'upgrade:quincy-x/stress-split/{0-distro/centos_8.stream_container_tools_crun'] Failure: HTTPSConnectionPool(host='shaman.ceph.com', port=443): Max retries exceeded with url: /api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=reef (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out')) 1 jobs: ['7329276'] Dead: None 3 jobs: ['7329236', '7329358', '7329172']