2024-04-10T15:55:38.588 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-10T15:55:38.588 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2024-04-10_14:18:30-upgrade-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-04-10_14:18:30-upgrade-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650716 --owner scheduled_yuriw@teuthology --description upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} --verbose -- /home/teuthworker/archive/yuriw-2024-04-10_14:18:30-upgrade-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650716/orig.config.yaml 2024-04-10T15:55:38.631 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T15:55:38.712 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-04-10_14:18:30-upgrade-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650716 branch: wip-yuri6-testing-2024-04-02-1310 description: upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} email: yweinste@redhat.com first_in_suite: false job_id: '7650716' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: 'install ceph/pacific latest ' name: yuriw-2024-04-10_14:18:30-upgrade-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri6-testing-2024-04-02-1310 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: bluestore warn on legacy statfs: false bluestore warn on no per pool omap: false mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon warn on osd down out interval zero: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - scrub mismatch - ScrubResult - wrongly marked - \(POOL_APP_NOT_ENABLED\) - \(SLOW_OPS\) - overall HEALTH_ - \(MON_MSGR2_NOT_ENABLED\) - slow request sha1: a5074d4516d566e9d8b6aec912f26afd099de101 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a5074d4516d566e9d8b6aec912f26afd099de101 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2024-04-02-1310 sha1: a5074d4516d566e9d8b6aec912f26afd099de101 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 seed: 7488 sha1: a5074d4516d566e9d8b6aec912f26afd099de101 sleep_before_teardown: 0 subset: 111/120000 suite: upgrade suite_branch: wip-yuri6-testing-2024-04-02-1310 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: a5074d4516d566e9d8b6aec912f26afd099de101 targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPoJye81yeSYgMy1VlVY1RK88dEAtH2lVdznujQAP3KA7z5oJtmvj+tRRcjNNQn0KqjoLVm9K069+sENHSly9jA= tasks: - install: branch: pacific exclude_packages: - librados3 - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - cephadm - ceph-volume extra_packages: - librados2 - print: '**** done installing pacific' - ceph: conf: global: mon warn on pool no app: false ms bind msgr2: false log-ignorelist: - overall HEALTH_ - \(FS_ - \(MDS_ - \(OSD_ - \(MON_DOWN\) - \(CACHE_POOL_ - \(POOL_ - \(MGR_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - Monitor daemon marked osd - Behind on trimming - Manager daemon - exec: osd.0: - ceph osd set-require-min-compat-client pacific - print: '**** done ceph' - print: '*** upgrading, no cephfs present' - exec: mon.a: - ceph fs dump - install.upgrade: mon.a: branch: reef - print: '**** done install.upgrade' - ceph.restart: daemons: - mon.* - mgr.* mon-health-to-clog: false wait-for-healthy: false - ceph.healthy: null - ceph.restart: daemons: - osd.* wait-for-healthy: false wait-for-osds-up: true - exec: mon.a: - ceph versions - ceph osd dump -f json-pretty - ceph fs dump - ceph osd require-osd-release quincy - for f in `ceph osd pool ls` ; do ceph osd pool set $f pg_autoscale_mode off ; done - ceph.healthy: null - print: '**** done ceph.restart' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-10_14:18:30 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-10T15:55:38.713 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa; will attempt to use it 2024-04-10T15:55:38.713 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_a5074d4516d566e9d8b6aec912f26afd099de101/qa/tasks 2024-04-10T15:55:38.713 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-10T15:55:38.715 INFO:teuthology.task.internal:Checking packages... 2024-04-10T15:55:38.738 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a5074d4516d566e9d8b6aec912f26afd099de101' 2024-04-10T15:55:38.738 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-10T15:55:38.738 INFO:teuthology.packaging:ref: None 2024-04-10T15:55:38.738 INFO:teuthology.packaging:tag: None 2024-04-10T15:55:38.738 INFO:teuthology.packaging:branch: wip-yuri6-testing-2024-04-02-1310 2024-04-10T15:55:38.738 INFO:teuthology.packaging:sha1: a5074d4516d566e9d8b6aec912f26afd099de101 2024-04-10T15:55:38.738 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri6-testing-2024-04-02-1310 2024-04-10T15:55:38.914 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2838.ga5074d45 2024-04-10T15:55:38.916 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-10T15:55:38.922 INFO:teuthology.task.internal:no buildpackages task found 2024-04-10T15:55:38.922 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-10T15:55:38.932 INFO:teuthology.task.internal:Saving configuration 2024-04-10T15:55:38.950 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-10T15:55:38.956 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T15:55:38.977 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:18:30-upgrade-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650716', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 15:50:07.787638', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPoJye81yeSYgMy1VlVY1RK88dEAtH2lVdznujQAP3KA7z5oJtmvj+tRRcjNNQn0KqjoLVm9K069+sENHSly9jA='} 2024-04-10T15:55:38.977 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-10T15:55:38.983 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-04-10T15:55:38.983 INFO:teuthology.run_tasks:Running task console_log... 2024-04-10T15:55:39.019 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2295a1f160>, signals=[15]) 2024-04-10T15:55:39.020 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-10T15:55:39.091 INFO:teuthology.task.internal:Opening connections... 2024-04-10T15:55:39.092 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-04-10T15:55:39.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T15:55:39.168 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-10T15:55:39.174 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-04-10T15:55:39.239 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-04-10T15:55:39.240 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-04-10T15:55:39.300 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2024-04-10T15:55:39.300 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8" 2024-04-10T15:55:39.301 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2024-04-10T15:55:39.301 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2024-04-10T15:55:39.301 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8" 2024-04-10T15:55:39.301 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8" 2024-04-10T15:55:39.301 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 8" 2024-04-10T15:55:39.301 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2024-04-10T15:55:39.301 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-04-10T15:55:39.301 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2024-04-10T15:55:39.301 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-10T15:55:39.301 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-04-10T15:55:39.301 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-10T15:55:39.302 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-04-10T15:55:39.326 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-10T15:55:39.334 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-10T15:55:39.339 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-10T15:55:39.339 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-04-10T15:55:39.359 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-10T15:55:39.364 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-10T15:55:39.364 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-04-10T15:55:39.431 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-10T15:55:39.462 INFO:teuthology.run_tasks:Running task kernel... 2024-04-10T15:55:39.476 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-10T15:55:39.476 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-10T15:55:39.477 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-10T15:55:39.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T15:55:39.496 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-04-10T15:55:39.555 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-489.el8.x86_64 2024-04-10T15:55:39.555 INFO:teuthology.task.kernel:Running kernel on smithi088: 4.18.0-489.el8.x86_64 2024-04-10T15:55:39.555 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2024-04-10T15:55:43.421 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - AppStream 21 MB/s | 28 MB 00:01 2024-04-10T15:55:49.796 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - BaseOS 7.7 MB/s | 10 MB 00:01 2024-04-10T15:55:51.788 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras 46 kB/s | 18 kB 00:00 2024-04-10T15:55:52.077 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 8 - Extras common packages 42 kB/s | 7.7 kB 00:00 2024-04-10T15:55:52.365 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2024-04-10T15:55:53.307 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2024-04-10T15:55:57.157 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 398 kB/s | 24 kB 00:00 2024-04-10T15:56:00.204 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-04-10T15:56:00.204 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-04-10T15:56:00.321 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-04-10T15:56:00.322 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-04-10T15:56:00.322 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-04-10T15:56:00.322 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-04-10T15:56:00.322 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-04-10T15:56:00.322 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 4.18.0-552.el8 baseos 10 M 2024-04-10T15:56:00.323 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-04-10T15:56:00.323 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core x86_64 4.18.0-552.el8 baseos 44 M 2024-04-10T15:56:00.323 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules x86_64 4.18.0-552.el8 baseos 36 M 2024-04-10T15:56:00.323 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T15:56:00.323 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-04-10T15:56:00.323 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-04-10T15:56:00.323 INFO:teuthology.orchestra.run.smithi088.stdout:Install 3 Packages 2024-04-10T15:56:00.323 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T15:56:00.323 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 90 M 2024-04-10T15:56:00.324 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 96 M 2024-04-10T15:56:00.324 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-04-10T15:56:01.617 INFO:teuthology.orchestra.run.smithi088.stdout:(1/3): kernel-4.18.0-552.el8.x86_64.rpm 9.4 MB/s | 10 MB 00:01 2024-04-10T15:56:02.708 INFO:teuthology.orchestra.run.smithi088.stdout:(2/3): kernel-modules-4.18.0-552.el8.x86_64.rpm 17 MB/s | 36 MB 00:02 2024-04-10T15:56:03.275 INFO:teuthology.orchestra.run.smithi088.stdout:(3/3): kernel-core-4.18.0-552.el8.x86_64.rpm 16 MB/s | 44 MB 00:02 2024-04-10T15:56:03.276 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-04-10T15:56:03.276 INFO:teuthology.orchestra.run.smithi088.stdout:Total 31 MB/s | 90 MB 00:02 2024-04-10T15:56:03.959 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-04-10T15:56:04.393 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-04-10T15:56:04.394 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-04-10T15:56:05.654 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-04-10T15:56:05.664 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-04-10T15:56:08.461 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-04-10T15:56:08.955 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T15:56:12.463 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 1/3 2024-04-10T15:56:12.865 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T15:56:16.299 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 2/3 2024-04-10T15:56:16.662 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T15:56:38.642 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-4.18.0-552.el8.x86_64 3/3 2024-04-10T15:56:38.666 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T15:56:39.023 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-4.18.0-552.el8.x86_64 3/3 2024-04-10T15:56:39.023 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-4.18.0-552.el8.x86_64 1/3 2024-04-10T15:56:39.024 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-core-4.18.0-552.el8.x86_64 2/3 2024-04-10T15:56:39.504 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-4.18.0-552.el8.x86_64 3/3 2024-04-10T15:56:39.504 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T15:56:39.505 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-04-10T15:56:39.505 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-4.18.0-552.el8.x86_64 kernel-core-4.18.0-552.el8.x86_64 2024-04-10T15:56:39.505 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-4.18.0-552.el8.x86_64 2024-04-10T15:56:39.505 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T15:56:39.505 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-04-10T15:56:39.643 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T15:56:39.863 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T15:56:39.863 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T15:56:39.863 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-552.el8.x86_64 2024-04-10T15:56:39.863 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-10T15:56:39.864 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-10T15:56:39.864 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-10T15:56:39.864 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-04-10T15:56:39.864 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-04-10T15:56:40.074 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-552.el8.x86_64 2024-04-10T15:56:40.074 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-552.el8.x86_64 2024-04-10T15:56:40.078 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2024-04-10T15:56:40.111 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.02 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 129.el8 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Group : System Environment/Base 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 9265586 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : kbuilder.bsys.centos.org 2024-04-10T15:56:40.112 INFO:teuthology.orchestra.run.smithi088.stdout:Relocations : (not relocatable) 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : CentOS BuildSystem 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-04-10T15:56:40.113 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2024-04-10T15:56:40.114 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-10T15:56:40.114 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 4.18.0-552.el8.x86_64 2024-04-10T15:56:40.114 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-10T15:56:41.568 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-04-10T15:56:44.027 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-04-10T15:56:44.029 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-04-10T15:56:44.070 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.aain34giZk 2024-04-10T15:56:44.071 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aain34giZk 2024-04-10T15:56:44.151 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.aain34giZk 2024-04-10T15:56:44.295 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.aain34giZk is 6KB 2024-04-10T15:56:44.345 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.aain34giZk 2024-04-10T15:56:44.362 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2024-04-10T15:56:44.442 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-04-10T15:56:44.442 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-04-10T15:56:44.442 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-04-10T15:56:44.442 INFO:teuthology.orchestra.run.smithi088.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64.conf 2024-04-10T15:56:44.443 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-552.el8.x86_64 2024-04-10T15:56:44.632 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-04-10T15:57:14.662 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-10T15:57:14.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T15:57:14.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T15:57:33.087 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-04-10T15:57:42.096 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T15:57:42.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T15:57:45.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-04-10T15:57:57.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T15:57:57.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T15:58:00.226 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-04-10T15:58:15.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T15:58:15.239 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T15:58:33.763 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-04-10T15:58:51.778 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T15:58:51.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-10T15:58:52.091 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-04-10T15:58:52.388 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-04-10T15:58:52.389 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-552.el8.x86_64"... 2024-04-10T15:58:52.389 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-04-10T15:58:52.447 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-552.el8.x86_64 2024-04-10T15:58:52.448 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-552.el8.x86_64 vs 4.18.0-552.el8.x86_64 2024-04-10T15:58:52.448 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-10T15:58:52.448 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-10T15:58:53.449 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-10T15:58:53.449 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-10T15:58:53.524 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-04-10T15:58:53.537 DEBUG:teuthology.parallel:result is None 2024-04-10T15:58:53.538 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-10T15:58:53.545 INFO:teuthology.task.internal:Creating test directory... 2024-04-10T15:58:53.545 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-10T15:58:53.565 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-10T15:58:53.573 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-10T15:58:53.581 INFO:teuthology.task.internal:Creating archive directory... 2024-04-10T15:58:53.581 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-10T15:58:53.640 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-10T15:58:53.648 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-10T15:58:53.649 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-10T15:58:53.711 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T15:58:53.721 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-10T15:58:53.722 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-10T15:58:53.730 INFO:teuthology.task.internal:Configuring sudo... 2024-04-10T15:58:53.730 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-10T15:58:53.791 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-10T15:58:53.801 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-10T15:58:53.801 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-10T15:58:53.850 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T15:58:53.935 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-10T15:58:53.982 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T15:58:54.065 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-10T15:58:54.109 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-04-10T15:58:54.109 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-10T15:58:54.175 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-04-10T15:58:54.281 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T15:58:54.617 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-10T15:58:54.625 INFO:teuthology.task.internal:Starting timer... 2024-04-10T15:58:54.625 INFO:teuthology.run_tasks:Running task pcp... 2024-04-10T15:58:54.668 INFO:teuthology.run_tasks:Running task selinux... 2024-04-10T15:58:54.677 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-04-10T15:58:54.678 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2024-04-10T15:58:54.775 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2024-04-10T15:58:54.777 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-10T15:58:54.777 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2024-04-10T15:58:54.808 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2024-04-10T15:58:54.808 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi088.front.sepia.ceph.com': 'permissive'} 2024-04-10T15:58:54.808 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T15:58:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T15:58:54.876 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-10T15:58:54.876 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2024-04-10T15:58:54.943 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-10T15:58:54.953 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-10T15:58:54.959 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-10T15:58:55.093 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-10T15:58:55.112 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-10T15:58:55.114 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-10T16:10:24.089 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-04-10_14:18:30-upgrade-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650716/ansible_failures.yaml 2024-04-10T16:10:24.091 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T16:10:24.321 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=51332c8be35d4accba6cad9dd0250f71 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 508, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 342, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 374, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/task/ansible.py", line 402, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2024-04-10T16:10:24.324 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-10T16:10:24.333 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-10T16:10:24.333 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-10T16:10:24.340 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-10T16:10:24.408 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-04-10T16:10:24.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:10:24.439 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-10T16:10:24.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-10T16:10:24.455 INFO:teuthology.task.internal:Duration was 689.829715 seconds 2024-04-10T16:10:24.455 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-10T16:10:24.463 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-10T16:10:24.463 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-10T16:10:24.526 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-10T16:10:24.770 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-10T16:10:24.771 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-04-10T16:10:24.771 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-10T16:10:24.793 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-10T16:10:24.794 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-10T16:10:24.873 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-10T16:10:24.874 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-10T16:10:24.975 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-10T16:10:24.984 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-10T16:10:24.984 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-10T16:10:25.010 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-10T16:10:25.022 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-10T16:10:25.076 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-04-10T16:10:25.093 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-10T16:10:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-10T16:10:25.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-10T16:10:25.159 INFO:teuthology.task.internal:Transferring archived files... 2024-04-10T16:10:25.160 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-04-10_14:18:30-upgrade-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650716/remote/smithi088 2024-04-10T16:10:25.161 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-10T16:10:25.258 INFO:teuthology.task.internal:Removing archive directory... 2024-04-10T16:10:25.259 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-10T16:10:25.276 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-10T16:10:25.286 INFO:teuthology.task.internal:Not uploading archives. 2024-04-10T16:10:25.286 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-10T16:10:25.345 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-10T16:10:25.345 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-10T16:10:25.362 INFO:teuthology.orchestra.run.smithi088.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 10 16:10 /home/ubuntu/cephtest 2024-04-10T16:10:25.364 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-10T16:10:25.404 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-10T16:10:25.421 INFO:teuthology.nuke:Checking targets against current locks 2024-04-10T16:10:25.448 DEBUG:teuthology.nuke:shortname: smithi088 2024-04-10T16:10:25.449 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-10T16:10:25.470 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-04-10_14:18:30-upgrade-wip-yuri6-testing-2024-04-02-1310-distro-default-smithi/7650716', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-10 15:50:07.787638', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPoJye81yeSYgMy1VlVY1RK88dEAtH2lVdznujQAP3KA7z5oJtmvj+tRRcjNNQn0KqjoLVm9K069+sENHSly9jA='} 2024-04-10T16:10:25.515 INFO:teuthology.orchestra.console.smithi088:Power off 2024-04-10T16:10:25.515 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-10T16:10:25.541 DEBUG:teuthology.orchestra.console.smithi088:power off output: Chassis Power Control: Down/Off 2024-04-10T16:10:25.541 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:10:25.564 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-04-10T16:10:29.566 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:10:29.676 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-04-10T16:10:33.680 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:10:33.790 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is on 2024-04-10T16:10:37.795 DEBUG:teuthology.orchestra.console.smithi088:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-10T16:10:37.905 DEBUG:teuthology.orchestra.console.smithi088:check power output: Chassis Power is off 2024-04-10T16:10:38.006 INFO:teuthology.orchestra.console.smithi088:Power off completed 2024-04-10T16:10:38.112 INFO:teuthology.run:Summary data: description: upgrade/cephfs/nofs/{bluestore-bitmap centos_8.stream conf/{client mds mgr mon osd} no-mds-cluster overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn pg_health} tasks/{0-pacific 1-upgrade}} duration: 689.8297147750854 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=51332c8be35d4accba6cad9dd0250f71 status: dead success: false 2024-04-10T16:10:38.112 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-10T16:10:38.195 INFO:teuthology.run:DEAD