2023-10-18T14:37:14.061 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-18T14:37:14.063 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} --archive /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910 --name pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi --owner scheduled_pdonnell@teuthology --verbose -- /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910/orig.config.yaml 2023-10-18T14:37:14.116 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:37:14.198 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910 branch: wip-batrick-testing-20231016.203825 description: fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} email: pdonnell@redhat.com first_in_suite: false job_id: '7430910' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: 'install ceph/octopus latest ' name: pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-batrick-testing-20231016.203825 ceph: cephfs: max_mds: 2 cluster-conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mon: mon op complaint time: 120 osd: osd op complaint time: 180 conf: global: bluestore warn on legacy statfs: false bluestore warn on no per pool omap: false mon pg warn min per osd: 0 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on osd down out interval zero: false osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - scrub mismatch - ScrubResult - wrongly marked - \(POOL_APP_NOT_ENABLED\) - \(SLOW_OPS\) - overall HEALTH_ - \(MON_MSGR2_NOT_ENABLED\) - slow request sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20231016.203825 sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - mgr.y - mds.a - mds.b - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - - client.0 - - client.1 seed: 7703 sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb sleep_before_teardown: 0 subset: 153/256 suite: fs suite_branch: wip-batrick-testing-20231016.203825 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_f7420d7f8e55ceea786dcbf536c338ce920c21eb/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK7Nx7z7bBDivepMKbAaOVYa5p8Sf8V5tBvDu0q/U1L27KGwOnmq4YEM5ACDjLuUNpGeGNpTqpNRlJOqJOBMCgU= smithi133.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZTfx0lPP2e1N15dq974CmzOEF3J5ouKLlVE/WIIiU6ZC0OfcqXT9Keq/nITGsirpSma2QGoVoBWaJcfyd9/Ag= smithi187.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8GY7oGtcmJGPyZNXquF//xl9vZRDCYrhk2eEwoCkIk/qyqZ5XqGxllDL8s+QAI6yUxEUfEx1HDkuhtQVLQZlI= tasks: - install: branch: octopus exclude_packages: - librados3 - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - cephadm - ceph-volume extra_packages: - librados2 - print: '**** done installing octopus' - ceph: conf: global: mon warn on pool no app: false ms bind msgr2: false log-ignorelist: - overall HEALTH_ - \(FS_ - \(MDS_ - \(OSD_ - \(MON_DOWN\) - \(CACHE_POOL_ - \(POOL_ - \(MGR_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - Monitor daemon marked osd - Behind on trimming - Manager daemon - exec: osd.0: - ceph osd set-require-min-compat-client octopus - print: '**** done ceph' - ceph-fuse: null - print: '**** done octopus client' - workunit: clients: all: - suites/fsstress.sh - print: '**** done fsstress' - mds_pre_upgrade: null - print: '**** done mds pre-upgrade sequence' - install.upgrade: mon.a: branch: quincy - print: '**** done install.upgrade the host' - ceph.restart: daemons: - mon.* - mgr.* mon-health-to-clog: false wait-for-healthy: false - ceph.healthy: null - ceph.restart: daemons: - osd.* wait-for-healthy: false wait-for-osds-up: true - ceph.stop: - mds.* - ceph.restart: daemons: - mds.* wait-for-healthy: false wait-for-osds-up: true - exec: mon.a: - ceph osd dump -f json-pretty - ceph versions - ceph osd require-osd-release quincy - for f in `ceph osd pool ls` ; do ceph osd pool set $f pg_autoscale_mode off ; done - ceph.healthy: null - print: '**** done ceph.restart' - workunit: clients: all: - suites/fsstress.sh - print: '**** done fsstress' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-16_23:59:58 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-18T14:37:14.199 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_f7420d7f8e55ceea786dcbf536c338ce920c21eb/qa; will attempt to use it 2023-10-18T14:37:14.200 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_f7420d7f8e55ceea786dcbf536c338ce920c21eb/qa/tasks 2023-10-18T14:37:14.200 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-18T14:37:14.202 INFO:teuthology.task.internal:Checking packages... 2023-10-18T14:37:14.226 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f7420d7f8e55ceea786dcbf536c338ce920c21eb' 2023-10-18T14:37:14.227 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-18T14:37:14.227 INFO:teuthology.packaging:ref: None 2023-10-18T14:37:14.227 INFO:teuthology.packaging:tag: None 2023-10-18T14:37:14.228 INFO:teuthology.packaging:branch: wip-batrick-testing-20231016.203825 2023-10-18T14:37:14.228 INFO:teuthology.packaging:sha1: f7420d7f8e55ceea786dcbf536c338ce920c21eb 2023-10-18T14:37:14.228 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-batrick-testing-20231016.203825 2023-10-18T14:37:14.460 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6786.gf7420d7f 2023-10-18T14:37:14.461 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-18T14:37:14.471 INFO:teuthology.task.internal:no buildpackages task found 2023-10-18T14:37:14.471 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-18T14:37:14.491 INFO:teuthology.task.internal:Saving configuration 2023-10-18T14:37:14.509 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-18T14:37:14.613 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:37:14.649 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:32:09.122889', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK7Nx7z7bBDivepMKbAaOVYa5p8Sf8V5tBvDu0q/U1L27KGwOnmq4YEM5ACDjLuUNpGeGNpTqpNRlJOqJOBMCgU='} 2023-10-18T14:37:14.681 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:32:09.120103', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZTfx0lPP2e1N15dq974CmzOEF3J5ouKLlVE/WIIiU6ZC0OfcqXT9Keq/nITGsirpSma2QGoVoBWaJcfyd9/Ag='} 2023-10-18T14:37:14.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:32:09.121609', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8GY7oGtcmJGPyZNXquF//xl9vZRDCYrhk2eEwoCkIk/qyqZ5XqGxllDL8s+QAI6yUxEUfEx1HDkuhtQVLQZlI='} 2023-10-18T14:37:14.707 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-18T14:37:14.721 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'mgr.y', 'mds.a', 'mds.b', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2023-10-18T14:37:14.721 INFO:teuthology.task.internal:roles: ubuntu@smithi133.front.sepia.ceph.com - ['client.0'] 2023-10-18T14:37:14.722 INFO:teuthology.task.internal:roles: ubuntu@smithi187.front.sepia.ceph.com - ['client.1'] 2023-10-18T14:37:14.722 INFO:teuthology.run_tasks:Running task console_log... 2023-10-18T14:37:14.855 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbe2ef13af0>, signals=[15]) 2023-10-18T14:37:14.856 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-18T14:37:14.865 INFO:teuthology.task.internal:Opening connections... 2023-10-18T14:37:14.865 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2023-10-18T14:37:14.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:37:14.942 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi133.front.sepia.ceph.com 2023-10-18T14:37:14.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:37:15.013 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi187.front.sepia.ceph.com 2023-10-18T14:37:15.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:37:15.082 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-18T14:37:15.093 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2023-10-18T14:37:15.116 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2023-10-18T14:37:15.117 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2023-10-18T14:37:15.178 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="CentOS Stream" 2023-10-18T14:37:15.178 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8" 2023-10-18T14:37:15.178 INFO:teuthology.orchestra.run.smithi078.stdout:ID="centos" 2023-10-18T14:37:15.179 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="rhel fedora" 2023-10-18T14:37:15.179 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8" 2023-10-18T14:37:15.179 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:37:15.179 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-18T14:37:15.179 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2023-10-18T14:37:15.179 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-18T14:37:15.180 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://centos.org/" 2023-10-18T14:37:15.180 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:37:15.180 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:37:15.180 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-18T14:37:15.181 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2023-10-18T14:37:15.203 DEBUG:teuthology.orchestra.run.smithi133:> uname -m 2023-10-18T14:37:15.225 INFO:teuthology.orchestra.run.smithi133.stdout:x86_64 2023-10-18T14:37:15.226 DEBUG:teuthology.orchestra.run.smithi133:> cat /etc/os-release 2023-10-18T14:37:15.285 INFO:teuthology.orchestra.run.smithi133.stdout:NAME="CentOS Stream" 2023-10-18T14:37:15.286 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION="8" 2023-10-18T14:37:15.286 INFO:teuthology.orchestra.run.smithi133.stdout:ID="centos" 2023-10-18T14:37:15.286 INFO:teuthology.orchestra.run.smithi133.stdout:ID_LIKE="rhel fedora" 2023-10-18T14:37:15.286 INFO:teuthology.orchestra.run.smithi133.stdout:VERSION_ID="8" 2023-10-18T14:37:15.286 INFO:teuthology.orchestra.run.smithi133.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:37:15.287 INFO:teuthology.orchestra.run.smithi133.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-18T14:37:15.287 INFO:teuthology.orchestra.run.smithi133.stdout:ANSI_COLOR="0;31" 2023-10-18T14:37:15.287 INFO:teuthology.orchestra.run.smithi133.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-18T14:37:15.287 INFO:teuthology.orchestra.run.smithi133.stdout:HOME_URL="https://centos.org/" 2023-10-18T14:37:15.287 INFO:teuthology.orchestra.run.smithi133.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:37:15.288 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:37:15.288 INFO:teuthology.orchestra.run.smithi133.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-18T14:37:15.288 INFO:teuthology.lock.ops:Updating smithi133.front.sepia.ceph.com on lock server 2023-10-18T14:37:15.311 DEBUG:teuthology.orchestra.run.smithi187:> uname -m 2023-10-18T14:37:15.333 INFO:teuthology.orchestra.run.smithi187.stdout:x86_64 2023-10-18T14:37:15.333 DEBUG:teuthology.orchestra.run.smithi187:> cat /etc/os-release 2023-10-18T14:37:15.394 INFO:teuthology.orchestra.run.smithi187.stdout:NAME="CentOS Stream" 2023-10-18T14:37:15.394 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION="8" 2023-10-18T14:37:15.395 INFO:teuthology.orchestra.run.smithi187.stdout:ID="centos" 2023-10-18T14:37:15.395 INFO:teuthology.orchestra.run.smithi187.stdout:ID_LIKE="rhel fedora" 2023-10-18T14:37:15.395 INFO:teuthology.orchestra.run.smithi187.stdout:VERSION_ID="8" 2023-10-18T14:37:15.395 INFO:teuthology.orchestra.run.smithi187.stdout:PLATFORM_ID="platform:el8" 2023-10-18T14:37:15.395 INFO:teuthology.orchestra.run.smithi187.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-18T14:37:15.396 INFO:teuthology.orchestra.run.smithi187.stdout:ANSI_COLOR="0;31" 2023-10-18T14:37:15.396 INFO:teuthology.orchestra.run.smithi187.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-18T14:37:15.396 INFO:teuthology.orchestra.run.smithi187.stdout:HOME_URL="https://centos.org/" 2023-10-18T14:37:15.396 INFO:teuthology.orchestra.run.smithi187.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-18T14:37:15.396 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-18T14:37:15.396 INFO:teuthology.orchestra.run.smithi187.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-18T14:37:15.397 INFO:teuthology.lock.ops:Updating smithi187.front.sepia.ceph.com on lock server 2023-10-18T14:37:15.419 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-18T14:37:15.430 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-18T14:37:15.438 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-18T14:37:15.438 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:37:15.441 DEBUG:teuthology.orchestra.run.smithi133:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:37:15.443 DEBUG:teuthology.orchestra.run.smithi187:> test '!' -e /home/ubuntu/cephtest 2023-10-18T14:37:15.463 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-18T14:37:15.470 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-18T14:37:15.470 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:37:15.502 DEBUG:teuthology.orchestra.run.smithi133:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:37:15.504 DEBUG:teuthology.orchestra.run.smithi187:> test -z $(ls -A /var/lib/ceph) 2023-10-18T14:37:15.533 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-18T14:37:15.614 INFO:teuthology.run_tasks:Running task kernel... 2023-10-18T14:37:15.631 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-18T14:37:15.632 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-18T14:37:15.632 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:37:15.633 DEBUG:teuthology.orchestra.run.smithi133:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:37:15.634 DEBUG:teuthology.orchestra.run.smithi187:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-18T14:37:15.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:37:15.654 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2023-10-18T14:37:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:37:15.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:37:15.657 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2023-10-18T14:37:15.657 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2023-10-18T14:37:15.715 INFO:teuthology.orchestra.run.smithi187.stdout:4.18.0-489.el8.x86_64 2023-10-18T14:37:15.715 INFO:teuthology.task.kernel:Running kernel on smithi187: 4.18.0-489.el8.x86_64 2023-10-18T14:37:15.715 DEBUG:teuthology.orchestra.run.smithi187:> sudo yum install -y kernel 2023-10-18T14:37:15.719 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-489.el8.x86_64 2023-10-18T14:37:15.719 INFO:teuthology.task.kernel:Running kernel on smithi078: 4.18.0-489.el8.x86_64 2023-10-18T14:37:15.719 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2023-10-18T14:37:15.720 INFO:teuthology.orchestra.run.smithi133.stdout:4.18.0-489.el8.x86_64 2023-10-18T14:37:15.721 INFO:teuthology.task.kernel:Running kernel on smithi133: 4.18.0-489.el8.x86_64 2023-10-18T14:37:15.721 DEBUG:teuthology.orchestra.run.smithi133:> sudo yum install -y kernel 2023-10-18T14:37:21.001 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - AppStream 12 MB/s | 33 MB 00:02 2023-10-18T14:37:21.005 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - AppStream 12 MB/s | 33 MB 00:02 2023-10-18T14:37:21.064 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 8 - AppStream 12 MB/s | 33 MB 00:02 2023-10-18T14:37:29.364 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 50 MB 00:02 2023-10-18T14:37:29.402 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - BaseOS 22 MB/s | 50 MB 00:02 2023-10-18T14:37:32.138 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - BaseOS 9.8 MB/s | 50 MB 00:05 2023-10-18T14:37:36.830 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 8 - Extras 53 kB/s | 18 kB 00:00 2023-10-18T14:37:36.834 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - Extras 57 kB/s | 18 kB 00:00 2023-10-18T14:37:37.401 INFO:teuthology.orchestra.run.smithi133.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2023-10-18T14:37:37.555 INFO:teuthology.orchestra.run.smithi187.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-10-18T14:37:37.728 INFO:teuthology.orchestra.run.smithi133.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-10-18T14:37:37.849 INFO:teuthology.orchestra.run.smithi187.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-10-18T14:37:38.718 INFO:teuthology.orchestra.run.smithi187.stdout:Extra Packages for Enterprise Linux 22 MB/s | 16 MB 00:00 2023-10-18T14:37:39.122 INFO:teuthology.orchestra.run.smithi133.stdout:Extra Packages for Enterprise Linux 13 MB/s | 16 MB 00:01 2023-10-18T14:37:39.830 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras 37 kB/s | 18 kB 00:00 2023-10-18T14:37:40.388 INFO:teuthology.orchestra.run.smithi078.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2023-10-18T14:37:40.682 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-10-18T14:37:41.467 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 25 MB/s | 16 MB 00:00 2023-10-18T14:37:42.516 INFO:teuthology.orchestra.run.smithi187.stdout:lab-extras 436 kB/s | 24 kB 00:00 2023-10-18T14:37:42.895 INFO:teuthology.orchestra.run.smithi133.stdout:lab-extras 446 kB/s | 24 kB 00:00 2023-10-18T14:37:45.341 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 270 kB/s | 24 kB 00:00 2023-10-18T14:37:46.067 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-18T14:37:46.067 INFO:teuthology.orchestra.run.smithi187.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-18T14:37:46.329 INFO:teuthology.orchestra.run.smithi187.stdout:Dependencies resolved. 2023-10-18T14:37:46.330 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-18T14:37:46.330 INFO:teuthology.orchestra.run.smithi187.stdout: Package Architecture Version Repository Size 2023-10-18T14:37:46.331 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-18T14:37:46.331 INFO:teuthology.orchestra.run.smithi187.stdout:Installing: 2023-10-18T14:37:46.331 INFO:teuthology.orchestra.run.smithi187.stdout: kernel x86_64 4.18.0-517.el8 baseos 10 M 2023-10-18T14:37:46.331 INFO:teuthology.orchestra.run.smithi187.stdout:Installing dependencies: 2023-10-18T14:37:46.331 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-core x86_64 4.18.0-517.el8 baseos 43 M 2023-10-18T14:37:46.331 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules x86_64 4.18.0-517.el8 baseos 36 M 2023-10-18T14:37:46.332 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-18T14:37:46.332 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction Summary 2023-10-18T14:37:46.332 INFO:teuthology.orchestra.run.smithi187.stdout:================================================================================ 2023-10-18T14:37:46.332 INFO:teuthology.orchestra.run.smithi187.stdout:Install 3 Packages 2023-10-18T14:37:46.332 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-18T14:37:46.333 INFO:teuthology.orchestra.run.smithi187.stdout:Total download size: 89 M 2023-10-18T14:37:46.333 INFO:teuthology.orchestra.run.smithi187.stdout:Installed size: 96 M 2023-10-18T14:37:46.334 INFO:teuthology.orchestra.run.smithi187.stdout:Downloading Packages: 2023-10-18T14:37:46.461 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-18T14:37:46.461 INFO:teuthology.orchestra.run.smithi133.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-18T14:37:46.723 INFO:teuthology.orchestra.run.smithi133.stdout:Dependencies resolved. 2023-10-18T14:37:46.723 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-10-18T14:37:46.724 INFO:teuthology.orchestra.run.smithi133.stdout: Package Architecture Version Repository Size 2023-10-18T14:37:46.724 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-10-18T14:37:46.724 INFO:teuthology.orchestra.run.smithi133.stdout:Installing: 2023-10-18T14:37:46.724 INFO:teuthology.orchestra.run.smithi133.stdout: kernel x86_64 4.18.0-517.el8 baseos 10 M 2023-10-18T14:37:46.724 INFO:teuthology.orchestra.run.smithi133.stdout:Installing dependencies: 2023-10-18T14:37:46.725 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-core x86_64 4.18.0-517.el8 baseos 43 M 2023-10-18T14:37:46.725 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules x86_64 4.18.0-517.el8 baseos 36 M 2023-10-18T14:37:46.725 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-18T14:37:46.725 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction Summary 2023-10-18T14:37:46.725 INFO:teuthology.orchestra.run.smithi133.stdout:================================================================================ 2023-10-18T14:37:46.725 INFO:teuthology.orchestra.run.smithi133.stdout:Install 3 Packages 2023-10-18T14:37:46.726 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-18T14:37:46.726 INFO:teuthology.orchestra.run.smithi133.stdout:Total download size: 89 M 2023-10-18T14:37:46.726 INFO:teuthology.orchestra.run.smithi133.stdout:Installed size: 96 M 2023-10-18T14:37:46.727 INFO:teuthology.orchestra.run.smithi133.stdout:Downloading Packages: 2023-10-18T14:37:47.050 INFO:teuthology.orchestra.run.smithi187.stdout:(1/3): kernel-4.18.0-517.el8.x86_64.rpm 19 MB/s | 10 MB 00:00 2023-10-18T14:37:47.754 INFO:teuthology.orchestra.run.smithi133.stdout:(1/3): kernel-4.18.0-517.el8.x86_64.rpm 12 MB/s | 10 MB 00:00 2023-10-18T14:37:48.734 INFO:teuthology.orchestra.run.smithi187.stdout:(2/3): kernel-modules-4.18.0-517.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2023-10-18T14:37:48.866 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-18T14:37:48.866 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-18T14:37:49.113 INFO:teuthology.orchestra.run.smithi133.stdout:(2/3): kernel-modules-4.18.0-517.el8.x86_64.rpm 16 MB/s | 36 MB 00:02 2023-10-18T14:37:49.124 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-10-18T14:37:49.125 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-10-18T14:37:49.125 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-10-18T14:37:49.126 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-10-18T14:37:49.126 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-10-18T14:37:49.126 INFO:teuthology.orchestra.run.smithi078.stdout: kernel x86_64 4.18.0-517.el8 baseos 10 M 2023-10-18T14:37:49.126 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-10-18T14:37:49.127 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-core x86_64 4.18.0-517.el8 baseos 43 M 2023-10-18T14:37:49.127 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules x86_64 4.18.0-517.el8 baseos 36 M 2023-10-18T14:37:49.127 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-18T14:37:49.127 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-10-18T14:37:49.127 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-10-18T14:37:49.127 INFO:teuthology.orchestra.run.smithi078.stdout:Install 3 Packages 2023-10-18T14:37:49.128 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-18T14:37:49.128 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 89 M 2023-10-18T14:37:49.128 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 96 M 2023-10-18T14:37:49.129 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-10-18T14:37:49.142 INFO:teuthology.orchestra.run.smithi187.stdout:(3/3): kernel-core-4.18.0-517.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2023-10-18T14:37:49.143 INFO:teuthology.orchestra.run.smithi187.stdout:-------------------------------------------------------------------------------- 2023-10-18T14:37:49.143 INFO:teuthology.orchestra.run.smithi187.stdout:Total 32 MB/s | 89 MB 00:02 2023-10-18T14:37:49.621 INFO:teuthology.orchestra.run.smithi133.stdout:(3/3): kernel-core-4.18.0-517.el8.x86_64.rpm 16 MB/s | 43 MB 00:02 2023-10-18T14:37:49.622 INFO:teuthology.orchestra.run.smithi133.stdout:-------------------------------------------------------------------------------- 2023-10-18T14:37:49.622 INFO:teuthology.orchestra.run.smithi133.stdout:Total 31 MB/s | 89 MB 00:02 2023-10-18T14:37:49.751 INFO:teuthology.orchestra.run.smithi078.stdout:(1/3): kernel-4.18.0-517.el8.x86_64.rpm 17 MB/s | 10 MB 00:00 2023-10-18T14:37:49.820 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction check 2023-10-18T14:37:50.237 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction check succeeded. 2023-10-18T14:37:50.237 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction test 2023-10-18T14:37:50.296 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction check 2023-10-18T14:37:50.720 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction check succeeded. 2023-10-18T14:37:50.721 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction test 2023-10-18T14:37:50.776 INFO:teuthology.orchestra.run.smithi078.stdout:(2/3): kernel-modules-4.18.0-517.el8.x86_64.rpm 22 MB/s | 36 MB 00:01 2023-10-18T14:37:51.218 INFO:teuthology.orchestra.run.smithi078.stdout:(3/3): kernel-core-4.18.0-517.el8.x86_64.rpm 21 MB/s | 43 MB 00:02 2023-10-18T14:37:51.219 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-10-18T14:37:51.219 INFO:teuthology.orchestra.run.smithi078.stdout:Total 42 MB/s | 89 MB 00:02 2023-10-18T14:37:51.458 INFO:teuthology.orchestra.run.smithi187.stdout:Transaction test succeeded. 2023-10-18T14:37:51.487 INFO:teuthology.orchestra.run.smithi187.stdout:Running transaction 2023-10-18T14:37:51.893 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-10-18T14:37:51.956 INFO:teuthology.orchestra.run.smithi133.stdout:Transaction test succeeded. 2023-10-18T14:37:51.978 INFO:teuthology.orchestra.run.smithi133.stdout:Running transaction 2023-10-18T14:37:52.316 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-10-18T14:37:52.317 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-10-18T14:37:53.543 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-10-18T14:37:53.565 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-10-18T14:37:54.261 INFO:teuthology.orchestra.run.smithi187.stdout: Preparing : 1/1 2023-10-18T14:37:54.694 INFO:teuthology.orchestra.run.smithi133.stdout: Preparing : 1/1 2023-10-18T14:37:54.867 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:37:55.261 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:37:56.295 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-10-18T14:37:56.906 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:37:58.426 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:37:58.810 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:37:58.833 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:37:59.394 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:38:00.601 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:38:00.955 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:38:02.216 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:38:02.538 INFO:teuthology.orchestra.run.smithi187.stdout: Installing : kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:02.775 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:38:03.057 INFO:teuthology.orchestra.run.smithi133.stdout: Installing : kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:04.334 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:38:04.612 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:24.428 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:24.453 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:24.816 INFO:teuthology.orchestra.run.smithi187.stdout: Running scriptlet: kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:24.816 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:38:24.816 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-core-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:38:24.883 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:24.908 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:25.263 INFO:teuthology.orchestra.run.smithi133.stdout: Running scriptlet: kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:25.263 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:38:25.263 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-core-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:38:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: Verifying : kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:25.584 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-18T14:38:25.584 INFO:teuthology.orchestra.run.smithi187.stdout:Installed: 2023-10-18T14:38:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-4.18.0-517.el8.x86_64 kernel-core-4.18.0-517.el8.x86_64 2023-10-18T14:38:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: kernel-modules-4.18.0-517.el8.x86_64 2023-10-18T14:38:25.585 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-18T14:38:25.585 INFO:teuthology.orchestra.run.smithi187.stdout:Complete! 2023-10-18T14:38:25.765 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:38:25.977 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:38:25.978 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:38:25.978 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-517.el8.x86_64 2023-10-18T14:38:25.978 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:38:25.979 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:38:25.979 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2023-10-18T14:38:25.979 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi187.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:38:25.979 DEBUG:teuthology.orchestra.run.smithi187:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:38:25.993 INFO:teuthology.orchestra.run.smithi133.stdout: Verifying : kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:25.994 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-18T14:38:25.994 INFO:teuthology.orchestra.run.smithi133.stdout:Installed: 2023-10-18T14:38:25.995 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-4.18.0-517.el8.x86_64 kernel-core-4.18.0-517.el8.x86_64 2023-10-18T14:38:25.995 INFO:teuthology.orchestra.run.smithi133.stdout: kernel-modules-4.18.0-517.el8.x86_64 2023-10-18T14:38:25.995 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-18T14:38:25.995 INFO:teuthology.orchestra.run.smithi133.stdout:Complete! 2023-10-18T14:38:26.188 INFO:teuthology.orchestra.run.smithi187.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:38:26.188 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:38:26.189 DEBUG:teuthology.orchestra.run.smithi187:> sudo rpm -qi grub2-tools 2023-10-18T14:38:26.222 INFO:teuthology.orchestra.run.smithi187.stdout:Name : grub2-tools 2023-10-18T14:38:26.223 INFO:teuthology.orchestra.run.smithi187.stdout:Epoch : 1 2023-10-18T14:38:26.223 INFO:teuthology.orchestra.run.smithi187.stdout:Version : 2.02 2023-10-18T14:38:26.223 INFO:teuthology.orchestra.run.smithi187.stdout:Release : 129.el8 2023-10-18T14:38:26.223 INFO:teuthology.orchestra.run.smithi187.stdout:Architecture: x86_64 2023-10-18T14:38:26.223 INFO:teuthology.orchestra.run.smithi187.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-18T14:38:26.224 INFO:teuthology.orchestra.run.smithi187.stdout:Group : System Environment/Base 2023-10-18T14:38:26.224 INFO:teuthology.orchestra.run.smithi187.stdout:Size : 9265586 2023-10-18T14:38:26.224 INFO:teuthology.orchestra.run.smithi187.stdout:License : GPLv3+ 2023-10-18T14:38:26.224 INFO:teuthology.orchestra.run.smithi187.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-18T14:38:26.224 INFO:teuthology.orchestra.run.smithi187.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-18T14:38:26.225 INFO:teuthology.orchestra.run.smithi187.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-18T14:38:26.225 INFO:teuthology.orchestra.run.smithi187.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-18T14:38:26.225 INFO:teuthology.orchestra.run.smithi187.stdout:Relocations : (not relocatable) 2023-10-18T14:38:26.225 INFO:teuthology.orchestra.run.smithi187.stdout:Packager : CentOS BuildSystem 2023-10-18T14:38:26.225 INFO:teuthology.orchestra.run.smithi187.stdout:Vendor : CentOS 2023-10-18T14:38:26.226 INFO:teuthology.orchestra.run.smithi187.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:38:26.226 INFO:teuthology.orchestra.run.smithi187.stdout:Summary : Support tools for GRUB. 2023-10-18T14:38:26.226 INFO:teuthology.orchestra.run.smithi187.stdout:Description : 2023-10-18T14:38:26.227 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-18T14:38:26.227 INFO:teuthology.orchestra.run.smithi187.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:38:26.227 INFO:teuthology.orchestra.run.smithi187.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:38:26.227 INFO:teuthology.orchestra.run.smithi187.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:38:26.228 INFO:teuthology.orchestra.run.smithi187.stdout:hardware devices. 2023-10-18T14:38:26.228 INFO:teuthology.orchestra.run.smithi187.stdout: 2023-10-18T14:38:26.228 INFO:teuthology.orchestra.run.smithi187.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:38:26.229 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:38:26.229 INFO:teuthology.task.kernel:Updating grub on smithi187 to boot 4.18.0-517.el8.x86_64 2023-10-18T14:38:26.229 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:38:26.270 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-core-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:26.292 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:26.625 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: kernel-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:26.626 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-4.18.0-517.el8.x86_64 1/3 2023-10-18T14:38:26.626 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-core-4.18.0-517.el8.x86_64 2/3 2023-10-18T14:38:26.719 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:38:26.943 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:38:26.944 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:38:26.944 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-517.el8.x86_64 2023-10-18T14:38:26.944 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:38:26.944 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:38:26.945 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-10-18T14:38:26.945 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi133.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:38:26.945 DEBUG:teuthology.orchestra.run.smithi133:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:38:27.154 INFO:teuthology.orchestra.run.smithi133.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:38:27.154 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:38:27.155 DEBUG:teuthology.orchestra.run.smithi133:> sudo rpm -qi grub2-tools 2023-10-18T14:38:27.188 INFO:teuthology.orchestra.run.smithi133.stdout:Name : grub2-tools 2023-10-18T14:38:27.189 INFO:teuthology.orchestra.run.smithi133.stdout:Epoch : 1 2023-10-18T14:38:27.189 INFO:teuthology.orchestra.run.smithi133.stdout:Version : 2.02 2023-10-18T14:38:27.189 INFO:teuthology.orchestra.run.smithi133.stdout:Release : 129.el8 2023-10-18T14:38:27.189 INFO:teuthology.orchestra.run.smithi133.stdout:Architecture: x86_64 2023-10-18T14:38:27.190 INFO:teuthology.orchestra.run.smithi133.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-18T14:38:27.190 INFO:teuthology.orchestra.run.smithi133.stdout:Group : System Environment/Base 2023-10-18T14:38:27.190 INFO:teuthology.orchestra.run.smithi133.stdout:Size : 9265586 2023-10-18T14:38:27.190 INFO:teuthology.orchestra.run.smithi133.stdout:License : GPLv3+ 2023-10-18T14:38:27.190 INFO:teuthology.orchestra.run.smithi133.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-18T14:38:27.191 INFO:teuthology.orchestra.run.smithi133.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-18T14:38:27.191 INFO:teuthology.orchestra.run.smithi133.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-18T14:38:27.191 INFO:teuthology.orchestra.run.smithi133.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-18T14:38:27.191 INFO:teuthology.orchestra.run.smithi133.stdout:Relocations : (not relocatable) 2023-10-18T14:38:27.191 INFO:teuthology.orchestra.run.smithi133.stdout:Packager : CentOS BuildSystem 2023-10-18T14:38:27.191 INFO:teuthology.orchestra.run.smithi133.stdout:Vendor : CentOS 2023-10-18T14:38:27.192 INFO:teuthology.orchestra.run.smithi133.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:38:27.192 INFO:teuthology.orchestra.run.smithi133.stdout:Summary : Support tools for GRUB. 2023-10-18T14:38:27.192 INFO:teuthology.orchestra.run.smithi133.stdout:Description : 2023-10-18T14:38:27.192 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-18T14:38:27.192 INFO:teuthology.orchestra.run.smithi133.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:38:27.192 INFO:teuthology.orchestra.run.smithi133.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:38:27.193 INFO:teuthology.orchestra.run.smithi133.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:38:27.193 INFO:teuthology.orchestra.run.smithi133.stdout:hardware devices. 2023-10-18T14:38:27.193 INFO:teuthology.orchestra.run.smithi133.stdout: 2023-10-18T14:38:27.193 INFO:teuthology.orchestra.run.smithi133.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:38:27.194 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:38:27.194 INFO:teuthology.task.kernel:Updating grub on smithi133 to boot 4.18.0-517.el8.x86_64 2023-10-18T14:38:27.195 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:38:27.329 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : kernel-modules-4.18.0-517.el8.x86_64 3/3 2023-10-18T14:38:27.329 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-18T14:38:27.330 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-10-18T14:38:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-4.18.0-517.el8.x86_64 kernel-core-4.18.0-517.el8.x86_64 2023-10-18T14:38:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: kernel-modules-4.18.0-517.el8.x86_64 2023-10-18T14:38:27.330 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-18T14:38:27.331 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-10-18T14:38:27.488 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:38:27.704 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:38:27.704 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:38:27.704 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-517.el8.x86_64 2023-10-18T14:38:27.704 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-18T14:38:27.705 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-18T14:38:27.705 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-10-18T14:38:27.705 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2023-10-18T14:38:27.705 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2023-10-18T14:38:27.913 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-4.18.0-517.el8.x86_64 2023-10-18T14:38:27.913 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-517.el8.x86_64 2023-10-18T14:38:27.914 DEBUG:teuthology.orchestra.run.smithi078:> sudo rpm -qi grub2-tools 2023-10-18T14:38:27.948 INFO:teuthology.orchestra.run.smithi078.stdout:Name : grub2-tools 2023-10-18T14:38:27.948 INFO:teuthology.orchestra.run.smithi078.stdout:Epoch : 1 2023-10-18T14:38:27.948 INFO:teuthology.orchestra.run.smithi078.stdout:Version : 2.02 2023-10-18T14:38:27.949 INFO:teuthology.orchestra.run.smithi078.stdout:Release : 129.el8 2023-10-18T14:38:27.949 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: x86_64 2023-10-18T14:38:27.949 INFO:teuthology.orchestra.run.smithi078.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-18T14:38:27.949 INFO:teuthology.orchestra.run.smithi078.stdout:Group : System Environment/Base 2023-10-18T14:38:27.949 INFO:teuthology.orchestra.run.smithi078.stdout:Size : 9265586 2023-10-18T14:38:27.950 INFO:teuthology.orchestra.run.smithi078.stdout:License : GPLv3+ 2023-10-18T14:38:27.950 INFO:teuthology.orchestra.run.smithi078.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-18T14:38:27.950 INFO:teuthology.orchestra.run.smithi078.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-18T14:38:27.950 INFO:teuthology.orchestra.run.smithi078.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-18T14:38:27.950 INFO:teuthology.orchestra.run.smithi078.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-18T14:38:27.951 INFO:teuthology.orchestra.run.smithi078.stdout:Relocations : (not relocatable) 2023-10-18T14:38:27.951 INFO:teuthology.orchestra.run.smithi078.stdout:Packager : CentOS BuildSystem 2023-10-18T14:38:27.951 INFO:teuthology.orchestra.run.smithi078.stdout:Vendor : CentOS 2023-10-18T14:38:27.951 INFO:teuthology.orchestra.run.smithi078.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-18T14:38:27.951 INFO:teuthology.orchestra.run.smithi078.stdout:Summary : Support tools for GRUB. 2023-10-18T14:38:27.952 INFO:teuthology.orchestra.run.smithi078.stdout:Description : 2023-10-18T14:38:27.952 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-18T14:38:27.952 INFO:teuthology.orchestra.run.smithi078.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-18T14:38:27.952 INFO:teuthology.orchestra.run.smithi078.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-18T14:38:27.952 INFO:teuthology.orchestra.run.smithi078.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-18T14:38:27.953 INFO:teuthology.orchestra.run.smithi078.stdout:hardware devices. 2023-10-18T14:38:27.953 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-10-18T14:38:27.953 INFO:teuthology.orchestra.run.smithi078.stdout:This subpackage provides tools for support of all platforms. 2023-10-18T14:38:27.954 INFO:teuthology.orchestra.run.smithi187.stderr:Generating grub configuration file ... 2023-10-18T14:38:27.954 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-18T14:38:27.955 INFO:teuthology.task.kernel:Updating grub on smithi078 to boot 4.18.0-517.el8.x86_64 2023-10-18T14:38:27.955 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-18T14:38:28.538 INFO:teuthology.orchestra.run.smithi133.stderr:Generating grub configuration file ... 2023-10-18T14:38:29.665 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2023-10-18T14:38:29.904 INFO:teuthology.orchestra.run.smithi187.stderr:done 2023-10-18T14:38:29.906 DEBUG:teuthology.orchestra.run.smithi187:> mktemp 2023-10-18T14:38:29.934 INFO:teuthology.orchestra.run.smithi187.stdout:/tmp/tmp.OJsn9VLAls 2023-10-18T14:38:29.935 DEBUG:teuthology.orchestra.run.smithi187:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OJsn9VLAls 2023-10-18T14:38:30.008 DEBUG:teuthology.orchestra.run.smithi187:> sudo chmod 0666 /tmp/tmp.OJsn9VLAls 2023-10-18T14:38:30.144 DEBUG:teuthology.orchestra.remote:smithi187:/tmp/tmp.OJsn9VLAls is 6KB 2023-10-18T14:38:30.192 DEBUG:teuthology.orchestra.run.smithi187:> rm -fr /tmp/tmp.OJsn9VLAls 2023-10-18T14:38:30.210 DEBUG:teuthology.orchestra.run.smithi187:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:38:30.285 INFO:teuthology.orchestra.run.smithi187.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-18T14:38:30.285 INFO:teuthology.orchestra.run.smithi187.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-18T14:38:30.285 INFO:teuthology.orchestra.run.smithi187.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-18T14:38:30.286 INFO:teuthology.orchestra.run.smithi187.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64.conf 2023-10-18T14:38:30.287 DEBUG:teuthology.orchestra.run.smithi187:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64 2023-10-18T14:38:30.443 DEBUG:teuthology.orchestra.run.smithi187:> sudo shutdown -r now 2023-10-18T14:38:30.820 INFO:teuthology.orchestra.run.smithi133.stderr:done 2023-10-18T14:38:30.822 DEBUG:teuthology.orchestra.run.smithi133:> mktemp 2023-10-18T14:38:30.844 INFO:teuthology.orchestra.run.smithi133.stdout:/tmp/tmp.HNLF2Ek84S 2023-10-18T14:38:30.845 DEBUG:teuthology.orchestra.run.smithi133:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.HNLF2Ek84S 2023-10-18T14:38:30.919 DEBUG:teuthology.orchestra.run.smithi133:> sudo chmod 0666 /tmp/tmp.HNLF2Ek84S 2023-10-18T14:38:31.068 DEBUG:teuthology.orchestra.remote:smithi133:/tmp/tmp.HNLF2Ek84S is 6KB 2023-10-18T14:38:31.117 DEBUG:teuthology.orchestra.run.smithi133:> rm -fr /tmp/tmp.HNLF2Ek84S 2023-10-18T14:38:31.135 DEBUG:teuthology.orchestra.run.smithi133:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:38:31.200 INFO:teuthology.orchestra.run.smithi133.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-18T14:38:31.201 INFO:teuthology.orchestra.run.smithi133.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-18T14:38:31.201 INFO:teuthology.orchestra.run.smithi133.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-18T14:38:31.201 INFO:teuthology.orchestra.run.smithi133.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64.conf 2023-10-18T14:38:31.202 DEBUG:teuthology.orchestra.run.smithi133:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64 2023-10-18T14:38:31.347 DEBUG:teuthology.orchestra.run.smithi133:> sudo shutdown -r now 2023-10-18T14:38:31.772 INFO:teuthology.orchestra.run.smithi078.stderr:done 2023-10-18T14:38:31.774 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2023-10-18T14:38:31.796 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.sE2PScmyQX 2023-10-18T14:38:31.796 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sE2PScmyQX 2023-10-18T14:38:31.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.sE2PScmyQX 2023-10-18T14:38:32.017 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.sE2PScmyQX is 6KB 2023-10-18T14:38:32.066 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.sE2PScmyQX 2023-10-18T14:38:32.083 DEBUG:teuthology.orchestra.run.smithi078:> sudo /bin/ls /boot/loader/entries || true 2023-10-18T14:38:32.155 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-18T14:38:32.156 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-18T14:38:32.156 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-18T14:38:32.156 INFO:teuthology.orchestra.run.smithi078.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64.conf 2023-10-18T14:38:32.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-517.el8.x86_64 2023-10-18T14:38:32.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2023-10-18T14:39:00.470 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-10-18T14:39:00.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-10-18T14:39:00.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:39:01.350 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-18T14:39:01.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-18T14:39:01.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:39:02.313 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-10-18T14:39:02.314 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-10-18T14:39:02.314 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:39:18.906 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-10-18T14:39:19.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-10-18T14:39:20.698 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2023-10-18T14:39:27.914 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-10-18T14:39:27.915 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:39:28.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-18T14:39:28.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:39:29.700 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-10-18T14:39:29.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:39:30.970 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-10-18T14:39:31.742 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-10-18T14:39:32.762 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2023-10-18T14:39:42.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-10-18T14:39:43.022 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:39:43.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-18T14:39:43.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:39:44.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-10-18T14:39:44.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:40:01.402 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.187 2023-10-18T14:40:02.170 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.133 2023-10-18T14:40:16.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-10-18T14:40:16.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi187.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:40:16.857 DEBUG:teuthology.orchestra.run.smithi078:> true 2023-10-18T14:40:17.171 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-18T14:40:17.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi133.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-18T14:40:17.714 DEBUG:teuthology.orchestra.run.smithi187:> true 2023-10-18T14:40:17.882 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-10-18T14:40:17.882 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-517.el8.x86_64"... 2023-10-18T14:40:17.882 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2023-10-18T14:40:18.199 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi187.front.sepia.ceph.com' 2023-10-18T14:40:18.200 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "4.18.0-517.el8.x86_64"... 2023-10-18T14:40:18.200 DEBUG:teuthology.orchestra.run.smithi187:> uname -r 2023-10-18T14:40:18.216 INFO:teuthology.orchestra.run.smithi187.stdout:4.18.0-517.el8.x86_64 2023-10-18T14:40:18.217 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-517.el8.x86_64 vs 4.18.0-517.el8.x86_64 2023-10-18T14:40:18.217 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:40:18.217 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-18T14:40:18.240 DEBUG:teuthology.orchestra.run.smithi133:> true 2023-10-18T14:40:18.290 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-517.el8.x86_64 2023-10-18T14:40:18.290 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-517.el8.x86_64 vs 4.18.0-517.el8.x86_64 2023-10-18T14:40:18.290 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:40:18.290 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-18T14:40:18.822 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi133.front.sepia.ceph.com' 2023-10-18T14:40:18.891 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-517.el8.x86_64"... 2023-10-18T14:40:18.891 DEBUG:teuthology.orchestra.run.smithi133:> uname -r 2023-10-18T14:40:18.910 INFO:teuthology.orchestra.run.smithi133.stdout:4.18.0-517.el8.x86_64 2023-10-18T14:40:18.911 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-517.el8.x86_64 vs 4.18.0-517.el8.x86_64 2023-10-18T14:40:18.911 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-18T14:40:18.911 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-18T14:40:19.218 INFO:teuthology.task.kernel:Enabling kdb on client.1... 2023-10-18T14:40:19.218 DEBUG:teuthology.orchestra.run.smithi187:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:40:19.289 INFO:teuthology.orchestra.run.smithi187.stdout:ttyS1 2023-10-18T14:40:19.291 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-18T14:40:19.292 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:40:19.301 DEBUG:teuthology.parallel:result is None 2023-10-18T14:40:19.373 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2023-10-18T14:40:19.557 DEBUG:teuthology.parallel:result is None 2023-10-18T14:40:19.912 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2023-10-18T14:40:19.913 DEBUG:teuthology.orchestra.run.smithi133:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-18T14:40:19.997 INFO:teuthology.orchestra.run.smithi133.stdout:ttyS1 2023-10-18T14:40:20.012 DEBUG:teuthology.parallel:result is None 2023-10-18T14:40:20.013 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-18T14:40:20.021 INFO:teuthology.task.internal:Creating test directory... 2023-10-18T14:40:20.021 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:40:20.024 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:40:20.026 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-18T14:40:20.048 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-18T14:40:20.055 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-18T14:40:20.065 INFO:teuthology.task.internal:Creating archive directory... 2023-10-18T14:40:20.065 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:40:20.081 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:40:20.086 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-18T14:40:20.135 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-18T14:40:20.142 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-18T14:40:20.142 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:40:20.150 DEBUG:teuthology.orchestra.run.smithi133:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:40:20.156 DEBUG:teuthology.orchestra.run.smithi187:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-18T14:40:20.181 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:40:20.191 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:40:20.193 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:40:20.203 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:40:20.207 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:40:20.218 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-18T14:40:20.220 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-18T14:40:20.227 INFO:teuthology.task.internal:Configuring sudo... 2023-10-18T14:40:20.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:40:20.233 DEBUG:teuthology.orchestra.run.smithi133:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:40:20.247 DEBUG:teuthology.orchestra.run.smithi187:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-18T14:40:20.290 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-18T14:40:20.299 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-18T14:40:20.300 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:40:20.303 DEBUG:teuthology.orchestra.run.smithi133:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:40:20.316 DEBUG:teuthology.orchestra.run.smithi187:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-18T14:40:20.350 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:40:20.402 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:40:20.466 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:40:20.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:40:20.551 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-10-18T14:40:20.551 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:40:20.618 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:40:20.664 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:40:20.721 DEBUG:teuthology.orchestra.run.smithi133:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:40:20.806 DEBUG:teuthology.orchestra.run.smithi133:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:40:20.847 DEBUG:teuthology.orchestra.run.smithi133:> set -ex 2023-10-18T14:40:20.847 DEBUG:teuthology.orchestra.run.smithi133:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:40:20.914 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:40:20.962 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-18T14:40:21.021 DEBUG:teuthology.orchestra.run.smithi187:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:40:21.110 DEBUG:teuthology.orchestra.run.smithi187:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-18T14:40:21.153 DEBUG:teuthology.orchestra.run.smithi187:> set -ex 2023-10-18T14:40:21.153 DEBUG:teuthology.orchestra.run.smithi187:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-18T14:40:21.223 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2023-10-18T14:40:21.225 DEBUG:teuthology.orchestra.run.smithi133:> sudo service rsyslog restart 2023-10-18T14:40:21.228 DEBUG:teuthology.orchestra.run.smithi187:> sudo service rsyslog restart 2023-10-18T14:40:21.282 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:40:21.286 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:40:21.336 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:40:21.804 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-18T14:40:21.812 INFO:teuthology.task.internal:Starting timer... 2023-10-18T14:40:21.812 INFO:teuthology.run_tasks:Running task pcp... 2023-10-18T14:40:21.822 INFO:teuthology.run_tasks:Running task selinux... 2023-10-18T14:40:21.831 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-18T14:40:21.832 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2023-10-18T14:40:21.931 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2023-10-18T14:40:21.933 DEBUG:teuthology.orchestra.run.smithi133:> sudo service auditd rotate 2023-10-18T14:40:22.001 INFO:teuthology.orchestra.run.smithi133.stdout:Rotating logs: 2023-10-18T14:40:22.003 DEBUG:teuthology.orchestra.run.smithi187:> sudo service auditd rotate 2023-10-18T14:40:22.083 INFO:teuthology.orchestra.run.smithi187.stdout:Rotating logs: 2023-10-18T14:40:22.084 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-18T14:40:22.085 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2023-10-18T14:40:22.114 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2023-10-18T14:40:22.114 DEBUG:teuthology.orchestra.run.smithi133:> /usr/sbin/getenforce 2023-10-18T14:40:22.135 INFO:teuthology.orchestra.run.smithi133.stdout:Permissive 2023-10-18T14:40:22.135 DEBUG:teuthology.orchestra.run.smithi187:> /usr/sbin/getenforce 2023-10-18T14:40:22.187 INFO:teuthology.orchestra.run.smithi187.stdout:Permissive 2023-10-18T14:40:22.188 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi133.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi187.front.sepia.ceph.com': 'permissive'} 2023-10-18T14:40:22.188 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:40:22.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:40:22.218 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:40:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:40:22.245 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:40:22.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:40:22.276 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-18T14:40:22.276 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:40:22.307 DEBUG:teuthology.orchestra.run.smithi133:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:40:22.344 DEBUG:teuthology.orchestra.run.smithi187:> sudo /usr/sbin/setenforce permissive 2023-10-18T14:40:22.379 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-18T14:40:22.389 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-18T14:40:22.395 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-18T14:40:22.530 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-18T14:40:22.545 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-18T14:40:22.546 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi133.front.sepia.ceph.com,smithi187.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-18T14:48:17.948 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910/ansible_failures.yaml 2023-10-18T14:48:17.951 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:48:18.034 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=8e4a2e13a12840c8ace298bf51322e34 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 503, in begin super(CephLab, self).begin() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 341, in begin self.execute_playbook() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 369, in execute_playbook self._handle_failure(command, status) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/task/ansible.py", line 397, in _handle_failure raise AnsibleFailedError(failures) teuthology.exceptions.AnsibleFailedError: [',', 'Bugzilla. In case of problems, contact the owner of this repository.', 'Enabling a Copr repository. Please note that this repository is not part', 'Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable', 'Please do not file bug reports about these packages in Fedora', 'The Fedora Project does not exercise any power over the contents of', 'and packages are not held to any quality or security level.', 'of the main distribution, and quality may vary.', 'this repository beyond the rules outlined in the Copr FAQ at'] 2023-10-18T14:48:18.036 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-18T14:48:18.048 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-18T14:48:18.048 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-18T14:48:18.061 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:48:18.155 DEBUG:teuthology.orchestra.run.smithi133:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:48:18.244 DEBUG:teuthology.orchestra.run.smithi187:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-18T14:48:18.333 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:48:18.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:48:18.369 DEBUG:teuthology.orchestra.run.smithi133:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:48:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:48:18.404 DEBUG:teuthology.orchestra.run.smithi187:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-18T14:48:18.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:48:18.438 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-18T14:48:18.447 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-18T14:48:18.461 INFO:teuthology.task.internal:Duration was 476.649104 seconds 2023-10-18T14:48:18.462 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-18T14:48:18.475 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-18T14:48:18.476 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:48:18.478 DEBUG:teuthology.orchestra.run.smithi133:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:48:18.481 DEBUG:teuthology.orchestra.run.smithi187:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-18T14:48:18.527 INFO:teuthology.orchestra.run.smithi133.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:48:18.528 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:48:18.529 INFO:teuthology.orchestra.run.smithi187.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-18T14:48:18.847 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-18T14:48:18.848 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2023-10-18T14:48:18.848 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:48:18.872 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi133.front.sepia.ceph.com 2023-10-18T14:48:18.873 DEBUG:teuthology.orchestra.run.smithi133:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:48:18.896 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi187.front.sepia.ceph.com 2023-10-18T14:48:18.897 DEBUG:teuthology.orchestra.run.smithi187:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-18T14:48:18.920 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-18T14:48:18.920 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:48:18.923 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:48:18.939 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-18T14:48:18.996 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-18T14:48:18.996 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:48:19.000 DEBUG:teuthology.orchestra.run.smithi133:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:48:19.015 DEBUG:teuthology.orchestra.run.smithi187:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-18T14:48:19.095 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-18T14:48:19.107 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-18T14:48:19.108 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:48:19.142 DEBUG:teuthology.orchestra.run.smithi133:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:48:19.171 DEBUG:teuthology.orchestra.run.smithi187:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-18T14:48:19.199 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-18T14:48:19.209 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:48:19.212 DEBUG:teuthology.orchestra.run.smithi133:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:48:19.214 DEBUG:teuthology.orchestra.run.smithi187:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:48:19.236 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2023-10-18T14:48:19.238 INFO:teuthology.orchestra.run.smithi133.stdout:kernel.core_pattern = core 2023-10-18T14:48:19.265 INFO:teuthology.orchestra.run.smithi187.stdout:kernel.core_pattern = core 2023-10-18T14:48:19.295 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:48:19.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:48:19.313 DEBUG:teuthology.orchestra.run.smithi133:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:48:19.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:48:19.331 DEBUG:teuthology.orchestra.run.smithi187:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-18T14:48:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-18T14:48:19.353 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-18T14:48:19.366 INFO:teuthology.task.internal:Transferring archived files... 2023-10-18T14:48:19.367 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910/remote/smithi078 2023-10-18T14:48:19.368 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:48:19.443 DEBUG:teuthology.misc:Transferring archived files from smithi133:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910/remote/smithi133 2023-10-18T14:48:19.444 DEBUG:teuthology.orchestra.run.smithi133:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:48:19.492 DEBUG:teuthology.misc:Transferring archived files from smithi187:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910/remote/smithi187 2023-10-18T14:48:19.493 DEBUG:teuthology.orchestra.run.smithi187:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-18T14:48:19.556 INFO:teuthology.task.internal:Removing archive directory... 2023-10-18T14:48:19.557 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:48:19.560 DEBUG:teuthology.orchestra.run.smithi133:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:48:19.562 DEBUG:teuthology.orchestra.run.smithi187:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-18T14:48:19.580 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-18T14:48:19.591 INFO:teuthology.task.internal:Not uploading archives. 2023-10-18T14:48:19.591 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-18T14:48:19.604 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-18T14:48:19.605 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:48:19.617 DEBUG:teuthology.orchestra.run.smithi133:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:48:19.620 DEBUG:teuthology.orchestra.run.smithi187:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-18T14:48:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:48 /home/ubuntu/cephtest 2023-10-18T14:48:19.635 INFO:teuthology.orchestra.run.smithi133.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:48 /home/ubuntu/cephtest 2023-10-18T14:48:19.638 INFO:teuthology.orchestra.run.smithi187.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 18 14:48 /home/ubuntu/cephtest 2023-10-18T14:48:19.640 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-18T14:48:19.649 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-18T14:48:19.679 INFO:teuthology.nuke:Checking targets against current locks 2023-10-18T14:48:19.706 DEBUG:teuthology.nuke:shortname: smithi078 2023-10-18T14:48:19.707 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:48:19.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:32:09.122889', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK7Nx7z7bBDivepMKbAaOVYa5p8Sf8V5tBvDu0q/U1L27KGwOnmq4YEM5ACDjLuUNpGeGNpTqpNRlJOqJOBMCgU='} 2023-10-18T14:48:19.736 DEBUG:teuthology.nuke:shortname: smithi133 2023-10-18T14:48:19.736 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:48:19.763 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi133.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:32:09.120103', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZTfx0lPP2e1N15dq974CmzOEF3J5ouKLlVE/WIIiU6ZC0OfcqXT9Keq/nITGsirpSma2QGoVoBWaJcfyd9/Ag='} 2023-10-18T14:48:19.765 DEBUG:teuthology.nuke:shortname: smithi187 2023-10-18T14:48:19.766 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-18T14:48:19.786 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi187.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-10-16_23:59:58-fs-wip-batrick-testing-20231016.203825-distro-default-smithi/7430910', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-18 14:32:09.121609', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP8GY7oGtcmJGPyZNXquF//xl9vZRDCYrhk2eEwoCkIk/qyqZ5XqGxllDL8s+QAI6yUxEUfEx1HDkuhtQVLQZlI='} 2023-10-18T14:48:19.800 INFO:teuthology.orchestra.console.smithi078:Power off 2023-10-18T14:48:19.802 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:48:19.818 INFO:teuthology.orchestra.console.smithi133:Power off 2023-10-18T14:48:19.819 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:48:19.832 DEBUG:teuthology.orchestra.console.smithi078:power off output: Chassis Power Control: Down/Off 2023-10-18T14:48:19.833 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:48:19.845 DEBUG:teuthology.orchestra.console.smithi133:power off output: Chassis Power Control: Down/Off 2023-10-18T14:48:19.846 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:48:19.861 INFO:teuthology.orchestra.console.smithi187:Power off 2023-10-18T14:48:19.862 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-18T14:48:19.872 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2023-10-18T14:48:19.873 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-10-18T14:48:19.885 DEBUG:teuthology.orchestra.console.smithi187:power off output: Chassis Power Control: Down/Off 2023-10-18T14:48:19.885 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:48:19.909 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-10-18T14:48:23.875 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:48:23.883 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:48:23.910 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:48:23.986 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is on 2023-10-18T14:48:23.993 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is on 2023-10-18T14:48:24.021 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is on 2023-10-18T14:48:27.990 DEBUG:teuthology.orchestra.console.smithi133:pexpect command: ipmitool -H smithi133.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:48:27.999 DEBUG:teuthology.orchestra.console.smithi078:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:48:28.022 DEBUG:teuthology.orchestra.console.smithi187:pexpect command: ipmitool -H smithi187.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-18T14:48:28.101 DEBUG:teuthology.orchestra.console.smithi133:check power output: Chassis Power is off 2023-10-18T14:48:28.108 DEBUG:teuthology.orchestra.console.smithi078:check power output: Chassis Power is off 2023-10-18T14:48:28.133 DEBUG:teuthology.orchestra.console.smithi187:check power output: Chassis Power is off 2023-10-18T14:48:28.202 INFO:teuthology.orchestra.console.smithi133:Power off completed 2023-10-18T14:48:28.209 INFO:teuthology.orchestra.console.smithi078:Power off completed 2023-10-18T14:48:28.234 INFO:teuthology.orchestra.console.smithi187:Power off completed 2023-10-18T14:48:28.341 INFO:teuthology.run:Summary data: description: fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_8.stream clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} duration: 476.64910364151 failure_reason: '['','', ''Bugzilla. In case of problems, contact the owner of this repository.'', ''Enabling a Copr repository. Please note that this repository is not part'', ''Error: Failed to connect to https://copr.fedorainfracloud.org/coprs/ceph/python3-asyncssh/repo/epel-8/dnf.repo?arch=x86_64: Network is unreachable'', ''Please do not file bug reports about these packages in Fedora'', ''The Fedora Project does not exercise any power over the contents of'', ''and packages are not held to any quality or security level.'', ''of the main distribution, and quality may vary.'', ''this repository beyond the rules outlined in the Copr FAQ at'']' owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=8e4a2e13a12840c8ace298bf51322e34 status: dead success: false 2023-10-18T14:48:28.342 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-18T14:48:28.430 INFO:teuthology.run:DEAD