2023-11-28T00:48:35.947 INFO:root:teuthology version: 0.0.1.dev205+g2442239 2023-11-28T00:48:35.948 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --description orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} --archive /home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380 -- /home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380/orig.config.yaml 2023-11-28T00:48:35.978 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T00:48:36.063 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380 branch: wip-yuri10-testing-2023-11-22-1112 description: orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} email: yweinste@redhat.com first_in_suite: false job_id: '7468380' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-yuri10-testing-2023-11-22-1112 ceph: conf: client: debug ms: 1 global: mon election default strategy: 1 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 369173db14b6995b2bd07c60ec5f63d01cf21631 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: cephadm-package install: ceph: flavor: default sha1: 369173db14b6995b2bd07c60ec5f63d01cf21631 extra_packages: - cephadm selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri10-testing-2023-11-22-1112 sha1: 369173db14b6995b2bd07c60ec5f63d01cf21631 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 1759 sha1: 369173db14b6995b2bd07c60ec5f63d01cf21631 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: wip-yuri10-testing-2023-11-22-1112 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 369173db14b6995b2bd07c60ec5f63d01cf21631 targets: smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:3.0 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 2442239d2653456406c25ae0c71b689c8f2657b6 timestamp: 2023-11-27_17:19:32 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-11-28T00:48:36.063 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa; will attempt to use it 2023-11-28T00:48:36.064 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa/tasks 2023-11-28T00:48:36.064 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-28T00:48:36.065 INFO:teuthology.task.internal:Checking packages... 2023-11-28T00:48:36.089 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '369173db14b6995b2bd07c60ec5f63d01cf21631' 2023-11-28T00:48:36.090 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-28T00:48:36.090 INFO:teuthology.packaging:ref: None 2023-11-28T00:48:36.090 INFO:teuthology.packaging:tag: None 2023-11-28T00:48:36.090 INFO:teuthology.packaging:branch: wip-yuri10-testing-2023-11-22-1112 2023-11-28T00:48:36.090 INFO:teuthology.packaging:sha1: 369173db14b6995b2bd07c60ec5f63d01cf21631 2023-11-28T00:48:36.090 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri10-testing-2023-11-22-1112 2023-11-28T00:48:36.315 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7498.g369173db 2023-11-28T00:48:36.315 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-28T00:48:36.322 INFO:teuthology.task.internal:no buildpackages task found 2023-11-28T00:48:36.323 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-28T00:48:36.338 INFO:teuthology.task.internal:Saving configuration 2023-11-28T00:48:36.350 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-28T00:48:36.361 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T00:48:36.389 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 00:42:52.078435', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-28T00:48:36.412 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 00:42:52.077023', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-28T00:48:36.412 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-28T00:48:36.422 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2023-11-28T00:48:36.422 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2023-11-28T00:48:36.423 INFO:teuthology.run_tasks:Running task console_log... 2023-11-28T00:48:36.500 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ffae1826f70>, signals=[15]) 2023-11-28T00:48:36.500 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-28T00:48:36.511 INFO:teuthology.task.internal:Opening connections... 2023-11-28T00:48:36.511 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2023-11-28T00:48:36.513 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T00:48:36.581 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2023-11-28T00:48:36.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T00:48:36.647 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-28T00:48:36.656 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2023-11-28T00:48:36.677 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2023-11-28T00:48:36.677 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2023-11-28T00:48:36.736 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Red Hat Enterprise Linux" 2023-11-28T00:48:36.736 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="8.6 (Ootpa)" 2023-11-28T00:48:36.736 INFO:teuthology.orchestra.run.smithi172.stdout:ID="rhel" 2023-11-28T00:48:36.737 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="fedora" 2023-11-28T00:48:36.737 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="8.6" 2023-11-28T00:48:36.737 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el8" 2023-11-28T00:48:36.737 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-28T00:48:36.737 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2023-11-28T00:48:36.737 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-28T00:48:36.737 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.redhat.com/" 2023-11-28T00:48:36.737 INFO:teuthology.orchestra.run.smithi172.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-28T00:48:36.738 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T00:48:36.738 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T00:48:36.738 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-28T00:48:36.738 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-28T00:48:36.738 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-28T00:48:36.738 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-28T00:48:36.739 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2023-11-28T00:48:36.766 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2023-11-28T00:48:36.786 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2023-11-28T00:48:36.786 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2023-11-28T00:48:36.844 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="Red Hat Enterprise Linux" 2023-11-28T00:48:36.845 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="8.6 (Ootpa)" 2023-11-28T00:48:36.845 INFO:teuthology.orchestra.run.smithi192.stdout:ID="rhel" 2023-11-28T00:48:36.845 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="fedora" 2023-11-28T00:48:36.845 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="8.6" 2023-11-28T00:48:36.845 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el8" 2023-11-28T00:48:36.845 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-11-28T00:48:36.846 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2023-11-28T00:48:36.846 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-11-28T00:48:36.846 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://www.redhat.com/" 2023-11-28T00:48:36.846 INFO:teuthology.orchestra.run.smithi192.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-11-28T00:48:36.847 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-28T00:48:36.847 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T00:48:36.847 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-28T00:48:36.847 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-11-28T00:48:36.847 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-11-28T00:48:36.847 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-11-28T00:48:36.848 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2023-11-28T00:48:36.866 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-28T00:48:36.876 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-28T00:48:36.884 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-28T00:48:36.884 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2023-11-28T00:48:36.887 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2023-11-28T00:48:36.906 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-28T00:48:36.914 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-28T00:48:36.914 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2023-11-28T00:48:36.946 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2023-11-28T00:48:36.985 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-28T00:48:37.042 INFO:teuthology.run_tasks:Running task kernel... 2023-11-28T00:48:37.058 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-28T00:48:37.059 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-28T00:48:37.059 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T00:48:37.060 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-28T00:48:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T00:48:37.078 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2023-11-28T00:48:37.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T00:48:37.079 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2023-11-28T00:48:37.137 INFO:teuthology.orchestra.run.smithi172.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-28T00:48:37.137 INFO:teuthology.task.kernel:Running kernel on smithi172: 4.18.0-372.9.1.el8.x86_64 2023-11-28T00:48:37.138 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2023-11-28T00:48:37.139 INFO:teuthology.orchestra.run.smithi192.stdout:4.18.0-372.9.1.el8.x86_64 2023-11-28T00:48:37.139 INFO:teuthology.task.kernel:Running kernel on smithi192: 4.18.0-372.9.1.el8.x86_64 2023-11-28T00:48:37.139 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2023-11-28T00:48:38.320 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T00:48:38.320 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-11-28T00:48:38.389 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T00:48:38.390 INFO:teuthology.orchestra.run.smithi192.stdout:Unable to read consumer identity 2023-11-28T00:48:39.778 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-11-28T00:48:39.903 INFO:teuthology.orchestra.run.smithi192.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-11-28T00:48:40.670 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2023-11-28T00:48:40.712 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-11-28T00:48:44.213 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 349 kB/s | 24 kB 00:00 2023-11-28T00:48:44.220 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 479 kB/s | 24 kB 00:00 2023-11-28T00:48:44.898 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-11-28T00:48:44.944 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T00:48:44.945 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-11-28T00:48:44.945 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T00:48:44.948 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-11-28T00:48:45.004 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T00:48:45.005 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-11-28T00:48:45.005 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T00:48:45.080 DEBUG:teuthology.orchestra.run.smithi192:> echo no | sudo yum reinstall kernel || true 2023-11-28T00:48:45.117 DEBUG:teuthology.orchestra.run.smithi172:> echo no | sudo yum reinstall kernel || true 2023-11-28T00:48:45.412 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T00:48:45.412 INFO:teuthology.orchestra.run.smithi192.stdout:Unable to read consumer identity 2023-11-28T00:48:45.446 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T00:48:45.447 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-11-28T00:48:45.673 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:01 ago on Tue 28 Nov 2023 12:48:44 AM UTC. 2023-11-28T00:48:45.708 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:01 ago on Tue 28 Nov 2023 12:48:44 AM UTC. 2023-11-28T00:48:45.713 INFO:teuthology.orchestra.run.smithi192.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-28T00:48:45.749 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-28T00:48:45.770 INFO:teuthology.orchestra.run.smithi192.stderr:Error: No packages marked for reinstall. 2023-11-28T00:48:45.788 INFO:teuthology.orchestra.run.smithi172.stderr:Error: No packages marked for reinstall. 2023-11-28T00:48:45.815 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum reinstall -y kernel || true 2023-11-28T00:48:45.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum reinstall -y kernel || true 2023-11-28T00:48:46.146 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T00:48:46.146 INFO:teuthology.orchestra.run.smithi192.stdout:Unable to read consumer identity 2023-11-28T00:48:46.166 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T00:48:46.166 INFO:teuthology.orchestra.run.smithi172.stdout:Unable to read consumer identity 2023-11-28T00:48:46.401 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:02 ago on Tue 28 Nov 2023 12:48:44 AM UTC. 2023-11-28T00:48:46.423 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:02 ago on Tue 28 Nov 2023 12:48:44 AM UTC. 2023-11-28T00:48:46.440 INFO:teuthology.orchestra.run.smithi192.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-28T00:48:46.464 INFO:teuthology.orchestra.run.smithi172.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-11-28T00:48:46.465 INFO:teuthology.orchestra.run.smithi192.stderr:Error: No packages marked for reinstall. 2023-11-28T00:48:46.486 INFO:teuthology.orchestra.run.smithi172.stderr:Error: No packages marked for reinstall. 2023-11-28T00:48:46.516 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2023-11-28T00:48:46.533 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2023-11-28T00:48:46.597 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-11-28T00:48:46.597 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-11-28T00:48:46.597 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-11-28T00:48:46.597 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-11-28T00:48:46.598 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-28T00:48:46.614 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-11-28T00:48:46.614 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-11-28T00:48:46.614 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-11-28T00:48:46.615 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-11-28T00:48:46.615 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-28T00:48:46.623 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2023-11-28T00:48:46.636 DEBUG:teuthology.parallel:result is None 2023-11-28T00:48:46.641 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2023-11-28T00:48:46.656 DEBUG:teuthology.parallel:result is None 2023-11-28T00:48:46.656 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-28T00:48:46.673 INFO:teuthology.task.internal:Creating test directory... 2023-11-28T00:48:46.673 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T00:48:46.698 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-28T00:48:46.715 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-28T00:48:46.726 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-28T00:48:46.739 INFO:teuthology.task.internal:Creating archive directory... 2023-11-28T00:48:46.740 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T00:48:46.757 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-28T00:48:46.781 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-28T00:48:46.792 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-28T00:48:46.792 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T00:48:46.814 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-28T00:48:46.850 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T00:48:46.850 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T00:48:46.859 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T00:48:46.860 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-28T00:48:46.862 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-28T00:48:46.873 INFO:teuthology.task.internal:Configuring sudo... 2023-11-28T00:48:46.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T00:48:46.903 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-28T00:48:46.929 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-28T00:48:46.942 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-28T00:48:46.942 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T00:48:46.970 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-28T00:48:46.987 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T00:48:47.068 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T00:48:47.117 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T00:48:47.200 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T00:48:47.243 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T00:48:47.243 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T00:48:47.309 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T00:48:47.350 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-28T00:48:47.400 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T00:48:47.484 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-28T00:48:47.525 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T00:48:47.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-28T00:48:47.592 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2023-11-28T00:48:47.595 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2023-11-28T00:48:47.652 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T00:48:47.690 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T00:48:48.082 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-28T00:48:48.091 INFO:teuthology.task.internal:Starting timer... 2023-11-28T00:48:48.091 INFO:teuthology.run_tasks:Running task pcp... 2023-11-28T00:48:48.108 INFO:teuthology.run_tasks:Running task selinux... 2023-11-28T00:48:48.122 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-28T00:48:48.123 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2023-11-28T00:48:48.195 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2023-11-28T00:48:48.196 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2023-11-28T00:48:48.253 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2023-11-28T00:48:48.255 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-28T00:48:48.255 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2023-11-28T00:48:48.277 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2023-11-28T00:48:48.277 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2023-11-28T00:48:48.295 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2023-11-28T00:48:48.295 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi172.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2023-11-28T00:48:48.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T00:48:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T00:48:48.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T00:48:48.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T00:48:48.371 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-28T00:48:48.371 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2023-11-28T00:48:48.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2023-11-28T00:48:48.443 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-28T00:48:48.454 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-11-28T00:48:48.454 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-28T00:48:48.478 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-28T00:48:48.479 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi172.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-28T00:57:52.802 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi172.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2023-11-28T00:57:52.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-11-28T00:57:52.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T00:57:52.874 DEBUG:teuthology.orchestra.run.smithi172:> true 2023-11-28T00:57:52.939 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2023-11-28T00:57:52.940 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-11-28T00:57:52.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-28T00:57:53.009 DEBUG:teuthology.orchestra.run.smithi192:> true 2023-11-28T00:57:53.081 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2023-11-28T00:57:53.082 INFO:teuthology.run_tasks:Running task clock... 2023-11-28T00:57:53.094 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-28T00:57:53.094 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T00:57:53.095 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T00:57:53.097 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-28T00:57:53.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T00:57:53.125 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T00:57:53.140 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T00:57:53.158 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-28T00:57:53.174 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-28T00:57:53.224 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2023-11-28T00:57:53.236 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2023-11-28T00:57:53.242 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2023-11-28T00:57:53.250 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T00:57:53.254 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2023-11-28T00:57:53.264 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T00:57:53.269 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-28T00:57:53.283 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-28T00:57:53.341 INFO:teuthology.orchestra.run.smithi172.stderr:bash: ntpq: command not found 2023-11-28T00:57:53.342 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T00:57:53.342 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2023-11-28T00:57:53.342 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T00:57:53.343 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T00:57:53.343 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T00:57:53.343 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T00:57:53.356 INFO:teuthology.orchestra.run.smithi192.stderr:bash: ntpq: command not found 2023-11-28T00:57:53.359 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T00:57:53.360 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2023-11-28T00:57:53.360 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T00:57:53.360 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T00:57:53.360 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T00:57:53.360 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T00:57:53.361 INFO:teuthology.run_tasks:Running task pexec... 2023-11-28T00:57:53.370 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-28T00:57:53.371 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi172.front.sepia.ceph.com 2023-11-28T00:57:53.371 DEBUG:teuthology.orchestra.run.smithi172:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-28T00:57:53.371 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi192.front.sepia.ceph.com 2023-11-28T00:57:53.371 DEBUG:teuthology.orchestra.run.smithi192:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-28T00:57:53.759 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T00:57:53.774 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T00:57:54.902 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:29 ago on Tue 28 Nov 2023 12:56:25 AM UTC. 2023-11-28T00:58:00.539 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T00:58:00.539 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T00:58:00.539 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2023-11-28T00:58:00.539 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T00:58:00.539 INFO:teuthology.orchestra.run.smithi192.stdout:Resetting modules: 2023-11-28T00:58:00.539 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools 2023-11-28T00:58:00.540 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T00:58:00.540 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T00:58:00.540 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T00:58:00.540 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T00:58:00.540 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:32 ago on Tue 28 Nov 2023 12:56:25 AM UTC. 2023-11-28T00:58:00.540 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:31 ago on Tue 28 Nov 2023 12:56:23 AM UTC. 2023-11-28T00:58:00.541 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T00:58:00.541 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T00:58:00.566 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2023-11-28T00:58:00.566 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T00:58:00.566 INFO:teuthology.orchestra.run.smithi172.stdout:Resetting modules: 2023-11-28T00:58:00.566 INFO:teuthology.orchestra.run.smithi172.stdout: container-tools 2023-11-28T00:58:00.567 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T00:58:00.567 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T00:58:00.567 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T00:58:00.567 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T00:58:00.567 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:34 ago on Tue 28 Nov 2023 12:56:23 AM UTC. 2023-11-28T00:58:05.510 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T00:58:05.961 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T00:58:05.967 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T00:58:06.421 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T00:58:07.572 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:42 ago on Tue 28 Nov 2023 12:56:25 AM UTC. 2023-11-28T00:58:08.068 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:44 ago on Tue 28 Nov 2023 12:56:23 AM UTC. 2023-11-28T00:58:09.273 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T00:58:09.366 INFO:teuthology.orchestra.run.smithi192.stdout:========================================================================================================================== 2023-11-28T00:58:09.366 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-11-28T00:58:09.366 INFO:teuthology.orchestra.run.smithi192.stdout:========================================================================================================================== 2023-11-28T00:58:09.366 INFO:teuthology.orchestra.run.smithi192.stdout:Installing group/module packages: 2023-11-28T00:58:09.366 INFO:teuthology.orchestra.run.smithi192.stdout: buildah x86_64 1.19.9-3.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 6.8 M 2023-11-28T00:58:09.366 INFO:teuthology.orchestra.run.smithi192.stdout: cockpit-podman noarch 29-2.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 1.1 M 2023-11-28T00:58:09.367 INFO:teuthology.orchestra.run.smithi192.stdout: crun x86_64 0.18-3.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 185 k 2023-11-28T00:58:09.367 INFO:teuthology.orchestra.run.smithi192.stdout: skopeo x86_64 1:1.2.4-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 6.4 M 2023-11-28T00:58:09.367 INFO:teuthology.orchestra.run.smithi192.stdout: toolbox x86_64 0.0.99.3-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-28T00:58:09.367 INFO:teuthology.orchestra.run.smithi192.stdout: udica noarch 0.2.4-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 51 k 2023-11-28T00:58:09.367 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-11-28T00:58:09.367 INFO:teuthology.orchestra.run.smithi192.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-28T00:58:09.367 INFO:teuthology.orchestra.run.smithi192.stdout:Downgrading: 2023-11-28T00:58:09.367 INFO:teuthology.orchestra.run.smithi192.stdout: conmon x86_64 2:2.0.26-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 51 k 2023-11-28T00:58:09.367 INFO:teuthology.orchestra.run.smithi192.stdout: container-selinux noarch 2:2.178.0-2.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 53 k 2023-11-28T00:58:09.367 INFO:teuthology.orchestra.run.smithi192.stdout: containernetworking-plugins x86_64 0.9.1-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 19 M 2023-11-28T00:58:09.367 INFO:teuthology.orchestra.run.smithi192.stdout: containers-common x86_64 1:1.2.4-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 91 k 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout: criu x86_64 3.15-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 511 k 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout: fuse-overlayfs x86_64 1.4.0-2.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 72 k 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout: libslirp x86_64 4.3.1-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 69 k 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout: podman x86_64 3.0.1-9.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 11 M 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout: podman-catatonit x86_64 3.0.1-9.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 322 k 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout: podman-docker noarch 3.0.1-9.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 56 k 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout: runc x86_64 1.0.0-73.rc95.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 2.9 M 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout: slirp4netns x86_64 1.1.8-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 51 k 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout:Installing module profiles: 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools/common 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout:Enabling module streams: 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout: container-tools 3.0 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T00:58:09.368 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T00:58:09.369 INFO:teuthology.orchestra.run.smithi192.stdout:========================================================================================================================== 2023-11-28T00:58:09.369 INFO:teuthology.orchestra.run.smithi192.stdout:Install 7 Packages 2023-11-28T00:58:09.369 INFO:teuthology.orchestra.run.smithi192.stdout:Downgrade 12 Packages 2023-11-28T00:58:09.369 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T00:58:09.369 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 51 M 2023-11-28T00:58:09.369 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T00:58:09.559 INFO:teuthology.orchestra.run.smithi192.stdout:(1/19): containers-common-1.2.4-1.module+el8.6. 481 kB/s | 91 kB 00:00 2023-11-28T00:58:09.584 INFO:teuthology.orchestra.run.smithi192.stdout:(2/19): slirp4netns-1.1.8-1.module+el8.6.0+1487 239 kB/s | 51 kB 00:00 2023-11-28T00:58:09.609 INFO:teuthology.orchestra.run.smithi192.stdout:(3/19): container-selinux-2.178.0-2.module+el8. 221 kB/s | 53 kB 00:00 2023-11-28T00:58:09.742 INFO:teuthology.orchestra.run.smithi192.stdout:(4/19): criu-3.15-1.module+el8.6.0+14874+644362 2.7 MB/s | 511 kB 00:00 2023-11-28T00:58:09.761 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T00:58:09.768 INFO:teuthology.orchestra.run.smithi192.stdout:(5/19): libslirp-4.3.1-1.module+el8.6.0+14874+6 376 kB/s | 69 kB 00:00 2023-11-28T00:58:09.844 INFO:teuthology.orchestra.run.smithi192.stdout:(6/19): runc-1.0.0-73.rc95.module+el8.6.0+14874 12 MB/s | 2.9 MB 00:00 2023-11-28T00:58:09.854 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================================================== 2023-11-28T00:58:09.854 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-11-28T00:58:09.854 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================================================== 2023-11-28T00:58:09.854 INFO:teuthology.orchestra.run.smithi172.stdout:Installing group/module packages: 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout: buildah x86_64 1.19.9-3.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 6.8 M 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout: cockpit-podman noarch 29-2.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 1.1 M 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout: crun x86_64 0.18-3.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 185 k 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout: skopeo x86_64 1:1.2.4-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 6.4 M 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout: toolbox x86_64 0.0.99.3-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout: udica noarch 0.2.4-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 51 k 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout:Downgrading: 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout: conmon x86_64 2:2.0.26-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 51 k 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout: container-selinux noarch 2:2.178.0-2.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 53 k 2023-11-28T00:58:09.855 INFO:teuthology.orchestra.run.smithi172.stdout: containernetworking-plugins x86_64 0.9.1-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 19 M 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout: containers-common x86_64 1:1.2.4-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 91 k 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout: criu x86_64 3.15-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 511 k 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout: fuse-overlayfs x86_64 1.4.0-2.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 72 k 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout: libslirp x86_64 4.3.1-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 69 k 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout: podman x86_64 3.0.1-9.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 11 M 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout: podman-catatonit x86_64 3.0.1-9.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 322 k 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout: podman-docker noarch 3.0.1-9.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 56 k 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout: runc x86_64 1.0.0-73.rc95.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 2.9 M 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout: slirp4netns x86_64 1.1.8-1.module+el8.6.0+14874+64436299 rhel-8-for-x86_64-appstream-rpms 51 k 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout:Installing module profiles: 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout: container-tools/common 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout:Enabling module streams: 2023-11-28T00:58:09.856 INFO:teuthology.orchestra.run.smithi172.stdout: container-tools 3.0 2023-11-28T00:58:09.857 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T00:58:09.857 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T00:58:09.857 INFO:teuthology.orchestra.run.smithi172.stdout:========================================================================================================================== 2023-11-28T00:58:09.857 INFO:teuthology.orchestra.run.smithi172.stdout:Install 7 Packages 2023-11-28T00:58:09.857 INFO:teuthology.orchestra.run.smithi172.stdout:Downgrade 12 Packages 2023-11-28T00:58:09.857 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T00:58:09.857 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 51 M 2023-11-28T00:58:09.857 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T00:58:09.860 INFO:teuthology.orchestra.run.smithi192.stdout:(7/19): fuse-overlayfs-1.4.0-2.module+el8.6.0+1 608 kB/s | 72 kB 00:00 2023-11-28T00:58:09.996 INFO:teuthology.orchestra.run.smithi172.stdout:(1/19): containers-common-1.2.4-1.module+el8.6. 660 kB/s | 91 kB 00:00 2023-11-28T00:58:10.027 INFO:teuthology.orchestra.run.smithi192.stdout:(8/19): conmon-2.0.26-1.module+el8.6.0+14874+64 306 kB/s | 51 kB 00:00 2023-11-28T00:58:10.030 INFO:teuthology.orchestra.run.smithi172.stdout:(2/19): slirp4netns-1.1.8-1.module+el8.6.0+1487 299 kB/s | 51 kB 00:00 2023-11-28T00:58:10.078 INFO:teuthology.orchestra.run.smithi192.stdout:(9/19): podman-docker-3.0.1-9.module+el8.6.0+14 239 kB/s | 56 kB 00:00 2023-11-28T00:58:10.088 INFO:teuthology.orchestra.run.smithi172.stdout:(3/19): container-selinux-2.178.0-2.module+el8. 230 kB/s | 53 kB 00:00 2023-11-28T00:58:10.245 INFO:teuthology.orchestra.run.smithi192.stdout:(10/19): podman-3.0.1-9.module+el8.6.0+14874+64 23 MB/s | 11 MB 00:00 2023-11-28T00:58:10.247 INFO:teuthology.orchestra.run.smithi172.stdout:(4/19): runc-1.0.0-73.rc95.module+el8.6.0+14874 18 MB/s | 2.9 MB 00:00 2023-11-28T00:58:10.270 INFO:teuthology.orchestra.run.smithi192.stdout:(11/19): podman-catatonit-3.0.1-9.module+el8.6. 1.6 MB/s | 322 kB 00:00 2023-11-28T00:58:10.289 INFO:teuthology.orchestra.run.smithi172.stdout:(5/19): criu-3.15-1.module+el8.6.0+14874+644362 1.7 MB/s | 511 kB 00:00 2023-11-28T00:58:10.345 INFO:teuthology.orchestra.run.smithi192.stdout:(12/19): yajl-2.1.0-10.el8.x86_64.rpm 407 kB/s | 41 kB 00:00 2023-11-28T00:58:10.364 INFO:teuthology.orchestra.run.smithi172.stdout:(6/19): libslirp-4.3.1-1.module+el8.6.0+14874+6 207 kB/s | 69 kB 00:00 2023-11-28T00:58:10.406 INFO:teuthology.orchestra.run.smithi172.stdout:(7/19): fuse-overlayfs-1.4.0-2.module+el8.6.0+1 453 kB/s | 72 kB 00:00 2023-11-28T00:58:10.504 INFO:teuthology.orchestra.run.smithi192.stdout:(13/19): toolbox-0.0.99.3-1.module+el8.6.0+1487 9.4 MB/s | 2.2 MB 00:00 2023-11-28T00:58:10.531 INFO:teuthology.orchestra.run.smithi172.stdout:(8/19): conmon-2.0.26-1.module+el8.6.0+14874+64 408 kB/s | 51 kB 00:00 2023-11-28T00:58:10.550 INFO:teuthology.orchestra.run.smithi172.stdout:(9/19): podman-docker-3.0.1-9.module+el8.6.0+14 304 kB/s | 56 kB 00:00 2023-11-28T00:58:10.638 INFO:teuthology.orchestra.run.smithi192.stdout:(14/19): skopeo-1.2.4-1.module+el8.6.0+14874+64 22 MB/s | 6.4 MB 00:00 2023-11-28T00:58:10.682 INFO:teuthology.orchestra.run.smithi172.stdout:(10/19): podman-3.0.1-9.module+el8.6.0+14874+64 28 MB/s | 11 MB 00:00 2023-11-28T00:58:10.716 INFO:teuthology.orchestra.run.smithi172.stdout:(11/19): podman-catatonit-3.0.1-9.module+el8.6. 1.9 MB/s | 322 kB 00:00 2023-11-28T00:58:10.808 INFO:teuthology.orchestra.run.smithi172.stdout:(12/19): yajl-2.1.0-10.el8.x86_64.rpm 325 kB/s | 41 kB 00:00 2023-11-28T00:58:10.917 INFO:teuthology.orchestra.run.smithi172.stdout:(13/19): toolbox-0.0.99.3-1.module+el8.6.0+1487 11 MB/s | 2.2 MB 00:00 2023-11-28T00:58:10.971 INFO:teuthology.orchestra.run.smithi192.stdout:(15/19): containernetworking-plugins-0.9.1-1.mo 20 MB/s | 19 MB 00:00 2023-11-28T00:58:11.055 INFO:teuthology.orchestra.run.smithi192.stdout:(16/19): buildah-1.19.9-3.module+el8.6.0+14874+ 12 MB/s | 6.8 MB 00:00 2023-11-28T00:58:11.097 INFO:teuthology.orchestra.run.smithi192.stdout:(17/19): cockpit-podman-29-2.module+el8.6.0+148 2.3 MB/s | 1.1 MB 00:00 2023-11-28T00:58:11.122 INFO:teuthology.orchestra.run.smithi192.stdout:(18/19): crun-0.18-3.module+el8.6.0+14874+64436 1.2 MB/s | 185 kB 00:00 2023-11-28T00:58:11.133 INFO:teuthology.orchestra.run.smithi172.stdout:(14/19): containernetworking-plugins-0.9.1-1.mo 31 MB/s | 19 MB 00:00 2023-11-28T00:58:11.230 INFO:teuthology.orchestra.run.smithi192.stdout:(19/19): udica-0.2.4-1.module+el8.6.0+14874+644 288 kB/s | 51 kB 00:00 2023-11-28T00:58:11.231 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T00:58:11.231 INFO:teuthology.orchestra.run.smithi192.stdout:Total 27 MB/s | 51 MB 00:01 2023-11-28T00:58:11.275 INFO:teuthology.orchestra.run.smithi172.stdout:(15/19): skopeo-1.2.4-1.module+el8.6.0+14874+64 14 MB/s | 6.4 MB 00:00 2023-11-28T00:58:11.317 INFO:teuthology.orchestra.run.smithi172.stdout:(16/19): cockpit-podman-29-2.module+el8.6.0+148 5.8 MB/s | 1.1 MB 00:00 2023-11-28T00:58:11.426 INFO:teuthology.orchestra.run.smithi172.stdout:(17/19): buildah-1.19.9-3.module+el8.6.0+14874+ 13 MB/s | 6.8 MB 00:00 2023-11-28T00:58:11.451 INFO:teuthology.orchestra.run.smithi172.stdout:(18/19): udica-0.2.4-1.module+el8.6.0+14874+644 379 kB/s | 51 kB 00:00 2023-11-28T00:58:11.476 INFO:teuthology.orchestra.run.smithi172.stdout:(19/19): crun-0.18-3.module+el8.6.0+14874+64436 922 kB/s | 185 kB 00:00 2023-11-28T00:58:11.477 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T00:58:11.477 INFO:teuthology.orchestra.run.smithi172.stdout:Total 31 MB/s | 51 MB 00:01 2023-11-28T00:58:11.710 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T00:58:11.751 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T00:58:11.751 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T00:58:11.953 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T00:58:11.997 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T00:58:11.997 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T00:58:12.185 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T00:58:12.189 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T00:58:12.418 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T00:58:12.422 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T00:58:13.397 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T00:58:13.450 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: fuse-overlayfs-1.4.0-2.module+el8.6.0+14874+64436299 1/1 2023-11-28T00:58:13.545 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T00:58:13.593 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: fuse-overlayfs-1.4.0-2.module+el8.6.0+14874+64436299 1/1 2023-11-28T00:58:13.640 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : fuse-overlayfs-1.4.0-2.module+el8.6.0+14874+644362 1/31 2023-11-28T00:58:13.649 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: fuse-overlayfs-1.4.0-2.module+el8.6.0+14874+644362 1/31 2023-11-28T00:58:13.668 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: container-selinux-2:2.178.0-2.module+el8.6.0+14874 2/31 2023-11-28T00:58:13.764 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : fuse-overlayfs-1.4.0-2.module+el8.6.0+14874+644362 1/31 2023-11-28T00:58:13.773 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: fuse-overlayfs-1.4.0-2.module+el8.6.0+14874+644362 1/31 2023-11-28T00:58:13.795 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: container-selinux-2:2.178.0-2.module+el8.6.0+14874 2/31 2023-11-28T00:58:13.840 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : container-selinux-2:2.178.0-2.module+el8.6.0+14874 2/31 2023-11-28T00:58:13.956 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : container-selinux-2:2.178.0-2.module+el8.6.0+14874 2/31 2023-11-28T00:58:31.206 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: container-selinux-2:2.178.0-2.module+el8.6.0+14874 2/31 2023-11-28T00:58:31.791 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: container-selinux-2:2.178.0-2.module+el8.6.0+14874 2/31 2023-11-28T00:58:32.991 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : yajl-2.1.0-10.el8.x86_64 3/31 2023-11-28T00:58:33.249 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : containernetworking-plugins-0.9.1-1.module+el8.6.0 4/31 2023-11-28T00:58:33.478 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : conmon-2:2.0.26-1.module+el8.6.0+14874+64436299.x8 5/31 2023-11-28T00:58:33.563 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : yajl-2.1.0-10.el8.x86_64 3/31 2023-11-28T00:58:33.652 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : libslirp-4.3.1-1.module+el8.6.0+14874+64436299.x86 6/31 2023-11-28T00:58:33.777 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : containernetworking-plugins-0.9.1-1.module+el8.6.0 4/31 2023-11-28T00:58:33.880 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : slirp4netns-1.1.8-1.module+el8.6.0+14874+64436299. 7/31 2023-11-28T00:58:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : containers-common-1:1.2.4-1.module+el8.6.0+14874+6 8/31 2023-11-28T00:58:33.881 INFO:teuthology.orchestra.run.smithi192.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2023-11-28T00:58:33.881 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T00:58:33.986 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : conmon-2:2.0.26-1.module+el8.6.0+14874+64436299.x8 5/31 2023-11-28T00:58:34.184 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : libslirp-4.3.1-1.module+el8.6.0+14874+64436299.x86 6/31 2023-11-28T00:58:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : slirp4netns-1.1.8-1.module+el8.6.0+14874+64436299. 7/31 2023-11-28T00:58:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : containers-common-1:1.2.4-1.module+el8.6.0+14874+6 8/31 2023-11-28T00:58:34.402 INFO:teuthology.orchestra.run.smithi172.stdout:warning: /etc/containers/registries.conf created as /etc/containers/registries.conf.rpmnew 2023-11-28T00:58:34.402 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T00:58:34.684 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : criu-3.15-1.module+el8.6.0+14874+64436299.x86_64 9/31 2023-11-28T00:58:34.912 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : runc-1.0.0-73.rc95.module+el8.6.0+14874+64436299.x 10/31 2023-11-28T00:58:35.224 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : criu-3.15-1.module+el8.6.0+14874+64436299.x86_64 9/31 2023-11-28T00:58:35.452 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : runc-1.0.0-73.rc95.module+el8.6.0+14874+64436299.x 10/31 2023-11-28T00:58:36.415 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : podman-catatonit-3.0.1-9.module+el8.6.0+14874+6443 11/31 2023-11-28T00:58:36.725 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : podman-3.0.1-9.module+el8.6.0+14874+64436299.x86_6 12/31 2023-11-28T00:58:36.908 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : podman-catatonit-3.0.1-9.module+el8.6.0+14874+6443 11/31 2023-11-28T00:58:37.105 INFO:teuthology.orchestra.run.smithi192.stdout: Downgrading : podman-docker-3.0.1-9.module+el8.6.0+14874+6443629 13/31 2023-11-28T00:58:37.214 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : podman-3.0.1-9.module+el8.6.0+14874+64436299.x86_6 12/31 2023-11-28T00:58:37.362 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : toolbox-0.0.99.3-1.module+el8.6.0+14874+64436299.x 14/31 2023-11-28T00:58:37.586 INFO:teuthology.orchestra.run.smithi172.stdout: Downgrading : podman-docker-3.0.1-9.module+el8.6.0+14874+6443629 13/31 2023-11-28T00:58:37.858 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : toolbox-0.0.99.3-1.module+el8.6.0+14874+64436299.x 14/31 2023-11-28T00:58:38.104 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cockpit-podman-29-2.module+el8.6.0+14874+64436299. 15/31 2023-11-28T00:58:38.586 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cockpit-podman-29-2.module+el8.6.0+14874+64436299. 15/31 2023-11-28T00:58:38.828 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : buildah-1.19.9-3.module+el8.6.0+14874+64436299.x86 16/31 2023-11-28T00:58:39.069 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : skopeo-1:1.2.4-1.module+el8.6.0+14874+64436299.x86 17/31 2023-11-28T00:58:39.291 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : buildah-1.19.9-3.module+el8.6.0+14874+64436299.x86 16/31 2023-11-28T00:58:39.307 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : crun-0.18-3.module+el8.6.0+14874+64436299.x86_64 18/31 2023-11-28T00:58:39.503 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : udica-0.2.4-1.module+el8.6.0+14874+64436299.noarch 19/31 2023-11-28T00:58:39.518 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : skopeo-1:1.2.4-1.module+el8.6.0+14874+64436299.x86 17/31 2023-11-28T00:58:39.670 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : podman-docker-2:4.1.1-7.module+el8.6.0+16889+9f154 20/31 2023-11-28T00:58:39.670 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : podman-2:4.1.1-7.module+el8.6.0+16889+9f154fa9.x86 21/31 2023-11-28T00:58:39.725 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : crun-0.18-3.module+el8.6.0+14874+64436299.x86_64 18/31 2023-11-28T00:58:39.918 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : udica-0.2.4-1.module+el8.6.0+14874+64436299.noarch 19/31 2023-11-28T00:58:39.973 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: podman-2:4.1.1-7.module+el8.6.0+16889+9f154fa9.x86 21/31 2023-11-28T00:58:40.032 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : podman-docker-2:4.1.1-7.module+el8.6.0+16889+9f154 20/31 2023-11-28T00:58:40.033 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : podman-2:4.1.1-7.module+el8.6.0+16889+9f154fa9.x86 21/31 2023-11-28T00:58:40.189 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : containers-common-2:1-27.module+el8.6.0+14877+f643 22/31 2023-11-28T00:58:40.189 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/31 2023-11-28T00:58:40.336 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: podman-2:4.1.1-7.module+el8.6.0+16889+9f154fa9.x86 21/31 2023-11-28T00:58:40.412 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/31 2023-11-28T00:58:40.527 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : containers-common-2:1-27.module+el8.6.0+14877+f643 22/31 2023-11-28T00:58:40.527 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/31 2023-11-28T00:58:40.553 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : podman-catatonit-2:4.1.1-7.module+el8.6.0+16889+9f 24/31 2023-11-28T00:58:40.696 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 25/31 2023-11-28T00:58:40.719 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: container-selinux-2:2.179.1-1.module+el8.6.0+14877 23/31 2023-11-28T00:58:40.827 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : podman-catatonit-2:4.1.1-7.module+el8.6.0+16889+9f 24/31 2023-11-28T00:58:40.870 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 26/31 2023-11-28T00:58:40.977 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 25/31 2023-11-28T00:58:41.039 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 27/31 2023-11-28T00:58:41.111 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 26/31 2023-11-28T00:58:41.221 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 28/31 2023-11-28T00:58:41.277 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 27/31 2023-11-28T00:58:41.353 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 29/31 2023-11-28T00:58:41.402 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 28/31 2023-11-28T00:58:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 30/31 2023-11-28T00:58:41.522 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : containernetworking-plugins-1:1.0.1-2.module+el8.6 31/31 2023-11-28T00:58:41.535 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 29/31 2023-11-28T00:58:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 30/31 2023-11-28T00:58:41.681 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : containernetworking-plugins-1:1.0.1-2.module+el8.6 31/31 2023-11-28T00:58:41.728 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: containernetworking-plugins-1:1.0.1-2.module+el8.6 31/31 2023-11-28T00:58:41.868 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: containernetworking-plugins-1:1.0.1-2.module+el8.6 31/31 2023-11-28T00:58:42.255 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: container-selinux-2:2.178.0-2.module+el8.6.0+14874 31/31 2023-11-28T00:58:42.396 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: container-selinux-2:2.178.0-2.module+el8.6.0+14874 31/31 2023-11-28T00:58:42.662 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: containernetworking-plugins-1:1.0.1-2.module+el8.6 31/31 2023-11-28T00:58:42.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : containers-common-1:1.2.4-1.module+el8.6.0+14874+6 1/31 2023-11-28T00:58:42.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : containers-common-2:1-27.module+el8.6.0+14877+f643 2/31 2023-11-28T00:58:42.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : container-selinux-2:2.178.0-2.module+el8.6.0+14874 3/31 2023-11-28T00:58:42.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : container-selinux-2:2.179.1-1.module+el8.6.0+14877 4/31 2023-11-28T00:58:42.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : slirp4netns-1.1.8-1.module+el8.6.0+14874+64436299. 5/31 2023-11-28T00:58:42.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 6/31 2023-11-28T00:58:42.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : criu-3.15-1.module+el8.6.0+14874+64436299.x86_64 7/31 2023-11-28T00:58:42.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 8/31 2023-11-28T00:58:42.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libslirp-4.3.1-1.module+el8.6.0+14874+64436299.x86 9/31 2023-11-28T00:58:42.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 10/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : runc-1.0.0-73.rc95.module+el8.6.0+14874+64436299.x 11/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 12/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-overlayfs-1.4.0-2.module+el8.6.0+14874+644362 13/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 14/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : podman-3.0.1-9.module+el8.6.0+14874+64436299.x86_6 15/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : podman-2:4.1.1-7.module+el8.6.0+16889+9f154fa9.x86 16/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : podman-docker-3.0.1-9.module+el8.6.0+14874+6443629 17/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : podman-docker-2:4.1.1-7.module+el8.6.0+16889+9f154 18/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : conmon-2:2.0.26-1.module+el8.6.0+14874+64436299.x8 19/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 20/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : containernetworking-plugins-0.9.1-1.module+el8.6.0 21/31 2023-11-28T00:58:42.664 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : containernetworking-plugins-1:1.0.1-2.module+el8.6 22/31 2023-11-28T00:58:42.665 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : podman-catatonit-3.0.1-9.module+el8.6.0+14874+6443 23/31 2023-11-28T00:58:42.665 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : podman-catatonit-2:4.1.1-7.module+el8.6.0+16889+9f 24/31 2023-11-28T00:58:42.665 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 25/31 2023-11-28T00:58:42.665 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : toolbox-0.0.99.3-1.module+el8.6.0+14874+64436299.x 26/31 2023-11-28T00:58:42.666 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : skopeo-1:1.2.4-1.module+el8.6.0+14874+64436299.x86 27/31 2023-11-28T00:58:42.666 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : buildah-1.19.9-3.module+el8.6.0+14874+64436299.x86 28/31 2023-11-28T00:58:42.666 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cockpit-podman-29-2.module+el8.6.0+14874+64436299. 29/31 2023-11-28T00:58:42.666 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : crun-0.18-3.module+el8.6.0+14874+64436299.x86_64 30/31 2023-11-28T00:58:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: containernetworking-plugins-1:1.0.1-2.module+el8.6 31/31 2023-11-28T00:58:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : containers-common-1:1.2.4-1.module+el8.6.0+14874+6 1/31 2023-11-28T00:58:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : containers-common-2:1-27.module+el8.6.0+14877+f643 2/31 2023-11-28T00:58:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : container-selinux-2:2.178.0-2.module+el8.6.0+14874 3/31 2023-11-28T00:58:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : container-selinux-2:2.179.1-1.module+el8.6.0+14877 4/31 2023-11-28T00:58:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : slirp4netns-1.1.8-1.module+el8.6.0+14874+64436299. 5/31 2023-11-28T00:58:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : slirp4netns-1.1.8-2.module+el8.6.0+14877+f643d2d6. 6/31 2023-11-28T00:58:42.775 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : criu-3.15-1.module+el8.6.0+14874+64436299.x86_64 7/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : criu-3.15-3.module+el8.6.0+14877+f643d2d6.x86_64 8/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libslirp-4.3.1-1.module+el8.6.0+14874+64436299.x86 9/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libslirp-4.4.0-1.module+el8.6.0+14877+f643d2d6.x86 10/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : runc-1.0.0-73.rc95.module+el8.6.0+14874+64436299.x 11/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : runc-1:1.0.3-2.module+el8.6.0+14877+f643d2d6.x86_6 12/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-overlayfs-1.4.0-2.module+el8.6.0+14874+644362 13/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-overlayfs-1.8.2-1.module+el8.6.0+14877+f643d2 14/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : podman-3.0.1-9.module+el8.6.0+14874+64436299.x86_6 15/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : podman-2:4.1.1-7.module+el8.6.0+16889+9f154fa9.x86 16/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : podman-docker-3.0.1-9.module+el8.6.0+14874+6443629 17/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : podman-docker-2:4.1.1-7.module+el8.6.0+16889+9f154 18/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : conmon-2:2.0.26-1.module+el8.6.0+14874+64436299.x8 19/31 2023-11-28T00:58:42.776 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : conmon-2:2.1.0-1.module+el8.6.0+14877+f643d2d6.x86 20/31 2023-11-28T00:58:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : containernetworking-plugins-0.9.1-1.module+el8.6.0 21/31 2023-11-28T00:58:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : containernetworking-plugins-1:1.0.1-2.module+el8.6 22/31 2023-11-28T00:58:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : podman-catatonit-3.0.1-9.module+el8.6.0+14874+6443 23/31 2023-11-28T00:58:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : podman-catatonit-2:4.1.1-7.module+el8.6.0+16889+9f 24/31 2023-11-28T00:58:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 25/31 2023-11-28T00:58:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : toolbox-0.0.99.3-1.module+el8.6.0+14874+64436299.x 26/31 2023-11-28T00:58:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : skopeo-1:1.2.4-1.module+el8.6.0+14874+64436299.x86 27/31 2023-11-28T00:58:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : buildah-1.19.9-3.module+el8.6.0+14874+64436299.x86 28/31 2023-11-28T00:58:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cockpit-podman-29-2.module+el8.6.0+14874+64436299. 29/31 2023-11-28T00:58:42.777 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : crun-0.18-3.module+el8.6.0+14874+64436299.x86_64 30/31 2023-11-28T00:58:44.141 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : udica-0.2.4-1.module+el8.6.0+14874+64436299.noarch 31/31 2023-11-28T00:58:44.141 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:18 ago on Tue 28 Nov 2023 12:56:25 AM UTC. 2023-11-28T00:58:44.256 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : udica-0.2.4-1.module+el8.6.0+14874+64436299.noarch 31/31 2023-11-28T00:58:44.256 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:20 ago on Tue 28 Nov 2023 12:56:23 AM UTC. 2023-11-28T00:58:51.377 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T00:58:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T00:58:51.570 INFO:teuthology.orchestra.run.smithi172.stdout:Downgraded: 2023-11-28T00:58:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: conmon-2:2.0.26-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: container-selinux-2:2.178.0-2.module+el8.6.0+14874+64436299.noarch 2023-11-28T00:58:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: containernetworking-plugins-0.9.1-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: containers-common-1:1.2.4-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: criu-3.15-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.570 INFO:teuthology.orchestra.run.smithi172.stdout: fuse-overlayfs-1.4.0-2.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: libslirp-4.3.1-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: podman-3.0.1-9.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: podman-catatonit-3.0.1-9.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: podman-docker-3.0.1-9.module+el8.6.0+14874+64436299.noarch 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: runc-1.0.0-73.rc95.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: slirp4netns-1.1.8-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: buildah-1.19.9-3.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: cockpit-podman-29-2.module+el8.6.0+14874+64436299.noarch 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: crun-0.18-3.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: skopeo-1:1.2.4-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.571 INFO:teuthology.orchestra.run.smithi172.stdout: toolbox-0.0.99.3-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: udica-0.2.4-1.module+el8.6.0+14874+64436299.noarch 2023-11-28T00:58:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-28T00:58:51.572 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T00:58:51.572 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T00:58:51.755 DEBUG:teuthology.parallel:result is None 2023-11-28T00:58:53.279 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T00:58:53.472 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T00:58:53.473 INFO:teuthology.orchestra.run.smithi192.stdout:Downgraded: 2023-11-28T00:58:53.473 INFO:teuthology.orchestra.run.smithi192.stdout: conmon-2:2.0.26-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.473 INFO:teuthology.orchestra.run.smithi192.stdout: container-selinux-2:2.178.0-2.module+el8.6.0+14874+64436299.noarch 2023-11-28T00:58:53.473 INFO:teuthology.orchestra.run.smithi192.stdout: containernetworking-plugins-0.9.1-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.473 INFO:teuthology.orchestra.run.smithi192.stdout: containers-common-1:1.2.4-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.473 INFO:teuthology.orchestra.run.smithi192.stdout: criu-3.15-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.473 INFO:teuthology.orchestra.run.smithi192.stdout: fuse-overlayfs-1.4.0-2.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.473 INFO:teuthology.orchestra.run.smithi192.stdout: libslirp-4.3.1-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.473 INFO:teuthology.orchestra.run.smithi192.stdout: podman-3.0.1-9.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.473 INFO:teuthology.orchestra.run.smithi192.stdout: podman-catatonit-3.0.1-9.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.473 INFO:teuthology.orchestra.run.smithi192.stdout: podman-docker-3.0.1-9.module+el8.6.0+14874+64436299.noarch 2023-11-28T00:58:53.474 INFO:teuthology.orchestra.run.smithi192.stdout: runc-1.0.0-73.rc95.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.474 INFO:teuthology.orchestra.run.smithi192.stdout: slirp4netns-1.1.8-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.474 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T00:58:53.474 INFO:teuthology.orchestra.run.smithi192.stdout: buildah-1.19.9-3.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.474 INFO:teuthology.orchestra.run.smithi192.stdout: cockpit-podman-29-2.module+el8.6.0+14874+64436299.noarch 2023-11-28T00:58:53.474 INFO:teuthology.orchestra.run.smithi192.stdout: crun-0.18-3.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.474 INFO:teuthology.orchestra.run.smithi192.stdout: skopeo-1:1.2.4-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.474 INFO:teuthology.orchestra.run.smithi192.stdout: toolbox-0.0.99.3-1.module+el8.6.0+14874+64436299.x86_64 2023-11-28T00:58:53.474 INFO:teuthology.orchestra.run.smithi192.stdout: udica-0.2.4-1.module+el8.6.0+14874+64436299.noarch 2023-11-28T00:58:53.474 INFO:teuthology.orchestra.run.smithi192.stdout: yajl-2.1.0-10.el8.x86_64 2023-11-28T00:58:53.475 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T00:58:53.475 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T00:58:53.640 DEBUG:teuthology.parallel:result is None 2023-11-28T00:58:53.640 INFO:teuthology.run_tasks:Running task install... 2023-11-28T00:58:53.651 DEBUG:teuthology.task.install:project ceph 2023-11-28T00:58:53.651 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '369173db14b6995b2bd07c60ec5f63d01cf21631'}, 'extra_packages': ['cephadm']} 2023-11-28T00:58:53.651 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '369173db14b6995b2bd07c60ec5f63d01cf21631'} 2023-11-28T00:58:53.651 INFO:teuthology.task.install:Using flavor: default 2023-11-28T00:58:53.659 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-28T00:58:53.659 INFO:teuthology.task.install:extra packages: [] 2023-11-28T00:58:53.659 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '369173db14b6995b2bd07c60ec5f63d01cf21631', 'tag': None, 'wait_for_package': False} 2023-11-28T00:58:53.660 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=369173db14b6995b2bd07c60ec5f63d01cf21631 2023-11-28T00:58:53.662 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '369173db14b6995b2bd07c60ec5f63d01cf21631', 'tag': None, 'wait_for_package': False} 2023-11-28T00:58:53.663 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=369173db14b6995b2bd07c60ec5f63d01cf21631 2023-11-28T00:58:53.808 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-11-22-1112/369173db14b6995b2bd07c60ec5f63d01cf21631/centos/8/flavors/default/ 2023-11-28T00:58:53.808 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7498.g369173db 2023-11-28T00:58:53.883 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-11-22-1112/369173db14b6995b2bd07c60ec5f63d01cf21631/centos/8/flavors/default/ 2023-11-28T00:58:53.883 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7498.g369173db 2023-11-28T00:58:53.974 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-11-22-1112/369173db14b6995b2bd07c60ec5f63d01cf21631/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-11-22-1112/369173db14b6995b2bd07c60ec5f63d01cf21631/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-11-22-1112/369173db14b6995b2bd07c60ec5f63d01cf21631/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T00:58:53.974 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T00:58:53.975 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T00:58:53.987 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-11-22-1112/369173db14b6995b2bd07c60ec5f63d01cf21631/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-11-22-1112/369173db14b6995b2bd07c60ec5f63d01cf21631/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-11-22-1112/369173db14b6995b2bd07c60ec5f63d01cf21631/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-28T00:58:53.988 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T00:58:53.988 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-28T00:58:54.003 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-28T00:58:54.004 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/369173db14b6995b2bd07c60ec5f63d01cf21631/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T00:58:54.022 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-28T00:58:54.022 DEBUG:teuthology.orchestra.run.smithi172:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/369173db14b6995b2bd07c60ec5f63d01cf21631/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-28T00:58:54.070 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T00:58:54.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-28T00:58:54.148 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T00:58:54.175 DEBUG:teuthology.orchestra.run.smithi172:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-28T00:58:54.214 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2023-11-28T00:58:54.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2023-11-28T00:58:54.243 INFO:teuthology.orchestra.run.smithi172.stdout:check_obsoletes = 1 2023-11-28T00:58:54.244 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum clean all 2023-11-28T00:58:54.589 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T00:58:54.618 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T00:58:55.788 INFO:teuthology.orchestra.run.smithi172.stdout:44 files removed 2023-11-28T00:58:55.822 INFO:teuthology.orchestra.run.smithi192.stdout:44 files removed 2023-11-28T00:58:55.841 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-radosgw 2023-11-28T00:58:55.876 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw 2023-11-28T00:58:56.176 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T00:58:56.211 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T00:58:57.664 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 327 kB/s | 81 kB 00:00 2023-11-28T00:58:57.778 INFO:teuthology.orchestra.run.smithi172.stdout:ceph packages for x86_64 224 kB/s | 81 kB 00:00 2023-11-28T00:58:57.965 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 35 kB/s | 9.5 kB 00:00 2023-11-28T00:58:58.112 INFO:teuthology.orchestra.run.smithi172.stdout:ceph noarch packages 32 kB/s | 9.5 kB 00:00 2023-11-28T00:58:58.282 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 5.1 kB/s | 1.5 kB 00:00 2023-11-28T00:58:58.421 INFO:teuthology.orchestra.run.smithi172.stdout:ceph source packages 5.2 kB/s | 1.5 kB 00:00 2023-11-28T00:58:58.441 INFO:teuthology.orchestra.run.smithi192.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-11-28T00:58:58.571 INFO:teuthology.orchestra.run.smithi172.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-11-28T00:59:00.089 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-11-28T00:59:00.611 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 16 MB 00:02 2023-11-28T00:59:03.706 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 501 kB/s | 24 kB 00:00 2023-11-28T00:59:04.125 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 507 kB/s | 24 kB 00:00 2023-11-28T00:59:04.816 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 51 MB/s | 53 MB 00:01 2023-11-28T00:59:05.252 INFO:teuthology.orchestra.run.smithi192.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 50 MB/s | 53 MB 00:01 2023-11-28T00:59:13.654 INFO:teuthology.orchestra.run.smithi192.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 MB/s | 47 MB 00:01 2023-11-28T00:59:13.709 INFO:teuthology.orchestra.run.smithi172.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 36 MB/s | 47 MB 00:01 2023-11-28T00:59:24.615 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T00:59:24.617 INFO:teuthology.orchestra.run.smithi192.stdout:==================================================================================================== 2023-11-28T00:59:24.617 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-11-28T00:59:24.617 INFO:teuthology.orchestra.run.smithi192.stdout:==================================================================================================== 2023-11-28T00:59:24.617 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T00:59:24.617 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:18.0.0-7498.g369173db.el8 ceph 14 M 2023-11-28T00:59:24.617 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2023-11-28T00:59:24.618 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:18.0.0-7498.g369173db.el8 ceph 3.8 M 2023-11-28T00:59:24.618 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:18.0.0-7498.g369173db.el8 ceph 3.7 M 2023-11-28T00:59:24.618 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-11-28T00:59:24.618 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:18.0.0-7498.g369173db.el8 ceph 5.4 M 2023-11-28T00:59:24.618 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:18.0.0-7498.g369173db.el8 ceph 23 M 2023-11-28T00:59:24.618 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:18.0.0-7498.g369173db.el8 ceph 24 k 2023-11-28T00:59:24.618 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-28T00:59:24.618 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:18.0.0-7498.g369173db.el8 ceph 839 k 2023-11-28T00:59:24.618 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-28T00:59:24.618 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-28T00:59:24.618 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:18.0.0-7498.g369173db.el8 ceph 517 k 2023-11-28T00:59:24.619 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-28T00:59:24.619 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:18.0.0-7498.g369173db.el8 ceph 6.9 M 2023-11-28T00:59:24.619 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-28T00:59:24.619 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-28T00:59:24.619 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:18.0.0-7498.g369173db.el8 ceph 45 k 2023-11-28T00:59:24.619 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:18.0.0-7498.g369173db.el8 ceph 128 k 2023-11-28T00:59:24.619 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:18.0.0-7498.g369173db.el8 ceph 197 k 2023-11-28T00:59:24.619 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-28T00:59:24.619 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:18.0.0-7498.g369173db.el8 ceph 371 k 2023-11-28T00:59:24.619 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:18.0.0-7498.g369173db.el8 ceph 364 k 2023-11-28T00:59:24.619 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:18.0.0-7498.g369173db.el8 ceph 107 k 2023-11-28T00:59:24.620 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-28T00:59:24.620 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T00:59:24.620 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T00:59:24.620 INFO:teuthology.orchestra.run.smithi192.stdout:==================================================================================================== 2023-11-28T00:59:24.620 INFO:teuthology.orchestra.run.smithi192.stdout:Install 21 Packages 2023-11-28T00:59:24.620 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 2 Packages 2023-11-28T00:59:24.620 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T00:59:24.621 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 62 M 2023-11-28T00:59:24.621 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T00:59:25.095 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T00:59:25.098 INFO:teuthology.orchestra.run.smithi172.stdout:==================================================================================================== 2023-11-28T00:59:25.098 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-11-28T00:59:25.098 INFO:teuthology.orchestra.run.smithi172.stdout:==================================================================================================== 2023-11-28T00:59:25.098 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T00:59:25.098 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw x86_64 2:18.0.0-7498.g369173db.el8 ceph 14 M 2023-11-28T00:59:25.098 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrading: 2023-11-28T00:59:25.098 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 x86_64 2:18.0.0-7498.g369173db.el8 ceph 3.8 M 2023-11-28T00:59:25.098 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1 x86_64 2:18.0.0-7498.g369173db.el8 ceph 3.7 M 2023-11-28T00:59:25.098 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base x86_64 2:18.0.0-7498.g369173db.el8 ceph 5.4 M 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common x86_64 2:18.0.0-7498.g369173db.el8 ceph 23 M 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux x86_64 2:18.0.0-7498.g369173db.el8 ceph 24 k 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2 x86_64 2:18.0.0-7498.g369173db.el8 ceph 839 k 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1 x86_64 2:18.0.0-7498.g369173db.el8 ceph 517 k 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2 x86_64 2:18.0.0-7498.g369173db.el8 ceph 6.9 M 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-28T00:59:25.099 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse x86_64 2:18.0.0-7498.g369173db.el8 ceph 45 k 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common x86_64 2:18.0.0-7498.g369173db.el8 ceph 128 k 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs x86_64 2:18.0.0-7498.g369173db.el8 ceph 197 k 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados x86_64 2:18.0.0-7498.g369173db.el8 ceph 371 k 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd x86_64 2:18.0.0-7498.g369173db.el8 ceph 364 k 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw x86_64 2:18.0.0-7498.g369173db.el8 ceph 107 k 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout:==================================================================================================== 2023-11-28T00:59:25.100 INFO:teuthology.orchestra.run.smithi172.stdout:Install 21 Packages 2023-11-28T00:59:25.101 INFO:teuthology.orchestra.run.smithi172.stdout:Upgrade 2 Packages 2023-11-28T00:59:25.101 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T00:59:25.101 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 62 M 2023-11-28T00:59:25.101 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T00:59:25.537 INFO:teuthology.orchestra.run.smithi192.stdout:(1/23): ceph-base-18.0.0-7498.g369173db.el8.x86 9.4 MB/s | 5.4 MB 00:00 2023-11-28T00:59:25.612 INFO:teuthology.orchestra.run.smithi192.stdout:(2/23): ceph-selinux-18.0.0-7498.g369173db.el8. 325 kB/s | 24 kB 00:00 2023-11-28T00:59:26.050 INFO:teuthology.orchestra.run.smithi172.stdout:(1/23): ceph-base-18.0.0-7498.g369173db.el8.x86 6.1 MB/s | 5.4 MB 00:00 2023-11-28T00:59:26.100 INFO:teuthology.orchestra.run.smithi172.stdout:(2/23): ceph-selinux-18.0.0-7498.g369173db.el8. 490 kB/s | 24 kB 00:00 2023-11-28T00:59:26.234 INFO:teuthology.orchestra.run.smithi172.stdout:(3/23): libcephfs2-18.0.0-7498.g369173db.el8.x8 6.1 MB/s | 839 kB 00:00 2023-11-28T00:59:26.317 INFO:teuthology.orchestra.run.smithi172.stdout:(4/23): libradosstriper1-18.0.0-7498.g369173db. 6.1 MB/s | 517 kB 00:00 2023-11-28T00:59:26.571 INFO:teuthology.orchestra.run.smithi192.stdout:(3/23): ceph-radosgw-18.0.0-7498.g369173db.el8. 8.7 MB/s | 14 MB 00:01 2023-11-28T00:59:26.596 INFO:teuthology.orchestra.run.smithi192.stdout:(4/23): libcephfs2-18.0.0-7498.g369173db.el8.x8 854 kB/s | 839 kB 00:00 2023-11-28T00:59:26.738 INFO:teuthology.orchestra.run.smithi192.stdout:(5/23): libradosstriper1-18.0.0-7498.g369173db. 3.0 MB/s | 517 kB 00:00 2023-11-28T00:59:26.846 INFO:teuthology.orchestra.run.smithi192.stdout:(6/23): python3-ceph-argparse-18.0.0-7498.g3691 412 kB/s | 45 kB 00:00 2023-11-28T00:59:26.951 INFO:teuthology.orchestra.run.smithi172.stdout:(5/23): librgw2-18.0.0-7498.g369173db.el8.x86_6 11 MB/s | 6.9 MB 00:00 2023-11-28T00:59:26.953 INFO:teuthology.orchestra.run.smithi192.stdout:(7/23): python3-ceph-common-18.0.0-7498.g369173 1.2 MB/s | 128 kB 00:00 2023-11-28T00:59:27.001 INFO:teuthology.orchestra.run.smithi172.stdout:(6/23): python3-ceph-argparse-18.0.0-7498.g3691 897 kB/s | 45 kB 00:00 2023-11-28T00:59:27.030 INFO:teuthology.orchestra.run.smithi192.stdout:(8/23): python3-cephfs-18.0.0-7498.g369173db.el 2.5 MB/s | 197 kB 00:00 2023-11-28T00:59:27.068 INFO:teuthology.orchestra.run.smithi172.stdout:(7/23): python3-ceph-common-18.0.0-7498.g369173 1.9 MB/s | 128 kB 00:00 2023-11-28T00:59:27.114 INFO:teuthology.orchestra.run.smithi192.stdout:(9/23): python3-rados-18.0.0-7498.g369173db.el8 4.4 MB/s | 371 kB 00:00 2023-11-28T00:59:27.169 INFO:teuthology.orchestra.run.smithi172.stdout:(8/23): python3-cephfs-18.0.0-7498.g369173db.el 1.9 MB/s | 197 kB 00:00 2023-11-28T00:59:27.206 INFO:teuthology.orchestra.run.smithi192.stdout:(10/23): python3-rbd-18.0.0-7498.g369173db.el8. 3.9 MB/s | 364 kB 00:00 2023-11-28T00:59:27.269 INFO:teuthology.orchestra.run.smithi172.stdout:(9/23): python3-rados-18.0.0-7498.g369173db.el8 3.6 MB/s | 371 kB 00:00 2023-11-28T00:59:27.290 INFO:teuthology.orchestra.run.smithi192.stdout:(11/23): python3-rgw-18.0.0-7498.g369173db.el8. 1.3 MB/s | 107 kB 00:00 2023-11-28T00:59:27.352 INFO:teuthology.orchestra.run.smithi172.stdout:(10/23): python3-rbd-18.0.0-7498.g369173db.el8. 4.3 MB/s | 364 kB 00:00 2023-11-28T00:59:27.419 INFO:teuthology.orchestra.run.smithi172.stdout:(11/23): python3-rgw-18.0.0-7498.g369173db.el8. 1.6 MB/s | 107 kB 00:00 2023-11-28T00:59:27.498 INFO:teuthology.orchestra.run.smithi192.stdout:(12/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2023-11-28T00:59:27.649 INFO:teuthology.orchestra.run.smithi192.stdout:(13/23): librgw2-18.0.0-7498.g369173db.el8.x86_ 6.6 MB/s | 6.9 MB 00:01 2023-11-28T00:59:27.780 INFO:teuthology.orchestra.run.smithi192.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 209 kB/s | 59 kB 00:00 2023-11-28T00:59:28.099 INFO:teuthology.orchestra.run.smithi192.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 167 kB/s | 75 kB 00:00 2023-11-28T00:59:28.150 INFO:teuthology.orchestra.run.smithi192.stdout:(16/23): thrift-0.13.0-2.el8.x86_64.rpm 4.7 MB/s | 1.7 MB 00:00 2023-11-28T00:59:28.212 INFO:teuthology.orchestra.run.smithi172.stdout:(12/23): ceph-common-18.0.0-7498.g369173db.el8. 7.4 MB/s | 23 MB 00:03 2023-11-28T00:59:28.309 INFO:teuthology.orchestra.run.smithi192.stdout:(17/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 225 kB/s | 47 kB 00:00 2023-11-28T00:59:28.334 INFO:teuthology.orchestra.run.smithi192.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2023-11-28T00:59:28.426 INFO:teuthology.orchestra.run.smithi192.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 377 kB/s | 44 kB 00:00 2023-11-28T00:59:28.493 INFO:teuthology.orchestra.run.smithi192.stdout:(20/23): librdkafka-0.11.4-3.el8.x86_64.rpm 2.2 MB/s | 354 kB 00:00 2023-11-28T00:59:28.620 INFO:teuthology.orchestra.run.smithi172.stdout:(13/23): ceph-radosgw-18.0.0-7498.g369173db.el8 4.0 MB/s | 14 MB 00:03 2023-11-28T00:59:28.645 INFO:teuthology.orchestra.run.smithi172.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 250 kB/s | 306 kB 00:01 2023-11-28T00:59:28.793 INFO:teuthology.orchestra.run.smithi192.stdout:(21/23): librados2-18.0.0-7498.g369173db.el8.x8 10 MB/s | 3.8 MB 00:00 2023-11-28T00:59:28.876 INFO:teuthology.orchestra.run.smithi192.stdout:(22/23): librbd1-18.0.0-7498.g369173db.el8.x86_ 9.7 MB/s | 3.7 MB 00:00 2023-11-28T00:59:29.079 INFO:teuthology.orchestra.run.smithi172.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 164 kB/s | 75 kB 00:00 2023-11-28T00:59:29.179 INFO:teuthology.orchestra.run.smithi172.stdout:(16/23): liboath-2.6.2-3.el8.x86_64.rpm 61 kB/s | 59 kB 00:00 2023-11-28T00:59:29.263 INFO:teuthology.orchestra.run.smithi172.stdout:(17/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 255 kB/s | 47 kB 00:00 2023-11-28T00:59:29.347 INFO:teuthology.orchestra.run.smithi172.stdout:(18/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2023-11-28T00:59:29.372 INFO:teuthology.orchestra.run.smithi172.stdout:(19/23): python3-prettytable-0.7.2-14.el8.noarc 405 kB/s | 44 kB 00:00 2023-11-28T00:59:29.489 INFO:teuthology.orchestra.run.smithi172.stdout:(20/23): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-11-28T00:59:29.698 INFO:teuthology.orchestra.run.smithi172.stdout:(21/23): librados2-18.0.0-7498.g369173db.el8.x8 12 MB/s | 3.8 MB 00:00 2023-11-28T00:59:29.906 INFO:teuthology.orchestra.run.smithi172.stdout:(22/23): librbd1-18.0.0-7498.g369173db.el8.x86_ 8.9 MB/s | 3.7 MB 00:00 2023-11-28T00:59:30.706 INFO:teuthology.orchestra.run.smithi172.stdout:(23/23): thrift-0.13.0-2.el8.x86_64.rpm 862 kB/s | 1.7 MB 00:02 2023-11-28T00:59:30.708 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T00:59:30.708 INFO:teuthology.orchestra.run.smithi172.stdout:Total 11 MB/s | 62 MB 00:05 2023-11-28T00:59:30.754 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T00:59:30.799 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T00:59:30.799 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T00:59:31.135 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T00:59:31.139 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T00:59:31.660 INFO:teuthology.orchestra.run.smithi192.stdout:(23/23): ceph-common-18.0.0-7498.g369173db.el8. 3.4 MB/s | 23 MB 00:06 2023-11-28T00:59:31.664 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T00:59:31.664 INFO:teuthology.orchestra.run.smithi192.stdout:Total 8.7 MB/s | 62 MB 00:07 2023-11-28T00:59:31.711 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T00:59:31.756 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T00:59:31.756 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T00:59:32.097 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T00:59:32.101 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T00:59:32.287 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T00:59:32.475 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-11-28T00:59:32.651 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-28T00:59:33.046 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-28T00:59:33.192 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librados2-2:18.0.0-7498.g369173db.el8.x86_64 3/25 2023-11-28T00:59:33.289 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-2:18.0.0-7498.g369173db.el8.x86_64 3/25 2023-11-28T00:59:33.352 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T00:59:33.473 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rados-2:18.0.0-7498.g369173db.el8.x86_64 4/25 2023-11-28T00:59:33.587 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-11-28T00:59:33.703 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2023-11-28T00:59:33.770 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-28T00:59:34.067 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs2-2:18.0.0-7498.g369173db.el8.x86_64 6/25 2023-11-28T00:59:34.177 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-11-28T00:59:34.370 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:18.0.0-7498.g369173db.el8.x86_64 3/25 2023-11-28T00:59:34.467 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:18.0.0-7498.g369173db.el8.x86_64 3/25 2023-11-28T00:59:34.483 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs2-2:18.0.0-7498.g369173db.el8.x86_64 6/25 2023-11-28T00:59:34.646 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:18.0.0-7498.g369173db.el8.x86_64 4/25 2023-11-28T00:59:34.675 INFO:teuthology.orchestra.run.smithi172.stdout: Upgrading : librbd1-2:18.0.0-7498.g369173db.el8.x86_64 7/25 2023-11-28T00:59:34.773 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-2:18.0.0-7498.g369173db.el8.x86_64 7/25 2023-11-28T00:59:34.873 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2023-11-28T00:59:34.925 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-28T00:59:34.988 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-28T00:59:35.078 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:18.0.0-7498.g369173db.el8.x86_64 6/25 2023-11-28T00:59:35.189 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/25 2023-11-28T00:59:35.327 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-argparse-2:18.0.0-7498.g369173db.el8. 10/25 2023-11-28T00:59:35.485 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:18.0.0-7498.g369173db.el8.x86_64 6/25 2023-11-28T00:59:35.493 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cephfs-2:18.0.0-7498.g369173db.el8.x86_64 11/25 2023-11-28T00:59:35.669 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:18.0.0-7498.g369173db.el8.x86_64 7/25 2023-11-28T00:59:35.678 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rbd-2:18.0.0-7498.g369173db.el8.x86_64 12/25 2023-11-28T00:59:35.743 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:18.0.0-7498.g369173db.el8.x86_64 7/25 2023-11-28T00:59:35.841 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libradosstriper1-2:18.0.0-7498.g369173db.el8.x86_6 13/25 2023-11-28T00:59:35.911 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7498.g369173db.el8.x86_6 13/25 2023-11-28T00:59:35.928 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-28T00:59:35.975 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-11-28T00:59:36.027 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2023-11-28T00:59:36.118 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/25 2023-11-28T00:59:36.233 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2023-11-28T00:59:36.255 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:18.0.0-7498.g369173db.el8. 10/25 2023-11-28T00:59:36.421 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:18.0.0-7498.g369173db.el8.x86_64 11/25 2023-11-28T00:59:36.623 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:18.0.0-7498.g369173db.el8.x86_64 12/25 2023-11-28T00:59:36.794 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:18.0.0-7498.g369173db.el8.x86_6 13/25 2023-11-28T00:59:36.864 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7498.g369173db.el8.x86_6 13/25 2023-11-28T00:59:37.028 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2023-11-28T00:59:37.046 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2023-11-28T00:59:37.208 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librgw2-2:18.0.0-7498.g369173db.el8.x86_64 17/25 2023-11-28T00:59:37.208 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2023-11-28T00:59:37.308 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librgw2-2:18.0.0-7498.g369173db.el8.x86_64 17/25 2023-11-28T00:59:37.518 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rgw-2:18.0.0-7498.g369173db.el8.x86_64 18/25 2023-11-28T00:59:37.675 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-ceph-common-2:18.0.0-7498.g369173db.el8.x8 19/25 2023-11-28T00:59:38.046 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2023-11-28T00:59:38.244 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:18.0.0-7498.g369173db.el8.x86_64 17/25 2023-11-28T00:59:38.304 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:18.0.0-7498.g369173db.el8.x86_64 17/25 2023-11-28T00:59:38.545 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:18.0.0-7498.g369173db.el8.x86_64 18/25 2023-11-28T00:59:38.711 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:18.0.0-7498.g369173db.el8.x8 19/25 2023-11-28T00:59:41.379 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 20/25 2023-11-28T00:59:41.582 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 20/25 2023-11-28T00:59:42.170 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 20/25 2023-11-28T00:59:42.424 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-base-2:18.0.0-7498.g369173db.el8.x86_64 21/25 2023-11-28T00:59:42.502 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-base-2:18.0.0-7498.g369173db.el8.x86_64 21/25 2023-11-28T00:59:42.506 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 20/25 2023-11-28T00:59:42.640 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-selinux-2:18.0.0-7498.g369173db.el8.x86_64 22/25 2023-11-28T00:59:42.752 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 20/25 2023-11-28T00:59:43.372 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 20/25 2023-11-28T00:59:43.654 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:18.0.0-7498.g369173db.el8.x86_64 21/25 2023-11-28T00:59:43.723 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:18.0.0-7498.g369173db.el8.x86_64 21/25 2023-11-28T00:59:43.900 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:18.0.0-7498.g369173db.el8.x86_64 22/25 2023-11-28T01:00:00.914 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7498.g369173db.el8.x86_64 22/25 2023-11-28T01:00:00.914 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /sys 2023-11-28T01:00:00.914 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /proc 2023-11-28T01:00:00.914 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2023-11-28T01:00:00.914 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /run 2023-11-28T01:00:00.915 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /mnt 2023-11-28T01:00:00.915 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /var/tmp 2023-11-28T01:00:00.915 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /home 2023-11-28T01:00:00.915 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /tmp 2023-11-28T01:00:00.915 INFO:teuthology.orchestra.run.smithi172.stdout:skipping the directory /dev 2023-11-28T01:00:00.915 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:00:02.370 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-radosgw-2:18.0.0-7498.g369173db.el8.x86_64 23/25 2023-11-28T01:00:02.391 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7498.g369173db.el8.x86_64 23/25 2023-11-28T01:00:02.391 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-28T01:00:02.589 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-28T01:00:02.589 INFO:teuthology.orchestra.run.smithi172.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-28T01:00:02.625 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7498.g369173db.el8.x86_64 22/25 2023-11-28T01:00:02.626 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2023-11-28T01:00:02.626 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2023-11-28T01:00:02.626 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /dev 2023-11-28T01:00:02.626 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /run 2023-11-28T01:00:02.626 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2023-11-28T01:00:02.626 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2023-11-28T01:00:02.626 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2023-11-28T01:00:02.626 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2023-11-28T01:00:02.626 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /dev 2023-11-28T01:00:02.626 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:00:03.186 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-28T01:00:03.186 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-base-2:18.0.0-7498.g369173db.el8.x86_64 1/25 2023-11-28T01:00:03.186 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 2/25 2023-11-28T01:00:03.186 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-radosgw-2:18.0.0-7498.g369173db.el8.x86_64 3/25 2023-11-28T01:00:03.186 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-selinux-2:18.0.0-7498.g369173db.el8.x86_64 4/25 2023-11-28T01:00:03.186 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs2-2:18.0.0-7498.g369173db.el8.x86_64 5/25 2023-11-28T01:00:03.186 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libradosstriper1-2:18.0.0-7498.g369173db.el8.x86_6 6/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librgw2-2:18.0.0-7498.g369173db.el8.x86_64 7/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7498.g369173db.el8. 8/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-ceph-common-2:18.0.0-7498.g369173db.el8.x8 9/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cephfs-2:18.0.0-7498.g369173db.el8.x86_64 10/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rados-2:18.0.0-7498.g369173db.el8.x86_64 11/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rbd-2:18.0.0-7498.g369173db.el8.x86_64 12/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rgw-2:18.0.0-7498.g369173db.el8.x86_64 13/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-11-28T01:00:03.187 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-11-28T01:00:03.188 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-11-28T01:00:03.188 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2023-11-28T01:00:03.188 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2023-11-28T01:00:03.188 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-2:18.0.0-7498.g369173db.el8.x86_64 22/25 2023-11-28T01:00:03.188 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-11-28T01:00:03.188 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-2:18.0.0-7498.g369173db.el8.x86_64 24/25 2023-11-28T01:00:04.053 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:18.0.0-7498.g369173db.el8.x86_64 23/25 2023-11-28T01:00:04.072 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7498.g369173db.el8.x86_64 23/25 2023-11-28T01:00:04.072 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-28T01:00:04.301 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-11-28T01:00:04.301 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-28T01:00:04.562 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-11-28T01:00:04.562 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:00:51 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:00:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-11-28T01:00:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:18.0.0-7498.g369173db.el8.x86_64 1/25 2023-11-28T01:00:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 2/25 2023-11-28T01:00:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:18.0.0-7498.g369173db.el8.x86_64 3/25 2023-11-28T01:00:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:18.0.0-7498.g369173db.el8.x86_64 4/25 2023-11-28T01:00:04.943 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:18.0.0-7498.g369173db.el8.x86_64 5/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:18.0.0-7498.g369173db.el8.x86_6 6/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:18.0.0-7498.g369173db.el8.x86_64 7/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7498.g369173db.el8. 8/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:18.0.0-7498.g369173db.el8.x8 9/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:18.0.0-7498.g369173db.el8.x86_64 10/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:18.0.0-7498.g369173db.el8.x86_64 11/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:18.0.0-7498.g369173db.el8.x86_64 12/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:18.0.0-7498.g369173db.el8.x86_64 13/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-11-28T01:00:04.944 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-11-28T01:00:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-11-28T01:00:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-11-28T01:00:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2023-11-28T01:00:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2023-11-28T01:00:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:18.0.0-7498.g369173db.el8.x86_64 22/25 2023-11-28T01:00:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-11-28T01:00:04.945 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:18.0.0-7498.g369173db.el8.x86_64 24/25 2023-11-28T01:00:06.334 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-11-28T01:00:06.335 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:53 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:00:13.950 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:00:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:00:14.138 INFO:teuthology.orchestra.run.smithi172.stdout:Upgraded: 2023-11-28T01:00:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: librados2-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: librbd1-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.138 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:00:14.138 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-radosgw-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-selinux-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs2-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: libradosstriper1-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: librgw2-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-28T01:00:14.139 INFO:teuthology.orchestra.run.smithi172.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-28T01:00:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cephfs-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-28T01:00:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rados-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rbd-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rgw-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-28T01:00:14.140 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:00:14.140 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:00:14.291 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-test 2023-11-28T01:00:14.640 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:00:15.858 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:00:16.044 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:00:16.044 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2023-11-28T01:00:16.044 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.044 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.044 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:00:16.044 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.044 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-28T01:00:16.045 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.046 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.046 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.046 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-28T01:00:16.046 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.046 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.046 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:16.046 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.13.0-2.el8.x86_64 2023-11-28T01:00:16.046 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:00:16.046 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:00:16.191 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test 2023-11-28T01:00:16.406 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:03 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:00:16.540 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:00:17.842 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:00:17.843 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2023-11-28T01:00:17.843 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-11-28T01:00:17.843 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2023-11-28T01:00:17.843 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:00:17.843 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test x86_64 2:18.0.0-7498.g369173db.el8 ceph 49 M 2023-11-28T01:00:17.843 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-11-28T01:00:17.843 INFO:teuthology.orchestra.run.smithi172.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-28T01:00:17.843 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite x86_64 2:18.0.0-7498.g369173db.el8 ceph 183 k 2023-11-28T01:00:17.843 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-28T01:00:17.844 INFO:teuthology.orchestra.run.smithi172.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-28T01:00:17.844 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-28T01:00:17.844 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:00:17.844 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:00:17.844 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================================ 2023-11-28T01:00:17.844 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2023-11-28T01:00:17.844 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:00:17.844 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 50 M 2023-11-28T01:00:17.844 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 223 M 2023-11-28T01:00:17.844 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:00:17.885 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 70 kB 00:00 2023-11-28T01:00:18.027 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.4 MB/s | 323 kB 00:00 2023-11-28T01:00:18.168 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2023-11-28T01:00:18.268 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): jq-1.6-3.el8.x86_64.rpm 2.0 MB/s | 202 kB 00:00 2023-11-28T01:00:18.294 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): libcephsqlite-18.0.0-7498.g369173db.el8. 407 kB/s | 183 kB 00:00 2023-11-28T01:00:18.315 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:05 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:00:19.694 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:00:19.696 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2023-11-28T01:00:19.696 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-11-28T01:00:19.696 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2023-11-28T01:00:19.696 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:00:19.696 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:18.0.0-7498.g369173db.el8 ceph 49 M 2023-11-28T01:00:19.696 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-11-28T01:00:19.696 INFO:teuthology.orchestra.run.smithi192.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-11-28T01:00:19.696 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:18.0.0-7498.g369173db.el8 ceph 183 k 2023-11-28T01:00:19.696 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-11-28T01:00:19.696 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-11-28T01:00:19.696 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-11-28T01:00:19.697 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:00:19.697 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:00:19.697 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2023-11-28T01:00:19.697 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2023-11-28T01:00:19.697 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:00:19.697 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 50 M 2023-11-28T01:00:19.698 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 223 M 2023-11-28T01:00:19.698 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:00:19.739 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 70 kB 00:00 2023-11-28T01:00:19.848 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2023-11-28T01:00:19.989 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): libcephsqlite-18.0.0-7498.g369173db.el8. 627 kB/s | 183 kB 00:00 2023-11-28T01:00:20.015 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2023-11-28T01:00:20.174 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): jq-1.6-3.el8.x86_64.rpm 1.1 MB/s | 202 kB 00:00 2023-11-28T01:00:20.327 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): ceph-test-18.0.0-7498.g369173db.el8.x86_ 20 MB/s | 49 MB 00:02 2023-11-28T01:00:20.329 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:00:20.329 INFO:teuthology.orchestra.run.smithi172.stdout:Total 20 MB/s | 50 MB 00:02 2023-11-28T01:00:20.367 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:00:20.389 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:00:20.389 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:00:20.653 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:00:20.656 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:00:21.096 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:00:26.797 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2023-11-28T01:00:26.797 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2023-11-28T01:00:26.797 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : jq-1.6-3.el8.x86_64 2/6 2023-11-28T01:00:26.797 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2023-11-28T01:00:26.797 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-11-28T01:00:26.797 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephsqlite-2:18.0.0-7498.g369173db.el8.x86_64 5/6 2023-11-28T01:00:26.797 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7498.g369173db.el8.x86_64 5/6 2023-11-28T01:00:26.954 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-test-2:18.0.0-7498.g369173db.el8.x86_64 6/6 2023-11-28T01:00:27.183 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-test-2:18.0.0-7498.g369173db.el8.x86_64 6/6 2023-11-28T01:00:27.184 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-test-2:18.0.0-7498.g369173db.el8.x86_64 1/6 2023-11-28T01:00:27.184 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephsqlite-2:18.0.0-7498.g369173db.el8.x86_64 2/6 2023-11-28T01:00:27.184 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-11-28T01:00:27.184 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2023-11-28T01:00:27.184 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-11-28T01:00:27.431 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): ceph-test-18.0.0-7498.g369173db.el8.x86_ 6.4 MB/s | 49 MB 00:07 2023-11-28T01:00:27.433 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:00:27.433 INFO:teuthology.orchestra.run.smithi192.stdout:Total 6.5 MB/s | 50 MB 00:07 2023-11-28T01:00:27.470 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:00:27.492 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:00:27.492 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:00:27.760 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:00:27.763 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:00:28.211 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:00:28.242 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : jq-1.6-3.el8.x86_64 6/6 2023-11-28T01:00:28.242 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:15 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:00:28.337 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2023-11-28T01:00:28.424 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2023-11-28T01:00:28.665 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : jq-1.6-3.el8.x86_64 2/6 2023-11-28T01:00:28.804 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2023-11-28T01:00:28.993 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-11-28T01:00:29.161 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:18.0.0-7498.g369173db.el8.x86_64 5/6 2023-11-28T01:00:33.470 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:00:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:00:33.650 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:00:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-test-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: jq-1.6-3.el8.x86_64 2023-11-28T01:00:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: libcephsqlite-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-28T01:00:33.650 INFO:teuthology.orchestra.run.smithi172.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-28T01:00:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-28T01:00:33.651 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:00:33.651 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:00:33.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph 2023-11-28T01:00:33.902 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7498.g369173db.el8.x86_64 5/6 2023-11-28T01:00:34.141 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:18.0.0-7498.g369173db.el8.x86_64 6/6 2023-11-28T01:00:34.141 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:00:34.381 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-2:18.0.0-7498.g369173db.el8.x86_64 6/6 2023-11-28T01:00:34.382 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:18.0.0-7498.g369173db.el8.x86_64 1/6 2023-11-28T01:00:34.382 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:18.0.0-7498.g369173db.el8.x86_64 2/6 2023-11-28T01:00:34.382 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-11-28T01:00:34.382 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2023-11-28T01:00:34.382 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-11-28T01:00:35.489 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : jq-1.6-3.el8.x86_64 6/6 2023-11-28T01:00:35.489 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:22 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:00:35.656 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:22 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:00:37.070 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:00:37.073 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================= 2023-11-28T01:00:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-11-28T01:00:37.073 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================= 2023-11-28T01:00:37.073 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:00:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: ceph x86_64 2:18.0.0-7498.g369173db.el8 ceph 6.4 k 2023-11-28T01:00:37.073 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-11-28T01:00:37.073 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds x86_64 2:18.0.0-7498.g369173db.el8 ceph 2.3 M 2023-11-28T01:00:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr x86_64 2:18.0.0-7498.g369173db.el8 ceph 1.6 M 2023-11-28T01:00:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 242 k 2023-11-28T01:00:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon x86_64 2:18.0.0-7498.g369173db.el8 ceph 4.6 M 2023-11-28T01:00:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd x86_64 2:18.0.0-7498.g369173db.el8 ceph 18 M 2023-11-28T01:00:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-28T01:00:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-28T01:00:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-28T01:00:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-28T01:00:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-28T01:00:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-28T01:00:37.074 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-28T01:00:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-28T01:00:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-28T01:00:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-28T01:00:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-28T01:00:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-28T01:00:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-28T01:00:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-28T01:00:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-28T01:00:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-28T01:00:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-28T01:00:37.075 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================= 2023-11-28T01:00:37.076 INFO:teuthology.orchestra.run.smithi172.stdout:Install 32 Packages 2023-11-28T01:00:37.077 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:00:37.077 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 30 M 2023-11-28T01:00:37.077 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 121 M 2023-11-28T01:00:37.077 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:00:37.236 INFO:teuthology.orchestra.run.smithi172.stdout:(1/32): ceph-18.0.0-7498.g369173db.el8.x86_64.r 50 kB/s | 6.4 kB 00:00 2023-11-28T01:00:37.411 INFO:teuthology.orchestra.run.smithi172.stdout:(2/32): ceph-mgr-18.0.0-7498.g369173db.el8.x86_ 5.4 MB/s | 1.6 MB 00:00 2023-11-28T01:00:37.487 INFO:teuthology.orchestra.run.smithi172.stdout:(3/32): ceph-mds-18.0.0-7498.g369173db.el8.x86_ 6.1 MB/s | 2.3 MB 00:00 2023-11-28T01:00:37.578 INFO:teuthology.orchestra.run.smithi172.stdout:(4/32): ceph-mgr-modules-core-18.0.0-7498.g3691 2.6 MB/s | 242 kB 00:00 2023-11-28T01:00:37.829 INFO:teuthology.orchestra.run.smithi172.stdout:(5/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 174 kB/s | 44 kB 00:00 2023-11-28T01:00:38.012 INFO:teuthology.orchestra.run.smithi172.stdout:(6/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 1.0 MB/s | 185 kB 00:00 2023-11-28T01:00:38.121 INFO:teuthology.orchestra.run.smithi172.stdout:(7/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.6 MB/s | 173 kB 00:00 2023-11-28T01:00:38.296 INFO:teuthology.orchestra.run.smithi172.stdout:(8/32): ceph-mon-18.0.0-7498.g369173db.el8.x86_ 4.4 MB/s | 4.6 MB 00:01 2023-11-28T01:00:38.321 INFO:teuthology.orchestra.run.smithi172.stdout:(9/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 1.9 MB/s | 384 kB 00:00 2023-11-28T01:00:38.513 INFO:teuthology.orchestra.run.smithi172.stdout:(10/32): ceph-osd-18.0.0-7498.g369173db.el8.x86 16 MB/s | 18 MB 00:01 2023-11-28T01:00:38.530 INFO:teuthology.orchestra.run.smithi172.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 84 kB/s | 18 kB 00:00 2023-11-28T01:00:38.597 INFO:teuthology.orchestra.run.smithi172.stdout:(12/32): python3-more-itertools-7.2.0-3.el8.noa 882 kB/s | 59 kB 00:00 2023-11-28T01:00:38.665 INFO:teuthology.orchestra.run.smithi172.stdout:(13/32): python3-jaraco-6.2-6.el8.noarch.rpm 30 kB/s | 11 kB 00:00 2023-11-28T01:00:38.690 INFO:teuthology.orchestra.run.smithi172.stdout:(14/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-11-28T01:00:38.740 INFO:teuthology.orchestra.run.smithi172.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 208 kB/s | 16 kB 00:00 2023-11-28T01:00:38.757 INFO:teuthology.orchestra.run.smithi172.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 280 kB/s | 19 kB 00:00 2023-11-28T01:00:38.774 INFO:teuthology.orchestra.run.smithi172.stdout:(17/32): python3-logutils-0.3.5-11.el8.noarch.r 190 kB/s | 49 kB 00:00 2023-11-28T01:00:38.824 INFO:teuthology.orchestra.run.smithi172.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 436 kB/s | 29 kB 00:00 2023-11-28T01:00:38.841 INFO:teuthology.orchestra.run.smithi172.stdout:(19/32): python3-singledispatch-3.4.0.3-18.el8. 237 kB/s | 24 kB 00:00 2023-11-28T01:00:38.858 INFO:teuthology.orchestra.run.smithi172.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 328 kB/s | 27 kB 00:00 2023-11-28T01:00:38.908 INFO:teuthology.orchestra.run.smithi172.stdout:(21/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.0 MB/s | 255 kB 00:00 2023-11-28T01:00:38.984 INFO:teuthology.orchestra.run.smithi172.stdout:(22/32): python3-zc-lockfile-2.0-2.el8.noarch.r 306 kB/s | 23 kB 00:00 2023-11-28T01:00:39.009 INFO:teuthology.orchestra.run.smithi172.stdout:(23/32): python3-webtest-2.0.33-1.el8.noarch.rp 567 kB/s | 85 kB 00:00 2023-11-28T01:00:39.076 INFO:teuthology.orchestra.run.smithi172.stdout:(24/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.0 MB/s | 251 kB 00:00 2023-11-28T01:00:39.143 INFO:teuthology.orchestra.run.smithi172.stdout:(25/32): python3-cryptography-3.2.1-5.el8.x86_6 4.1 MB/s | 559 kB 00:00 2023-11-28T01:00:39.168 INFO:teuthology.orchestra.run.smithi172.stdout:(26/32): python3-pycparser-2.14-14.el8.noarch.r 593 kB/s | 109 kB 00:00 2023-11-28T01:00:39.227 INFO:teuthology.orchestra.run.smithi172.stdout:(27/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.5 MB/s | 238 kB 00:00 2023-11-28T01:00:39.252 INFO:teuthology.orchestra.run.smithi172.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 463 kB/s | 39 kB 00:00 2023-11-28T01:00:39.277 INFO:teuthology.orchestra.run.smithi172.stdout:(29/32): python3-pytz-2017.2-9.el8.noarch.rpm 404 kB/s | 54 kB 00:00 2023-11-28T01:00:39.378 INFO:teuthology.orchestra.run.smithi172.stdout:(30/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.6 MB/s | 457 kB 00:00 2023-11-28T01:00:39.403 INFO:teuthology.orchestra.run.smithi172.stdout:(31/32): python3-mako-1.0.6-13.el8.noarch.rpm 894 kB/s | 157 kB 00:00 2023-11-28T01:00:39.453 INFO:teuthology.orchestra.run.smithi172.stdout:(32/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 591 kB/s | 103 kB 00:00 2023-11-28T01:00:39.457 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:00:39.457 INFO:teuthology.orchestra.run.smithi172.stdout:Total 13 MB/s | 30 MB 00:02 2023-11-28T01:00:39.552 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:00:39.590 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:00:39.590 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:00:39.805 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:00:39.808 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:00:40.896 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:00:41.119 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-28T01:00:41.422 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-28T01:00:41.667 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 3/32 2023-11-28T01:00:41.898 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2023-11-28T01:00:42.101 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/32 2023-11-28T01:00:42.291 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 6/32 2023-11-28T01:00:42.515 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2023-11-28T01:00:42.896 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 8/32 2023-11-28T01:00:43.174 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 9/32 2023-11-28T01:00:43.354 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2023-11-28T01:00:43.538 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2023-11-28T01:00:43.720 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2023-11-28T01:00:43.979 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-11-28T01:00:44.161 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-11-28T01:00:44.350 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-11-28T01:00:44.506 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-11-28T01:00:44.665 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-11-28T01:00:44.793 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-11-28T01:00:45.046 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-11-28T01:00:45.281 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-11-28T01:00:45.467 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-28T01:00:45.721 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-28T01:00:45.992 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-28T01:00:46.196 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-11-28T01:00:46.505 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-11-28T01:00:46.818 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-28T01:00:46.836 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:00:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:00:47.017 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:00:47.017 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: jq-1.6-3.el8.x86_64 2023-11-28T01:00:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:00:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-11-28T01:00:47.018 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-1.el8.x86_64 2023-11-28T01:00:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-11-28T01:00:47.019 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:00:47.019 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:00:47.148 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph 2023-11-28T01:00:47.197 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7498.g369173db.el8. 27/32 2023-11-28T01:00:47.346 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-2:18.0.0-7498.g369173db.el8.x86_64 28/32 2023-11-28T01:00:47.484 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:00:48.968 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7498.g369173db.el8.x86_64 28/32 2023-11-28T01:00:49.137 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-osd-2:18.0.0-7498.g369173db.el8.x86_64 29/32 2023-11-28T01:00:49.278 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:36 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:00:49.602 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-osd-2:18.0.0-7498.g369173db.el8.x86_64 29/32 2023-11-28T01:00:49.745 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mon-2:18.0.0-7498.g369173db.el8.x86_64 30/32 2023-11-28T01:00:49.962 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mon-2:18.0.0-7498.g369173db.el8.x86_64 30/32 2023-11-28T01:00:50.095 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mds-2:18.0.0-7498.g369173db.el8.x86_64 31/32 2023-11-28T01:00:50.114 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mds-2:18.0.0-7498.g369173db.el8.x86_64 31/32 2023-11-28T01:00:50.259 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-2:18.0.0-7498.g369173db.el8.x86_64 32/32 2023-11-28T01:00:50.637 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-2:18.0.0-7498.g369173db.el8.x86_64 32/32 2023-11-28T01:00:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-2:18.0.0-7498.g369173db.el8.x86_64 1/32 2023-11-28T01:00:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mds-2:18.0.0-7498.g369173db.el8.x86_64 2/32 2023-11-28T01:00:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-2:18.0.0-7498.g369173db.el8.x86_64 3/32 2023-11-28T01:00:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mon-2:18.0.0-7498.g369173db.el8.x86_64 4/32 2023-11-28T01:00:50.638 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-osd-2:18.0.0-7498.g369173db.el8.x86_64 5/32 2023-11-28T01:00:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7498.g369173db.el8. 6/32 2023-11-28T01:00:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-11-28T01:00:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-11-28T01:00:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-11-28T01:00:50.639 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-11-28T01:00:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-11-28T01:00:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-11-28T01:00:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-11-28T01:00:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-11-28T01:00:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-11-28T01:00:50.640 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-11-28T01:00:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-11-28T01:00:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-11-28T01:00:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-11-28T01:00:50.641 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-11-28T01:00:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-11-28T01:00:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-11-28T01:00:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-11-28T01:00:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-11-28T01:00:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2023-11-28T01:00:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 26/32 2023-11-28T01:00:50.642 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 27/32 2023-11-28T01:00:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/32 2023-11-28T01:00:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2023-11-28T01:00:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 30/32 2023-11-28T01:00:50.643 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 31/32 2023-11-28T01:00:50.717 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:00:50.721 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================================= 2023-11-28T01:00:50.721 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-11-28T01:00:50.721 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================================= 2023-11-28T01:00:50.721 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:00:50.721 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:18.0.0-7498.g369173db.el8 ceph 6.4 k 2023-11-28T01:00:50.722 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-11-28T01:00:50.722 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:18.0.0-7498.g369173db.el8 ceph 2.3 M 2023-11-28T01:00:50.722 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:18.0.0-7498.g369173db.el8 ceph 1.6 M 2023-11-28T01:00:50.722 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 242 k 2023-11-28T01:00:50.722 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:18.0.0-7498.g369173db.el8 ceph 4.6 M 2023-11-28T01:00:50.722 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:18.0.0-7498.g369173db.el8 ceph 18 M 2023-11-28T01:00:50.722 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-28T01:00:50.722 INFO:teuthology.orchestra.run.smithi192.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-28T01:00:50.723 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-11-28T01:00:50.723 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-28T01:00:50.723 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-28T01:00:50.723 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-11-28T01:00:50.723 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-28T01:00:50.723 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-28T01:00:50.723 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-28T01:00:50.723 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-11-28T01:00:50.724 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-11-28T01:00:50.724 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-28T01:00:50.724 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-28T01:00:50.724 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-28T01:00:50.724 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-11-28T01:00:50.724 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-11-28T01:00:50.724 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-11-28T01:00:50.725 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-28T01:00:50.725 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-28T01:00:50.725 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-28T01:00:50.725 INFO:teuthology.orchestra.run.smithi192.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-28T01:00:50.725 INFO:teuthology.orchestra.run.smithi192.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-28T01:00:50.725 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-28T01:00:50.726 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-28T01:00:50.726 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-11-28T01:00:50.726 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-28T01:00:50.726 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:00:50.726 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:00:50.726 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================================= 2023-11-28T01:00:50.726 INFO:teuthology.orchestra.run.smithi192.stdout:Install 32 Packages 2023-11-28T01:00:50.726 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:00:50.727 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 30 M 2023-11-28T01:00:50.727 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 121 M 2023-11-28T01:00:50.727 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:00:51.339 INFO:teuthology.orchestra.run.smithi192.stdout:(1/32): ceph-mgr-18.0.0-7498.g369173db.el8.x86_ 5.7 MB/s | 1.6 MB 00:00 2023-11-28T01:00:51.364 INFO:teuthology.orchestra.run.smithi192.stdout:(2/32): ceph-18.0.0-7498.g369173db.el8.x86_64.r 21 kB/s | 6.4 kB 00:00 2023-11-28T01:00:52.088 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 32/32 2023-11-28T01:00:52.088 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:39 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:00:52.240 INFO:teuthology.orchestra.run.smithi192.stdout:(3/32): ceph-osd-18.0.0-7498.g369173db.el8.x86_ 21 MB/s | 18 MB 00:00 2023-11-28T01:00:52.281 INFO:teuthology.orchestra.run.smithi192.stdout:(4/32): ceph-mgr-modules-core-18.0.0-7498.g3691 5.7 MB/s | 242 kB 00:00 2023-11-28T01:00:52.374 INFO:teuthology.orchestra.run.smithi192.stdout:(5/32): ceph-mds-18.0.0-7498.g369173db.el8.x86_ 1.8 MB/s | 2.3 MB 00:01 2023-11-28T01:00:52.540 INFO:teuthology.orchestra.run.smithi192.stdout:(6/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 169 kB/s | 44 kB 00:00 2023-11-28T01:00:52.790 INFO:teuthology.orchestra.run.smithi192.stdout:(7/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 444 kB/s | 185 kB 00:00 2023-11-28T01:00:52.949 INFO:teuthology.orchestra.run.smithi192.stdout:(8/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 422 kB/s | 173 kB 00:00 2023-11-28T01:00:53.083 INFO:teuthology.orchestra.run.smithi192.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2023-11-28T01:00:53.241 INFO:teuthology.orchestra.run.smithi192.stdout:(10/32): python3-cherrypy-18.4.0-1.el8.noarch.r 853 kB/s | 384 kB 00:00 2023-11-28T01:00:53.267 INFO:teuthology.orchestra.run.smithi192.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 96 kB/s | 18 kB 00:00 2023-11-28T01:00:53.475 INFO:teuthology.orchestra.run.smithi192.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 209 kB/s | 49 kB 00:00 2023-11-28T01:00:53.509 INFO:teuthology.orchestra.run.smithi192.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 243 kB/s | 59 kB 00:00 2023-11-28T01:00:53.709 INFO:teuthology.orchestra.run.smithi192.stdout:(14/32): python3-portend-2.6-1.el8.noarch.rpm 78 kB/s | 16 kB 00:00 2023-11-28T01:00:53.893 INFO:teuthology.orchestra.run.smithi192.stdout:(15/32): python3-simplegeneric-0.8.1-17.el8.noa 102 kB/s | 19 kB 00:00 2023-11-28T01:00:54.018 INFO:teuthology.orchestra.run.smithi192.stdout:(16/32): python3-pecan-1.3.2-9.el8.noarch.rpm 521 kB/s | 283 kB 00:00 2023-11-28T01:00:54.144 INFO:teuthology.orchestra.run.smithi192.stdout:(17/32): ceph-mon-18.0.0-7498.g369173db.el8.x86 1.7 MB/s | 4.6 MB 00:02 2023-11-28T01:00:54.162 INFO:teuthology.orchestra.run.smithi192.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 89 kB/s | 24 kB 00:00 2023-11-28T01:00:54.319 INFO:teuthology.orchestra.run.smithi192.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 97 kB/s | 29 kB 00:00 2023-11-28T01:00:54.394 INFO:teuthology.orchestra.run.smithi192.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 110 kB/s | 27 kB 00:00 2023-11-28T01:00:54.620 INFO:teuthology.orchestra.run.smithi192.stdout:(21/32): python3-webtest-2.0.33-1.el8.noarch.rp 379 kB/s | 85 kB 00:00 2023-11-28T01:00:54.745 INFO:teuthology.orchestra.run.smithi192.stdout:(22/32): python3-waitress-1.4.3-1.el8.noarch.rp 437 kB/s | 255 kB 00:00 2023-11-28T01:00:54.862 INFO:teuthology.orchestra.run.smithi192.stdout:(23/32): python3-zc-lockfile-2.0-2.el8.noarch.r 95 kB/s | 23 kB 00:00 2023-11-28T01:00:54.895 INFO:teuthology.orchestra.run.smithi192.stdout:(24/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 436 kB/s | 251 kB 00:00 2023-11-28T01:00:54.921 INFO:teuthology.orchestra.run.smithi192.stdout:(25/32): python3-pycparser-2.14-14.el8.noarch.r 621 kB/s | 109 kB 00:00 2023-11-28T01:00:55.076 INFO:teuthology.orchestra.run.smithi192.stdout:(26/32): python3-pytz-2017.2-9.el8.noarch.rpm 361 kB/s | 54 kB 00:00 2023-11-28T01:00:55.113 INFO:teuthology.orchestra.run.smithi192.stdout:(27/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.1 MB/s | 238 kB 00:00 2023-11-28T01:00:55.146 INFO:teuthology.orchestra.run.smithi192.stdout:(28/32): python3-cryptography-3.2.1-5.el8.x86_6 1.9 MB/s | 559 kB 00:00 2023-11-28T01:00:55.172 INFO:teuthology.orchestra.run.smithi192.stdout:(29/32): python3-markupsafe-0.23-19.el8.x86_64. 407 kB/s | 39 kB 00:00 2023-11-28T01:00:55.222 INFO:teuthology.orchestra.run.smithi192.stdout:(30/32): python3-werkzeug-0.12.2-4.el8.noarch.r 5.9 MB/s | 457 kB 00:00 2023-11-28T01:00:55.247 INFO:teuthology.orchestra.run.smithi192.stdout:(31/32): python3-mako-1.0.6-13.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2023-11-28T01:00:55.306 INFO:teuthology.orchestra.run.smithi192.stdout:(32/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 774 kB/s | 103 kB 00:00 2023-11-28T01:00:55.308 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:00:55.309 INFO:teuthology.orchestra.run.smithi192.stdout:Total 6.6 MB/s | 30 MB 00:04 2023-11-28T01:00:55.397 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:00:55.435 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:00:55.435 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:00:55.646 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:00:55.649 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:00:56.757 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:00:56.978 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-28T01:00:57.265 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-28T01:00:57.511 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 3/32 2023-11-28T01:00:57.717 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2023-11-28T01:00:57.940 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/32 2023-11-28T01:00:58.168 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 6/32 2023-11-28T01:00:58.400 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2023-11-28T01:00:58.770 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 8/32 2023-11-28T01:00:59.076 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 9/32 2023-11-28T01:00:59.223 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2023-11-28T01:00:59.432 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2023-11-28T01:00:59.631 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2023-11-28T01:00:59.883 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-11-28T01:01:00.055 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-11-28T01:01:00.269 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-11-28T01:01:00.415 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-11-28T01:01:00.592 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-11-28T01:01:00.729 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-11-28T01:01:00.964 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-11-28T01:01:01.031 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:01:01.176 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-11-28T01:01:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:01.219 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:01:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:01.219 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mds-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:01:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mon-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-osd-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-28T01:01:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-28T01:01:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-28T01:01:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-28T01:01:01.220 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-portend-2.6-1.el8.noarch 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-28T01:01:01.221 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-28T01:01:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-28T01:01:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-28T01:01:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-28T01:01:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-28T01:01:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-28T01:01:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-28T01:01:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-28T01:01:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-28T01:01:01.222 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:01.222 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:01:01.374 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-base 2023-11-28T01:01:01.412 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-28T01:01:01.672 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-28T01:01:01.726 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:01:01.918 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-28T01:01:02.156 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-11-28T01:01:02.456 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-11-28T01:01:02.752 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-28T01:01:03.102 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7498.g369173db.el8. 27/32 2023-11-28T01:01:03.284 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:18.0.0-7498.g369173db.el8.x86_64 28/32 2023-11-28T01:01:03.325 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:50 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:04.657 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-base-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:01:04.765 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:01:04.766 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-11-28T01:01:04.766 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:01:04.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install cephadm 2023-11-28T01:01:04.980 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7498.g369173db.el8.x86_64 28/32 2023-11-28T01:01:05.221 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:01:06.300 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:18.0.0-7498.g369173db.el8.x86_64 29/32 2023-11-28T01:01:06.686 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:53 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:06.767 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:18.0.0-7498.g369173db.el8.x86_64 29/32 2023-11-28T01:01:06.922 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:18.0.0-7498.g369173db.el8.x86_64 30/32 2023-11-28T01:01:07.139 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:18.0.0-7498.g369173db.el8.x86_64 30/32 2023-11-28T01:01:07.289 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:18.0.0-7498.g369173db.el8.x86_64 31/32 2023-11-28T01:01:07.307 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:18.0.0-7498.g369173db.el8.x86_64 31/32 2023-11-28T01:01:07.462 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:18.0.0-7498.g369173db.el8.x86_64 32/32 2023-11-28T01:01:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-2:18.0.0-7498.g369173db.el8.x86_64 32/32 2023-11-28T01:01:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:18.0.0-7498.g369173db.el8.x86_64 1/32 2023-11-28T01:01:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:18.0.0-7498.g369173db.el8.x86_64 2/32 2023-11-28T01:01:07.819 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:18.0.0-7498.g369173db.el8.x86_64 3/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:18.0.0-7498.g369173db.el8.x86_64 4/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:18.0.0-7498.g369173db.el8.x86_64 5/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7498.g369173db.el8. 6/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-11-28T01:01:07.820 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 26/32 2023-11-28T01:01:07.821 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 27/32 2023-11-28T01:01:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/32 2023-11-28T01:01:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2023-11-28T01:01:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 30/32 2023-11-28T01:01:07.822 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 31/32 2023-11-28T01:01:08.045 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:01:08.046 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:01:08.046 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-11-28T01:01:08.046 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:01:08.046 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:01:08.046 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 411 k 2023-11-28T01:01:08.046 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:08.046 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:01:08.046 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:01:08.046 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-11-28T01:01:08.046 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:08.047 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 411 k 2023-11-28T01:01:08.047 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 2.1 M 2023-11-28T01:01:08.047 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:01:08.700 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm-18.0.0-7498.g369173db.el8.noarch.rpm 629 kB/s | 411 kB 00:00 2023-11-28T01:01:08.701 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:01:08.701 INFO:teuthology.orchestra.run.smithi172.stdout:Total 627 kB/s | 411 kB 00:00 2023-11-28T01:01:08.701 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:01:08.709 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:01:08.709 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:01:08.712 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:01:08.716 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:01:08.761 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:01:09.298 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 32/32 2023-11-28T01:01:09.298 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:56 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:10.462 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:01:10.625 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : cephadm-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:01:10.793 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: cephadm-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:01:11.890 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : cephadm-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:01:11.890 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:01:58 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:16.497 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:01:16.684 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:16.684 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:01:16.684 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:16.684 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:16.685 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:16.685 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:01:16.685 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:16.685 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:16.685 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-28T01:01:16.685 INFO:teuthology.orchestra.run.smithi192.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-28T01:01:16.685 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-11-28T01:01:16.685 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-28T01:01:16.685 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.0.6-13.el8.noarch 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-2.6-1.el8.noarch 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-28T01:01:16.686 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2017.2-9.el8.noarch 2023-11-28T01:01:16.687 INFO:teuthology.orchestra.run.smithi192.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-28T01:01:16.687 INFO:teuthology.orchestra.run.smithi192.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-28T01:01:16.687 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-28T01:01:16.687 INFO:teuthology.orchestra.run.smithi192.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-28T01:01:16.687 INFO:teuthology.orchestra.run.smithi192.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-28T01:01:16.687 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-28T01:01:16.687 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-28T01:01:16.687 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-28T01:01:16.687 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-28T01:01:16.687 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:16.687 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:01:16.837 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base 2023-11-28T01:01:17.183 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:01:18.691 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:05 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:19.947 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:01:20.055 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:01:20.056 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-11-28T01:01:20.056 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:01:20.101 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:01:20.175 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install cephadm 2023-11-28T01:01:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:20.281 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:01:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: cephadm-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:01:20.281 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:20.281 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:01:20.415 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-immutable-object-cache 2023-11-28T01:01:20.519 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:01:20.776 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:01:22.002 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:08 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:22.353 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:09 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:23.425 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:01:23.426 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:01:23.426 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-11-28T01:01:23.426 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:01:23.426 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:01:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 411 k 2023-11-28T01:01:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:23.427 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:01:23.427 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:01:23.427 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-11-28T01:01:23.427 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:23.427 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 411 k 2023-11-28T01:01:23.427 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.1 M 2023-11-28T01:01:23.427 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:01:23.689 INFO:teuthology.orchestra.run.smithi192.stdout:cephadm-18.0.0-7498.g369173db.el8.noarch.rpm 1.5 MB/s | 411 kB 00:00 2023-11-28T01:01:23.690 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:01:23.690 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.5 MB/s | 411 kB 00:00 2023-11-28T01:01:23.690 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:01:23.698 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:01:23.698 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:01:23.701 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:01:23.705 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:01:23.752 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:01:23.784 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:01:23.784 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:01:23.784 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-11-28T01:01:23.784 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:01:23.784 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:01:23.784 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7498.g369173db.el8 ceph 171 k 2023-11-28T01:01:23.784 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:23.785 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:01:23.785 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:01:23.785 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-11-28T01:01:23.785 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:23.785 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 171 k 2023-11-28T01:01:23.785 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 524 k 2023-11-28T01:01:23.785 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:01:23.975 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-immutable-object-cache-18.0.0-7498.g369173 901 kB/s | 171 kB 00:00 2023-11-28T01:01:23.976 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:01:23.976 INFO:teuthology.orchestra.run.smithi172.stdout:Total 896 kB/s | 171 kB 00:00 2023-11-28T01:01:23.976 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:01:23.987 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:01:23.987 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:01:24.040 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:01:24.043 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:01:24.160 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:01:24.347 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7498.g369173db. 1/1 2023-11-28T01:01:24.701 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7498.g369173db. 1/1 2023-11-28T01:01:25.576 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:01:25.739 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:01:25.767 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7498.g369173db. 1/1 2023-11-28T01:01:25.767 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:12 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:25.907 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:01:26.937 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:01:26.937 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:13 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:36.216 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:01:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:36.395 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:01:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-immutable-object-cache-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:36.395 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:36.396 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:01:36.526 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr 2023-11-28T01:01:36.871 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:01:37.232 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:01:37.413 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:37.413 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:01:37.413 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:01:37.413 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:37.413 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:01:38.319 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache 2023-11-28T01:01:38.554 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:25 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:38.691 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:01:39.830 INFO:teuthology.orchestra.run.smithi172.stdout:Package ceph-mgr-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:01:39.938 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:01:39.939 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-11-28T01:01:39.939 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:01:40.054 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-dashboard 2023-11-28T01:01:40.119 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:27 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:40.397 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:01:41.476 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:01:41.476 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:01:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-11-28T01:01:41.477 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:01:41.477 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:01:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7498.g369173db.el8 ceph 171 k 2023-11-28T01:01:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:41.477 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:01:41.477 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:01:41.477 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-11-28T01:01:41.477 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:41.478 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 171 k 2023-11-28T01:01:41.478 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 524 k 2023-11-28T01:01:41.478 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:01:41.790 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-18.0.0-7498.g369173 547 kB/s | 171 kB 00:00 2023-11-28T01:01:41.791 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:01:41.791 INFO:teuthology.orchestra.run.smithi192.stdout:Total 545 kB/s | 171 kB 00:00 2023-11-28T01:01:41.791 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:01:41.802 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:01:41.803 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:01:41.855 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:01:41.859 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:01:41.934 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:28 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:41.977 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:01:42.167 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7498.g369173db. 1/1 2023-11-28T01:01:42.523 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7498.g369173db. 1/1 2023-11-28T01:01:43.291 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:01:43.291 INFO:teuthology.orchestra.run.smithi172.stdout:=================================================================================================== 2023-11-28T01:01:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-11-28T01:01:43.292 INFO:teuthology.orchestra.run.smithi172.stdout:=================================================================================================== 2023-11-28T01:01:43.292 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:01:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 1.8 M 2023-11-28T01:01:43.292 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-11-28T01:01:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 28 k 2023-11-28T01:01:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 15 k 2023-11-28T01:01:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-28T01:01:43.292 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-28T01:01:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-28T01:01:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:43.293 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:01:43.293 INFO:teuthology.orchestra.run.smithi172.stdout:=================================================================================================== 2023-11-28T01:01:43.293 INFO:teuthology.orchestra.run.smithi172.stdout:Install 6 Packages 2023-11-28T01:01:43.293 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:43.294 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 2.1 M 2023-11-28T01:01:43.294 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 8.6 M 2023-11-28T01:01:43.294 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:01:43.546 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7498.g369173db. 1/1 2023-11-28T01:01:43.546 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:30 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:43.602 INFO:teuthology.orchestra.run.smithi172.stdout:(1/6): ceph-prometheus-alerts-18.0.0-7498.g3691 103 kB/s | 15 kB 00:00 2023-11-28T01:01:43.619 INFO:teuthology.orchestra.run.smithi172.stdout:(2/6): ceph-grafana-dashboards-18.0.0-7498.g369 171 kB/s | 28 kB 00:00 2023-11-28T01:01:43.753 INFO:teuthology.orchestra.run.smithi172.stdout:(3/6): ceph-mgr-dashboard-18.0.0-7498.g369173db 5.9 MB/s | 1.8 MB 00:00 2023-11-28T01:01:43.895 INFO:teuthology.orchestra.run.smithi172.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 320 kB/s | 43 kB 00:00 2023-11-28T01:01:44.087 INFO:teuthology.orchestra.run.smithi172.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 69 kB/s | 34 kB 00:00 2023-11-28T01:01:44.295 INFO:teuthology.orchestra.run.smithi172.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 291 kB/s | 196 kB 00:00 2023-11-28T01:01:44.298 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:01:44.299 INFO:teuthology.orchestra.run.smithi172.stdout:Total 2.1 MB/s | 2.1 MB 00:01 2023-11-28T01:01:44.311 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:01:44.320 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:01:44.320 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:01:44.379 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:01:44.383 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:01:44.679 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:01:44.861 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-11-28T01:01:45.058 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-11-28T01:01:45.246 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-11-28T01:01:45.368 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7498.g369173db.el8.n 4/6 2023-11-28T01:01:45.704 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7498.g369173db.el8. 5/6 2023-11-28T01:01:45.953 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7498.g369173db.el8.noarc 6/6 2023-11-28T01:01:46.175 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7498.g369173db.el8.noarc 6/6 2023-11-28T01:01:46.175 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7498.g369173db.el8. 1/6 2023-11-28T01:01:46.175 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7498.g369173db.el8.noarc 2/6 2023-11-28T01:01:46.176 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7498.g369173db.el8.n 3/6 2023-11-28T01:01:46.176 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-11-28T01:01:46.176 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-11-28T01:01:47.203 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-11-28T01:01:47.203 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:34 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:48.633 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:01:48.812 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:48.812 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:01:48.812 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:01:48.812 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:48.812 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:01:48.945 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr 2023-11-28T01:01:49.292 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:01:50.731 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:37 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:52.012 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-mgr-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:01:52.120 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:01:52.121 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-11-28T01:01:52.121 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:01:52.134 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:01:52.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-dashboard 2023-11-28T01:01:52.316 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:52.317 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:01:52.317 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-grafana-dashboards-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:01:52.317 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:01:52.317 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-prometheus-alerts-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:01:52.317 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-28T01:01:52.317 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-28T01:01:52.317 INFO:teuthology.orchestra.run.smithi172.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-28T01:01:52.317 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:52.317 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:01:52.454 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-28T01:01:52.587 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:01:52.801 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:01:54.104 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:41 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:54.241 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:41 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:55.466 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:01:55.467 INFO:teuthology.orchestra.run.smithi192.stdout:=================================================================================================== 2023-11-28T01:01:55.467 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-11-28T01:01:55.467 INFO:teuthology.orchestra.run.smithi192.stdout:=================================================================================================== 2023-11-28T01:01:55.467 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:01:55.467 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 1.8 M 2023-11-28T01:01:55.467 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-11-28T01:01:55.467 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 28 k 2023-11-28T01:01:55.467 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 15 k 2023-11-28T01:01:55.467 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-28T01:01:55.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-28T01:01:55.468 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-28T01:01:55.468 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:55.468 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:01:55.468 INFO:teuthology.orchestra.run.smithi192.stdout:=================================================================================================== 2023-11-28T01:01:55.468 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2023-11-28T01:01:55.468 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:01:55.469 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 2.1 M 2023-11-28T01:01:55.469 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 8.6 M 2023-11-28T01:01:55.469 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:01:55.604 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:01:55.605 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================================= 2023-11-28T01:01:55.605 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-11-28T01:01:55.605 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================================= 2023-11-28T01:01:55.605 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:01:55.605 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 7.4 M 2023-11-28T01:01:55.606 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-11-28T01:01:55.606 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-28T01:01:55.606 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-28T01:01:55.606 INFO:teuthology.orchestra.run.smithi172.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-28T01:01:55.606 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-28T01:01:55.606 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-28T01:01:55.606 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-28T01:01:55.606 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-28T01:01:55.606 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:55.606 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:01:55.606 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================================= 2023-11-28T01:01:55.607 INFO:teuthology.orchestra.run.smithi172.stdout:Install 8 Packages 2023-11-28T01:01:55.607 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:01:55.607 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 35 M 2023-11-28T01:01:55.607 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 210 M 2023-11-28T01:01:55.607 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:01:55.744 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): ceph-grafana-dashboards-18.0.0-7498.g369 196 kB/s | 28 kB 00:00 2023-11-28T01:01:55.758 INFO:teuthology.orchestra.run.smithi172.stdout:(1/8): libquadmath-8.5.0-10.1.el8_6.x86_64.rpm 1.1 MB/s | 171 kB 00:00 2023-11-28T01:01:55.825 INFO:teuthology.orchestra.run.smithi172.stdout:(2/8): libgfortran-8.5.0-10.1.el8_6.x86_64.rpm 2.9 MB/s | 644 kB 00:00 2023-11-28T01:01:55.853 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): ceph-prometheus-alerts-18.0.0-7498.g3691 60 kB/s | 15 kB 00:00 2023-11-28T01:01:55.970 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 149 kB/s | 34 kB 00:00 2023-11-28T01:01:56.092 INFO:teuthology.orchestra.run.smithi172.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 14 MB/s | 3.7 MB 00:00 2023-11-28T01:01:56.103 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 320 kB/s | 43 kB 00:00 2023-11-28T01:01:56.234 INFO:teuthology.orchestra.run.smithi172.stdout:(4/8): ceph-mgr-diskprediction-local-18.0.0-749 12 MB/s | 7.4 MB 00:00 2023-11-28T01:01:56.260 INFO:teuthology.orchestra.run.smithi172.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.3 MB/s | 225 kB 00:00 2023-11-28T01:01:56.287 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 453 kB/s | 196 kB 00:00 2023-11-28T01:01:56.353 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): ceph-mgr-dashboard-18.0.0-7498.g369173db 2.3 MB/s | 1.8 MB 00:00 2023-11-28T01:01:56.356 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:01:56.356 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.3 MB/s | 2.1 MB 00:00 2023-11-28T01:01:56.367 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:01:56.376 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:01:56.376 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:01:56.410 INFO:teuthology.orchestra.run.smithi172.stdout:(6/8): python3-scipy-1.0.0-21.module+el8.5.0+10 21 MB/s | 14 MB 00:00 2023-11-28T01:01:56.433 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:01:56.437 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:01:56.535 INFO:teuthology.orchestra.run.smithi172.stdout:(7/8): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-11-28T01:01:56.627 INFO:teuthology.orchestra.run.smithi172.stdout:(8/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 12 MB/s | 4.9 MB 00:00 2023-11-28T01:01:56.628 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:01:56.628 INFO:teuthology.orchestra.run.smithi172.stdout:Total 35 MB/s | 35 MB 00:01 2023-11-28T01:01:56.722 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:01:56.867 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:01:56.885 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:01:56.885 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:01:56.899 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-11-28T01:01:57.070 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-11-28T01:01:57.178 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:01:57.182 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:01:57.268 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-11-28T01:01:57.381 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7498.g369173db.el8.n 4/6 2023-11-28T01:01:57.716 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7498.g369173db.el8. 5/6 2023-11-28T01:01:57.719 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:01:57.885 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-11-28T01:01:57.899 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7498.g369173db.el8.noarc 6/6 2023-11-28T01:01:58.013 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-11-28T01:01:58.153 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7498.g369173db.el8.noarc 6/6 2023-11-28T01:01:58.153 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7498.g369173db.el8. 1/6 2023-11-28T01:01:58.153 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7498.g369173db.el8.noarc 2/6 2023-11-28T01:01:58.153 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7498.g369173db.el8.n 3/6 2023-11-28T01:01:58.153 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-11-28T01:01:58.153 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-11-28T01:01:58.157 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-11-28T01:01:58.778 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-11-28T01:01:58.933 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-11-28T01:01:59.193 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-11-28T01:01:59.193 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:46 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:01:59.533 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-11-28T01:01:59.741 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : openblas-0.3.15-3.el8.x86_64 4/8 2023-11-28T01:02:00.421 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 4/8 2023-11-28T01:02:00.727 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-28T01:02:00.915 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-28T01:02:02.682 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-28T01:02:03.925 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 7/8 2023-11-28T01:02:04.099 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7498.g369173d 8/8 2023-11-28T01:02:04.358 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7498.g369173d 8/8 2023-11-28T01:02:04.359 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7498.g369173d 1/8 2023-11-28T01:02:04.359 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-11-28T01:02:04.359 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 3/8 2023-11-28T01:02:04.359 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 4/8 2023-11-28T01:02:04.359 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-28T01:02:04.359 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-28T01:02:04.359 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 7/8 2023-11-28T01:02:05.414 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 8/8 2023-11-28T01:02:05.415 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:02:52 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:05.792 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:02:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:05.977 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:02:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:02:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:02:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:02:05.977 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-28T01:02:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-28T01:02:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-28T01:02:05.978 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:05.978 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:02:06.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-28T01:02:06.462 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:02:08.045 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:55 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:09.420 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:02:09.421 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================================================= 2023-11-28T01:02:09.421 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-11-28T01:02:09.421 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================================================= 2023-11-28T01:02:09.421 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:02:09.421 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 7.4 M 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================================================= 2023-11-28T01:02:09.422 INFO:teuthology.orchestra.run.smithi192.stdout:Install 8 Packages 2023-11-28T01:02:09.423 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:09.423 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 35 M 2023-11-28T01:02:09.423 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 210 M 2023-11-28T01:02:09.423 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:02:09.620 INFO:teuthology.orchestra.run.smithi192.stdout:(1/8): libgfortran-8.5.0-10.1.el8_6.x86_64.rpm 3.2 MB/s | 644 kB 00:00 2023-11-28T01:02:09.661 INFO:teuthology.orchestra.run.smithi192.stdout:(2/8): libquadmath-8.5.0-10.1.el8_6.x86_64.rpm 720 kB/s | 171 kB 00:00 2023-11-28T01:02:09.837 INFO:teuthology.orchestra.run.smithi192.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 21 MB/s | 3.7 MB 00:00 2023-11-28T01:02:09.979 INFO:teuthology.orchestra.run.smithi192.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.6 MB/s | 225 kB 00:00 2023-11-28T01:02:10.146 INFO:teuthology.orchestra.run.smithi192.stdout:(5/8): python3-scipy-1.0.0-21.module+el8.5.0+10 26 MB/s | 14 MB 00:00 2023-11-28T01:02:10.246 INFO:teuthology.orchestra.run.smithi192.stdout:(6/8): ceph-mgr-diskprediction-local-18.0.0-749 9.0 MB/s | 7.4 MB 00:00 2023-11-28T01:02:10.338 INFO:teuthology.orchestra.run.smithi192.stdout:(7/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 14 MB/s | 4.9 MB 00:00 2023-11-28T01:02:10.555 INFO:teuthology.orchestra.run.smithi192.stdout:(8/8): openblas-0.3.15-3.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2023-11-28T01:02:10.556 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:02:10.556 INFO:teuthology.orchestra.run.smithi192.stdout:Total 31 MB/s | 35 MB 00:01 2023-11-28T01:02:10.795 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:02:10.814 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:02:10.814 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:02:11.106 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:02:11.110 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:02:11.679 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:02:11.812 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-11-28T01:02:11.952 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-11-28T01:02:12.102 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:02:12.121 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-11-28T01:02:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:12.289 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:02:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:02:12.289 INFO:teuthology.orchestra.run.smithi172.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-28T01:02:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-28T01:02:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-28T01:02:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-28T01:02:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-28T01:02:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-28T01:02:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-28T01:02:12.290 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:12.290 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:02:12.438 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-rook 2023-11-28T01:02:12.723 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-11-28T01:02:12.784 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:02:12.921 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-11-28T01:02:13.514 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-11-28T01:02:13.720 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.15-3.el8.x86_64 4/8 2023-11-28T01:02:14.341 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:01 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:14.398 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 4/8 2023-11-28T01:02:14.724 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-28T01:02:14.895 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-28T01:02:15.764 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:02:15.766 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================== 2023-11-28T01:02:15.766 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-11-28T01:02:15.766 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================== 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 50 k 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-28T01:02:15.767 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-28T01:02:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-28T01:02:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-28T01:02:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-28T01:02:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:15.768 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:02:15.768 INFO:teuthology.orchestra.run.smithi172.stdout:======================================================================================================== 2023-11-28T01:02:15.768 INFO:teuthology.orchestra.run.smithi172.stdout:Install 13 Packages 2023-11-28T01:02:15.768 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:15.769 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 1.9 M 2023-11-28T01:02:15.769 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 25 M 2023-11-28T01:02:15.769 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:02:16.165 INFO:teuthology.orchestra.run.smithi172.stdout:(1/13): ceph-mgr-rook-18.0.0-7498.g369173db.el8 138 kB/s | 50 kB 00:00 2023-11-28T01:02:16.673 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-28T01:02:16.757 INFO:teuthology.orchestra.run.smithi172.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 17 kB/s | 16 kB 00:00 2023-11-28T01:02:16.833 INFO:teuthology.orchestra.run.smithi172.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 32 kB/s | 33 kB 00:01 2023-11-28T01:02:17.400 INFO:teuthology.orchestra.run.smithi172.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 113 kB/s | 64 kB 00:00 2023-11-28T01:02:17.425 INFO:teuthology.orchestra.run.smithi172.stdout:(5/13): python3-google-auth-1.1.1-10.el8.noarch 64 kB/s | 81 kB 00:01 2023-11-28T01:02:17.634 INFO:teuthology.orchestra.run.smithi172.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 204 kB/s | 43 kB 00:00 2023-11-28T01:02:17.743 INFO:teuthology.orchestra.run.smithi172.stdout:(7/13): python3-websocket-client-0.56.0-5.el8.n 179 kB/s | 61 kB 00:00 2023-11-28T01:02:17.802 INFO:teuthology.orchestra.run.smithi172.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 928 kB/s | 155 kB 00:00 2023-11-28T01:02:17.899 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 7/8 2023-11-28T01:02:17.927 INFO:teuthology.orchestra.run.smithi172.stdout:(9/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 212 kB/s | 27 kB 00:00 2023-11-28T01:02:17.977 INFO:teuthology.orchestra.run.smithi172.stdout:(10/13): python3-pyasn1-modules-0.3.7-6.el8.noa 471 kB/s | 110 kB 00:00 2023-11-28T01:02:18.044 INFO:teuthology.orchestra.run.smithi172.stdout:(11/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2023-11-28T01:02:18.062 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7498.g369173d 8/8 2023-11-28T01:02:18.152 INFO:teuthology.orchestra.run.smithi172.stdout:(12/13): python3-jsonpointer-1.10-11.el8.noarch 114 kB/s | 20 kB 00:00 2023-11-28T01:02:18.194 INFO:teuthology.orchestra.run.smithi172.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 831 kB/s | 1.2 MB 00:01 2023-11-28T01:02:18.197 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:02:18.198 INFO:teuthology.orchestra.run.smithi172.stdout:Total 815 kB/s | 1.9 MB 00:02 2023-11-28T01:02:18.265 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:02:18.282 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:02:18.282 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:02:18.351 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7498.g369173d 8/8 2023-11-28T01:02:18.351 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7498.g369173d 1/8 2023-11-28T01:02:18.351 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-11-28T01:02:18.351 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 3/8 2023-11-28T01:02:18.351 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 4/8 2023-11-28T01:02:18.351 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-28T01:02:18.351 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-28T01:02:18.351 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 7/8 2023-11-28T01:02:18.353 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:02:18.357 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:02:18.845 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:02:19.028 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-28T01:02:19.256 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-11-28T01:02:19.402 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 8/8 2023-11-28T01:02:19.402 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:06 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:19.418 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-11-28T01:02:19.569 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-11-28T01:02:19.790 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-11-28T01:02:20.025 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-28T01:02:20.199 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-28T01:02:20.412 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-11-28T01:02:20.594 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-28T01:02:20.835 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-11-28T01:02:21.920 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-11-28T01:02:22.147 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-11-28T01:02:22.287 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-rook-2:18.0.0-7498.g369173db.el8.noarch 13/13 2023-11-28T01:02:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7498.g369173db.el8.noarch 13/13 2023-11-28T01:02:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7498.g369173db.el8.noarch 1/13 2023-11-28T01:02:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-11-28T01:02:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-11-28T01:02:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-11-28T01:02:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-11-28T01:02:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-11-28T01:02:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-11-28T01:02:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2023-11-28T01:02:22.528 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2023-11-28T01:02:22.529 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-11-28T01:02:22.529 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 11/13 2023-11-28T01:02:22.529 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2023-11-28T01:02:23.684 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-11-28T01:02:23.684 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:10 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:25.753 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:02:25.938 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:25.938 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:02:25.938 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:02:25.938 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-11-28T01:02:25.939 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-11-28T01:02:25.939 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.15-3.el8.x86_64 2023-11-28T01:02:25.939 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-11-28T01:02:25.939 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-28T01:02:25.939 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-28T01:02:25.939 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-11-28T01:02:25.940 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:25.940 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:02:26.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-rook 2023-11-28T01:02:26.443 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:02:27.860 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:14 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:29.293 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:02:29.295 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================================== 2023-11-28T01:02:29.295 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-11-28T01:02:29.295 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================================== 2023-11-28T01:02:29.295 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:02:29.295 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 50 k 2023-11-28T01:02:29.295 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:29.296 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:02:29.297 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================================== 2023-11-28T01:02:29.297 INFO:teuthology.orchestra.run.smithi192.stdout:Install 13 Packages 2023-11-28T01:02:29.297 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:29.297 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.9 M 2023-11-28T01:02:29.297 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 25 M 2023-11-28T01:02:29.297 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:02:29.679 INFO:teuthology.orchestra.run.smithi192.stdout:(1/13): ceph-mgr-rook-18.0.0-7498.g369173db.el8 293 kB/s | 50 kB 00:00 2023-11-28T01:02:29.713 INFO:teuthology.orchestra.run.smithi192.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 80 kB/s | 16 kB 00:00 2023-11-28T01:02:29.746 INFO:teuthology.orchestra.run.smithi192.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 138 kB/s | 33 kB 00:00 2023-11-28T01:02:29.981 INFO:teuthology.orchestra.run.smithi192.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 274 kB/s | 64 kB 00:00 2023-11-28T01:02:29.998 INFO:teuthology.orchestra.run.smithi192.stdout:(5/13): python3-google-auth-1.1.1-10.el8.noarch 255 kB/s | 81 kB 00:00 2023-11-28T01:02:30.048 INFO:teuthology.orchestra.run.smithi192.stdout:(6/13): python3-websocket-client-0.56.0-5.el8.n 915 kB/s | 61 kB 00:00 2023-11-28T01:02:30.123 INFO:teuthology.orchestra.run.smithi192.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 341 kB/s | 43 kB 00:00 2023-11-28T01:02:30.157 INFO:teuthology.orchestra.run.smithi192.stdout:(8/13): python3-kubernetes-11.0.0-6.el8.noarch. 2.6 MB/s | 1.2 MB 00:00 2023-11-28T01:02:30.182 INFO:teuthology.orchestra.run.smithi192.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.1 MB/s | 155 kB 00:00 2023-11-28T01:02:30.241 INFO:teuthology.orchestra.run.smithi192.stdout:(10/13): python3-pyasn1-modules-0.3.7-6.el8.noa 939 kB/s | 110 kB 00:00 2023-11-28T01:02:30.283 INFO:teuthology.orchestra.run.smithi192.stdout:(11/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.2 MB/s | 126 kB 00:00 2023-11-28T01:02:30.299 INFO:teuthology.orchestra.run.smithi192.stdout:(12/13): python3-jsonpatch-1.21-2.el8.noarch.rp 186 kB/s | 27 kB 00:00 2023-11-28T01:02:30.333 INFO:teuthology.orchestra.run.smithi192.stdout:(13/13): python3-jsonpointer-1.10-11.el8.noarch 217 kB/s | 20 kB 00:00 2023-11-28T01:02:30.334 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:02:30.334 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.9 MB/s | 1.9 MB 00:01 2023-11-28T01:02:30.396 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:02:30.412 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:02:30.412 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:02:30.482 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:02:30.485 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:02:30.987 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:02:31.192 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-28T01:02:31.462 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-11-28T01:02:31.616 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-11-28T01:02:31.791 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-11-28T01:02:31.986 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-11-28T01:02:32.115 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:02:32.247 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-28T01:02:32.294 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:32.295 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:02:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-rook-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:02:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-28T01:02:32.296 INFO:teuthology.orchestra.run.smithi172.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-28T01:02:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-28T01:02:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-28T01:02:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-28T01:02:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-28T01:02:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-28T01:02:32.297 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-28T01:02:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-28T01:02:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-28T01:02:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-28T01:02:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-28T01:02:32.298 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:32.298 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:02:32.420 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-28T01:02:32.436 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-mgr-cephadm 2023-11-28T01:02:32.624 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-11-28T01:02:32.788 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:02:32.789 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-28T01:02:32.995 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-11-28T01:02:34.064 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-11-28T01:02:34.294 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-11-28T01:02:34.441 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:21 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:34.475 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:18.0.0-7498.g369173db.el8.noarch 13/13 2023-11-28T01:02:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7498.g369173db.el8.noarch 13/13 2023-11-28T01:02:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7498.g369173db.el8.noarch 1/13 2023-11-28T01:02:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-11-28T01:02:34.697 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-11-28T01:02:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-11-28T01:02:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-11-28T01:02:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-11-28T01:02:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-11-28T01:02:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2023-11-28T01:02:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2023-11-28T01:02:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-11-28T01:02:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 11/13 2023-11-28T01:02:34.698 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2023-11-28T01:02:35.809 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:02:35.810 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================================== 2023-11-28T01:02:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-11-28T01:02:35.810 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================================== 2023-11-28T01:02:35.810 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:02:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 133 k 2023-11-28T01:02:35.810 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-11-28T01:02:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-28T01:02:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-28T01:02:35.810 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-28T01:02:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-28T01:02:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:35.811 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:02:35.811 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================================== 2023-11-28T01:02:35.811 INFO:teuthology.orchestra.run.smithi172.stdout:Install 5 Packages 2023-11-28T01:02:35.811 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:35.811 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 6.0 M 2023-11-28T01:02:35.812 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 26 M 2023-11-28T01:02:35.812 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:02:35.826 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-11-28T01:02:35.826 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:22 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:36.531 INFO:teuthology.orchestra.run.smithi172.stdout:(1/5): ceph-mgr-cephadm-18.0.0-7498.g369173db.e 351 kB/s | 133 kB 00:00 2023-11-28T01:02:36.573 INFO:teuthology.orchestra.run.smithi172.stdout:(2/5): python3-natsort-7.1.1-2.el8.noarch.rpm 143 kB/s | 60 kB 00:00 2023-11-28T01:02:36.706 INFO:teuthology.orchestra.run.smithi172.stdout:(3/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.0 MB/s | 538 kB 00:00 2023-11-28T01:02:36.815 INFO:teuthology.orchestra.run.smithi172.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 718 kB/s | 476 kB 00:00 2023-11-28T01:02:36.882 INFO:teuthology.orchestra.run.smithi172.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 16 MB/s | 4.8 MB 00:00 2023-11-28T01:02:36.883 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:02:36.885 INFO:teuthology.orchestra.run.smithi172.stdout:Total 5.6 MB/s | 6.0 MB 00:01 2023-11-28T01:02:36.937 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:02:36.948 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:02:36.949 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:02:37.027 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:02:37.031 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:02:37.997 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:02:38.370 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2023-11-28T01:02:38.602 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2023-11-28T01:02:38.921 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-11-28T01:02:39.207 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-11-28T01:02:39.434 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7498.g369173db.el8.noarch 5/5 2023-11-28T01:02:39.671 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7498.g369173db.el8.noarch 5/5 2023-11-28T01:02:39.671 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7498.g369173db.el8.noarch 1/5 2023-11-28T01:02:39.671 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-11-28T01:02:39.672 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-11-28T01:02:39.672 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-11-28T01:02:40.687 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/5 2023-11-28T01:02:40.687 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:27 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:41.143 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:02:41.326 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:41.327 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:02:41.327 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:02:41.327 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-28T01:02:41.327 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-28T01:02:41.327 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-28T01:02:41.327 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-28T01:02:41.327 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-28T01:02:41.327 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-28T01:02:41.327 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-28T01:02:41.327 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-28T01:02:41.327 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-28T01:02:41.328 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-28T01:02:41.328 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-28T01:02:41.328 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-28T01:02:41.328 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:41.328 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:02:41.461 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-cephadm 2023-11-28T01:02:41.815 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:02:43.423 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:30 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:44.780 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:02:44.781 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================================== 2023-11-28T01:02:44.781 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-11-28T01:02:44.781 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================================== 2023-11-28T01:02:44.781 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:02:44.781 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 133 k 2023-11-28T01:02:44.781 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-11-28T01:02:44.781 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-11-28T01:02:44.782 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-11-28T01:02:44.782 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-11-28T01:02:44.782 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-11-28T01:02:44.782 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:44.782 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:02:44.782 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================================== 2023-11-28T01:02:44.782 INFO:teuthology.orchestra.run.smithi192.stdout:Install 5 Packages 2023-11-28T01:02:44.782 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:02:44.783 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 6.0 M 2023-11-28T01:02:44.783 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 26 M 2023-11-28T01:02:44.783 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:02:45.233 INFO:teuthology.orchestra.run.smithi192.stdout:(1/5): ceph-mgr-cephadm-18.0.0-7498.g369173db.e 455 kB/s | 133 kB 00:00 2023-11-28T01:02:45.408 INFO:teuthology.orchestra.run.smithi192.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 3.0 MB/s | 538 kB 00:00 2023-11-28T01:02:45.483 INFO:teuthology.orchestra.run.smithi192.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 111 kB/s | 60 kB 00:00 2023-11-28T01:02:45.633 INFO:teuthology.orchestra.run.smithi192.stdout:(4/5): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2023-11-28T01:02:46.246 INFO:teuthology.orchestra.run.smithi192.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 365 kB/s | 476 kB 00:01 2023-11-28T01:02:46.249 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:02:46.249 INFO:teuthology.orchestra.run.smithi192.stdout:Total 4.1 MB/s | 6.0 MB 00:01 2023-11-28T01:02:46.306 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:02:46.318 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:02:46.318 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:02:46.414 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:02:46.417 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:02:48.625 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:02:48.994 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2023-11-28T01:02:49.258 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2023-11-28T01:02:49.577 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-11-28T01:02:49.821 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:02:49.847 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-11-28T01:02:50.002 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:50.003 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:02:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-cephadm-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:02:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-28T01:02:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-28T01:02:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-28T01:02:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-28T01:02:50.003 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:50.003 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:02:50.065 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7498.g369173db.el8.noarch 5/5 2023-11-28T01:02:50.142 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-fuse 2023-11-28T01:02:50.282 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7498.g369173db.el8.noarch 5/5 2023-11-28T01:02:50.283 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7498.g369173db.el8.noarch 1/5 2023-11-28T01:02:50.283 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-11-28T01:02:50.283 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-11-28T01:02:50.283 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-11-28T01:02:50.498 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:02:51.320 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/5 2023-11-28T01:02:51.320 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:38 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:52.067 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:39 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:02:53.456 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:02:53.456 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:02:53.457 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-11-28T01:02:53.457 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:02:53.457 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:02:53.457 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse 2023-11-28T01:02:53.457 INFO:teuthology.orchestra.run.smithi172.stdout: x86_64 2:18.0.0-7498.g369173db.el8 ceph 921 k 2023-11-28T01:02:53.457 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2023-11-28T01:02:53.457 INFO:teuthology.orchestra.run.smithi172.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-28T01:02:53.458 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:53.458 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:02:53.458 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:02:53.458 INFO:teuthology.orchestra.run.smithi172.stdout:Install 2 Packages 2023-11-28T01:02:53.458 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:02:53.459 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 1.0 M 2023-11-28T01:02:53.459 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 3.1 M 2023-11-28T01:02:53.459 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:02:53.684 INFO:teuthology.orchestra.run.smithi172.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 367 kB/s | 83 kB 00:00 2023-11-28T01:02:54.384 INFO:teuthology.orchestra.run.smithi172.stdout:(2/2): ceph-fuse-18.0.0-7498.g369173db.el8.x86_ 995 kB/s | 921 kB 00:00 2023-11-28T01:02:54.384 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:02:54.385 INFO:teuthology.orchestra.run.smithi172.stdout:Total 1.1 MB/s | 1.0 MB 00:00 2023-11-28T01:02:54.396 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:02:54.417 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:02:54.418 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:02:54.491 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:02:54.495 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:02:54.673 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:02:54.944 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2023-11-28T01:02:55.121 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-fuse-2:18.0.0-7498.g369173db.el8.x86_64 2/2 2023-11-28T01:02:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7498.g369173db.el8.x86_64 2/2 2023-11-28T01:02:55.440 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-fuse-2:18.0.0-7498.g369173db.el8.x86_64 1/2 2023-11-28T01:02:56.800 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2023-11-28T01:02:56.801 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:03:43 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:00.433 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:03:00.615 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:00.615 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:03:00.615 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:03:00.616 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-11-28T01:03:00.616 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.5.1-7.el8.noarch 2023-11-28T01:03:00.616 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-11-28T01:03:00.616 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-11-28T01:03:00.616 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:00.616 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:03:00.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse 2023-11-28T01:03:01.126 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:03:03.036 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:50 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:10.809 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:03:10.809 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:10.809 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:03:10.809 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-fuse-2:18.0.0-7498.g369173db.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2023-11-28T01:03:10.809 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:10.809 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:03:10.810 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install ceph-volume 2023-11-28T01:03:10.811 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout: x86_64 2:18.0.0-7498.g369173db.el8 ceph 921 k 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:10.812 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.0 M 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 3.1 M 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 565 kB/s | 83 kB 00:00 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-fuse-18.0.0-7498.g369173db.el8.x86_ 1.1 MB/s | 921 kB 00:00 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.2 MB/s | 1.0 MB 00:00 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:03:10.813 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:03:10.814 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:03:10.814 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2023-11-28T01:03:10.814 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:18.0.0-7498.g369173db.el8.x86_64 2/2 2023-11-28T01:03:10.814 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7498.g369173db.el8.x86_64 2/2 2023-11-28T01:03:10.814 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:18.0.0-7498.g369173db.el8.x86_64 1/2 2023-11-28T01:03:10.814 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2023-11-28T01:03:10.841 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:54 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:11.212 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:03:13.066 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:00 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:14.461 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:03:14.462 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:03:14.462 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repository Size 2023-11-28T01:03:14.462 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:03:14.463 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:03:14.463 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 262 k 2023-11-28T01:03:14.463 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:14.463 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:03:14.463 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:03:14.463 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-11-28T01:03:14.463 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:14.463 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 262 k 2023-11-28T01:03:14.463 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 1.3 M 2023-11-28T01:03:14.464 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:03:14.823 INFO:teuthology.orchestra.run.smithi172.stdout:ceph-volume-18.0.0-7498.g369173db.el8.noarch.rp 726 kB/s | 262 kB 00:00 2023-11-28T01:03:14.824 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:03:14.824 INFO:teuthology.orchestra.run.smithi172.stdout:Total 724 kB/s | 262 kB 00:00 2023-11-28T01:03:14.824 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:03:14.835 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:03:14.835 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:03:14.854 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:03:14.856 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:03:15.023 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:03:15.294 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : ceph-volume-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:03:15.640 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: ceph-volume-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:03:16.723 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : ceph-volume-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:03:16.724 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:03 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:17.253 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:03:17.433 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:17.433 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:03:17.433 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:18.0.0-7498.g369173db.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2023-11-28T01:03:17.433 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:17.433 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:03:17.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-volume 2023-11-28T01:03:17.933 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:03:19.663 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:06 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:21.050 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:03:21.051 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:21.051 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2023-11-28T01:03:21.051 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:21.051 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:03:21.052 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:18.0.0-7498.g369173db.el8 ceph-noarch 262 k 2023-11-28T01:03:21.052 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:21.052 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:03:21.052 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:21.052 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-11-28T01:03:21.052 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:21.052 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 262 k 2023-11-28T01:03:21.052 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.3 M 2023-11-28T01:03:21.052 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:03:21.388 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-volume-18.0.0-7498.g369173db.el8.noarch.rp 779 kB/s | 262 kB 00:00 2023-11-28T01:03:21.388 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:03:21.389 INFO:teuthology.orchestra.run.smithi192.stdout:Total 776 kB/s | 262 kB 00:00 2023-11-28T01:03:21.389 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:03:21.400 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:03:21.400 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:03:21.417 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:03:21.420 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:03:21.585 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:03:21.852 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:03:22.191 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:03:23.295 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:18.0.0-7498.g369173db.el8.noarch 1/1 2023-11-28T01:03:23.295 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:10 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:27.334 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:03:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:27.516 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:03:27.516 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-volume-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:03:27.517 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:27.517 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:03:27.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados-devel 2023-11-28T01:03:28.013 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:03:29.761 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:16 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:31.192 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:03:31.193 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:03:31.193 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-11-28T01:03:31.193 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:03:31.193 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:03:31.193 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel x86_64 2:18.0.0-7498.g369173db.el8 ceph 139 k 2023-11-28T01:03:31.193 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:31.193 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:03:31.193 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:03:31.193 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-11-28T01:03:31.193 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:31.194 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 139 k 2023-11-28T01:03:31.194 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 513 k 2023-11-28T01:03:31.194 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:03:31.402 INFO:teuthology.orchestra.run.smithi172.stdout:librados-devel-18.0.0-7498.g369173db.el8.x86_64 662 kB/s | 139 kB 00:00 2023-11-28T01:03:31.402 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:03:31.403 INFO:teuthology.orchestra.run.smithi172.stdout:Total 658 kB/s | 139 kB 00:00 2023-11-28T01:03:31.403 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:03:31.409 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:03:31.409 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:03:31.463 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:03:31.466 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:03:31.578 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:03:31.739 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : librados-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:31.958 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: librados-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:32.419 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:03:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:32.603 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:03:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:18.0.0-7498.g369173db.el8.noarch 2023-11-28T01:03:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:32.603 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:03:32.775 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel 2023-11-28T01:03:33.039 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : librados-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:33.039 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:20 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:33.132 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:03:34.674 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:21 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:36.042 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:03:36.043 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:36.043 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-11-28T01:03:36.043 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:36.043 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:03:36.043 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:18.0.0-7498.g369173db.el8 ceph 139 k 2023-11-28T01:03:36.043 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:36.043 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:03:36.044 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:36.044 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-11-28T01:03:36.044 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:36.044 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 139 k 2023-11-28T01:03:36.044 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 513 k 2023-11-28T01:03:36.045 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:03:36.351 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-18.0.0-7498.g369173db.el8.x86_64 453 kB/s | 139 kB 00:00 2023-11-28T01:03:36.352 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:03:36.352 INFO:teuthology.orchestra.run.smithi192.stdout:Total 451 kB/s | 139 kB 00:00 2023-11-28T01:03:36.352 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:03:36.357 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:03:36.357 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:03:36.412 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:03:36.415 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:03:36.547 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:03:36.706 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:36.938 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:38.016 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:38.016 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:25 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:41.923 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:03:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:42.104 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:03:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: librados-devel-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:03:42.104 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:42.104 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:03:42.235 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs2 2023-11-28T01:03:42.591 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:03:44.146 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:31 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:45.433 INFO:teuthology.orchestra.run.smithi172.stdout:Package libcephfs2-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:03:45.540 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:03:45.542 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-11-28T01:03:45.542 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:03:45.655 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install libcephfs-devel 2023-11-28T01:03:46.012 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:03:46.190 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:03:46.375 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:46.376 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:03:46.376 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:03:46.376 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:46.376 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:03:46.524 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2 2023-11-28T01:03:46.878 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:03:47.678 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:34 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:48.682 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:35 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:49.065 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:03:49.065 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel x86_64 2:18.0.0-7498.g369173db.el8 ceph 31 k 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 31 k 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 139 k 2023-11-28T01:03:49.066 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:03:49.237 INFO:teuthology.orchestra.run.smithi172.stdout:libcephfs-devel-18.0.0-7498.g369173db.el8.x86_6 182 kB/s | 31 kB 00:00 2023-11-28T01:03:49.238 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:03:49.238 INFO:teuthology.orchestra.run.smithi172.stdout:Total 180 kB/s | 31 kB 00:00 2023-11-28T01:03:49.238 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:03:49.242 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:03:49.242 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:03:49.252 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:03:49.256 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:03:49.310 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:03:49.510 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : libcephfs-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:49.740 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:49.932 INFO:teuthology.orchestra.run.smithi192.stdout:Package libcephfs2-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:03:50.041 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:03:50.042 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-11-28T01:03:50.042 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:03:50.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs-devel 2023-11-28T01:03:50.525 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:03:50.737 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : libcephfs-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:50.737 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:37 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:52.193 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:39 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:53.553 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:03:53.553 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:53.553 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-11-28T01:03:53.553 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:53.553 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:03:53.553 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:18.0.0-7498.g369173db.el8 ceph 31 k 2023-11-28T01:03:53.554 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:53.554 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:03:53.554 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:03:53.554 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-11-28T01:03:53.554 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:03:53.555 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 31 k 2023-11-28T01:03:53.555 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 139 k 2023-11-28T01:03:53.555 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:03:53.896 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs-devel-18.0.0-7498.g369173db.el8.x86_6 91 kB/s | 31 kB 00:00 2023-11-28T01:03:53.896 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:03:53.896 INFO:teuthology.orchestra.run.smithi192.stdout:Total 90 kB/s | 31 kB 00:00 2023-11-28T01:03:53.897 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:03:53.901 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:03:53.901 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:03:53.911 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:03:53.915 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:03:53.972 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:03:54.128 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:54.322 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:55.306 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:03:55.306 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:42 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:03:59.911 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:04:00.091 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:00.091 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:04:00.091 INFO:teuthology.orchestra.run.smithi172.stdout: libcephfs-devel-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:04:00.091 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:00.091 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:04:00.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librados2 2023-11-28T01:04:00.576 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:02.234 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:49 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:03.512 INFO:teuthology.orchestra.run.smithi172.stdout:Package librados2-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:03.621 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:04:03.622 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-11-28T01:04:03.622 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:04:03.736 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install librbd1 2023-11-28T01:04:04.092 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:05.454 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:04:05.634 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:04:05.634 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:04:05.634 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:04:05.634 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:04:05.634 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:04:05.766 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2 2023-11-28T01:04:05.987 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:52 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:06.120 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:07.273 INFO:teuthology.orchestra.run.smithi172.stdout:Package librbd1-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:07.384 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:04:07.385 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-11-28T01:04:07.386 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:04:07.488 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rados 2023-11-28T01:04:07.668 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:54 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:07.846 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:08.917 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:09.028 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:04:09.029 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-11-28T01:04:09.029 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:04:09.143 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1 2023-11-28T01:04:09.399 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:56 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:09.494 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:10.738 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rados-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:10.847 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:04:10.848 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-11-28T01:04:10.849 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:04:10.966 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rgw 2023-11-28T01:04:11.179 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:58 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:11.323 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:12.462 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:12.573 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:04:12.575 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-11-28T01:04:12.575 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:04:12.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados 2023-11-28T01:04:12.831 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:04:59 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:13.045 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:14.081 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rgw-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:14.188 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:04:14.189 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-11-28T01:04:14.190 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:04:14.298 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-cephfs 2023-11-28T01:04:14.559 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:01 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:14.653 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:15.824 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rados-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:15.934 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:04:15.935 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-11-28T01:04:15.935 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:04:16.051 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw 2023-11-28T01:04:16.154 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:03 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:16.404 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:17.418 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-cephfs-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:17.530 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:04:17.530 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-11-28T01:04:17.530 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:04:17.660 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install python3-rbd 2023-11-28T01:04:17.898 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:04 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:18.013 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:19.153 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rgw-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:19.263 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:04:19.264 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-11-28T01:04:19.264 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:04:19.387 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs 2023-11-28T01:04:19.440 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:06 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:19.740 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:20.695 INFO:teuthology.orchestra.run.smithi172.stdout:Package python3-rbd-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:20.805 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:04:20.806 INFO:teuthology.orchestra.run.smithi172.stdout:Nothing to do. 2023-11-28T01:04:20.806 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:04:20.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-fuse 2023-11-28T01:04:21.278 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:21.356 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:08 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:22.642 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-cephfs-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:22.747 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:04:22.748 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-11-28T01:04:22.749 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:04:22.809 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:09 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:22.869 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd 2023-11-28T01:04:23.226 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:24.188 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:04:24.189 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:04:24.189 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2023-11-28T01:04:24.189 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:04:24.189 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:04:24.189 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse x86_64 2:18.0.0-7498.g369173db.el8 ceph 94 k 2023-11-28T01:04:24.189 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:24.189 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:04:24.189 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:04:24.190 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-11-28T01:04:24.190 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:24.190 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 94 k 2023-11-28T01:04:24.190 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 280 k 2023-11-28T01:04:24.190 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:04:24.621 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-fuse-18.0.0-7498.g369173db.el8.x86_64.rpm 217 kB/s | 94 kB 00:00 2023-11-28T01:04:24.621 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:04:24.621 INFO:teuthology.orchestra.run.smithi172.stdout:Total 217 kB/s | 94 kB 00:00 2023-11-28T01:04:24.622 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:04:24.632 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:04:24.632 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:04:24.687 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:04:24.690 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:04:24.796 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:04:24.946 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-fuse-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:24.962 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:11 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:25.178 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:26.221 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-fuse-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:26.221 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:13 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:26.234 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rbd-2:18.0.0-7498.g369173db.el8.x86_64 is already installed. 2023-11-28T01:04:26.346 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:04:26.347 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2023-11-28T01:04:26.347 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:04:26.468 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse 2023-11-28T01:04:26.822 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:28.895 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:15 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:30.328 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:04:30.329 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:04:30.329 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2023-11-28T01:04:30.329 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:04:30.329 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:04:30.329 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:18.0.0-7498.g369173db.el8 ceph 94 k 2023-11-28T01:04:30.329 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:04:30.329 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:04:30.329 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:04:30.330 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-11-28T01:04:30.330 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:04:30.330 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 94 k 2023-11-28T01:04:30.330 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 280 k 2023-11-28T01:04:30.330 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:04:30.622 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-18.0.0-7498.g369173db.el8.x86_64.rpm 320 kB/s | 94 kB 00:00 2023-11-28T01:04:30.623 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:04:30.623 INFO:teuthology.orchestra.run.smithi192.stdout:Total 319 kB/s | 94 kB 00:00 2023-11-28T01:04:30.623 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:04:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:04:30.634 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:04:30.688 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:04:30.692 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:04:30.800 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:04:30.964 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:31.202 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:32.272 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:32.272 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:19 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:37.719 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:04:37.900 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:37.900 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:04:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-fuse-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:04:37.901 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:37.901 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:04:38.021 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-mirror 2023-11-28T01:04:38.379 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:40.099 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:27 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:41.490 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:04:41.494 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:04:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: Package Arch Version Repo Size 2023-11-28T01:04:41.494 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:04:41.494 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:04:41.494 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror x86_64 2:18.0.0-7498.g369173db.el8 ceph 3.6 M 2023-11-28T01:04:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:41.495 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:04:41.495 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:04:41.495 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-11-28T01:04:41.495 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:41.495 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 3.6 M 2023-11-28T01:04:41.495 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 17 M 2023-11-28T01:04:41.495 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:04:42.072 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-mirror-18.0.0-7498.g369173db.el8.x86_64.rpm 6.2 MB/s | 3.6 MB 00:00 2023-11-28T01:04:42.073 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:04:42.073 INFO:teuthology.orchestra.run.smithi172.stdout:Total 6.2 MB/s | 3.6 MB 00:00 2023-11-28T01:04:42.074 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:04:42.086 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:04:42.086 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:04:42.155 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:04:42.158 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:04:42.620 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:04:42.796 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-mirror-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:43.169 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:43.244 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:04:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:04:43.427 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:04:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:04:43.427 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:04:43.427 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:04:43.572 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror 2023-11-28T01:04:43.929 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:44.195 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-mirror-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:44.195 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:31 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:45.620 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:32 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:47.004 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:04:47.005 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:04:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2023-11-28T01:04:47.006 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:04:47.006 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:04:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:18.0.0-7498.g369173db.el8 ceph 3.6 M 2023-11-28T01:04:47.006 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:04:47.007 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:04:47.007 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:04:47.007 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-11-28T01:04:47.007 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:04:47.007 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.6 M 2023-11-28T01:04:47.007 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 17 M 2023-11-28T01:04:47.008 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:04:48.666 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-18.0.0-7498.g369173db.el8.x86_64.rpm 2.2 MB/s | 3.6 MB 00:01 2023-11-28T01:04:48.667 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:04:48.667 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.2 MB/s | 3.6 MB 00:01 2023-11-28T01:04:48.668 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:04:48.682 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:04:48.682 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:04:48.754 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:04:48.758 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:04:49.211 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:04:49.406 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:49.746 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:50.814 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:50.814 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:37 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:55.143 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:04:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:55.325 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:04:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-mirror-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:04:55.325 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:55.325 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:04:55.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum -y install rbd-nbd 2023-11-28T01:04:55.817 INFO:teuthology.orchestra.run.smithi172.stdout:Updating Subscription Management repositories. 2023-11-28T01:04:57.364 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:44 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:04:58.753 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2023-11-28T01:04:58.753 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:04:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2023-11-28T01:04:58.754 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:04:58.754 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2023-11-28T01:04:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd x86_64 2:18.0.0-7498.g369173db.el8 ceph 181 k 2023-11-28T01:04:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:58.754 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2023-11-28T01:04:58.754 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2023-11-28T01:04:58.754 INFO:teuthology.orchestra.run.smithi172.stdout:Install 1 Package 2023-11-28T01:04:58.754 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:04:58.755 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 181 k 2023-11-28T01:04:58.755 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 572 k 2023-11-28T01:04:58.755 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2023-11-28T01:04:59.058 INFO:teuthology.orchestra.run.smithi172.stdout:rbd-nbd-18.0.0-7498.g369173db.el8.x86_64.rpm 597 kB/s | 181 kB 00:00 2023-11-28T01:04:59.074 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:04:59.074 INFO:teuthology.orchestra.run.smithi172.stdout:Total 595 kB/s | 181 kB 00:00 2023-11-28T01:04:59.074 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2023-11-28T01:04:59.074 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2023-11-28T01:04:59.074 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2023-11-28T01:04:59.124 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2023-11-28T01:04:59.130 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2023-11-28T01:04:59.239 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2023-11-28T01:04:59.579 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:04:59.590 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : rbd-nbd-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:59.760 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:04:59.760 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:04:59.761 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:04:59.761 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:04:59.761 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:04:59.831 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:04:59.895 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd 2023-11-28T01:05:00.251 INFO:teuthology.orchestra.run.smithi192.stdout:Updating Subscription Management repositories. 2023-11-28T01:05:00.889 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : rbd-nbd-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:05:00.889 INFO:teuthology.orchestra.run.smithi172.stdout:Last metadata expiration check: 0:05:47 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:05:01.958 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:48 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:05:03.321 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2023-11-28T01:05:03.322 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:05:03.322 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2023-11-28T01:05:03.322 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:05:03.322 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2023-11-28T01:05:03.322 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:18.0.0-7498.g369173db.el8 ceph 181 k 2023-11-28T01:05:03.322 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:05:03.323 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2023-11-28T01:05:03.323 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2023-11-28T01:05:03.323 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2023-11-28T01:05:03.323 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:05:03.323 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 181 k 2023-11-28T01:05:03.323 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 572 k 2023-11-28T01:05:03.323 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2023-11-28T01:05:03.578 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-18.0.0-7498.g369173db.el8.x86_64.rpm 710 kB/s | 181 kB 00:00 2023-11-28T01:05:03.579 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:05:03.579 INFO:teuthology.orchestra.run.smithi192.stdout:Total 706 kB/s | 181 kB 00:00 2023-11-28T01:05:03.579 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2023-11-28T01:05:03.590 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2023-11-28T01:05:03.590 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2023-11-28T01:05:03.644 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2023-11-28T01:05:03.647 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2023-11-28T01:05:03.762 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2023-11-28T01:05:03.909 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:05:04.124 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:05:05.218 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:18.0.0-7498.g369173db.el8.x86_64 1/1 2023-11-28T01:05:05.218 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:52 ago on Tue 28 Nov 2023 12:59:13 AM UTC. 2023-11-28T01:05:11.139 INFO:teuthology.orchestra.run.smithi172.stdout:Installed products updated. 2023-11-28T01:05:11.321 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:05:11.321 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2023-11-28T01:05:11.321 INFO:teuthology.orchestra.run.smithi172.stdout: rbd-nbd-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:05:11.321 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:05:11.321 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2023-11-28T01:05:11.456 DEBUG:teuthology.parallel:result is None 2023-11-28T01:05:15.671 INFO:teuthology.orchestra.run.smithi192.stdout:Installed products updated. 2023-11-28T01:05:15.855 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:05:15.855 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2023-11-28T01:05:15.855 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:18.0.0-7498.g369173db.el8.x86_64 2023-11-28T01:05:15.855 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:05:15.855 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2023-11-28T01:05:16.029 DEBUG:teuthology.parallel:result is None 2023-11-28T01:05:16.030 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=369173db14b6995b2bd07c60ec5f63d01cf21631 2023-11-28T01:05:16.172 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T01:05:16.198 INFO:teuthology.orchestra.run.smithi172.stdout:18.0.0-7498.g369173db.el8 2023-11-28T01:05:16.199 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7498.g369173db.el8 2023-11-28T01:05:16.199 INFO:teuthology.task.install:The correct ceph version 18.0.0-7498.g369173db is installed. 2023-11-28T01:05:16.199 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=369173db14b6995b2bd07c60ec5f63d01cf21631 2023-11-28T01:05:16.340 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-28T01:05:16.365 INFO:teuthology.orchestra.run.smithi192.stdout:18.0.0-7498.g369173db.el8 2023-11-28T01:05:16.366 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7498.g369173db.el8 2023-11-28T01:05:16.366 INFO:teuthology.task.install:The correct ceph version 18.0.0-7498.g369173db is installed. 2023-11-28T01:05:16.366 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-28T01:05:16.366 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:05:16.366 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T01:05:16.400 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T01:05:16.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-28T01:05:16.432 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-28T01:05:16.433 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:05:16.433 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T01:05:16.471 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T01:05:16.541 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T01:05:16.541 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2023-11-28T01:05:16.567 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-28T01:05:16.633 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-28T01:05:16.634 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:05:16.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T01:05:16.662 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T01:05:16.730 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T01:05:16.731 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-28T01:05:16.756 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-28T01:05:16.822 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-28T01:05:16.823 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:05:16.823 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T01:05:16.851 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T01:05:16.919 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T01:05:16.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2023-11-28T01:05:16.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-28T01:05:17.011 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-28T01:05:17.093 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 1, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '369173db14b6995b2bd07c60ec5f63d01cf21631', 'cephadm_mode': 'cephadm-package'} 2023-11-28T01:05:17.093 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 2023-11-28T01:05:17.094 INFO:tasks.cephadm:Cluster fsid is 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:05:17.094 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-28T01:05:17.094 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.172', 'mon.c': '[v2:172.21.15.172:3301,v1:172.21.15.172:6790]', 'mon.b': '172.21.15.192'} 2023-11-28T01:05:17.094 INFO:tasks.cephadm:First mon is mon.a on smithi172 2023-11-28T01:05:17.094 INFO:tasks.cephadm:First mgr is y 2023-11-28T01:05:17.094 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-28T01:05:17.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo hostname $(hostname -s) 2023-11-28T01:05:17.123 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2023-11-28T01:05:17.164 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 on all hosts... 2023-11-28T01:05:17.165 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 pull 2023-11-28T01:05:17.167 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 pull 2023-11-28T01:05:18.155 INFO:teuthology.orchestra.run.smithi172.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631... 2023-11-28T01:05:18.387 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631... 2023-11-28T01:06:35.873 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2023-11-28T01:06:35.873 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_version": "ceph version 18.0.0-7498-g369173db (369173db14b6995b2bd07c60ec5f63d01cf21631) squid (dev)", 2023-11-28T01:06:35.873 INFO:teuthology.orchestra.run.smithi172.stdout: "image_id": "7cc7cb22378ea340aa30f458321cde9f9aaeaa13a42c56b1e5524abb2fc9827f", 2023-11-28T01:06:35.873 INFO:teuthology.orchestra.run.smithi172.stdout: "repo_digests": [ 2023-11-28T01:06:35.874 INFO:teuthology.orchestra.run.smithi172.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe7adc4523e418af299591f3c6d30928eb22748e2d0084aed8a14ab0695eaa4d" 2023-11-28T01:06:35.874 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2023-11-28T01:06:35.874 INFO:teuthology.orchestra.run.smithi172.stdout:} 2023-11-28T01:06:49.620 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2023-11-28T01:06:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 18.0.0-7498-g369173db (369173db14b6995b2bd07c60ec5f63d01cf21631) squid (dev)", 2023-11-28T01:06:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "7cc7cb22378ea340aa30f458321cde9f9aaeaa13a42c56b1e5524abb2fc9827f", 2023-11-28T01:06:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2023-11-28T01:06:49.620 INFO:teuthology.orchestra.run.smithi192.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:fe7adc4523e418af299591f3c6d30928eb22748e2d0084aed8a14ab0695eaa4d" 2023-11-28T01:06:49.621 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2023-11-28T01:06:49.621 INFO:teuthology.orchestra.run.smithi192.stdout:} 2023-11-28T01:06:49.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph 2023-11-28T01:06:49.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2023-11-28T01:06:49.750 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /etc/ceph 2023-11-28T01:06:49.778 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2023-11-28T01:06:49.819 INFO:tasks.cephadm:Writing seed config... 2023-11-28T01:06:49.820 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-28T01:06:49.820 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-28T01:06:49.820 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-28T01:06:49.820 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2023-11-28T01:06:49.820 INFO:tasks.cephadm: override: [global] ms type = async 2023-11-28T01:06:49.821 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-28T01:06:49.821 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-28T01:06:49.821 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-28T01:06:49.821 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2023-11-28T01:06:49.821 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-28T01:06:49.821 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-28T01:06:49.821 INFO:tasks.cephadm: override: [osd] osd class default list = * 2023-11-28T01:06:49.821 INFO:tasks.cephadm: override: [osd] osd class load list = * 2023-11-28T01:06:49.821 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-11-28T01:06:49.822 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:06:49.822 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-28T01:06:49.840 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 31861c10-8d8a-11ee-95a2-87774f69a715 mon election default strategy = 1 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2023-11-28T01:06:49.841 DEBUG:teuthology.orchestra.run.smithi172:mon.a> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.a.service 2023-11-28T01:06:49.883 DEBUG:teuthology.orchestra.run.smithi172:mgr.y> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mgr.y.service 2023-11-28T01:06:49.910 INFO:journalctl@ceph.mon.a.smithi172.stdout:-- Logs begin at Tue 2023-11-28 00:47:41 UTC. -- 2023-11-28T01:06:49.911 INFO:tasks.cephadm:Bootstrapping... 2023-11-28T01:06:49.911 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 -v bootstrap --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.172 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-28T01:06:49.937 INFO:journalctl@ceph.mgr.y.smithi172.stdout:-- Logs begin at Tue 2023-11-28 00:47:41 UTC. -- 2023-11-28T01:06:50.109 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2023-11-28T01:06:50.110 INFO:teuthology.orchestra.run.smithi172.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631', '-v', 'bootstrap', '--fsid', '31861c10-8d8a-11ee-95a2-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.172', '--skip-admin-label'] 2023-11-28T01:06:50.168 INFO:teuthology.orchestra.run.smithi172.stdout:/bin/podman: stdout 3.0.1 2023-11-28T01:06:50.169 INFO:teuthology.orchestra.run.smithi172.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-11-28T01:06:50.169 INFO:teuthology.orchestra.run.smithi172.stdout:Verifying podman|docker is present... 2023-11-28T01:06:50.229 INFO:teuthology.orchestra.run.smithi172.stdout:/bin/podman: stdout 3.0.1 2023-11-28T01:06:50.229 INFO:teuthology.orchestra.run.smithi172.stdout:Verifying lvm2 is present... 2023-11-28T01:06:50.229 INFO:teuthology.orchestra.run.smithi172.stdout:Verifying time synchronization is in place... 2023-11-28T01:06:50.238 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-11-28T01:06:50.238 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-11-28T01:06:50.241 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-11-28T01:06:50.241 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout inactive 2023-11-28T01:06:50.247 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout enabled 2023-11-28T01:06:50.253 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout active 2023-11-28T01:06:50.253 INFO:teuthology.orchestra.run.smithi172.stdout:Unit chronyd.service is enabled and running 2023-11-28T01:06:50.253 INFO:teuthology.orchestra.run.smithi172.stdout:Repeating the final host check... 2023-11-28T01:06:50.314 INFO:teuthology.orchestra.run.smithi172.stdout:/bin/podman: stdout 3.0.1 2023-11-28T01:06:50.314 INFO:teuthology.orchestra.run.smithi172.stdout:podman (/bin/podman) version 3.0.1 is present 2023-11-28T01:06:50.314 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl is present 2023-11-28T01:06:50.314 INFO:teuthology.orchestra.run.smithi172.stdout:lvcreate is present 2023-11-28T01:06:50.320 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-11-28T01:06:50.320 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-11-28T01:06:50.325 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-11-28T01:06:50.326 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout inactive 2023-11-28T01:06:50.331 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout enabled 2023-11-28T01:06:50.337 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout active 2023-11-28T01:06:50.338 INFO:teuthology.orchestra.run.smithi172.stdout:Unit chronyd.service is enabled and running 2023-11-28T01:06:50.338 INFO:teuthology.orchestra.run.smithi172.stdout:Host looks OK 2023-11-28T01:06:50.338 INFO:teuthology.orchestra.run.smithi172.stdout:Cluster fsid: 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:06:50.338 INFO:teuthology.orchestra.run.smithi172.stdout:Acquiring lock 140298953274536 on /run/cephadm/31861c10-8d8a-11ee-95a2-87774f69a715.lock 2023-11-28T01:06:50.338 INFO:teuthology.orchestra.run.smithi172.stdout:Lock 140298953274536 acquired on /run/cephadm/31861c10-8d8a-11ee-95a2-87774f69a715.lock 2023-11-28T01:06:50.338 INFO:teuthology.orchestra.run.smithi172.stdout:Verifying IP 172.21.15.172 port 3300 ... 2023-11-28T01:06:50.339 INFO:teuthology.orchestra.run.smithi172.stdout:Verifying IP 172.21.15.172 port 6789 ... 2023-11-28T01:06:50.339 INFO:teuthology.orchestra.run.smithi172.stdout:Base mon IP(s) is [172.21.15.172:3300, 172.21.15.172:6789], mon addrv is [v2:172.21.15.172:3300,v1:172.21.15.172:6789] 2023-11-28T01:06:50.342 INFO:teuthology.orchestra.run.smithi172.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2023-11-28T01:06:50.342 INFO:teuthology.orchestra.run.smithi172.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.172 metric 100 2023-11-28T01:06:50.345 INFO:teuthology.orchestra.run.smithi172.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-11-28T01:06:50.345 INFO:teuthology.orchestra.run.smithi172.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2023-11-28T01:06:50.345 INFO:teuthology.orchestra.run.smithi172.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2023-11-28T01:06:50.347 INFO:teuthology.orchestra.run.smithi172.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-28T01:06:50.348 INFO:teuthology.orchestra.run.smithi172.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-11-28T01:06:50.348 INFO:teuthology.orchestra.run.smithi172.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-11-28T01:06:50.348 INFO:teuthology.orchestra.run.smithi172.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-11-28T01:06:50.348 INFO:teuthology.orchestra.run.smithi172.stdout:/sbin/ip: stdout inet6 fe80::f68f:48af:2004:9cca/64 scope link noprefixroute 2023-11-28T01:06:50.348 INFO:teuthology.orchestra.run.smithi172.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-11-28T01:06:50.349 INFO:teuthology.orchestra.run.smithi172.stdout:Mon IP `172.21.15.172` is in CIDR network `172.21.0.0/20` 2023-11-28T01:06:50.349 INFO:teuthology.orchestra.run.smithi172.stdout:Mon IP `172.21.15.172` is in CIDR network `172.21.0.0/20` 2023-11-28T01:06:50.349 INFO:teuthology.orchestra.run.smithi172.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-11-28T01:06:50.349 INFO:teuthology.orchestra.run.smithi172.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-28T01:06:50.350 INFO:teuthology.orchestra.run.smithi172.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631... 2023-11-28T01:06:50.752 INFO:teuthology.orchestra.run.smithi172.stdout:/bin/podman: stdout 7cc7cb22378ea340aa30f458321cde9f9aaeaa13a42c56b1e5524abb2fc9827f 2023-11-28T01:06:50.752 INFO:teuthology.orchestra.run.smithi172.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631... 2023-11-28T01:06:50.753 INFO:teuthology.orchestra.run.smithi172.stdout:/bin/podman: stderr Getting image source signatures 2023-11-28T01:06:50.753 INFO:teuthology.orchestra.run.smithi172.stdout:/bin/podman: stderr Copying blob sha256:3d22195da913bce20c81704c00a9cc88d1593e21a55d76d50b2cbe16d0bcd02c 2023-11-28T01:06:50.753 INFO:teuthology.orchestra.run.smithi172.stdout:/bin/podman: stderr Copying blob sha256:6e043562390995b4d66f5d024d621ebdd5ed686f9aa0f5304b2a4423b709787c 2023-11-28T01:06:50.753 INFO:teuthology.orchestra.run.smithi172.stdout:/bin/podman: stderr Copying config sha256:7cc7cb22378ea340aa30f458321cde9f9aaeaa13a42c56b1e5524abb2fc9827f 2023-11-28T01:06:50.753 INFO:teuthology.orchestra.run.smithi172.stdout:/bin/podman: stderr Writing manifest to image destination 2023-11-28T01:06:50.753 INFO:teuthology.orchestra.run.smithi172.stdout:/bin/podman: stderr Storing signatures 2023-11-28T01:06:51.756 INFO:teuthology.orchestra.run.smithi172.stdout:ceph: stdout ceph version 18.0.0-7498-g369173db (369173db14b6995b2bd07c60ec5f63d01cf21631) squid (dev) 2023-11-28T01:06:51.756 INFO:teuthology.orchestra.run.smithi172.stdout:Ceph version: ceph version 18.0.0-7498-g369173db (369173db14b6995b2bd07c60ec5f63d01cf21631) squid (dev) 2023-11-28T01:06:51.757 INFO:teuthology.orchestra.run.smithi172.stdout:Extracting ceph user uid/gid from container image... 2023-11-28T01:06:52.593 INFO:teuthology.orchestra.run.smithi172.stdout:stat: stdout 167 167 2023-11-28T01:06:52.593 INFO:teuthology.orchestra.run.smithi172.stdout:Creating initial keys... 2023-11-28T01:06:53.505 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph-authtool: stdout AQAsPWVlNCPNNxAAiyd7pvanRG/A6OPylmitIQ== 2023-11-28T01:06:54.350 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph-authtool: stdout AQAtPWVl1D8ENBAAElJFuQ3gPNkZru2dh+E7Cw== 2023-11-28T01:06:55.184 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph-authtool: stdout AQAuPWVlEgjiJxAAl6FRvKc+c+BTmq4NJfNE+Q== 2023-11-28T01:06:55.185 INFO:teuthology.orchestra.run.smithi172.stdout:Creating initial monmap... 2023-11-28T01:06:56.027 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-28T01:06:56.027 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2023-11-28T01:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-28T01:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool for a [v2:172.21.15.172:3300,v1:172.21.15.172:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-28T01:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout:setting min_mon_release = quincy 2023-11-28T01:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/monmaptool: set fsid to 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-28T01:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:06:56.028 INFO:teuthology.orchestra.run.smithi172.stdout:Creating mon... 2023-11-28T01:06:56.932 INFO:teuthology.orchestra.run.smithi172.stdout:create mon.a on 2023-11-28T01:06:57.058 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-28T01:06:57.143 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-28T01:06:57.258 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-31861c10-8d8a-11ee-95a2-87774f69a715.target → /etc/systemd/system/ceph-31861c10-8d8a-11ee-95a2-87774f69a715.target. 2023-11-28T01:06:57.258 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-31861c10-8d8a-11ee-95a2-87774f69a715.target → /etc/systemd/system/ceph-31861c10-8d8a-11ee-95a2-87774f69a715.target. 2023-11-28T01:06:57.436 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.a 2023-11-28T01:06:57.436 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr Failed to reset failed state of unit ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.a.service: Unit ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.a.service not loaded. 2023-11-28T01:06:57.524 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-31861c10-8d8a-11ee-95a2-87774f69a715.target.wants/ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.a.service → /etc/systemd/system/ceph-31861c10-8d8a-11ee-95a2-87774f69a715@.service. 2023-11-28T01:06:57.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:06:57 smithi172 systemd[1]: Starting Ceph mon.a for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:06:58.073 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-28T01:06:58.073 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout disabled 2023-11-28T01:06:58.080 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-28T01:06:58.080 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout inactive 2023-11-28T01:06:58.081 INFO:teuthology.orchestra.run.smithi172.stdout:firewalld.service is not enabled 2023-11-28T01:06:58.081 INFO:teuthology.orchestra.run.smithi172.stdout:Not possible to enable service . firewalld.service is not available 2023-11-28T01:06:58.081 INFO:teuthology.orchestra.run.smithi172.stdout:Waiting for mon to start... 2023-11-28T01:06:58.081 INFO:teuthology.orchestra.run.smithi172.stdout:Waiting for mon... 2023-11-28T01:06:58.330 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:06:58 smithi172 bash[99611]: f87355fef77c657dba14c86836cc8c0cf5c9a1a890c3b012e452de010044df24 2023-11-28T01:06:58.330 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:06:58 smithi172 systemd[1]: Started Ceph mon.a for 31861c10-8d8a-11ee-95a2-87774f69a715. 2023-11-28T01:06:58.331 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:06:58 smithi172 ceph-mon[99676]: mkfs 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:06:58.331 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:06:58 smithi172 ceph-mon[99676]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-28T01:06:59.265 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout cluster: 2023-11-28T01:06:59.265 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout id: 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:06:59.265 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-11-28T01:06:59.265 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:06:59.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout services: 2023-11-28T01:06:59.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.519407s) 2023-11-28T01:06:59.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-11-28T01:06:59.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-11-28T01:06:59.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:06:59.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout data: 2023-11-28T01:06:59.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-11-28T01:06:59.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-11-28T01:06:59.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-11-28T01:06:59.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout pgs: 2023-11-28T01:06:59.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:06:59.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.525+0000 7f4b4d687700 1 Processor -- start 2023-11-28T01:06:59.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.525+0000 7f4b4d687700 1 -- start start 2023-11-28T01:06:59.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.525+0000 7f4b4d687700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b48078ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:06:59.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.525+0000 7f4b4d687700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48079170 con 0x7f4b4807a740 2023-11-28T01:06:59.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.525+0000 7f4b46ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b48078ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:06:59.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.525+0000 7f4b46ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b48078ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49100/0 (socket says 172.21.15.172:49100) 2023-11-28T01:06:59.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.525+0000 7f4b46ffd700 1 -- 172.21.15.172:0/1155222266 learned_addr learned my addr 172.21.15.172:0/1155222266 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:06:59.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.525+0000 7f4b46ffd700 1 -- 172.21.15.172:0/1155222266 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b480792b0 con 0x7f4b4807a740 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.526+0000 7f4b46ffd700 1 --2- 172.21.15.172:0/1155222266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b48078ba0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4b300098a0 tx=0x7f4b30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97306679f87073e4 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.526+0000 7f4b45ffb700 1 -- 172.21.15.172:0/1155222266 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b30004160 con 0x7f4b4807a740 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.526+0000 7f4b45ffb700 1 -- 172.21.15.172:0/1155222266 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b300042c0 con 0x7f4b4807a740 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.526+0000 7f4b45ffb700 1 -- 172.21.15.172:0/1155222266 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b300045a0 con 0x7f4b4807a740 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.526+0000 7f4b4d687700 1 -- 172.21.15.172:0/1155222266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 msgr2=0x7f4b48078ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.526+0000 7f4b4d687700 1 --2- 172.21.15.172:0/1155222266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b48078ba0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4b300098a0 tx=0x7f4b30009bb0 comp rx=0 tx=0).stop 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.526+0000 7f4b4d687700 1 -- 172.21.15.172:0/1155222266 shutdown_connections 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.526+0000 7f4b4d687700 1 --2- 172.21.15.172:0/1155222266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b48078ba0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.526+0000 7f4b4d687700 1 -- 172.21.15.172:0/1155222266 >> 172.21.15.172:0/1155222266 conn(0x7f4b48100700 msgr2=0x7f4b48102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.526+0000 7f4b4d687700 1 -- 172.21.15.172:0/1155222266 shutdown_connections 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.526+0000 7f4b4d687700 1 -- 172.21.15.172:0/1155222266 wait complete. 2023-11-28T01:06:59.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.527+0000 7f4b4d687700 1 Processor -- start 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.527+0000 7f4b4d687700 1 -- start start 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.527+0000 7f4b4d687700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b4819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.527+0000 7f4b4d687700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4819ae20 con 0x7f4b4807a740 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.528+0000 7f4b46ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b4819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.528+0000 7f4b46ffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b4819a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49102/0 (socket says 172.21.15.172:49102) 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.528+0000 7f4b46ffd700 1 -- 172.21.15.172:0/1126302443 learned_addr learned my addr 172.21.15.172:0/1126302443 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.528+0000 7f4b46ffd700 1 -- 172.21.15.172:0/1126302443 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b30009550 con 0x7f4b4807a740 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.528+0000 7f4b46ffd700 1 --2- 172.21.15.172:0/1126302443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b4819a8e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4b30004000 tx=0x7f4b30004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.528+0000 7f4b3ffff700 1 -- 172.21.15.172:0/1126302443 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b30011630 con 0x7f4b4807a740 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.528+0000 7f4b3ffff700 1 -- 172.21.15.172:0/1126302443 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b30011790 con 0x7f4b4807a740 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.528+0000 7f4b4d687700 1 -- 172.21.15.172:0/1126302443 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4819b020 con 0x7f4b4807a740 2023-11-28T01:06:59.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.528+0000 7f4b3ffff700 1 -- 172.21.15.172:0/1126302443 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b30011a30 con 0x7f4b4807a740 2023-11-28T01:06:59.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.528+0000 7f4b4d687700 1 -- 172.21.15.172:0/1126302443 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4819b4c0 con 0x7f4b4807a740 2023-11-28T01:06:59.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.529+0000 7f4b3ffff700 1 -- 172.21.15.172:0/1126302443 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f4b30021070 con 0x7f4b4807a740 2023-11-28T01:06:59.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.529+0000 7f4b3ffff700 1 -- 172.21.15.172:0/1126302443 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4b3001b440 con 0x7f4b4807a740 2023-11-28T01:06:59.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.529+0000 7f4b4d687700 1 -- 172.21.15.172:0/1126302443 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b48194690 con 0x7f4b4807a740 2023-11-28T01:06:59.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.531+0000 7f4b3ffff700 1 -- 172.21.15.172:0/1126302443 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f4b30011b90 con 0x7f4b4807a740 2023-11-28T01:06:59.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.602+0000 7f4b4d687700 1 -- 172.21.15.172:0/1126302443 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f4b48079620 con 0x7f4b4807a740 2023-11-28T01:06:59.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.603+0000 7f4b3ffff700 1 -- 172.21.15.172:0/1126302443 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f4b30011d30 con 0x7f4b4807a740 2023-11-28T01:06:59.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.603+0000 7f4b4d687700 1 -- 172.21.15.172:0/1126302443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 msgr2=0x7f4b4819a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:06:59.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.603+0000 7f4b4d687700 1 --2- 172.21.15.172:0/1126302443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b4819a8e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4b30004000 tx=0x7f4b30004750 comp rx=0 tx=0).stop 2023-11-28T01:06:59.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.603+0000 7f4b4d687700 1 -- 172.21.15.172:0/1126302443 shutdown_connections 2023-11-28T01:06:59.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.603+0000 7f4b4d687700 1 --2- 172.21.15.172:0/1126302443 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4b4807a740 0x7f4b4819a8e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:06:59.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.603+0000 7f4b4d687700 1 -- 172.21.15.172:0/1126302443 >> 172.21.15.172:0/1126302443 conn(0x7f4b48100700 msgr2=0x7f4b48102ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:06:59.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.603+0000 7f4b4d687700 1 -- 172.21.15.172:0/1126302443 shutdown_connections 2023-11-28T01:06:59.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:58.603+0000 7f4b4d687700 1 -- 172.21.15.172:0/1126302443 wait complete. 2023-11-28T01:06:59.271 INFO:teuthology.orchestra.run.smithi172.stdout:mon is available 2023-11-28T01:06:59.271 INFO:teuthology.orchestra.run.smithi172.stdout:Assimilating anything we can from ceph.conf... 2023-11-28T01:06:59.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:06:59 smithi172 ceph-mon[99676]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-28T01:06:59.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:06:59 smithi172 ceph-mon[99676]: monmap e1: 1 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-11-28T01:06:59.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:06:59 smithi172 ceph-mon[99676]: fsmap 2023-11-28T01:06:59.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:06:59 smithi172 ceph-mon[99676]: osdmap e1: 0 total, 0 up, 0 in 2023-11-28T01:06:59.434 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:06:59 smithi172 ceph-mon[99676]: mgrmap e1: no daemons active 2023-11-28T01:06:59.434 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:06:59 smithi172 ceph-mon[99676]: from='client.? 172.21.15.172:0/1126302443' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-28T01:07:00.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:07:00.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout [global] 2023-11-28T01:07:00.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout fsid = 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:07:00.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.172:3300,v1:172.21.15.172:6789] 2023-11-28T01:07:00.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-11-28T01:07:00.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-11-28T01:07:00.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-11-28T01:07:00.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-11-28T01:07:00.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:07:00.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout [mgr] 2023-11-28T01:07:00.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-11-28T01:07:00.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:07:00.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout [osd] 2023-11-28T01:07:00.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-11-28T01:07:00.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-11-28T01:07:00.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.672+0000 7fa51fac5700 1 Processor -- start 2023-11-28T01:07:00.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.672+0000 7fa51fac5700 1 -- start start 2023-11-28T01:07:00.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.672+0000 7fa51fac5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa5181059d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:00.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.672+0000 7fa51fac5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518105fa0 con 0x7fa5181055b0 2023-11-28T01:07:00.297 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.672+0000 7fa51d861700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa5181059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:00.297 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.672+0000 7fa51d861700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa5181059d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49104/0 (socket says 172.21.15.172:49104) 2023-11-28T01:07:00.297 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.672+0000 7fa51d861700 1 -- 172.21.15.172:0/25494737 learned_addr learned my addr 172.21.15.172:0/25494737 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:00.297 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51d861700 1 -- 172.21.15.172:0/25494737 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5181060e0 con 0x7fa5181055b0 2023-11-28T01:07:00.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51d861700 1 --2- 172.21.15.172:0/25494737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa5181059d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa5140098a0 tx=0x7fa514009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ea00613bd76383 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:00.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51c85f700 1 -- 172.21.15.172:0/25494737 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa514004160 con 0x7fa5181055b0 2023-11-28T01:07:00.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51c85f700 1 -- 172.21.15.172:0/25494737 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5140042c0 con 0x7fa5181055b0 2023-11-28T01:07:00.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51c85f700 1 -- 172.21.15.172:0/25494737 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5140045a0 con 0x7fa5181055b0 2023-11-28T01:07:00.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51fac5700 1 -- 172.21.15.172:0/25494737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 msgr2=0x7fa5181059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:00.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51fac5700 1 --2- 172.21.15.172:0/25494737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa5181059d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa5140098a0 tx=0x7fa514009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:00.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51fac5700 1 -- 172.21.15.172:0/25494737 shutdown_connections 2023-11-28T01:07:00.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51fac5700 1 --2- 172.21.15.172:0/25494737 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa5181059d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:00.299 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51fac5700 1 -- 172.21.15.172:0/25494737 >> 172.21.15.172:0/25494737 conn(0x7fa5180fed00 msgr2=0x7fa518101180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:00.299 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51fac5700 1 -- 172.21.15.172:0/25494737 shutdown_connections 2023-11-28T01:07:00.299 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.673+0000 7fa51fac5700 1 -- 172.21.15.172:0/25494737 wait complete. 2023-11-28T01:07:00.299 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.674+0000 7fa51fac5700 1 Processor -- start 2023-11-28T01:07:00.299 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.674+0000 7fa51fac5700 1 -- start start 2023-11-28T01:07:00.299 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.674+0000 7fa51fac5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa518103bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:00.299 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.674+0000 7fa51fac5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa518104100 con 0x7fa5181055b0 2023-11-28T01:07:00.299 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.674+0000 7fa51d861700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa518103bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:00.299 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.674+0000 7fa51d861700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa518103bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49106/0 (socket says 172.21.15.172:49106) 2023-11-28T01:07:00.300 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.674+0000 7fa51d861700 1 -- 172.21.15.172:0/450020916 learned_addr learned my addr 172.21.15.172:0/450020916 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:00.300 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.675+0000 7fa51d861700 1 -- 172.21.15.172:0/450020916 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa514009550 con 0x7fa5181055b0 2023-11-28T01:07:00.300 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.675+0000 7fa51d861700 1 --2- 172.21.15.172:0/450020916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa518103bc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa514004000 tx=0x7fa514004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:00.300 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.675+0000 7fa50effd700 1 -- 172.21.15.172:0/450020916 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa514011630 con 0x7fa5181055b0 2023-11-28T01:07:00.300 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.675+0000 7fa50effd700 1 -- 172.21.15.172:0/450020916 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa514011790 con 0x7fa5181055b0 2023-11-28T01:07:00.300 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.675+0000 7fa50effd700 1 -- 172.21.15.172:0/450020916 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa514011a30 con 0x7fa5181055b0 2023-11-28T01:07:00.300 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.675+0000 7fa51fac5700 1 -- 172.21.15.172:0/450020916 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa51819b440 con 0x7fa5181055b0 2023-11-28T01:07:00.300 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.675+0000 7fa51fac5700 1 -- 172.21.15.172:0/450020916 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa51819b880 con 0x7fa5181055b0 2023-11-28T01:07:00.300 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.675+0000 7fa50effd700 1 -- 172.21.15.172:0/450020916 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7fa514021070 con 0x7fa5181055b0 2023-11-28T01:07:00.301 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.675+0000 7fa50effd700 1 -- 172.21.15.172:0/450020916 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa51401b440 con 0x7fa5181055b0 2023-11-28T01:07:00.301 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.676+0000 7fa51fac5700 1 -- 172.21.15.172:0/450020916 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa518191220 con 0x7fa5181055b0 2023-11-28T01:07:00.301 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.678+0000 7fa50effd700 1 -- 172.21.15.172:0/450020916 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7fa514011b90 con 0x7fa5181055b0 2023-11-28T01:07:00.301 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.744+0000 7fa51fac5700 1 -- 172.21.15.172:0/450020916 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa518104470 con 0x7fa5181055b0 2023-11-28T01:07:00.301 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.747+0000 7fa50effd700 1 -- 172.21.15.172:0/450020916 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+340 (secure 0 0 0) 0x7fa51401a460 con 0x7fa5181055b0 2023-11-28T01:07:00.301 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.747+0000 7fa50effd700 1 -- 172.21.15.172:0/450020916 <== mon.0 v2:172.21.15.172:3300/0 8 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7fa5140338c0 con 0x7fa5181055b0 2023-11-28T01:07:00.301 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.748+0000 7fa51fac5700 1 -- 172.21.15.172:0/450020916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 msgr2=0x7fa518103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:00.301 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.748+0000 7fa51fac5700 1 --2- 172.21.15.172:0/450020916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa518103bc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa514004000 tx=0x7fa514004750 comp rx=0 tx=0).stop 2023-11-28T01:07:00.302 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.748+0000 7fa51fac5700 1 -- 172.21.15.172:0/450020916 shutdown_connections 2023-11-28T01:07:00.302 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.748+0000 7fa51fac5700 1 --2- 172.21.15.172:0/450020916 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa5181055b0 0x7fa518103bc0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:00.302 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.748+0000 7fa51fac5700 1 -- 172.21.15.172:0/450020916 >> 172.21.15.172:0/450020916 conn(0x7fa5180fed00 msgr2=0x7fa51818e360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:00.302 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.748+0000 7fa51fac5700 1 -- 172.21.15.172:0/450020916 shutdown_connections 2023-11-28T01:07:00.302 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:06:59.748+0000 7fa51fac5700 1 -- 172.21.15.172:0/450020916 wait complete. 2023-11-28T01:07:00.302 INFO:teuthology.orchestra.run.smithi172.stdout:Generating new minimal ceph.conf... 2023-11-28T01:07:00.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:00 smithi172 ceph-mon[99676]: from='client.? 172.21.15.172:0/450020916' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-28T01:07:00.474 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:00 smithi172 ceph-mon[99676]: from='client.? 172.21.15.172:0/450020916' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-28T01:07:01.387 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.721+0000 7f13f3de0700 1 Processor -- start 2023-11-28T01:07:01.387 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.722+0000 7f13f3de0700 1 -- start start 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.722+0000 7f13f3de0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec104450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.722+0000 7f13f3de0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec104a20 con 0x7f13ec104030 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.722+0000 7f13f1b7c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec104450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.722+0000 7f13f1b7c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec104450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49108/0 (socket says 172.21.15.172:49108) 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.722+0000 7f13f1b7c700 1 -- 172.21.15.172:0/1142657663 learned_addr learned my addr 172.21.15.172:0/1142657663 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.722+0000 7f13f1b7c700 1 -- 172.21.15.172:0/1142657663 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec105230 con 0x7f13ec104030 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.723+0000 7f13f1b7c700 1 --2- 172.21.15.172:0/1142657663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec104450 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0098a0 tx=0x7f13dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d3de1efbeac62c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.723+0000 7f13f0b7a700 1 -- 172.21.15.172:0/1142657663 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13dc004160 con 0x7f13ec104030 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.723+0000 7f13f0b7a700 1 -- 172.21.15.172:0/1142657663 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f13dc0036a0 con 0x7f13ec104030 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.723+0000 7f13f0b7a700 1 -- 172.21.15.172:0/1142657663 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13dc0117c0 con 0x7f13ec104030 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.723+0000 7f13f3de0700 1 -- 172.21.15.172:0/1142657663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 msgr2=0x7f13ec104450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.723+0000 7f13f3de0700 1 --2- 172.21.15.172:0/1142657663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec104450 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f13dc0098a0 tx=0x7f13dc009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.723+0000 7f13f3de0700 1 -- 172.21.15.172:0/1142657663 shutdown_connections 2023-11-28T01:07:01.388 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.723+0000 7f13f3de0700 1 --2- 172.21.15.172:0/1142657663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec104450 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.723+0000 7f13f3de0700 1 -- 172.21.15.172:0/1142657663 >> 172.21.15.172:0/1142657663 conn(0x7f13ec0ff590 msgr2=0x7f13ec101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.724+0000 7f13f3de0700 1 -- 172.21.15.172:0/1142657663 shutdown_connections 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.724+0000 7f13f3de0700 1 -- 172.21.15.172:0/1142657663 wait complete. 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.724+0000 7f13f3de0700 1 Processor -- start 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.724+0000 7f13f3de0700 1 -- start start 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13f3de0700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec197cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13f3de0700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec198200 con 0x7f13ec104030 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13f1b7c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec197cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13f1b7c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec197cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49110/0 (socket says 172.21.15.172:49110) 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13f1b7c700 1 -- 172.21.15.172:0/163438035 learned_addr learned my addr 172.21.15.172:0/163438035 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13f1b7c700 1 -- 172.21.15.172:0/163438035 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13dc009550 con 0x7f13ec104030 2023-11-28T01:07:01.389 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13f1b7c700 1 --2- 172.21.15.172:0/163438035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec197cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009870 tx=0x7f13dc011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13e2ffd700 1 -- 172.21.15.172:0/163438035 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13dc0040a0 con 0x7f13ec104030 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13e2ffd700 1 -- 172.21.15.172:0/163438035 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f13dc01b440 con 0x7f13ec104030 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13f3de0700 1 -- 172.21.15.172:0/163438035 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec198400 con 0x7f13ec104030 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13e2ffd700 1 -- 172.21.15.172:0/163438035 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13dc0193d0 con 0x7f13ec104030 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13f3de0700 1 -- 172.21.15.172:0/163438035 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec1988a0 con 0x7f13ec104030 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13e2ffd700 1 -- 172.21.15.172:0/163438035 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f13dc019610 con 0x7f13ec104030 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.725+0000 7f13e2ffd700 1 -- 172.21.15.172:0/163438035 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f13dc024e20 con 0x7f13ec104030 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.726+0000 7f13f3de0700 1 -- 172.21.15.172:0/163438035 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ec105f80 con 0x7f13ec104030 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.728+0000 7f13e2ffd700 1 -- 172.21.15.172:0/163438035 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f13dc029060 con 0x7f13ec104030 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.796+0000 7f13f3de0700 1 -- 172.21.15.172:0/163438035 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f13ec062c00 con 0x7f13ec104030 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.796+0000 7f13e2ffd700 1 -- 172.21.15.172:0/163438035 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f13dc032090 con 0x7f13ec104030 2023-11-28T01:07:01.390 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.797+0000 7f13f3de0700 1 -- 172.21.15.172:0/163438035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 msgr2=0x7f13ec197cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:01.391 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.797+0000 7f13f3de0700 1 --2- 172.21.15.172:0/163438035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec197cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009870 tx=0x7f13dc011ae0 comp rx=0 tx=0).stop 2023-11-28T01:07:01.391 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.797+0000 7f13f3de0700 1 -- 172.21.15.172:0/163438035 shutdown_connections 2023-11-28T01:07:01.391 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.797+0000 7f13f3de0700 1 --2- 172.21.15.172:0/163438035 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f13ec104030 0x7f13ec197cc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:01.391 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.797+0000 7f13f3de0700 1 -- 172.21.15.172:0/163438035 >> 172.21.15.172:0/163438035 conn(0x7f13ec0ff590 msgr2=0x7f13ec100180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:01.391 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.797+0000 7f13f3de0700 1 -- 172.21.15.172:0/163438035 shutdown_connections 2023-11-28T01:07:01.391 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:00.797+0000 7f13f3de0700 1 -- 172.21.15.172:0/163438035 wait complete. 2023-11-28T01:07:01.391 INFO:teuthology.orchestra.run.smithi172.stdout:Restarting the monitor... 2023-11-28T01:07:01.392 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:01 smithi172 ceph-mon[99676]: from='client.? 172.21.15.172:0/163438035' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:07:01.758 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:01 smithi172 systemd[1]: Stopping Ceph mon.a for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:07:01.758 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:01 smithi172 conmon[99652]: 2023-11-28T01:07:01.631+0000 7f0c60894700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:07:01.758 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:01 smithi172 conmon[99652]: 2023-11-28T01:07:01.631+0000 7f0c60894700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-11-28T01:07:02.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 bash[100237]: f87355fef77c657dba14c86836cc8c0cf5c9a1a890c3b012e452de010044df24 2023-11-28T01:07:02.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 systemd[1]: ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.a.service: Succeeded. 2023-11-28T01:07:02.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 systemd[1]: Stopped Ceph mon.a for 31861c10-8d8a-11ee-95a2-87774f69a715. 2023-11-28T01:07:02.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 systemd[1]: Starting Ceph mon.a for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:07:02.880 INFO:teuthology.orchestra.run.smithi172.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-11-28T01:07:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: set uid:gid to 167:167 (ceph:ceph) 2023-11-28T01:07:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: ceph version 18.0.0-7498-g369173db (369173db14b6995b2bd07c60ec5f63d01cf21631) squid (dev), process ceph-mon, pid 7 2023-11-28T01:07:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: pidfile_write: ignore empty --pid-file 2023-11-28T01:07:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: load: jerasure load: lrc 2023-11-28T01:07:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: RocksDB version: 7.9.2 2023-11-28T01:07:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Git sha 0 2023-11-28T01:07:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Compile date 2023-11-22 22:33:43 2023-11-28T01:07:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: DB SUMMARY 2023-11-28T01:07:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: DB Session ID: K71DYCYUG2D4OHWEMY86 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: CURRENT file: CURRENT 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: IDENTITY file: IDENTITY 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 86277 ; 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.error_if_exists: 0 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.create_if_missing: 0 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.paranoid_checks: 1 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.env: 0x5606f45e0a60 2023-11-28T01:07:03.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.fs: PosixFileSystem 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.info_log: 0x5606f57b9740 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_file_opening_threads: 16 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.statistics: (nil) 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.use_fsync: 0 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_log_file_size: 0 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.keep_log_file_num: 1000 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.recycle_log_file_num: 0 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.allow_fallocate: 1 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.allow_mmap_reads: 0 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.allow_mmap_writes: 0 2023-11-28T01:07:03.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.use_direct_reads: 0 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.create_missing_column_families: 0 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.db_log_dir: 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.wal_dir: 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.advise_random_on_open: 1 2023-11-28T01:07:03.057 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.db_write_buffer_size: 0 2023-11-28T01:07:03.058 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.write_buffer_manager: 0x5606f576d4a0 2023-11-28T01:07:03.058 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-28T01:07:03.058 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-28T01:07:03.058 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-28T01:07:03.058 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.rate_limiter: (nil) 2023-11-28T01:07:03.058 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-28T01:07:03.058 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.wal_recovery_mode: 2 2023-11-28T01:07:03.058 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.enable_thread_tracking: 0 2023-11-28T01:07:03.058 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.enable_pipelined_write: 0 2023-11-28T01:07:03.058 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.unordered_write: 0 2023-11-28T01:07:03.058 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.row_cache: None 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.wal_filter: None 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.allow_ingest_behind: 0 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.two_write_queues: 0 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.manual_wal_flush: 0 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.wal_compression: 0 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.atomic_flush: 0 2023-11-28T01:07:03.059 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-28T01:07:03.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-28T01:07:03.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-28T01:07:03.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.log_readahead_size: 0 2023-11-28T01:07:03.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-28T01:07:03.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.best_efforts_recovery: 0 2023-11-28T01:07:03.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-28T01:07:03.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-28T01:07:03.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.allow_data_in_errors: 0 2023-11-28T01:07:03.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.db_host_id: __hostname__ 2023-11-28T01:07:03.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-28T01:07:03.060 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_background_jobs: 2 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_background_compactions: -1 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_subcompactions: 1 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_total_wal_size: 0 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_open_files: -1 2023-11-28T01:07:03.061 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bytes_per_sync: 0 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_readahead_size: 0 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_background_flushes: -1 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Compression algorithms supported: 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: kZSTD supported: 0 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: kXpressCompression supported: 0 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: kLZ4HCCompression supported: 1 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: kZlibCompression supported: 1 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: kSnappyCompression supported: 1 2023-11-28T01:07:03.062 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: kLZ4Compression supported: 1 2023-11-28T01:07:03.063 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: kBZip2Compression supported: 0 2023-11-28T01:07:03.063 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-28T01:07:03.063 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-28T01:07:03.063 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-11-28T01:07:03.063 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-28T01:07:03.063 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-28T01:07:03.063 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.merge_operator: 2023-11-28T01:07:03.063 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_filter: None 2023-11-28T01:07:03.063 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_filter_factory: None 2023-11-28T01:07:03.064 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.sst_partitioner_factory: None 2023-11-28T01:07:03.064 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-28T01:07:03.064 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-28T01:07:03.064 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5606f57b9860) 2023-11-28T01:07:03.064 INFO:journalctl@ceph.mon.a.smithi172.stdout: cache_index_and_filter_blocks: 1 2023-11-28T01:07:03.064 INFO:journalctl@ceph.mon.a.smithi172.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-28T01:07:03.064 INFO:journalctl@ceph.mon.a.smithi172.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-28T01:07:03.064 INFO:journalctl@ceph.mon.a.smithi172.stdout: pin_top_level_index_and_filter: 1 2023-11-28T01:07:03.064 INFO:journalctl@ceph.mon.a.smithi172.stdout: index_type: 0 2023-11-28T01:07:03.064 INFO:journalctl@ceph.mon.a.smithi172.stdout: data_block_index_type: 0 2023-11-28T01:07:03.065 INFO:journalctl@ceph.mon.a.smithi172.stdout: index_shortening: 1 2023-11-28T01:07:03.065 INFO:journalctl@ceph.mon.a.smithi172.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-28T01:07:03.065 INFO:journalctl@ceph.mon.a.smithi172.stdout: checksum: 4 2023-11-28T01:07:03.065 INFO:journalctl@ceph.mon.a.smithi172.stdout: no_block_cache: 0 2023-11-28T01:07:03.065 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache: 0x5606f580f090 2023-11-28T01:07:03.065 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache_name: BinnedLRUCache 2023-11-28T01:07:03.065 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache_options: 2023-11-28T01:07:03.065 INFO:journalctl@ceph.mon.a.smithi172.stdout: capacity : 536870912 2023-11-28T01:07:03.065 INFO:journalctl@ceph.mon.a.smithi172.stdout: num_shard_bits : 4 2023-11-28T01:07:03.065 INFO:journalctl@ceph.mon.a.smithi172.stdout: strict_capacity_limit : 0 2023-11-28T01:07:03.065 INFO:journalctl@ceph.mon.a.smithi172.stdout: high_pri_pool_ratio: 0.000 2023-11-28T01:07:03.066 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_cache_compressed: (nil) 2023-11-28T01:07:03.066 INFO:journalctl@ceph.mon.a.smithi172.stdout: persistent_cache: (nil) 2023-11-28T01:07:03.066 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_size: 4096 2023-11-28T01:07:03.066 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_size_deviation: 10 2023-11-28T01:07:03.066 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_restart_interval: 16 2023-11-28T01:07:03.066 INFO:journalctl@ceph.mon.a.smithi172.stdout: index_block_restart_interval: 1 2023-11-28T01:07:03.066 INFO:journalctl@ceph.mon.a.smithi172.stdout: metadata_block_size: 4096 2023-11-28T01:07:03.066 INFO:journalctl@ceph.mon.a.smithi172.stdout: partition_filters: 0 2023-11-28T01:07:03.066 INFO:journalctl@ceph.mon.a.smithi172.stdout: use_delta_encoding: 1 2023-11-28T01:07:03.066 INFO:journalctl@ceph.mon.a.smithi172.stdout: filter_policy: bloomfilter 2023-11-28T01:07:03.066 INFO:journalctl@ceph.mon.a.smithi172.stdout: whole_key_filtering: 1 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout: verify_compression: 0 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout: read_amp_bytes_per_bit: 0 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout: format_version: 5 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout: enable_index_compression: 1 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout: block_align: 0 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout: max_auto_readahead_size: 262144 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout: prepopulate_block_cache: 0 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout: initial_auto_readahead_size: 8192 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout: num_file_reads_for_auto_readahead: 2 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.write_buffer_size: 33554432 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_write_buffer_number: 2 2023-11-28T01:07:03.067 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compression: NoCompression 2023-11-28T01:07:03.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bottommost_compression: Disabled 2023-11-28T01:07:03.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.prefix_extractor: nullptr 2023-11-28T01:07:03.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-28T01:07:03.068 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.num_levels: 7 2023-11-28T01:07:03.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-28T01:07:03.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-28T01:07:03.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-28T01:07:03.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-28T01:07:03.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-28T01:07:03.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-28T01:07:03.069 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-28T01:07:03.070 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-28T01:07:03.070 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-28T01:07:03.070 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-28T01:07:03.070 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-28T01:07:03.070 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compression_opts.level: 32767 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compression_opts.strategy: 0 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compression_opts.enabled: false 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-28T01:07:03.071 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.target_file_size_base: 67108864 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-28T01:07:03.072 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.arena_block_size: 1048576 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.disable_auto_compactions: 0 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 bash[100562]: e5cbc746eb8f5a1ee9362b369e24779a2c030505c07b7ec96f9e153afcbba06a 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-28T01:07:03.073 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.table_properties_collectors: 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.inplace_update_support: 0 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-28T01:07:03.074 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.bloom_locality: 0 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.max_successive_merges: 0 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.paranoid_file_checks: 0 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.force_consistency_checks: 1 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.report_bg_io_stats: 0 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.ttl: 2592000 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.enable_blob_files: false 2023-11-28T01:07:03.075 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.min_blob_size: 0 2023-11-28T01:07:03.076 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.blob_file_size: 268435456 2023-11-28T01:07:03.076 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-28T01:07:03.076 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-28T01:07:03.076 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-28T01:07:03.076 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-28T01:07:03.076 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-28T01:07:03.076 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.blob_file_starting_level: 0 2023-11-28T01:07:03.076 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-28T01:07:03.076 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-11-28T01:07:03.076 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-11-28T01:07:03.076 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 84639a58-14a9-4fd3-9f63-fde8e7908cd0 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: EVENT_LOG_v1 {"time_micros": 1701133622868260, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: EVENT_LOG_v1 {"time_micros": 1701133622869100, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 82744, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 260, "table_properties": {"data_size": 80932, "index_size": 222, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10461, "raw_average_key_size": 47, "raw_value_size": 75004, "raw_average_value_size": 342, "num_data_blocks": 10, "num_entries": 219, "num_filter_entries": 219, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1701133622, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "84639a58-14a9-4fd3-9f63-fde8e7908cd0", "db_session_id": "K71DYCYUG2D4OHWEMY86", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: EVENT_LOG_v1 {"time_micros": 1701133622869169, "job": 1, "event": "recovery_finished"} 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5606f58a8000 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: DB pointer 0x5606f5896000 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** DB Stats ** 2023-11-28T01:07:03.077 INFO:journalctl@ceph.mon.a.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-28T01:07:03.078 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-28T01:07:03.078 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-28T01:07:03.078 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-28T01:07:03.078 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-28T01:07:03.078 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-28T01:07:03.078 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-28T01:07:03.078 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2023-11-28T01:07:03.078 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** Compaction Stats [default] ** 2023-11-28T01:07:03.078 INFO:journalctl@ceph.mon.a.smithi172.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-28T01:07:03.078 INFO:journalctl@ceph.mon.a.smithi172.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-28T01:07:03.078 INFO:journalctl@ceph.mon.a.smithi172.stdout: L0 2/0 82.64 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: Sum 2/0 82.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 146.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** Compaction Stats [default] ** 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 146.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-28T01:07:03.079 INFO:journalctl@ceph.mon.a.smithi172.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout: Cumulative compaction: 0.00 GB write, 14.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout: Interval compaction: 0.00 GB write, 14.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout: Block cache BinnedLRUCache@0x5606f580f090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-28T01:07:03.080 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: starting mon.a rank 0 at public addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] at bind addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: mon.a@-1(???) e1 preinit fsid 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: mon.a@-1(???).mds e1 new map 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: mon.a@-1(???).mds e1 print_map 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout: e1 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout: legacy client fscid: -1 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout: 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout: No filesystems configured 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-28T01:07:03.081 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-28T01:07:03.082 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-28T01:07:03.082 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-28T01:07:03.082 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 systemd[1]: Started Ceph mon.a for 31861c10-8d8a-11ee-95a2-87774f69a715. 2023-11-28T01:07:03.082 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: mon.a is new leader, mons a in quorum (ranks 0) 2023-11-28T01:07:03.082 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: monmap e1: 1 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-11-28T01:07:03.082 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: fsmap 2023-11-28T01:07:03.082 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: osdmap e1: 0 total, 0 up, 0 in 2023-11-28T01:07:03.082 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:02 smithi172 ceph-mon[100627]: mgrmap e1: no daemons active 2023-11-28T01:07:04.033 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.362+0000 7f0cddf50700 1 Processor -- start 2023-11-28T01:07:04.038 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.362+0000 7f0cddf50700 1 -- start start 2023-11-28T01:07:04.038 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.362+0000 7f0cddf50700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:04.038 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.362+0000 7f0cddf50700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd8078710 con 0x7f0cd8105bf0 2023-11-28T01:07:04.038 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.362+0000 7f0cd77fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:04.038 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.362+0000 7f0cd77fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8107fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49112/0 (socket says 172.21.15.172:49112) 2023-11-28T01:07:04.039 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.362+0000 7f0cd77fe700 1 -- 172.21.15.172:0/1750615280 learned_addr learned my addr 172.21.15.172:0/1750615280 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:04.039 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.363+0000 7f0cd77fe700 1 -- 172.21.15.172:0/1750615280 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd8108520 con 0x7f0cd8105bf0 2023-11-28T01:07:04.039 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.363+0000 7f0cd77fe700 1 --2- 172.21.15.172:0/1750615280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8107fe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80098a0 tx=0x7f0cc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ed5dbd8f98cc2f server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:04.039 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.363+0000 7f0cd67fc700 1 -- 172.21.15.172:0/1750615280 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cc8004160 con 0x7f0cd8105bf0 2023-11-28T01:07:04.039 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.363+0000 7f0cd67fc700 1 -- 172.21.15.172:0/1750615280 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f0cc80036a0 con 0x7f0cd8105bf0 2023-11-28T01:07:04.039 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.363+0000 7f0cd67fc700 1 -- 172.21.15.172:0/1750615280 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cc80117c0 con 0x7f0cd8105bf0 2023-11-28T01:07:04.039 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.363+0000 7f0cddf50700 1 -- 172.21.15.172:0/1750615280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 msgr2=0x7f0cd8107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:04.040 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.363+0000 7f0cddf50700 1 --2- 172.21.15.172:0/1750615280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8107fe0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0cc80098a0 tx=0x7f0cc8009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:04.040 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.363+0000 7f0cddf50700 1 -- 172.21.15.172:0/1750615280 shutdown_connections 2023-11-28T01:07:04.040 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.363+0000 7f0cddf50700 1 --2- 172.21.15.172:0/1750615280 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8107fe0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:04.040 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.363+0000 7f0cddf50700 1 -- 172.21.15.172:0/1750615280 >> 172.21.15.172:0/1750615280 conn(0x7f0cd80ff5c0 msgr2=0x7f0cd8101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:04.040 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.363+0000 7f0cddf50700 1 -- 172.21.15.172:0/1750615280 shutdown_connections 2023-11-28T01:07:04.041 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.364+0000 7f0cddf50700 1 -- 172.21.15.172:0/1750615280 wait complete. 2023-11-28T01:07:04.041 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.364+0000 7f0cddf50700 1 Processor -- start 2023-11-28T01:07:04.041 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.364+0000 7f0cddf50700 1 -- start start 2023-11-28T01:07:04.041 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.364+0000 7f0cddf50700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:04.041 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.364+0000 7f0cddf50700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd8199280 con 0x7f0cd8105bf0 2023-11-28T01:07:04.041 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cd77fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:04.041 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cd77fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8198d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49114/0 (socket says 172.21.15.172:49114) 2023-11-28T01:07:04.041 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cd77fe700 1 -- 172.21.15.172:0/847635536 learned_addr learned my addr 172.21.15.172:0/847635536 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:04.041 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cd77fe700 1 -- 172.21.15.172:0/847635536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc8009550 con 0x7f0cd8105bf0 2023-11-28T01:07:04.041 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cd77fe700 1 --2- 172.21.15.172:0/847635536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8198d40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009130 tx=0x7f0cc80119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:04.041 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cd4ff9700 1 -- 172.21.15.172:0/847635536 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cc8004180 con 0x7f0cd8105bf0 2023-11-28T01:07:04.042 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cd4ff9700 1 -- 172.21.15.172:0/847635536 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f0cc80042e0 con 0x7f0cd8105bf0 2023-11-28T01:07:04.042 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cddf50700 1 -- 172.21.15.172:0/847635536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd8199480 con 0x7f0cd8105bf0 2023-11-28T01:07:04.042 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cd4ff9700 1 -- 172.21.15.172:0/847635536 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0cc80193f0 con 0x7f0cd8105bf0 2023-11-28T01:07:04.042 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cddf50700 1 -- 172.21.15.172:0/847635536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd8199920 con 0x7f0cd8105bf0 2023-11-28T01:07:04.042 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cd4ff9700 1 -- 172.21.15.172:0/847635536 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f0cc80196b0 con 0x7f0cd8105bf0 2023-11-28T01:07:04.042 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.365+0000 7f0cd4ff9700 1 -- 172.21.15.172:0/847635536 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0cc801a3d0 con 0x7f0cd8105bf0 2023-11-28T01:07:04.042 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.366+0000 7f0cddf50700 1 -- 172.21.15.172:0/847635536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8062c00 con 0x7f0cd8105bf0 2023-11-28T01:07:04.042 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.368+0000 7f0cd4ff9700 1 -- 172.21.15.172:0/847635536 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f0cc8029060 con 0x7f0cd8105bf0 2023-11-28T01:07:04.042 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.435+0000 7f0cddf50700 1 -- 172.21.15.172:0/847635536 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f0cd8109420 con 0x7f0cd8105bf0 2023-11-28T01:07:04.043 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.437+0000 7f0cd4ff9700 1 -- 172.21.15.172:0/847635536 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0cc80330c0 con 0x7f0cd8105bf0 2023-11-28T01:07:04.043 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.437+0000 7f0cddf50700 1 -- 172.21.15.172:0/847635536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 msgr2=0x7f0cd8198d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:04.043 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.437+0000 7f0cddf50700 1 --2- 172.21.15.172:0/847635536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8198d40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0cc8009130 tx=0x7f0cc80119c0 comp rx=0 tx=0).stop 2023-11-28T01:07:04.043 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.437+0000 7f0cddf50700 1 -- 172.21.15.172:0/847635536 shutdown_connections 2023-11-28T01:07:04.043 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.437+0000 7f0cddf50700 1 --2- 172.21.15.172:0/847635536 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0cd8105bf0 0x7f0cd8198d40 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:04.043 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.437+0000 7f0cddf50700 1 -- 172.21.15.172:0/847635536 >> 172.21.15.172:0/847635536 conn(0x7f0cd80ff5c0 msgr2=0x7f0cd8100140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:04.043 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.437+0000 7f0cddf50700 1 -- 172.21.15.172:0/847635536 shutdown_connections 2023-11-28T01:07:04.043 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:03.437+0000 7f0cddf50700 1 -- 172.21.15.172:0/847635536 wait complete. 2023-11-28T01:07:04.043 INFO:teuthology.orchestra.run.smithi172.stdout:Wrote config to /etc/ceph/ceph.conf 2023-11-28T01:07:04.066 INFO:teuthology.orchestra.run.smithi172.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-28T01:07:04.066 INFO:teuthology.orchestra.run.smithi172.stdout:Creating mgr... 2023-11-28T01:07:04.067 INFO:teuthology.orchestra.run.smithi172.stdout:Verifying port 0.0.0.0:9283 ... 2023-11-28T01:07:04.067 INFO:teuthology.orchestra.run.smithi172.stdout:Verifying port 0.0.0.0:8765 ... 2023-11-28T01:07:04.292 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mgr.y 2023-11-28T01:07:04.293 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr Failed to reset failed state of unit ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mgr.y.service: Unit ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mgr.y.service not loaded. 2023-11-28T01:07:04.380 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-31861c10-8d8a-11ee-95a2-87774f69a715.target.wants/ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-31861c10-8d8a-11ee-95a2-87774f69a715@.service. 2023-11-28T01:07:04.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:04 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/847635536' entity='client.admin' 2023-11-28T01:07:04.553 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:04 smithi172 systemd[1]: Starting Ceph mgr.y for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:07:04.987 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-28T01:07:04.987 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout disabled 2023-11-28T01:07:04.995 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-28T01:07:04.995 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout inactive 2023-11-28T01:07:04.995 INFO:teuthology.orchestra.run.smithi172.stdout:firewalld.service is not enabled 2023-11-28T01:07:04.995 INFO:teuthology.orchestra.run.smithi172.stdout:Not possible to enable service . firewalld.service is not available 2023-11-28T01:07:05.003 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-28T01:07:05.003 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout disabled 2023-11-28T01:07:05.011 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-28T01:07:05.011 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout inactive 2023-11-28T01:07:05.011 INFO:teuthology.orchestra.run.smithi172.stdout:firewalld.service is not enabled 2023-11-28T01:07:05.011 INFO:teuthology.orchestra.run.smithi172.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-11-28T01:07:05.011 INFO:teuthology.orchestra.run.smithi172.stdout:Waiting for mgr to start... 2023-11-28T01:07:05.011 INFO:teuthology.orchestra.run.smithi172.stdout:Waiting for mgr... 2023-11-28T01:07:05.241 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:04 smithi172 bash[101008]: 79434f533d8478be26d7d308fd73aa52d6ac693f5dc86477619011b77da626c4 2023-11-28T01:07:05.241 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:04 smithi172 systemd[1]: Started Ceph mgr.y for 31861c10-8d8a-11ee-95a2-87774f69a715. 2023-11-28T01:07:05.242 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:05 smithi172 conmon[101049]: 2023-11-28T01:07:05.090+0000 7f3757b62200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T01:07:05.242 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:05 smithi172 conmon[101049]: 2023-11-28T01:07:05.173+0000 7f3757b62200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T01:07:05.544 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:05 smithi172 conmon[101049]: 2023-11-28T01:07:05.269+0000 7f3757b62200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T01:07:05.544 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:05 smithi172 conmon[101049]: 2023-11-28T01:07:05.365+0000 7f3757b62200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T01:07:05.544 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:05 smithi172 conmon[101049]: 2023-11-28T01:07:05.543+0000 7f3757b62200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T01:07:05.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:05 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/860705530' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-28T01:07:06.168 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:07:06.168 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout { 2023-11-28T01:07:06.168 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "fsid": "31861c10-8d8a-11ee-95a2-87774f69a715", 2023-11-28T01:07:06.168 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "health": { 2023-11-28T01:07:06.168 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-28T01:07:06.168 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-28T01:07:06.168 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-28T01:07:06.169 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:06.169 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-28T01:07:06.169 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-28T01:07:06.169 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 0 2023-11-28T01:07:06.169 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:06.169 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-28T01:07:06.169 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "a" 2023-11-28T01:07:06.169 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:06.169 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-11-28T01:07:06.169 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-28T01:07:06.169 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:06.170 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2023-11-28T01:07:06.170 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-28T01:07:06.170 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:06.170 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-28T01:07:06.170 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:06.170 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-28T01:07:06.170 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-28T01:07:06.170 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-28T01:07:06.177 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-28T01:07:06.178 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-28T01:07:06.178 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-28T01:07:06.178 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:06.178 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-28T01:07:06.178 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-28T01:07:06.178 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-28T01:07:06.178 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-28T01:07:06.178 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-28T01:07:06.178 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-28T01:07:06.179 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-28T01:07:06.180 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-28T01:07:06.180 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-28T01:07:06.180 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-28T01:07:06.186 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "restful" 2023-11-28T01:07:06.187 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:06.188 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-28T01:07:06.188 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:06.188 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-28T01:07:06.188 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:06.188 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "modified": "2023-11-28T01:06:58.086438+0000", 2023-11-28T01:07:06.188 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-28T01:07:06.188 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:06.188 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-28T01:07:06.188 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout } 2023-11-28T01:07:06.188 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.462+0000 7f857156d700 1 Processor -- start 2023-11-28T01:07:06.188 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.462+0000 7f857156d700 1 -- start start 2023-11-28T01:07:06.189 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.462+0000 7f857156d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c107fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:06.189 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.462+0000 7f857156d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c077e80 con 0x7f856c105bd0 2023-11-28T01:07:06.189 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.463+0000 7f856affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c107fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:06.189 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.463+0000 7f856affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c107fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49120/0 (socket says 172.21.15.172:49120) 2023-11-28T01:07:06.189 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.463+0000 7f856affd700 1 -- 172.21.15.172:0/4029473535 learned_addr learned my addr 172.21.15.172:0/4029473535 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:06.189 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.463+0000 7f856affd700 1 -- 172.21.15.172:0/4029473535 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c108500 con 0x7f856c105bd0 2023-11-28T01:07:06.189 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.463+0000 7f856affd700 1 --2- 172.21.15.172:0/4029473535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c107fc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f855c0098a0 tx=0x7f855c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee55cc2acc4a8a7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:06.189 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.463+0000 7f8569ffb700 1 -- 172.21.15.172:0/4029473535 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f855c004160 con 0x7f856c105bd0 2023-11-28T01:07:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.463+0000 7f8569ffb700 1 -- 172.21.15.172:0/4029473535 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f855c0036a0 con 0x7f856c105bd0 2023-11-28T01:07:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.463+0000 7f8569ffb700 1 -- 172.21.15.172:0/4029473535 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f855c0117c0 con 0x7f856c105bd0 2023-11-28T01:07:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.464+0000 7f857156d700 1 -- 172.21.15.172:0/4029473535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 msgr2=0x7f856c107fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.464+0000 7f857156d700 1 --2- 172.21.15.172:0/4029473535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c107fc0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f855c0098a0 tx=0x7f855c009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.464+0000 7f857156d700 1 -- 172.21.15.172:0/4029473535 shutdown_connections 2023-11-28T01:07:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.464+0000 7f857156d700 1 --2- 172.21.15.172:0/4029473535 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c107fc0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.464+0000 7f857156d700 1 -- 172.21.15.172:0/4029473535 >> 172.21.15.172:0/4029473535 conn(0x7f856c0ff580 msgr2=0x7f856c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.464+0000 7f857156d700 1 -- 172.21.15.172:0/4029473535 shutdown_connections 2023-11-28T01:07:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.464+0000 7f857156d700 1 -- 172.21.15.172:0/4029473535 wait complete. 2023-11-28T01:07:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.464+0000 7f857156d700 1 Processor -- start 2023-11-28T01:07:06.190 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.465+0000 7f857156d700 1 -- start start 2023-11-28T01:07:06.191 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.465+0000 7f857156d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c19afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:06.191 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.465+0000 7f857156d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856c19b4f0 con 0x7f856c105bd0 2023-11-28T01:07:06.191 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.465+0000 7f856affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c19afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:06.191 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.465+0000 7f856affd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c19afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49122/0 (socket says 172.21.15.172:49122) 2023-11-28T01:07:06.191 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.465+0000 7f856affd700 1 -- 172.21.15.172:0/860705530 learned_addr learned my addr 172.21.15.172:0/860705530 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:06.191 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.465+0000 7f856affd700 1 -- 172.21.15.172:0/860705530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855c009550 con 0x7f856c105bd0 2023-11-28T01:07:06.191 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.465+0000 7f856affd700 1 --2- 172.21.15.172:0/860705530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c19afb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f855c011cb0 tx=0x7f855c011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:06.191 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.465+0000 7f8553fff700 1 -- 172.21.15.172:0/860705530 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f855c0042b0 con 0x7f856c105bd0 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.465+0000 7f8553fff700 1 -- 172.21.15.172:0/860705530 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f855c004410 con 0x7f856c105bd0 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.465+0000 7f857156d700 1 -- 172.21.15.172:0/860705530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f856c19b6f0 con 0x7f856c105bd0 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.466+0000 7f8553fff700 1 -- 172.21.15.172:0/860705530 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f855c0193f0 con 0x7f856c105bd0 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.466+0000 7f857156d700 1 -- 172.21.15.172:0/860705530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f856c19bb90 con 0x7f856c105bd0 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.466+0000 7f8553fff700 1 -- 172.21.15.172:0/860705530 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f855c019550 con 0x7f856c105bd0 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.466+0000 7f8553fff700 1 -- 172.21.15.172:0/860705530 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f855c01a3d0 con 0x7f856c105bd0 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.467+0000 7f857156d700 1 -- 172.21.15.172:0/860705530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856c062c00 con 0x7f856c105bd0 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.469+0000 7f8553fff700 1 -- 172.21.15.172:0/860705530 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f855c029060 con 0x7f856c105bd0 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.541+0000 7f857156d700 1 -- 172.21.15.172:0/860705530 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f856c109370 con 0x7f856c105bd0 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.542+0000 7f8553fff700 1 -- 172.21.15.172:0/860705530 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f855c01f070 con 0x7f856c105bd0 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.543+0000 7f857156d700 1 -- 172.21.15.172:0/860705530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 msgr2=0x7f856c19afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:06.192 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.543+0000 7f857156d700 1 --2- 172.21.15.172:0/860705530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c19afb0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f855c011cb0 tx=0x7f855c011d90 comp rx=0 tx=0).stop 2023-11-28T01:07:06.193 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.543+0000 7f857156d700 1 -- 172.21.15.172:0/860705530 shutdown_connections 2023-11-28T01:07:06.193 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.543+0000 7f857156d700 1 --2- 172.21.15.172:0/860705530 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f856c105bd0 0x7f856c19afb0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:06.193 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.543+0000 7f857156d700 1 -- 172.21.15.172:0/860705530 >> 172.21.15.172:0/860705530 conn(0x7f856c0ff580 msgr2=0x7f856c100180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:06.193 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.543+0000 7f857156d700 1 -- 172.21.15.172:0/860705530 shutdown_connections 2023-11-28T01:07:06.193 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:05.543+0000 7f857156d700 1 -- 172.21.15.172:0/860705530 wait complete. 2023-11-28T01:07:06.193 INFO:teuthology.orchestra.run.smithi172.stdout:mgr not available, waiting (1/15)... 2023-11-28T01:07:06.254 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:05 smithi172 conmon[101049]: 2023-11-28T01:07:05.930+0000 7f3757b62200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T01:07:06.254 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:06 smithi172 conmon[101049]: 2023-11-28T01:07:06.065+0000 7f3757b62200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T01:07:06.254 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:06 smithi172 conmon[101049]: 2023-11-28T01:07:06.155+0000 7f3757b62200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T01:07:06.553 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:06 smithi172 conmon[101049]: 2023-11-28T01:07:06.252+0000 7f3757b62200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T01:07:07.053 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:06 smithi172 conmon[101049]: 2023-11-28T01:07:06.671+0000 7f3757b62200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T01:07:07.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:07 smithi172 conmon[101049]: 2023-11-28T01:07:07.336+0000 7f3757b62200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T01:07:07.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:07 smithi172 conmon[101049]: 2023-11-28T01:07:07.418+0000 7f3757b62200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T01:07:08.473 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:08 smithi172 conmon[101049]: 2023-11-28T01:07:08.184+0000 7f3757b62200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T01:07:08.474 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:08 smithi172 conmon[101049]: 2023-11-28T01:07:08.284+0000 7f3757b62200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T01:07:08.736 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:08 smithi172 conmon[101049]: 2023-11-28T01:07:08.471+0000 7f3757b62200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T01:07:09.022 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:08 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/177536075' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-28T01:07:09.023 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:08 smithi172 conmon[101049]: 2023-11-28T01:07:08.914+0000 7f3757b62200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T01:07:09.265 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:07:09.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout { 2023-11-28T01:07:09.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "fsid": "31861c10-8d8a-11ee-95a2-87774f69a715", 2023-11-28T01:07:09.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "health": { 2023-11-28T01:07:09.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-28T01:07:09.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-28T01:07:09.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-28T01:07:09.266 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:09.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-28T01:07:09.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-28T01:07:09.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 0 2023-11-28T01:07:09.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:09.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-28T01:07:09.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "a" 2023-11-28T01:07:09.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:09.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-11-28T01:07:09.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-28T01:07:09.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:09.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2023-11-28T01:07:09.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-28T01:07:09.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:09.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-28T01:07:09.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:09.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-28T01:07:09.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-28T01:07:09.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-28T01:07:09.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-28T01:07:09.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-28T01:07:09.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-28T01:07:09.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:09.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-28T01:07:09.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-28T01:07:09.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-28T01:07:09.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-28T01:07:09.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-28T01:07:09.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-28T01:07:09.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-28T01:07:09.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-28T01:07:09.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-28T01:07:09.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:09.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-28T01:07:09.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:09.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-28T01:07:09.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-28T01:07:09.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:09.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-28T01:07:09.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-28T01:07:09.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-28T01:07:09.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-28T01:07:09.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-28T01:07:09.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-28T01:07:09.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "restful" 2023-11-28T01:07:09.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:09.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-28T01:07:09.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:09.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-28T01:07:09.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:09.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "modified": "2023-11-28T01:06:58.086438+0000", 2023-11-28T01:07:09.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-28T01:07:09.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:09.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-28T01:07:09.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout } 2023-11-28T01:07:09.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.624+0000 7f5d76af8700 1 Processor -- start 2023-11-28T01:07:09.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.624+0000 7f5d76af8700 1 -- start start 2023-11-28T01:07:09.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.625+0000 7f5d76af8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d70103e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:09.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.625+0000 7f5d76af8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d70077e40 con 0x7f5d70103a00 2023-11-28T01:07:09.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.625+0000 7f5d74894700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d70103e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:09.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.625+0000 7f5d74894700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d70103e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49124/0 (socket says 172.21.15.172:49124) 2023-11-28T01:07:09.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.625+0000 7f5d74894700 1 -- 172.21.15.172:0/461730679 learned_addr learned my addr 172.21.15.172:0/461730679 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:09.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.625+0000 7f5d74894700 1 -- 172.21.15.172:0/461730679 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d70104360 con 0x7f5d70103a00 2023-11-28T01:07:09.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.625+0000 7f5d74894700 1 --2- 172.21.15.172:0/461730679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d70103e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5d640098a0 tx=0x7f5d64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ce9db50c6d5a0ca server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:09.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.626+0000 7f5d6f7fe700 1 -- 172.21.15.172:0/461730679 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d64004160 con 0x7f5d70103a00 2023-11-28T01:07:09.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.626+0000 7f5d6f7fe700 1 -- 172.21.15.172:0/461730679 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f5d640036a0 con 0x7f5d70103a00 2023-11-28T01:07:09.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.626+0000 7f5d6f7fe700 1 -- 172.21.15.172:0/461730679 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d640117c0 con 0x7f5d70103a00 2023-11-28T01:07:09.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.626+0000 7f5d76af8700 1 -- 172.21.15.172:0/461730679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 msgr2=0x7f5d70103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:09.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.626+0000 7f5d76af8700 1 --2- 172.21.15.172:0/461730679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d70103e20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5d640098a0 tx=0x7f5d64009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:09.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.626+0000 7f5d76af8700 1 -- 172.21.15.172:0/461730679 shutdown_connections 2023-11-28T01:07:09.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.626+0000 7f5d76af8700 1 --2- 172.21.15.172:0/461730679 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d70103e20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:09.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.626+0000 7f5d76af8700 1 -- 172.21.15.172:0/461730679 >> 172.21.15.172:0/461730679 conn(0x7f5d700ff580 msgr2=0x7f5d70101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:09.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.626+0000 7f5d76af8700 1 -- 172.21.15.172:0/461730679 shutdown_connections 2023-11-28T01:07:09.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.626+0000 7f5d76af8700 1 -- 172.21.15.172:0/461730679 wait complete. 2023-11-28T01:07:09.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.627+0000 7f5d76af8700 1 Processor -- start 2023-11-28T01:07:09.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.627+0000 7f5d76af8700 1 -- start start 2023-11-28T01:07:09.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.627+0000 7f5d76af8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d7019afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:09.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.627+0000 7f5d76af8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7019b520 con 0x7f5d70103a00 2023-11-28T01:07:09.279 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.627+0000 7f5d74894700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d7019afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:09.279 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.627+0000 7f5d74894700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d7019afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49126/0 (socket says 172.21.15.172:49126) 2023-11-28T01:07:09.279 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.627+0000 7f5d74894700 1 -- 172.21.15.172:0/177536075 learned_addr learned my addr 172.21.15.172:0/177536075 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:09.279 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.628+0000 7f5d74894700 1 -- 172.21.15.172:0/177536075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d64009550 con 0x7f5d70103a00 2023-11-28T01:07:09.279 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.628+0000 7f5d74894700 1 --2- 172.21.15.172:0/177536075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d7019afe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d64009870 tx=0x7f5d64011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:09.279 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.628+0000 7f5d6dffb700 1 -- 172.21.15.172:0/177536075 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d64004180 con 0x7f5d70103a00 2023-11-28T01:07:09.280 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.628+0000 7f5d6dffb700 1 -- 172.21.15.172:0/177536075 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f5d640042e0 con 0x7f5d70103a00 2023-11-28T01:07:09.280 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.628+0000 7f5d6dffb700 1 -- 172.21.15.172:0/177536075 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d6401bea0 con 0x7f5d70103a00 2023-11-28T01:07:09.280 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.628+0000 7f5d76af8700 1 -- 172.21.15.172:0/177536075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7019b720 con 0x7f5d70103a00 2023-11-28T01:07:09.280 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.628+0000 7f5d76af8700 1 -- 172.21.15.172:0/177536075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7019bbc0 con 0x7f5d70103a00 2023-11-28T01:07:09.280 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.628+0000 7f5d6dffb700 1 -- 172.21.15.172:0/177536075 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f5d6401a4e0 con 0x7f5d70103a00 2023-11-28T01:07:09.280 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.628+0000 7f5d6dffb700 1 -- 172.21.15.172:0/177536075 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5d64023ac0 con 0x7f5d70103a00 2023-11-28T01:07:09.281 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.629+0000 7f5d76af8700 1 -- 172.21.15.172:0/177536075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d70194ce0 con 0x7f5d70103a00 2023-11-28T01:07:09.281 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.631+0000 7f5d6dffb700 1 -- 172.21.15.172:0/177536075 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f5d64028060 con 0x7f5d70103a00 2023-11-28T01:07:09.281 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.703+0000 7f5d76af8700 1 -- 172.21.15.172:0/177536075 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5d701050b0 con 0x7f5d70103a00 2023-11-28T01:07:09.281 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.703+0000 7f5d6dffb700 1 -- 172.21.15.172:0/177536075 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f5d64004450 con 0x7f5d70103a00 2023-11-28T01:07:09.281 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.704+0000 7f5d76af8700 1 -- 172.21.15.172:0/177536075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 msgr2=0x7f5d7019afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:09.282 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.704+0000 7f5d76af8700 1 --2- 172.21.15.172:0/177536075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d7019afe0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d64009870 tx=0x7f5d64011d10 comp rx=0 tx=0).stop 2023-11-28T01:07:09.282 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.705+0000 7f5d76af8700 1 -- 172.21.15.172:0/177536075 shutdown_connections 2023-11-28T01:07:09.282 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.705+0000 7f5d76af8700 1 --2- 172.21.15.172:0/177536075 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5d70103a00 0x7f5d7019afe0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:09.282 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.705+0000 7f5d76af8700 1 -- 172.21.15.172:0/177536075 >> 172.21.15.172:0/177536075 conn(0x7f5d700ff580 msgr2=0x7f5d70191e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:09.282 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.705+0000 7f5d76af8700 1 -- 172.21.15.172:0/177536075 shutdown_connections 2023-11-28T01:07:09.282 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:08.705+0000 7f5d76af8700 1 -- 172.21.15.172:0/177536075 wait complete. 2023-11-28T01:07:09.283 INFO:teuthology.orchestra.run.smithi172.stdout:mgr not available, waiting (2/15)... 2023-11-28T01:07:09.305 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:09 smithi172 conmon[101049]: 2023-11-28T01:07:09.003+0000 7f3757b62200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T01:07:09.305 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:09 smithi172 conmon[101049]: 2023-11-28T01:07:09.150+0000 7f3757b62200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T01:07:09.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:09 smithi172 conmon[101049]: 2023-11-28T01:07:09.417+0000 7f3757b62200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T01:07:10.437 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:10 smithi172 conmon[101049]: 2023-11-28T01:07:10.113+0000 7f3757b62200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T01:07:10.437 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:10 smithi172 conmon[101049]: 2023-11-28T01:07:10.244+0000 7f3757b62200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T01:07:10.766 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:10 smithi172 conmon[101049]: 2023-11-28T01:07:10.435+0000 7f3757b62200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T01:07:10.766 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:10 smithi172 conmon[101049]: 2023-11-28T01:07:10.547+0000 7f3757b62200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T01:07:11.053 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:10 smithi172 conmon[101049]: 2023-11-28T01:07:10.763+0000 7f3757b62200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T01:07:11.720 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:11 smithi172 conmon[101049]: 2023-11-28T01:07:11.653+0000 7f3757b62200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T01:07:12.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: Activating manager daemon y 2023-11-28T01:07:12.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: mgrmap e2: y(active, starting, since 0.003386s) 2023-11-28T01:07:12.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: from='mgr.14100 172.21.15.172:0/4087041603' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-28T01:07:12.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: from='mgr.14100 172.21.15.172:0/4087041603' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-28T01:07:12.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: from='mgr.14100 172.21.15.172:0/4087041603' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-28T01:07:12.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: from='mgr.14100 172.21.15.172:0/4087041603' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:07:12.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: from='mgr.14100 172.21.15.172:0/4087041603' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-28T01:07:12.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: Manager daemon y is now available 2023-11-28T01:07:12.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: from='mgr.14100 172.21.15.172:0/4087041603' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-11-28T01:07:12.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: from='mgr.14100 172.21.15.172:0/4087041603' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-11-28T01:07:12.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: from='mgr.14100 172.21.15.172:0/4087041603' entity='mgr.y' 2023-11-28T01:07:12.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: from='mgr.14100 172.21.15.172:0/4087041603' entity='mgr.y' 2023-11-28T01:07:12.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:11 smithi172 ceph-mon[100627]: from='mgr.14100 172.21.15.172:0/4087041603' entity='mgr.y' 2023-11-28T01:07:12.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:07:12.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout { 2023-11-28T01:07:12.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "fsid": "31861c10-8d8a-11ee-95a2-87774f69a715", 2023-11-28T01:07:12.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "health": { 2023-11-28T01:07:12.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-28T01:07:12.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-28T01:07:12.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-28T01:07:12.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:12.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 0 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "a" 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-28T01:07:12.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:12.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-28T01:07:12.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:12.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-28T01:07:12.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-28T01:07:12.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-28T01:07:12.288 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-28T01:07:12.288 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-28T01:07:12.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-28T01:07:12.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:12.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-28T01:07:12.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-28T01:07:12.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-28T01:07:12.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-28T01:07:12.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-28T01:07:12.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-28T01:07:12.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-28T01:07:12.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-28T01:07:12.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-28T01:07:12.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:12.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-28T01:07:12.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:12.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-28T01:07:12.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-28T01:07:12.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:12.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-28T01:07:12.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "available": false, 2023-11-28T01:07:12.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-28T01:07:12.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-28T01:07:12.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-28T01:07:12.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-28T01:07:12.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "restful" 2023-11-28T01:07:12.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:12.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-28T01:07:12.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:12.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-28T01:07:12.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "modified": "2023-11-28T01:06:58.086438+0000", 2023-11-28T01:07:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-28T01:07:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-28T01:07:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout } 2023-11-28T01:07:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.715+0000 7f1c3356c700 1 Processor -- start 2023-11-28T01:07:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.715+0000 7f1c3356c700 1 -- start start 2023-11-28T01:07:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.716+0000 7f1c3356c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c103e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.716+0000 7f1c3356c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c077e40 con 0x7f1c2c103a00 2023-11-28T01:07:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.716+0000 7f1c31308700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c103e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.716+0000 7f1c31308700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c103e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49142/0 (socket says 172.21.15.172:49142) 2023-11-28T01:07:12.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.716+0000 7f1c31308700 1 -- 172.21.15.172:0/443359485 learned_addr learned my addr 172.21.15.172:0/443359485 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:12.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.716+0000 7f1c31308700 1 -- 172.21.15.172:0/443359485 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2c104360 con 0x7f1c2c103a00 2023-11-28T01:07:12.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.716+0000 7f1c31308700 1 --2- 172.21.15.172:0/443359485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c103e20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009e80 tx=0x7f1c2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec8e41502c8e95e8 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:12.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.716+0000 7f1c2bfff700 1 -- 172.21.15.172:0/443359485 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c2400e070 con 0x7f1c2c103a00 2023-11-28T01:07:12.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.717+0000 7f1c2bfff700 1 -- 172.21.15.172:0/443359485 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f1c24003e50 con 0x7f1c2c103a00 2023-11-28T01:07:12.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.717+0000 7f1c2bfff700 1 -- 172.21.15.172:0/443359485 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c24004240 con 0x7f1c2c103a00 2023-11-28T01:07:12.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.717+0000 7f1c3356c700 1 -- 172.21.15.172:0/443359485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 msgr2=0x7f1c2c103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:12.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.717+0000 7f1c3356c700 1 --2- 172.21.15.172:0/443359485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c103e20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1c24009e80 tx=0x7f1c2400b330 comp rx=0 tx=0).stop 2023-11-28T01:07:12.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.717+0000 7f1c3356c700 1 -- 172.21.15.172:0/443359485 shutdown_connections 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.717+0000 7f1c3356c700 1 --2- 172.21.15.172:0/443359485 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c103e20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.717+0000 7f1c3356c700 1 -- 172.21.15.172:0/443359485 >> 172.21.15.172:0/443359485 conn(0x7f1c2c0ff580 msgr2=0x7f1c2c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.718+0000 7f1c3356c700 1 -- 172.21.15.172:0/443359485 shutdown_connections 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.718+0000 7f1c3356c700 1 -- 172.21.15.172:0/443359485 wait complete. 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.718+0000 7f1c3356c700 1 Processor -- start 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.718+0000 7f1c3356c700 1 -- start start 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.718+0000 7f1c3356c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c19af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.718+0000 7f1c3356c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2c19b440 con 0x7f1c2c103a00 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.718+0000 7f1c31308700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c19af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.718+0000 7f1c31308700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c19af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49144/0 (socket says 172.21.15.172:49144) 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.718+0000 7f1c31308700 1 -- 172.21.15.172:0/941369 learned_addr learned my addr 172.21.15.172:0/941369 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:12.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.719+0000 7f1c31308700 1 -- 172.21.15.172:0/941369 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c24009b30 con 0x7f1c2c103a00 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.719+0000 7f1c31308700 1 --2- 172.21.15.172:0/941369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c19af00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400bf50 tx=0x7f1c240043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.722+0000 7f1c2a7fc700 1 -- 172.21.15.172:0/941369 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c2400e040 con 0x7f1c2c103a00 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.722+0000 7f1c2a7fc700 1 -- 172.21.15.172:0/941369 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f1c2401e880 con 0x7f1c2c103a00 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.722+0000 7f1c3356c700 1 -- 172.21.15.172:0/941369 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2c19b640 con 0x7f1c2c103a00 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.722+0000 7f1c2a7fc700 1 -- 172.21.15.172:0/941369 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c2401c710 con 0x7f1c2c103a00 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.722+0000 7f1c3356c700 1 -- 172.21.15.172:0/941369 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2c19bae0 con 0x7f1c2c103a00 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.722+0000 7f1c2a7fc700 1 -- 172.21.15.172:0/941369 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 2) v1 ==== 45475+0+0 (secure 0 0 0) 0x7f1c2401c8c0 con 0x7f1c2c103a00 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.723+0000 7f1c2a7fc700 1 -- 172.21.15.172:0/941369 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1c2404c7d0 con 0x7f1c2c103a00 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.723+0000 7f1c3356c700 1 -- 172.21.15.172:0/941369 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2c194c80 con 0x7f1c2c103a00 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.725+0000 7f1c2a7fc700 1 -- 172.21.15.172:0/941369 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f1c24017080 con 0x7f1c2c103a00 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.794+0000 7f1c3356c700 1 -- 172.21.15.172:0/941369 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1c2c1050b0 con 0x7f1c2c103a00 2023-11-28T01:07:12.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.794+0000 7f1c2a7fc700 1 -- 172.21.15.172:0/941369 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f1c24026b50 con 0x7f1c2c103a00 2023-11-28T01:07:12.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.795+0000 7f1c3356c700 1 -- 172.21.15.172:0/941369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 msgr2=0x7f1c2c19af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:12.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.795+0000 7f1c3356c700 1 --2- 172.21.15.172:0/941369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c19af00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1c2400bf50 tx=0x7f1c240043b0 comp rx=0 tx=0).stop 2023-11-28T01:07:12.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.795+0000 7f1c3356c700 1 -- 172.21.15.172:0/941369 shutdown_connections 2023-11-28T01:07:12.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.795+0000 7f1c3356c700 1 --2- 172.21.15.172:0/941369 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1c2c103a00 0x7f1c2c19af00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:12.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.795+0000 7f1c3356c700 1 -- 172.21.15.172:0/941369 >> 172.21.15.172:0/941369 conn(0x7f1c2c0ff580 msgr2=0x7f1c2c191dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:12.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.795+0000 7f1c3356c700 1 -- 172.21.15.172:0/941369 shutdown_connections 2023-11-28T01:07:12.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:11.795+0000 7f1c3356c700 1 -- 172.21.15.172:0/941369 wait complete. 2023-11-28T01:07:12.296 INFO:teuthology.orchestra.run.smithi172.stdout:mgr not available, waiting (3/15)... 2023-11-28T01:07:13.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:12 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/941369' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-28T01:07:13.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:12 smithi172 ceph-mon[100627]: mgrmap e3: y(active, since 1.00677s) 2023-11-28T01:07:15.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:14 smithi172 ceph-mon[100627]: mgrmap e4: y(active, since 2s) 2023-11-28T01:07:15.459 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:07:15.459 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout { 2023-11-28T01:07:15.459 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "fsid": "31861c10-8d8a-11ee-95a2-87774f69a715", 2023-11-28T01:07:15.459 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "health": { 2023-11-28T01:07:15.459 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-11-28T01:07:15.459 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-11-28T01:07:15.459 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 0 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "a" 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "monmap": { 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:15.460 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2023-11-28T01:07:15.461 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-11-28T01:07:15.461 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:15.461 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-11-28T01:07:15.461 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:15.461 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-11-28T01:07:15.461 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-11-28T01:07:15.461 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-11-28T01:07:15.461 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-11-28T01:07:15.461 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-11-28T01:07:15.461 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-11-28T01:07:15.462 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:15.462 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-11-28T01:07:15.462 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-11-28T01:07:15.462 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-11-28T01:07:15.462 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-11-28T01:07:15.462 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-11-28T01:07:15.462 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-11-28T01:07:15.462 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-11-28T01:07:15.462 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-11-28T01:07:15.462 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-11-28T01:07:15.462 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "modules": [ 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "iostat", 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "nfs", 2023-11-28T01:07:15.463 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "restful" 2023-11-28T01:07:15.464 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ], 2023-11-28T01:07:15.464 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-28T01:07:15.464 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:15.464 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-11-28T01:07:15.464 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-11-28T01:07:15.464 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "modified": "2023-11-28T01:06:58.086438+0000", 2023-11-28T01:07:15.464 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "services": {} 2023-11-28T01:07:15.464 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout }, 2023-11-28T01:07:15.464 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-11-28T01:07:15.464 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout } 2023-11-28T01:07:15.464 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.722+0000 7f3ca3729700 1 Processor -- start 2023-11-28T01:07:15.465 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.722+0000 7f3ca3729700 1 -- start start 2023-11-28T01:07:15.465 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.723+0000 7f3ca3729700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c107f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:15.465 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.723+0000 7f3ca3729700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c077ec0 con 0x7f3c9c105b90 2023-11-28T01:07:15.465 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.723+0000 7f3ca14c5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:15.465 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.723+0000 7f3ca14c5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c107f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49156/0 (socket says 172.21.15.172:49156) 2023-11-28T01:07:15.465 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.723+0000 7f3ca14c5700 1 -- 172.21.15.172:0/4203635663 learned_addr learned my addr 172.21.15.172:0/4203635663 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:15.465 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.723+0000 7f3ca14c5700 1 -- 172.21.15.172:0/4203635663 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c1084c0 con 0x7f3c9c105b90 2023-11-28T01:07:15.465 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.723+0000 7f3ca14c5700 1 --2- 172.21.15.172:0/4203635663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c107f80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3c900098a0 tx=0x7f3c90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549a6ccb7a523b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:15.465 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.723+0000 7f3c8ffff700 1 -- 172.21.15.172:0/4203635663 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c90004160 con 0x7f3c9c105b90 2023-11-28T01:07:15.465 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.723+0000 7f3c8ffff700 1 -- 172.21.15.172:0/4203635663 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f3c900036a0 con 0x7f3c9c105b90 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.723+0000 7f3c8ffff700 1 -- 172.21.15.172:0/4203635663 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c900117c0 con 0x7f3c9c105b90 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.724+0000 7f3ca3729700 1 -- 172.21.15.172:0/4203635663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 msgr2=0x7f3c9c107f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.724+0000 7f3ca3729700 1 --2- 172.21.15.172:0/4203635663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c107f80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3c900098a0 tx=0x7f3c90009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.724+0000 7f3ca3729700 1 -- 172.21.15.172:0/4203635663 shutdown_connections 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.724+0000 7f3ca3729700 1 --2- 172.21.15.172:0/4203635663 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c107f80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.724+0000 7f3ca3729700 1 -- 172.21.15.172:0/4203635663 >> 172.21.15.172:0/4203635663 conn(0x7f3c9c0ff580 msgr2=0x7f3c9c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.724+0000 7f3ca3729700 1 -- 172.21.15.172:0/4203635663 shutdown_connections 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.724+0000 7f3ca3729700 1 -- 172.21.15.172:0/4203635663 wait complete. 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.725+0000 7f3ca3729700 1 Processor -- start 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.725+0000 7f3ca3729700 1 -- start start 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.725+0000 7f3ca3729700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c1053c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:15.466 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.725+0000 7f3ca3729700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c105900 con 0x7f3c9c105b90 2023-11-28T01:07:15.467 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.725+0000 7f3ca14c5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c1053c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:15.467 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.725+0000 7f3ca14c5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c1053c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49158/0 (socket says 172.21.15.172:49158) 2023-11-28T01:07:15.467 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.725+0000 7f3ca14c5700 1 -- 172.21.15.172:0/1093554141 learned_addr learned my addr 172.21.15.172:0/1093554141 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:15.467 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.725+0000 7f3ca14c5700 1 -- 172.21.15.172:0/1093554141 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c90009550 con 0x7f3c9c105b90 2023-11-28T01:07:15.467 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.725+0000 7f3ca14c5700 1 --2- 172.21.15.172:0/1093554141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c1053c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009870 tx=0x7f3c90011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:15.467 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.726+0000 7f3c8e7fc700 1 -- 172.21.15.172:0/1093554141 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c90004180 con 0x7f3c9c105b90 2023-11-28T01:07:15.469 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.726+0000 7f3c8e7fc700 1 -- 172.21.15.172:0/1093554141 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f3c900042e0 con 0x7f3c9c105b90 2023-11-28T01:07:15.470 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.726+0000 7f3c8e7fc700 1 -- 172.21.15.172:0/1093554141 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3c9001bea0 con 0x7f3c9c105b90 2023-11-28T01:07:15.470 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.726+0000 7f3ca3729700 1 -- 172.21.15.172:0/1093554141 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9c103b40 con 0x7f3c9c105b90 2023-11-28T01:07:15.470 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.726+0000 7f3ca3729700 1 -- 172.21.15.172:0/1093554141 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9c103fe0 con 0x7f3c9c105b90 2023-11-28T01:07:15.470 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.726+0000 7f3c8e7fc700 1 -- 172.21.15.172:0/1093554141 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 4) v1 ==== 45708+0+0 (secure 0 0 0) 0x7f3c9001a4e0 con 0x7f3c9c105b90 2023-11-28T01:07:15.470 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.727+0000 7f3c8e7fc700 1 --2- 172.21.15.172:0/1093554141 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f3c88038b90 0x7f3c8803b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:15.470 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.727+0000 7f3c8e7fc700 1 -- 172.21.15.172:0/1093554141 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3c9004c940 con 0x7f3c9c105b90 2023-11-28T01:07:15.470 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.727+0000 7f3ca3729700 1 -- 172.21.15.172:0/1093554141 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9c062c00 con 0x7f3c9c105b90 2023-11-28T01:07:15.470 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.727+0000 7f3ca0cc4700 1 --2- 172.21.15.172:0/1093554141 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f3c88038b90 0x7f3c8803b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:15.470 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.727+0000 7f3ca0cc4700 1 --2- 172.21.15.172:0/1093554141 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f3c88038b90 0x7f3c8803b050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3c98006fd0 tx=0x7f3c98006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:15.470 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.732+0000 7f3c8e7fc700 1 -- 172.21.15.172:0/1093554141 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f3c90018530 con 0x7f3c9c105b90 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.996+0000 7f3ca3729700 1 -- 172.21.15.172:0/1093554141 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3c9c1091a0 con 0x7f3c9c105b90 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.996+0000 7f3c8e7fc700 1 -- 172.21.15.172:0/1093554141 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f3c9c1091a0 con 0x7f3c9c105b90 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.998+0000 7f3ca3729700 1 -- 172.21.15.172:0/1093554141 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f3c88038b90 msgr2=0x7f3c8803b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.998+0000 7f3ca3729700 1 --2- 172.21.15.172:0/1093554141 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f3c88038b90 0x7f3c8803b050 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3c98006fd0 tx=0x7f3c98006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.998+0000 7f3ca3729700 1 -- 172.21.15.172:0/1093554141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 msgr2=0x7f3c9c1053c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.998+0000 7f3ca3729700 1 --2- 172.21.15.172:0/1093554141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c1053c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3c90009870 tx=0x7f3c90011d10 comp rx=0 tx=0).stop 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.998+0000 7f3ca3729700 1 -- 172.21.15.172:0/1093554141 shutdown_connections 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.998+0000 7f3ca3729700 1 --2- 172.21.15.172:0/1093554141 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f3c88038b90 0x7f3c8803b050 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.998+0000 7f3ca3729700 1 --2- 172.21.15.172:0/1093554141 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f3c9c105b90 0x7f3c9c1053c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.998+0000 7f3ca3729700 1 -- 172.21.15.172:0/1093554141 >> 172.21.15.172:0/1093554141 conn(0x7f3c9c0ff580 msgr2=0x7f3c9c100210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.998+0000 7f3ca3729700 1 -- 172.21.15.172:0/1093554141 shutdown_connections 2023-11-28T01:07:15.471 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:14.998+0000 7f3ca3729700 1 -- 172.21.15.172:0/1093554141 wait complete. 2023-11-28T01:07:15.472 INFO:teuthology.orchestra.run.smithi172.stdout:mgr is available 2023-11-28T01:07:16.246 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:15 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/1093554141' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-28T01:07:16.596 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:07:16.596 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout [global] 2023-11-28T01:07:16.596 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout fsid = 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:07:16.596 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-11-28T01:07:16.596 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-11-28T01:07:16.596 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-11-28T01:07:16.596 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-11-28T01:07:16.596 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:07:16.596 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout [mgr] 2023-11-28T01:07:16.596 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout [osd] 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.889+0000 7f1587efc700 1 Processor -- start 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.889+0000 7f1587efc700 1 -- start start 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.889+0000 7f1587efc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f1580103e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.889+0000 7f1587efc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15800786d0 con 0x7f1580103a10 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.889+0000 7f1585c98700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f1580103e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.889+0000 7f1585c98700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f1580103e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49162/0 (socket says 172.21.15.172:49162) 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.889+0000 7f1585c98700 1 -- 172.21.15.172:0/4245551198 learned_addr learned my addr 172.21.15.172:0/4245551198 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:16.597 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.889+0000 7f1585c98700 1 -- 172.21.15.172:0/4245551198 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1580104370 con 0x7f1580103a10 2023-11-28T01:07:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.890+0000 7f1585c98700 1 --2- 172.21.15.172:0/4245551198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f1580103e30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f157c0098a0 tx=0x7f157c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2cc23be2589cb6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.890+0000 7f1584c96700 1 -- 172.21.15.172:0/4245551198 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f157c004160 con 0x7f1580103a10 2023-11-28T01:07:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.890+0000 7f1584c96700 1 -- 172.21.15.172:0/4245551198 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f157c0036a0 con 0x7f1580103a10 2023-11-28T01:07:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.890+0000 7f1584c96700 1 -- 172.21.15.172:0/4245551198 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f157c0117c0 con 0x7f1580103a10 2023-11-28T01:07:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.890+0000 7f1587efc700 1 -- 172.21.15.172:0/4245551198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 msgr2=0x7f1580103e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.890+0000 7f1587efc700 1 --2- 172.21.15.172:0/4245551198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f1580103e30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f157c0098a0 tx=0x7f157c009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.890+0000 7f1587efc700 1 -- 172.21.15.172:0/4245551198 shutdown_connections 2023-11-28T01:07:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.890+0000 7f1587efc700 1 --2- 172.21.15.172:0/4245551198 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f1580103e30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.890+0000 7f1587efc700 1 -- 172.21.15.172:0/4245551198 >> 172.21.15.172:0/4245551198 conn(0x7f15800ff590 msgr2=0x7f1580101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.890+0000 7f1587efc700 1 -- 172.21.15.172:0/4245551198 shutdown_connections 2023-11-28T01:07:16.598 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.890+0000 7f1587efc700 1 -- 172.21.15.172:0/4245551198 wait complete. 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.891+0000 7f1587efc700 1 Processor -- start 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.891+0000 7f1587efc700 1 -- start start 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.891+0000 7f1587efc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f158019af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.891+0000 7f1587efc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158019b480 con 0x7f1580103a10 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.891+0000 7f1585c98700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f158019af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.891+0000 7f1585c98700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f158019af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49164/0 (socket says 172.21.15.172:49164) 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.891+0000 7f1585c98700 1 -- 172.21.15.172:0/4121700823 learned_addr learned my addr 172.21.15.172:0/4121700823 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.892+0000 7f1585c98700 1 -- 172.21.15.172:0/4121700823 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c009550 con 0x7f1580103a10 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.892+0000 7f1585c98700 1 --2- 172.21.15.172:0/4121700823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f158019af40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f157c009130 tx=0x7f157c0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.892+0000 7f1576ffd700 1 -- 172.21.15.172:0/4121700823 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f157c004180 con 0x7f1580103a10 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.892+0000 7f1576ffd700 1 -- 172.21.15.172:0/4121700823 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f157c0042e0 con 0x7f1580103a10 2023-11-28T01:07:16.599 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.892+0000 7f1576ffd700 1 -- 172.21.15.172:0/4121700823 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f157c0193f0 con 0x7f1580103a10 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.892+0000 7f1587efc700 1 -- 172.21.15.172:0/4121700823 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158019b680 con 0x7f1580103a10 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.892+0000 7f1587efc700 1 -- 172.21.15.172:0/4121700823 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158019bb20 con 0x7f1580103a10 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.893+0000 7f1587efc700 1 -- 172.21.15.172:0/4121700823 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1580194cf0 con 0x7f1580103a10 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.893+0000 7f1576ffd700 1 -- 172.21.15.172:0/4121700823 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 4) v1 ==== 45708+0+0 (secure 0 0 0) 0x7f157c01b440 con 0x7f1580103a10 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.893+0000 7f1576ffd700 1 --2- 172.21.15.172:0/4121700823 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f156c038b40 0x7f156c03b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.893+0000 7f1576ffd700 1 -- 172.21.15.172:0/4121700823 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f157c04cb90 con 0x7f1580103a10 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.893+0000 7f1585497700 1 --2- 172.21.15.172:0/4121700823 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f156c038b40 0x7f156c03b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.894+0000 7f1585497700 1 --2- 172.21.15.172:0/4121700823 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f156c038b40 0x7f156c03b000 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1570006fd0 tx=0x7f1570006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:15.898+0000 7f1576ffd700 1 -- 172.21.15.172:0/4121700823 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f157c022540 con 0x7f1580103a10 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.083+0000 7f1587efc700 1 -- 172.21.15.172:0/4121700823 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f15801050c0 con 0x7f1580103a10 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.083+0000 7f1576ffd700 1 -- 172.21.15.172:0/4121700823 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (secure 0 0 0) 0x7f157c01f020 con 0x7f1580103a10 2023-11-28T01:07:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.085+0000 7f1587efc700 1 -- 172.21.15.172:0/4121700823 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f156c038b40 msgr2=0x7f156c03b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:16.601 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.085+0000 7f1587efc700 1 --2- 172.21.15.172:0/4121700823 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f156c038b40 0x7f156c03b000 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1570006fd0 tx=0x7f1570006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:16.601 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.085+0000 7f1587efc700 1 -- 172.21.15.172:0/4121700823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 msgr2=0x7f158019af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:16.601 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.085+0000 7f1587efc700 1 --2- 172.21.15.172:0/4121700823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f158019af40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f157c009130 tx=0x7f157c0119c0 comp rx=0 tx=0).stop 2023-11-28T01:07:16.601 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.085+0000 7f1587efc700 1 -- 172.21.15.172:0/4121700823 shutdown_connections 2023-11-28T01:07:16.601 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.085+0000 7f1587efc700 1 --2- 172.21.15.172:0/4121700823 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f156c038b40 0x7f156c03b000 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:16.601 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.085+0000 7f1587efc700 1 --2- 172.21.15.172:0/4121700823 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1580103a10 0x7f158019af40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:16.601 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.086+0000 7f1587efc700 1 -- 172.21.15.172:0/4121700823 >> 172.21.15.172:0/4121700823 conn(0x7f15800ff590 msgr2=0x7f1580191e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:16.601 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.086+0000 7f1587efc700 1 -- 172.21.15.172:0/4121700823 shutdown_connections 2023-11-28T01:07:16.601 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:16.086+0000 7f1587efc700 1 -- 172.21.15.172:0/4121700823 wait complete. 2023-11-28T01:07:16.601 INFO:teuthology.orchestra.run.smithi172.stdout:Enabling cephadm module... 2023-11-28T01:07:17.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:16 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/4121700823' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-28T01:07:18.143 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:17 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2686193034' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-28T01:07:18.144 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:17 smithi172 conmon[101049]: ignoring --setuser ceph since I am not root 2023-11-28T01:07:18.144 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:17 smithi172 conmon[101049]: ignoring --setgroup ceph since I am not root 2023-11-28T01:07:18.144 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:18 smithi172 conmon[101049]: 2023-11-28T01:07:18.055+0000 7fc263ce5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T01:07:18.144 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:18 smithi172 conmon[101049]: 2023-11-28T01:07:18.138+0000 7fc263ce5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T01:07:18.499 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:18 smithi172 conmon[101049]: 2023-11-28T01:07:18.231+0000 7fc263ce5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T01:07:18.499 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:18 smithi172 conmon[101049]: 2023-11-28T01:07:18.325+0000 7fc263ce5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T01:07:18.581 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.014+0000 7f4099ddc700 1 Processor -- start 2023-11-28T01:07:18.581 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.014+0000 7f4099ddc700 1 -- start start 2023-11-28T01:07:18.582 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.015+0000 7f4099ddc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:18.582 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.015+0000 7f4099ddc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4094078710 con 0x7f4094105bf0 2023-11-28T01:07:18.582 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.015+0000 7f40937fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:18.583 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.015+0000 7f40937fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094107fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49168/0 (socket says 172.21.15.172:49168) 2023-11-28T01:07:18.583 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.015+0000 7f40937fe700 1 -- 172.21.15.172:0/3179881764 learned_addr learned my addr 172.21.15.172:0/3179881764 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:18.583 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.015+0000 7f40937fe700 1 -- 172.21.15.172:0/3179881764 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4094108520 con 0x7f4094105bf0 2023-11-28T01:07:18.584 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.015+0000 7f40937fe700 1 --2- 172.21.15.172:0/3179881764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094107fe0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f407c0098a0 tx=0x7f407c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4e8276537476ca9 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:18.584 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.015+0000 7f40927fc700 1 -- 172.21.15.172:0/3179881764 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f407c004160 con 0x7f4094105bf0 2023-11-28T01:07:18.584 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.015+0000 7f40927fc700 1 -- 172.21.15.172:0/3179881764 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f407c0036a0 con 0x7f4094105bf0 2023-11-28T01:07:18.585 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.015+0000 7f40927fc700 1 -- 172.21.15.172:0/3179881764 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f407c0117c0 con 0x7f4094105bf0 2023-11-28T01:07:18.585 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.016+0000 7f4099ddc700 1 -- 172.21.15.172:0/3179881764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 msgr2=0x7f4094107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:18.585 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.016+0000 7f4099ddc700 1 --2- 172.21.15.172:0/3179881764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094107fe0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f407c0098a0 tx=0x7f407c009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:18.585 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.016+0000 7f4099ddc700 1 -- 172.21.15.172:0/3179881764 shutdown_connections 2023-11-28T01:07:18.586 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.016+0000 7f4099ddc700 1 --2- 172.21.15.172:0/3179881764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094107fe0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:18.586 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.016+0000 7f4099ddc700 1 -- 172.21.15.172:0/3179881764 >> 172.21.15.172:0/3179881764 conn(0x7f40940ff5c0 msgr2=0x7f4094101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:18.586 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.016+0000 7f4099ddc700 1 -- 172.21.15.172:0/3179881764 shutdown_connections 2023-11-28T01:07:18.586 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.016+0000 7f4099ddc700 1 -- 172.21.15.172:0/3179881764 wait complete. 2023-11-28T01:07:18.586 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.017+0000 7f4099ddc700 1 Processor -- start 2023-11-28T01:07:18.586 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.017+0000 7f4099ddc700 1 -- start start 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.017+0000 7f4099ddc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094198d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.017+0000 7f4099ddc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4094199280 con 0x7f4094105bf0 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.017+0000 7f40937fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094198d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.017+0000 7f40937fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094198d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49170/0 (socket says 172.21.15.172:49170) 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.017+0000 7f40937fe700 1 -- 172.21.15.172:0/2686193034 learned_addr learned my addr 172.21.15.172:0/2686193034 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.017+0000 7f40937fe700 1 -- 172.21.15.172:0/2686193034 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c009550 con 0x7f4094105bf0 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.017+0000 7f40937fe700 1 --2- 172.21.15.172:0/2686193034 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094198d40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f407c009130 tx=0x7f407c0119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.018+0000 7f4090ff9700 1 -- 172.21.15.172:0/2686193034 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f407c004180 con 0x7f4094105bf0 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.018+0000 7f4099ddc700 1 -- 172.21.15.172:0/2686193034 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4094199480 con 0x7f4094105bf0 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.018+0000 7f4090ff9700 1 -- 172.21.15.172:0/2686193034 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f407c0042e0 con 0x7f4094105bf0 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.018+0000 7f4090ff9700 1 -- 172.21.15.172:0/2686193034 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f407c0193f0 con 0x7f4094105bf0 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.018+0000 7f4099ddc700 1 -- 172.21.15.172:0/2686193034 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4094199920 con 0x7f4094105bf0 2023-11-28T01:07:18.587 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.018+0000 7f4090ff9700 1 -- 172.21.15.172:0/2686193034 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 4) v1 ==== 45708+0+0 (secure 0 0 0) 0x7f407c0196b0 con 0x7f4094105bf0 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.019+0000 7f4090ff9700 1 --2- 172.21.15.172:0/2686193034 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f4080038af0 0x7f408003afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.019+0000 7f4090ff9700 1 -- 172.21.15.172:0/2686193034 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f407c01b870 con 0x7f4094105bf0 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.019+0000 7f4092ffd700 1 --2- 172.21.15.172:0/2686193034 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f4080038af0 0x7f408003afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.019+0000 7f4099ddc700 1 -- 172.21.15.172:0/2686193034 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4094062c00 con 0x7f4094105bf0 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.019+0000 7f4092ffd700 1 --2- 172.21.15.172:0/2686193034 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f4080038af0 0x7f408003afb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4084006fd0 tx=0x7f4084006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.024+0000 7f4090ff9700 1 -- 172.21.15.172:0/2686193034 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f407c01f020 con 0x7f4094105bf0 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.264+0000 7f4099ddc700 1 -- 172.21.15.172:0/2686193034 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f40941091a0 con 0x7f4094105bf0 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.885+0000 7f4090ff9700 1 -- 172.21.15.172:0/2686193034 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f407c04b090 con 0x7f4094105bf0 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.886+0000 7f4090ff9700 1 -- 172.21.15.172:0/2686193034 <== mon.0 v2:172.21.15.172:3300/0 8 ==== mgrmap(e 5) v1 ==== 45719+0+0 (secure 0 0 0) 0x7f407c020d20 con 0x7f4094105bf0 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.888+0000 7f4099ddc700 1 -- 172.21.15.172:0/2686193034 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f4080038af0 msgr2=0x7f408003afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.888+0000 7f4099ddc700 1 --2- 172.21.15.172:0/2686193034 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f4080038af0 0x7f408003afb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4084006fd0 tx=0x7f4084006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.888+0000 7f4099ddc700 1 -- 172.21.15.172:0/2686193034 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 msgr2=0x7f4094198d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:18.588 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.888+0000 7f4099ddc700 1 --2- 172.21.15.172:0/2686193034 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094198d40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f407c009130 tx=0x7f407c0119c0 comp rx=0 tx=0).stop 2023-11-28T01:07:18.589 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.888+0000 7f4099ddc700 1 -- 172.21.15.172:0/2686193034 shutdown_connections 2023-11-28T01:07:18.589 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.888+0000 7f4099ddc700 1 --2- 172.21.15.172:0/2686193034 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f4080038af0 0x7f408003afb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:18.589 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.888+0000 7f4099ddc700 1 --2- 172.21.15.172:0/2686193034 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4094105bf0 0x7f4094198d40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:18.589 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.888+0000 7f4099ddc700 1 -- 172.21.15.172:0/2686193034 >> 172.21.15.172:0/2686193034 conn(0x7f40940ff5c0 msgr2=0x7f4094100140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:18.589 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.888+0000 7f4099ddc700 1 -- 172.21.15.172:0/2686193034 shutdown_connections 2023-11-28T01:07:18.589 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:17.888+0000 7f4099ddc700 1 -- 172.21.15.172:0/2686193034 wait complete. 2023-11-28T01:07:18.754 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:18 smithi172 conmon[101049]: 2023-11-28T01:07:18.497+0000 7fc263ce5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T01:07:19.038 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:18 smithi172 conmon[101049]: 2023-11-28T01:07:18.897+0000 7fc263ce5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T01:07:19.039 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:18 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2686193034' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-28T01:07:19.039 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:18 smithi172 ceph-mon[100627]: mgrmap e5: y(active, since 6s) 2023-11-28T01:07:19.299 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:19 smithi172 conmon[101049]: 2023-11-28T01:07:19.035+0000 7fc263ce5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T01:07:19.299 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:19 smithi172 conmon[101049]: 2023-11-28T01:07:19.126+0000 7fc263ce5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T01:07:19.299 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:19 smithi172 conmon[101049]: 2023-11-28T01:07:19.223+0000 7fc263ce5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T01:07:19.753 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout { 2023-11-28T01:07:19.753 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-11-28T01:07:19.753 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-28T01:07:19.753 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "active_name": "y", 2023-11-28T01:07:19.753 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-11-28T01:07:19.753 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout } 2023-11-28T01:07:19.754 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.013+0000 7fe937bfc700 1 Processor -- start 2023-11-28T01:07:19.754 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.014+0000 7fe937bfc700 1 -- start start 2023-11-28T01:07:19.754 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.014+0000 7fe937bfc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe9300782f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:19.754 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.014+0000 7fe937bfc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9300788c0 con 0x7fe930079e90 2023-11-28T01:07:19.754 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.014+0000 7fe935998700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe9300782f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:19.754 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.014+0000 7fe935998700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe9300782f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49184/0 (socket says 172.21.15.172:49184) 2023-11-28T01:07:19.754 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.014+0000 7fe935998700 1 -- 172.21.15.172:0/1794609778 learned_addr learned my addr 172.21.15.172:0/1794609778 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:19.754 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.014+0000 7fe935998700 1 -- 172.21.15.172:0/1794609778 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930078a00 con 0x7fe930079e90 2023-11-28T01:07:19.754 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.014+0000 7fe935998700 1 --2- 172.21.15.172:0/1794609778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe9300782f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0098a0 tx=0x7fe92c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac5e8682323153c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:19.754 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.014+0000 7fe934996700 1 -- 172.21.15.172:0/1794609778 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe92c004160 con 0x7fe930079e90 2023-11-28T01:07:19.754 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.015+0000 7fe934996700 1 -- 172.21.15.172:0/1794609778 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7fe92c0036a0 con 0x7fe930079e90 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.015+0000 7fe934996700 1 -- 172.21.15.172:0/1794609778 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe92c0117c0 con 0x7fe930079e90 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.015+0000 7fe937bfc700 1 -- 172.21.15.172:0/1794609778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 msgr2=0x7fe9300782f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.015+0000 7fe937bfc700 1 --2- 172.21.15.172:0/1794609778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe9300782f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe92c0098a0 tx=0x7fe92c009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.015+0000 7fe937bfc700 1 -- 172.21.15.172:0/1794609778 shutdown_connections 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.015+0000 7fe937bfc700 1 --2- 172.21.15.172:0/1794609778 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe9300782f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.015+0000 7fe937bfc700 1 -- 172.21.15.172:0/1794609778 >> 172.21.15.172:0/1794609778 conn(0x7fe9301006f0 msgr2=0x7fe930102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.015+0000 7fe937bfc700 1 -- 172.21.15.172:0/1794609778 shutdown_connections 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.015+0000 7fe937bfc700 1 -- 172.21.15.172:0/1794609778 wait complete. 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.016+0000 7fe937bfc700 1 Processor -- start 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.016+0000 7fe937bfc700 1 -- start start 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.016+0000 7fe937bfc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe93019afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:19.755 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.016+0000 7fe937bfc700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93019b520 con 0x7fe930079e90 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.016+0000 7fe935998700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe93019afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.016+0000 7fe935998700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe93019afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49186/0 (socket says 172.21.15.172:49186) 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.016+0000 7fe935998700 1 -- 172.21.15.172:0/3685184311 learned_addr learned my addr 172.21.15.172:0/3685184311 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.016+0000 7fe935998700 1 -- 172.21.15.172:0/3685184311 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c009550 con 0x7fe930079e90 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.017+0000 7fe935998700 1 --2- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe93019afe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe92c009870 tx=0x7fe92c011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.017+0000 7fe926ffd700 1 -- 172.21.15.172:0/3685184311 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe92c004180 con 0x7fe930079e90 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.017+0000 7fe926ffd700 1 -- 172.21.15.172:0/3685184311 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7fe92c0042e0 con 0x7fe930079e90 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.017+0000 7fe937bfc700 1 -- 172.21.15.172:0/3685184311 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93019b720 con 0x7fe930079e90 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.017+0000 7fe926ffd700 1 -- 172.21.15.172:0/3685184311 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe92c01bea0 con 0x7fe930079e90 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.017+0000 7fe937bfc700 1 -- 172.21.15.172:0/3685184311 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93019bbc0 con 0x7fe930079e90 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.018+0000 7fe937bfc700 1 -- 172.21.15.172:0/3685184311 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93010b420 con 0x7fe930079e90 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.018+0000 7fe926ffd700 1 -- 172.21.15.172:0/3685184311 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 45719+0+0 (secure 0 0 0) 0x7fe92c01b440 con 0x7fe930079e90 2023-11-28T01:07:19.756 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.018+0000 7fe926ffd700 1 --2- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7fe91c038be0 0x7fe91c03b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.018+0000 7fe926ffd700 1 -- 172.21.15.172:0/3685184311 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe92c04c710 con 0x7fe930079e90 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.018+0000 7fe935197700 1 -- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7fe91c038be0 msgr2=0x7fe91c03b0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/618480980 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.018+0000 7fe935197700 1 --2- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7fe91c038be0 0x7fe91c03b0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.023+0000 7fe926ffd700 1 -- 172.21.15.172:0/3685184311 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7fe92c004450 con 0x7fe930079e90 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.219+0000 7fe935197700 1 -- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7fe91c038be0 msgr2=0x7fe91c03b0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/618480980 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.219+0000 7fe935197700 1 --2- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7fe91c038be0 0x7fe91c03b0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.271+0000 7fe937bfc700 1 -- 172.21.15.172:0/3685184311 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe93019e5f0 con 0x7fe930079e90 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.272+0000 7fe926ffd700 1 -- 172.21.15.172:0/3685184311 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fe92c019b70 con 0x7fe930079e90 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.274+0000 7fe937bfc700 1 -- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7fe91c038be0 msgr2=0x7fe91c03b0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.274+0000 7fe937bfc700 1 --2- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7fe91c038be0 0x7fe91c03b0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.274+0000 7fe937bfc700 1 -- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 msgr2=0x7fe93019afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.274+0000 7fe937bfc700 1 --2- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe93019afe0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe92c009870 tx=0x7fe92c011d10 comp rx=0 tx=0).stop 2023-11-28T01:07:19.757 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.274+0000 7fe937bfc700 1 -- 172.21.15.172:0/3685184311 shutdown_connections 2023-11-28T01:07:19.758 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.274+0000 7fe937bfc700 1 --2- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7fe91c038be0 0x7fe91c03b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:19.758 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.274+0000 7fe937bfc700 1 --2- 172.21.15.172:0/3685184311 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe930079e90 0x7fe93019afe0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:19.758 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.274+0000 7fe937bfc700 1 -- 172.21.15.172:0/3685184311 >> 172.21.15.172:0/3685184311 conn(0x7fe9301006f0 msgr2=0x7fe930101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:19.758 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.274+0000 7fe937bfc700 1 -- 172.21.15.172:0/3685184311 shutdown_connections 2023-11-28T01:07:19.758 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:19.274+0000 7fe937bfc700 1 -- 172.21.15.172:0/3685184311 wait complete. 2023-11-28T01:07:19.758 INFO:teuthology.orchestra.run.smithi172.stdout:Waiting for the mgr to restart... 2023-11-28T01:07:19.758 INFO:teuthology.orchestra.run.smithi172.stdout:Waiting for mgr epoch 5... 2023-11-28T01:07:19.954 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:19 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/3685184311' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-28T01:07:19.954 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:19 smithi172 conmon[101049]: 2023-11-28T01:07:19.666+0000 7fc263ce5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T01:07:20.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:20 smithi172 conmon[101049]: 2023-11-28T01:07:20.362+0000 7fc263ce5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T01:07:20.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:20 smithi172 conmon[101049]: 2023-11-28T01:07:20.444+0000 7fc263ce5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T01:07:21.475 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:21 smithi172 conmon[101049]: 2023-11-28T01:07:21.212+0000 7fc263ce5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T01:07:21.476 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:21 smithi172 conmon[101049]: 2023-11-28T01:07:21.302+0000 7fc263ce5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T01:07:21.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:21 smithi172 conmon[101049]: 2023-11-28T01:07:21.473+0000 7fc263ce5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T01:07:22.303 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:21 smithi172 conmon[101049]: 2023-11-28T01:07:21.887+0000 7fc263ce5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T01:07:22.303 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:21 smithi172 conmon[101049]: 2023-11-28T01:07:21.973+0000 7fc263ce5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T01:07:22.303 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:22 smithi172 conmon[101049]: 2023-11-28T01:07:22.120+0000 7fc263ce5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T01:07:22.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:22 smithi172 conmon[101049]: 2023-11-28T01:07:22.381+0000 7fc263ce5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T01:07:23.404 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:23 smithi172 conmon[101049]: 2023-11-28T01:07:23.076+0000 7fc263ce5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T01:07:23.404 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:23 smithi172 conmon[101049]: 2023-11-28T01:07:23.207+0000 7fc263ce5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T01:07:23.735 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:23 smithi172 conmon[101049]: 2023-11-28T01:07:23.401+0000 7fc263ce5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T01:07:23.735 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:23 smithi172 conmon[101049]: 2023-11-28T01:07:23.515+0000 7fc263ce5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T01:07:24.053 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:23 smithi172 conmon[101049]: 2023-11-28T01:07:23.733+0000 7fc263ce5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T01:07:25.053 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:24 smithi172 conmon[101049]: 2023-11-28T01:07:24.590+0000 7fc263ce5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T01:07:25.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: Active manager daemon y restarted 2023-11-28T01:07:25.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: Activating manager daemon y 2023-11-28T01:07:25.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: osdmap e2: 0 total, 0 up, 0 in 2023-11-28T01:07:25.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: mgrmap e6: y(active, starting, since 0.00341781s) 2023-11-28T01:07:25.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:07:25.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-28T01:07:25.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-28T01:07:25.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-28T01:07:25.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-28T01:07:25.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: Manager daemon y is now available 2023-11-28T01:07:25.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:25.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:25.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:07:25.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:24 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:07:26.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:25 smithi172 ceph-mon[100627]: Found migration_current of "None". Setting to last migration. 2023-11-28T01:07:26.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:25 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:07:26.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:25 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-11-28T01:07:26.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:25 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-11-28T01:07:26.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:25 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:26.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:25 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:26.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:25 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:07:26.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:25 smithi172 ceph-mon[100627]: mgrmap e7: y(active, since 1.00616s) 2023-11-28T01:07:26.146 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout { 2023-11-28T01:07:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-11-28T01:07:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "initialized": true 2023-11-28T01:07:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout } 2023-11-28T01:07:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.239+0000 7f649c8fe700 1 Processor -- start 2023-11-28T01:07:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.239+0000 7f649c8fe700 1 -- start start 2023-11-28T01:07:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.240+0000 7f649c8fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498103830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.240+0000 7f649c8fe700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6498103e00 con 0x7f6498103410 2023-11-28T01:07:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.240+0000 7f649659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498103830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.240+0000 7f649659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498103830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49194/0 (socket says 172.21.15.172:49194) 2023-11-28T01:07:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.240+0000 7f649659c700 1 -- 172.21.15.172:0/1273977708 learned_addr learned my addr 172.21.15.172:0/1273977708 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:26.153 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.241+0000 7f649659c700 1 -- 172.21.15.172:0/1273977708 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64981063a0 con 0x7f6498103410 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.241+0000 7f649659c700 1 --2- 172.21.15.172:0/1273977708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498103830 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f64880098a0 tx=0x7f6488009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81363e061b6d4fd1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.241+0000 7f649559a700 1 -- 172.21.15.172:0/1273977708 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6488004160 con 0x7f6498103410 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.241+0000 7f649559a700 1 -- 172.21.15.172:0/1273977708 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f64880036a0 con 0x7f6498103410 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.241+0000 7f649559a700 1 -- 172.21.15.172:0/1273977708 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64880117c0 con 0x7f6498103410 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.242+0000 7f649c8fe700 1 -- 172.21.15.172:0/1273977708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 msgr2=0x7f6498103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.242+0000 7f649c8fe700 1 --2- 172.21.15.172:0/1273977708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498103830 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f64880098a0 tx=0x7f6488009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.242+0000 7f649c8fe700 1 -- 172.21.15.172:0/1273977708 shutdown_connections 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.242+0000 7f649c8fe700 1 --2- 172.21.15.172:0/1273977708 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498103830 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.242+0000 7f649c8fe700 1 -- 172.21.15.172:0/1273977708 >> 172.21.15.172:0/1273977708 conn(0x7f64980fed00 msgr2=0x7f6498101180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.242+0000 7f649c8fe700 1 -- 172.21.15.172:0/1273977708 shutdown_connections 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.242+0000 7f649c8fe700 1 -- 172.21.15.172:0/1273977708 wait complete. 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.243+0000 7f649c8fe700 1 Processor -- start 2023-11-28T01:07:26.154 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.243+0000 7f649c8fe700 1 -- start start 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.243+0000 7f649c8fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.243+0000 7f649c8fe700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6498199350 con 0x7f6498103410 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.243+0000 7f649659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.243+0000 7f649659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49196/0 (socket says 172.21.15.172:49196) 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.243+0000 7f649659c700 1 -- 172.21.15.172:0/2172006446 learned_addr learned my addr 172.21.15.172:0/2172006446 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.243+0000 7f649659c700 1 -- 172.21.15.172:0/2172006446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6488009550 con 0x7f6498103410 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.243+0000 7f649659c700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498198e10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6488009fb0 tx=0x7f6488011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.244+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64880042b0 con 0x7f6498103410 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.244+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f6488004410 con 0x7f6498103410 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.244+0000 7f649c8fe700 1 -- 172.21.15.172:0/2172006446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6498199550 con 0x7f6498103410 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.244+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64880193f0 con 0x7f6498103410 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.244+0000 7f649c8fe700 1 -- 172.21.15.172:0/2172006446 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64981999f0 con 0x7f6498103410 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.245+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 5) v1 ==== 45719+0+0 (secure 0 0 0) 0x7f6488019550 con 0x7f6498103410 2023-11-28T01:07:26.155 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.245+0000 7f64877fe700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 0x7f648003b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.245+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 --> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f648003b760 con 0x7f6480038b90 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.245+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f648804cb90 con 0x7f6498103410 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.245+0000 7f6495d9b700 1 -- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 msgr2=0x7f648003b050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/618480980 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.245+0000 7f6495d9b700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 0x7f648003b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.446+0000 7f6495d9b700 1 -- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 msgr2=0x7f648003b050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/618480980 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.446+0000 7f6495d9b700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 0x7f648003b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.847+0000 7f6495d9b700 1 -- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 msgr2=0x7f648003b050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/618480980 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:20.847+0000 7f6495d9b700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 0x7f648003b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:21.648+0000 7f6495d9b700 1 -- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 msgr2=0x7f648003b050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/618480980 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:21.648+0000 7f6495d9b700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 0x7f648003b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:23.249+0000 7f6495d9b700 1 -- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 msgr2=0x7f648003b050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/618480980 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:23.249+0000 7f6495d9b700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 0x7f648003b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T01:07:26.156 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:24.605+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mgrmap(e 6) v1 ==== 45486+0+0 (secure 0 0 0) 0x7f64880198f0 con 0x7f6498103410 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:24.605+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 msgr2=0x7f648003b050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:24.605+0000 7f64877fe700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 0x7f648003b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.606+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mgrmap(e 7) v1 ==== 45613+0+0 (secure 0 0 0) 0x7f64880019f0 con 0x7f6498103410 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.606+0000 7f64877fe700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f648003c470 0x7f648003e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.606+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 --> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f648003b760 con 0x7f648003c470 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.607+0000 7f6495d9b700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f648003c470 0x7f648003e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.607+0000 7f6495d9b700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f648003c470 0x7f648003e860 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f648c003a10 tx=0x7f648c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.608+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 <== mgr.14120 v2:172.21.15.172:6800/2738031896 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f648003b760 con 0x7f648003c470 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 -- 172.21.15.172:0/2172006446 --> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f6498199b80 con 0x7f648003c470 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f64877fe700 1 -- 172.21.15.172:0/2172006446 <== mgr.14120 v2:172.21.15.172:6800/2738031896 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f648c00a050 con 0x7f648003c470 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 -- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f648003c470 msgr2=0x7f648003e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f648003c470 0x7f648003e860 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f648c003a10 tx=0x7f648c0092b0 comp rx=0 tx=0).stop 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 -- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 msgr2=0x7f6498198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498198e10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6488009fb0 tx=0x7f6488011d60 comp rx=0 tx=0).stop 2023-11-28T01:07:26.157 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 -- 172.21.15.172:0/2172006446 shutdown_connections 2023-11-28T01:07:26.158 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f648003c470 0x7f648003e860 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:26.158 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6498103410 0x7f6498198e10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:26.158 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 --2- 172.21.15.172:0/2172006446 >> [v2:172.21.15.172:6800/618480980,v1:172.21.15.172:6801/618480980] conn(0x7f6480038b90 0x7f648003b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:26.158 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 -- 172.21.15.172:0/2172006446 >> 172.21.15.172:0/2172006446 conn(0x7f64980fed00 msgr2=0x7f64980ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:26.158 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 -- 172.21.15.172:0/2172006446 shutdown_connections 2023-11-28T01:07:26.158 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:25.615+0000 7f649c8fe700 1 -- 172.21.15.172:0/2172006446 wait complete. 2023-11-28T01:07:26.158 INFO:teuthology.orchestra.run.smithi172.stdout:mgr epoch 5 is available 2023-11-28T01:07:26.158 INFO:teuthology.orchestra.run.smithi172.stdout:Setting orchestrator backend to cephadm... 2023-11-28T01:07:26.944 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:26 smithi172 ceph-mon[100627]: [28/Nov/2023:01:07:25] ENGINE Bus STARTING 2023-11-28T01:07:26.945 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:26 smithi172 ceph-mon[100627]: [28/Nov/2023:01:07:25] ENGINE Serving on http://172.21.15.172:8765 2023-11-28T01:07:26.945 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:26 smithi172 ceph-mon[100627]: [28/Nov/2023:01:07:25] ENGINE Serving on https://172.21.15.172:7150 2023-11-28T01:07:26.945 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:26 smithi172 ceph-mon[100627]: [28/Nov/2023:01:07:25] ENGINE Bus STARTED 2023-11-28T01:07:26.945 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:26 smithi172 ceph-mon[100627]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-28T01:07:26.945 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:26 smithi172 ceph-mon[100627]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-28T01:07:27.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f77f20700 1 Processor -- start 2023-11-28T01:07:27.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f77f20700 1 -- start start 2023-11-28T01:07:27.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f77f20700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:27.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f77f20700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f70078710 con 0x7f2f70105bf0 2023-11-28T01:07:27.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f75cbc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:27.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f75cbc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70107fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49248/0 (socket says 172.21.15.172:49248) 2023-11-28T01:07:27.293 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f75cbc700 1 -- 172.21.15.172:0/3359855494 learned_addr learned my addr 172.21.15.172:0/3359855494 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:27.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f75cbc700 1 -- 172.21.15.172:0/3359855494 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f70108520 con 0x7f2f70105bf0 2023-11-28T01:07:27.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f75cbc700 1 --2- 172.21.15.172:0/3359855494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70107fe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0098a0 tx=0x7f2f6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29166476d5e25b32 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:27.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f74cba700 1 -- 172.21.15.172:0/3359855494 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f6c004160 con 0x7f2f70105bf0 2023-11-28T01:07:27.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f74cba700 1 -- 172.21.15.172:0/3359855494 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f2f6c0036a0 con 0x7f2f70105bf0 2023-11-28T01:07:27.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f74cba700 1 -- 172.21.15.172:0/3359855494 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f6c0117c0 con 0x7f2f70105bf0 2023-11-28T01:07:27.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f77f20700 1 -- 172.21.15.172:0/3359855494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 msgr2=0x7f2f70107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:27.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f77f20700 1 --2- 172.21.15.172:0/3359855494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70107fe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c0098a0 tx=0x7f2f6c009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:27.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f77f20700 1 -- 172.21.15.172:0/3359855494 shutdown_connections 2023-11-28T01:07:27.294 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f77f20700 1 --2- 172.21.15.172:0/3359855494 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70107fe0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:27.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.595+0000 7f2f77f20700 1 -- 172.21.15.172:0/3359855494 >> 172.21.15.172:0/3359855494 conn(0x7f2f700ff5c0 msgr2=0x7f2f70101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:27.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.596+0000 7f2f77f20700 1 -- 172.21.15.172:0/3359855494 shutdown_connections 2023-11-28T01:07:27.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.596+0000 7f2f77f20700 1 -- 172.21.15.172:0/3359855494 wait complete. 2023-11-28T01:07:27.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.596+0000 7f2f77f20700 1 Processor -- start 2023-11-28T01:07:27.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.596+0000 7f2f77f20700 1 -- start start 2023-11-28T01:07:27.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.596+0000 7f2f77f20700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70105420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:27.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.596+0000 7f2f77f20700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f70105960 con 0x7f2f70105bf0 2023-11-28T01:07:27.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.596+0000 7f2f75cbc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:27.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.596+0000 7f2f75cbc700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49250/0 (socket says 172.21.15.172:49250) 2023-11-28T01:07:27.295 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.596+0000 7f2f75cbc700 1 -- 172.21.15.172:0/3353106498 learned_addr learned my addr 172.21.15.172:0/3353106498 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.597+0000 7f2f75cbc700 1 -- 172.21.15.172:0/3353106498 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c009550 con 0x7f2f70105bf0 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.597+0000 7f2f75cbc700 1 --2- 172.21.15.172:0/3353106498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70105420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c006b20 tx=0x7f2f6c011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.597+0000 7f2f66ffd700 1 -- 172.21.15.172:0/3353106498 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f6c004180 con 0x7f2f70105bf0 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.597+0000 7f2f66ffd700 1 -- 172.21.15.172:0/3353106498 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f2f6c0042e0 con 0x7f2f70105bf0 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.597+0000 7f2f77f20700 1 -- 172.21.15.172:0/3353106498 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f70103ba0 con 0x7f2f70105bf0 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.597+0000 7f2f66ffd700 1 -- 172.21.15.172:0/3353106498 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f6c01be80 con 0x7f2f70105bf0 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.597+0000 7f2f77f20700 1 -- 172.21.15.172:0/3353106498 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f70104040 con 0x7f2f70105bf0 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.598+0000 7f2f66ffd700 1 -- 172.21.15.172:0/3353106498 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 7) v1 ==== 45613+0+0 (secure 0 0 0) 0x7f2f6c02c3f0 con 0x7f2f70105bf0 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.598+0000 7f2f66ffd700 1 --2- 172.21.15.172:0/3353106498 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f2f5c038a70 0x7f2f5c03af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.598+0000 7f2f66ffd700 1 -- 172.21.15.172:0/3353106498 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2f6c04cc20 con 0x7f2f70105bf0 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.598+0000 7f2f77f20700 1 -- 172.21.15.172:0/3353106498 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f70062c00 con 0x7f2f70105bf0 2023-11-28T01:07:27.296 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.598+0000 7f2f754bb700 1 --2- 172.21.15.172:0/3353106498 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f2f5c038a70 0x7f2f5c03af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:27.297 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.598+0000 7f2f754bb700 1 --2- 172.21.15.172:0/3353106498 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f2f5c038a70 0x7f2f5c03af30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f60006fd0 tx=0x7f2f60006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:27.297 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.604+0000 7f2f66ffd700 1 -- 172.21.15.172:0/3353106498 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f2f6c018940 con 0x7f2f70105bf0 2023-11-28T01:07:27.297 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.657+0000 7f2f66ffd700 1 -- 172.21.15.172:0/3353106498 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mgrmap(e 8) v1 ==== 45719+0+0 (secure 0 0 0) 0x7f2f6c029d50 con 0x7f2f70105bf0 2023-11-28T01:07:27.297 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.807+0000 7f2f77f20700 1 -- 172.21.15.172:0/3353106498 --> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f70100850 con 0x7f2f5c038a70 2023-11-28T01:07:27.297 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.812+0000 7f2f66ffd700 1 -- 172.21.15.172:0/3353106498 <== mgr.14120 v2:172.21.15.172:6800/2738031896 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2f70100850 con 0x7f2f5c038a70 2023-11-28T01:07:27.297 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.815+0000 7f2f77f20700 1 -- 172.21.15.172:0/3353106498 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f2f5c038a70 msgr2=0x7f2f5c03af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:27.297 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.815+0000 7f2f77f20700 1 --2- 172.21.15.172:0/3353106498 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f2f5c038a70 0x7f2f5c03af30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f60006fd0 tx=0x7f2f60006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:27.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.815+0000 7f2f77f20700 1 -- 172.21.15.172:0/3353106498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 msgr2=0x7f2f70105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:27.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.815+0000 7f2f77f20700 1 --2- 172.21.15.172:0/3353106498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70105420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c006b20 tx=0x7f2f6c011d10 comp rx=0 tx=0).stop 2023-11-28T01:07:27.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.815+0000 7f2f77f20700 1 -- 172.21.15.172:0/3353106498 shutdown_connections 2023-11-28T01:07:27.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.815+0000 7f2f77f20700 1 --2- 172.21.15.172:0/3353106498 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f2f5c038a70 0x7f2f5c03af30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:27.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.815+0000 7f2f77f20700 1 --2- 172.21.15.172:0/3353106498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2f70105bf0 0x7f2f70105420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:27.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.815+0000 7f2f77f20700 1 -- 172.21.15.172:0/3353106498 >> 172.21.15.172:0/3353106498 conn(0x7f2f700ff5c0 msgr2=0x7f2f70100140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:27.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.815+0000 7f2f77f20700 1 -- 172.21.15.172:0/3353106498 shutdown_connections 2023-11-28T01:07:27.298 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:26.815+0000 7f2f77f20700 1 -- 172.21.15.172:0/3353106498 wait complete. 2023-11-28T01:07:27.966 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:27 smithi172 ceph-mon[100627]: mgrmap e8: y(active, since 2s) 2023-11-28T01:07:27.969 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:27 smithi172 ceph-mon[100627]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:07:27.969 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:27 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:27.969 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:27 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:07:28.425 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout value unchanged 2023-11-28T01:07:28.425 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.739+0000 7fdf8edf9700 1 Processor -- start 2023-11-28T01:07:28.425 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.739+0000 7fdf8edf9700 1 -- start start 2023-11-28T01:07:28.425 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.739+0000 7fdf8edf9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88105ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:28.425 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.739+0000 7fdf8edf9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf88106090 con 0x7fdf881056a0 2023-11-28T01:07:28.425 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.739+0000 7fdf8cb95700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88105ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:28.425 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.739+0000 7fdf8cb95700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88105ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49254/0 (socket says 172.21.15.172:49254) 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.739+0000 7fdf8cb95700 1 -- 172.21.15.172:0/2190985001 learned_addr learned my addr 172.21.15.172:0/2190985001 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.740+0000 7fdf8cb95700 1 -- 172.21.15.172:0/2190985001 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf88104270 con 0x7fdf881056a0 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.740+0000 7fdf8cb95700 1 --2- 172.21.15.172:0/2190985001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88105ac0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0098a0 tx=0x7fdf7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8f97177bd47b4e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.740+0000 7fdf877fe700 1 -- 172.21.15.172:0/2190985001 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf7c004160 con 0x7fdf881056a0 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.740+0000 7fdf877fe700 1 -- 172.21.15.172:0/2190985001 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7fdf7c0036a0 con 0x7fdf881056a0 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.740+0000 7fdf877fe700 1 -- 172.21.15.172:0/2190985001 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf7c0117c0 con 0x7fdf881056a0 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.741+0000 7fdf8edf9700 1 -- 172.21.15.172:0/2190985001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 msgr2=0x7fdf88105ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.741+0000 7fdf8edf9700 1 --2- 172.21.15.172:0/2190985001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88105ac0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c0098a0 tx=0x7fdf7c009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.741+0000 7fdf8edf9700 1 -- 172.21.15.172:0/2190985001 shutdown_connections 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.741+0000 7fdf8edf9700 1 --2- 172.21.15.172:0/2190985001 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88105ac0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.741+0000 7fdf8edf9700 1 -- 172.21.15.172:0/2190985001 >> 172.21.15.172:0/2190985001 conn(0x7fdf880fed00 msgr2=0x7fdf88101180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.741+0000 7fdf8edf9700 1 -- 172.21.15.172:0/2190985001 shutdown_connections 2023-11-28T01:07:28.426 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.741+0000 7fdf8edf9700 1 -- 172.21.15.172:0/2190985001 wait complete. 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.741+0000 7fdf8edf9700 1 Processor -- start 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.741+0000 7fdf8edf9700 1 -- start start 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.741+0000 7fdf8edf9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88198df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.742+0000 7fdf8edf9700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf88199330 con 0x7fdf881056a0 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.742+0000 7fdf8cb95700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88198df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.742+0000 7fdf8cb95700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88198df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49256/0 (socket says 172.21.15.172:49256) 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.742+0000 7fdf8cb95700 1 -- 172.21.15.172:0/153455455 learned_addr learned my addr 172.21.15.172:0/153455455 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.742+0000 7fdf8cb95700 1 -- 172.21.15.172:0/153455455 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c009550 con 0x7fdf881056a0 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.742+0000 7fdf8cb95700 1 --2- 172.21.15.172:0/153455455 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88198df0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c011cb0 tx=0x7fdf7c011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.742+0000 7fdf85ffb700 1 -- 172.21.15.172:0/153455455 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf7c0042b0 con 0x7fdf881056a0 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.742+0000 7fdf85ffb700 1 -- 172.21.15.172:0/153455455 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7fdf7c004410 con 0x7fdf881056a0 2023-11-28T01:07:28.427 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.742+0000 7fdf8edf9700 1 -- 172.21.15.172:0/153455455 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf88199530 con 0x7fdf881056a0 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.742+0000 7fdf85ffb700 1 -- 172.21.15.172:0/153455455 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf7c0193f0 con 0x7fdf881056a0 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.742+0000 7fdf8edf9700 1 -- 172.21.15.172:0/153455455 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf881999d0 con 0x7fdf881056a0 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.743+0000 7fdf85ffb700 1 -- 172.21.15.172:0/153455455 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 8) v1 ==== 45719+0+0 (secure 0 0 0) 0x7fdf7c01b440 con 0x7fdf881056a0 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.743+0000 7fdf8edf9700 1 -- 172.21.15.172:0/153455455 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf88062c00 con 0x7fdf881056a0 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.743+0000 7fdf85ffb700 1 --2- 172.21.15.172:0/153455455 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fdf78038b90 0x7fdf7803b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.743+0000 7fdf85ffb700 1 -- 172.21.15.172:0/153455455 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdf7c04cba0 con 0x7fdf881056a0 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.744+0000 7fdf87fff700 1 --2- 172.21.15.172:0/153455455 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fdf78038b90 0x7fdf7803b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.744+0000 7fdf87fff700 1 --2- 172.21.15.172:0/153455455 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fdf78038b90 0x7fdf7803b050 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdf74006fd0 tx=0x7fdf74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.748+0000 7fdf85ffb700 1 -- 172.21.15.172:0/153455455 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7fdf7c0229b0 con 0x7fdf881056a0 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.939+0000 7fdf8edf9700 1 -- 172.21.15.172:0/153455455 --> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf88100080 con 0x7fdf78038b90 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.940+0000 7fdf85ffb700 1 -- 172.21.15.172:0/153455455 <== mgr.14120 v2:172.21.15.172:6800/2738031896 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fdf88100080 con 0x7fdf78038b90 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.942+0000 7fdf8edf9700 1 -- 172.21.15.172:0/153455455 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fdf78038b90 msgr2=0x7fdf7803b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:28.428 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.942+0000 7fdf8edf9700 1 --2- 172.21.15.172:0/153455455 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fdf78038b90 0x7fdf7803b050 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdf74006fd0 tx=0x7fdf74006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:28.429 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.942+0000 7fdf8edf9700 1 -- 172.21.15.172:0/153455455 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 msgr2=0x7fdf88198df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:28.429 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.942+0000 7fdf8edf9700 1 --2- 172.21.15.172:0/153455455 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88198df0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c011cb0 tx=0x7fdf7c011d90 comp rx=0 tx=0).stop 2023-11-28T01:07:28.429 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.942+0000 7fdf8edf9700 1 -- 172.21.15.172:0/153455455 shutdown_connections 2023-11-28T01:07:28.429 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.942+0000 7fdf8edf9700 1 --2- 172.21.15.172:0/153455455 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fdf78038b90 0x7fdf7803b050 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:28.429 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.942+0000 7fdf8edf9700 1 --2- 172.21.15.172:0/153455455 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdf881056a0 0x7fdf88198df0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:28.429 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.942+0000 7fdf8edf9700 1 -- 172.21.15.172:0/153455455 >> 172.21.15.172:0/153455455 conn(0x7fdf880fed00 msgr2=0x7fdf880ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:28.429 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.942+0000 7fdf8edf9700 1 -- 172.21.15.172:0/153455455 shutdown_connections 2023-11-28T01:07:28.429 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:27.942+0000 7fdf8edf9700 1 -- 172.21.15.172:0/153455455 wait complete. 2023-11-28T01:07:28.429 INFO:teuthology.orchestra.run.smithi172.stdout:Generating ssh key... 2023-11-28T01:07:29.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:28 smithi172 ceph-mon[100627]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:07:30.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:29 smithi172 ceph-mon[100627]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:07:30.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:29 smithi172 ceph-mon[100627]: Generating ssh key... 2023-11-28T01:07:30.304 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: Generating public/private rsa key pair. 2023-11-28T01:07:30.304 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: Your identification has been saved in /tmp/tmputv63j14/key. 2023-11-28T01:07:30.304 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: Your public key has been saved in /tmp/tmputv63j14/key.pub. 2023-11-28T01:07:30.304 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: The key fingerprint is: 2023-11-28T01:07:30.305 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: SHA256:bclnPEIoHYPBvxOhTkZEjiOpXJeqTJuvzgmIHQvNp/I ceph-31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:07:30.305 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: The key's randomart image is: 2023-11-28T01:07:30.305 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: +---[RSA 3072]----+ 2023-11-28T01:07:30.305 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: | ++oo | 2023-11-28T01:07:30.305 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: | . o=..+ | 2023-11-28T01:07:30.305 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: | o.o+ooo.. | 2023-11-28T01:07:30.306 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: |.+..o.+.o+ o | 2023-11-28T01:07:30.306 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: |oo+..+ So* = | 2023-11-28T01:07:30.306 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: |=o+= . o. + . | 2023-11-28T01:07:30.306 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: |==+ . | 2023-11-28T01:07:30.306 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: |ooo | 2023-11-28T01:07:30.306 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: |.=E. | 2023-11-28T01:07:30.307 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:30 smithi172 conmon[101049]: +----[SHA256]-----+ 2023-11-28T01:07:30.525 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.847+0000 7fad5829d700 1 Processor -- start 2023-11-28T01:07:30.525 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.847+0000 7fad5829d700 1 -- start start 2023-11-28T01:07:30.525 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.847+0000 7fad5829d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad501044c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:30.526 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.847+0000 7fad5829d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad50104a90 con 0x7fad501040a0 2023-11-28T01:07:30.526 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.847+0000 7fad56039700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad501044c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:30.526 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.847+0000 7fad56039700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad501044c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49260/0 (socket says 172.21.15.172:49260) 2023-11-28T01:07:30.526 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.847+0000 7fad56039700 1 -- 172.21.15.172:0/1671478337 learned_addr learned my addr 172.21.15.172:0/1671478337 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:30.526 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.848+0000 7fad56039700 1 -- 172.21.15.172:0/1671478337 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad501052a0 con 0x7fad501040a0 2023-11-28T01:07:30.526 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.848+0000 7fad56039700 1 --2- 172.21.15.172:0/1671478337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad501044c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad400098a0 tx=0x7fad40009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7717e143a52eac10 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:30.526 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.848+0000 7fad55037700 1 -- 172.21.15.172:0/1671478337 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad40004160 con 0x7fad501040a0 2023-11-28T01:07:30.526 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.848+0000 7fad55037700 1 -- 172.21.15.172:0/1671478337 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7fad400036a0 con 0x7fad501040a0 2023-11-28T01:07:30.526 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.848+0000 7fad55037700 1 -- 172.21.15.172:0/1671478337 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad400117c0 con 0x7fad501040a0 2023-11-28T01:07:30.526 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.849+0000 7fad5829d700 1 -- 172.21.15.172:0/1671478337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 msgr2=0x7fad501044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:30.526 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.849+0000 7fad5829d700 1 --2- 172.21.15.172:0/1671478337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad501044c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fad400098a0 tx=0x7fad40009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.849+0000 7fad5829d700 1 -- 172.21.15.172:0/1671478337 shutdown_connections 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.849+0000 7fad5829d700 1 --2- 172.21.15.172:0/1671478337 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad501044c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.849+0000 7fad5829d700 1 -- 172.21.15.172:0/1671478337 >> 172.21.15.172:0/1671478337 conn(0x7fad500ff620 msgr2=0x7fad50101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.849+0000 7fad5829d700 1 -- 172.21.15.172:0/1671478337 shutdown_connections 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.849+0000 7fad5829d700 1 -- 172.21.15.172:0/1671478337 wait complete. 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.849+0000 7fad5829d700 1 Processor -- start 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.849+0000 7fad5829d700 1 -- start start 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.849+0000 7fad5829d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad50197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.849+0000 7fad5829d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad501981c0 con 0x7fad501040a0 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.850+0000 7fad56039700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad50197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.850+0000 7fad56039700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad50197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49262/0 (socket says 172.21.15.172:49262) 2023-11-28T01:07:30.527 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.850+0000 7fad56039700 1 -- 172.21.15.172:0/3420676709 learned_addr learned my addr 172.21.15.172:0/3420676709 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:30.530 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.850+0000 7fad56039700 1 -- 172.21.15.172:0/3420676709 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad40009550 con 0x7fad501040a0 2023-11-28T01:07:30.530 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.850+0000 7fad56039700 1 --2- 172.21.15.172:0/3420676709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad50197c80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad40009870 tx=0x7fad40011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:30.530 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.850+0000 7fad477fe700 1 -- 172.21.15.172:0/3420676709 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad400040a0 con 0x7fad501040a0 2023-11-28T01:07:30.530 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.850+0000 7fad477fe700 1 -- 172.21.15.172:0/3420676709 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7fad4001b440 con 0x7fad501040a0 2023-11-28T01:07:30.530 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.850+0000 7fad5829d700 1 -- 172.21.15.172:0/3420676709 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad501983c0 con 0x7fad501040a0 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.850+0000 7fad477fe700 1 -- 172.21.15.172:0/3420676709 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad400193d0 con 0x7fad501040a0 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.850+0000 7fad5829d700 1 -- 172.21.15.172:0/3420676709 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad501987a0 con 0x7fad501040a0 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.851+0000 7fad477fe700 1 -- 172.21.15.172:0/3420676709 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 8) v1 ==== 45719+0+0 (secure 0 0 0) 0x7fad400195c0 con 0x7fad501040a0 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.851+0000 7fad477fe700 1 --2- 172.21.15.172:0/3420676709 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fad3c038be0 0x7fad3c03b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.851+0000 7fad477fe700 1 -- 172.21.15.172:0/3420676709 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fad4004cb50 con 0x7fad501040a0 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.851+0000 7fad5829d700 1 -- 172.21.15.172:0/3420676709 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad50198bc0 con 0x7fad501040a0 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.851+0000 7fad55838700 1 --2- 172.21.15.172:0/3420676709 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fad3c038be0 0x7fad3c03b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.852+0000 7fad55838700 1 --2- 172.21.15.172:0/3420676709 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fad3c038be0 0x7fad3c03b0a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fad4c006fd0 tx=0x7fad4c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:28.856+0000 7fad477fe700 1 -- 172.21.15.172:0/3420676709 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7fad4001f080 con 0x7fad501040a0 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:29.045+0000 7fad5829d700 1 -- 172.21.15.172:0/3420676709 --> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fad50100900 con 0x7fad3c038be0 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.006+0000 7fad477fe700 1 -- 172.21.15.172:0/3420676709 <== mgr.14120 v2:172.21.15.172:6800/2738031896 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fad50100900 con 0x7fad3c038be0 2023-11-28T01:07:30.531 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.008+0000 7fad5829d700 1 -- 172.21.15.172:0/3420676709 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fad3c038be0 msgr2=0x7fad3c03b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:30.532 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.008+0000 7fad5829d700 1 --2- 172.21.15.172:0/3420676709 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fad3c038be0 0x7fad3c03b0a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fad4c006fd0 tx=0x7fad4c006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:30.532 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.008+0000 7fad5829d700 1 -- 172.21.15.172:0/3420676709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 msgr2=0x7fad50197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:30.532 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.008+0000 7fad5829d700 1 --2- 172.21.15.172:0/3420676709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad50197c80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad40009870 tx=0x7fad40011ae0 comp rx=0 tx=0).stop 2023-11-28T01:07:30.532 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.008+0000 7fad5829d700 1 -- 172.21.15.172:0/3420676709 shutdown_connections 2023-11-28T01:07:30.532 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.008+0000 7fad5829d700 1 --2- 172.21.15.172:0/3420676709 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fad3c038be0 0x7fad3c03b0a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:30.532 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.008+0000 7fad5829d700 1 --2- 172.21.15.172:0/3420676709 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fad501040a0 0x7fad50197c80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:30.532 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.008+0000 7fad5829d700 1 -- 172.21.15.172:0/3420676709 >> 172.21.15.172:0/3420676709 conn(0x7fad500ff620 msgr2=0x7fad501001f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:30.532 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.008+0000 7fad5829d700 1 -- 172.21.15.172:0/3420676709 shutdown_connections 2023-11-28T01:07:30.532 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.008+0000 7fad5829d700 1 -- 172.21.15.172:0/3420676709 wait complete. 2023-11-28T01:07:31.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:31 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:31.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:31 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:31.733 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1em39VpKJ+k4B3HCHf6AkpnGfMaeKjpQr3ijKMh8xqa2565AiVrnR0vUa6CXjJpwPS/EqODlNIaeJS8DX+8PADG3mgNFXZXD1uHvIZ15Wv9iL8M2ZgzxExwf00780dCQEkiTqBdqXAoXlkjXTsZk5a01+E/y3hAefwsiPNSixl+mc9VTOldRKOwf+83rEMmp4l5I+pZlMViajGbVwgX8zOneHJiclodaFKqW8KzB1mBDelDmVWawe5QXBkKjrxX7QA8/L05KkM00dhmEcgsIwDTupfBf66WrmrNDaZ/XQA4easCP6lAZ5lh5SSGlggSd8U4joehzu++8dmAgtErKuPsjiDCVGSYZBv29D8SuEyC7zdn90ISKAzXFL7lcVLTQeCgTbbsGelK3jc5N8zrBRQW6hs3n7MxUQsCK9mAa5AmXwTZaMgFVCRjPOsmb8o2D/aTx8WyDI8LefPy2rXWj4QMIClW1qUNLEJ/vXto+uZKSz3lz+0YBGAVO0E2hKy3M= ceph-31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:07:31.733 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.934+0000 7f902f5d8700 1 Processor -- start 2023-11-28T01:07:31.733 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.934+0000 7f902f5d8700 1 -- start start 2023-11-28T01:07:31.733 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.934+0000 7f902f5d8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9028103a00 0x7f9028103e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:31.733 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.934+0000 7f902f5d8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9028078750 con 0x7f9028103a00 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.935+0000 7f902d374700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9028103a00 0x7f9028103e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.935+0000 7f902d374700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9028103a00 0x7f9028103e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49266/0 (socket says 172.21.15.172:49266) 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.935+0000 7f902d374700 1 -- 172.21.15.172:0/1313403126 learned_addr learned my addr 172.21.15.172:0/1313403126 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.935+0000 7f902d374700 1 -- 172.21.15.172:0/1313403126 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9028104360 con 0x7f9028103a00 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.935+0000 7f902d374700 1 --2- 172.21.15.172:0/1313403126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9028103a00 0x7f9028103e20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f90180098a0 tx=0x7f9018009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e2ee187d6f78294 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.935+0000 7f901ffff700 1 -- 172.21.15.172:0/1313403126 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9018004160 con 0x7f9028103a00 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.935+0000 7f901ffff700 1 -- 172.21.15.172:0/1313403126 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f90180036a0 con 0x7f9028103a00 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.935+0000 7f901ffff700 1 -- 172.21.15.172:0/1313403126 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f90180117c0 con 0x7f9028103a00 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.936+0000 7f902f5d8700 1 -- 172.21.15.172:0/1313403126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9028103a00 msgr2=0x7f9028103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.936+0000 7f902f5d8700 1 --2- 172.21.15.172:0/1313403126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9028103a00 0x7f9028103e20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f90180098a0 tx=0x7f9018009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.936+0000 7f902f5d8700 1 -- 172.21.15.172:0/1313403126 shutdown_connections 2023-11-28T01:07:31.734 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.936+0000 7f902f5d8700 1 --2- 172.21.15.172:0/1313403126 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f9028103a00 0x7f9028103e20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:31.735 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.936+0000 7f902f5d8700 1 -- 172.21.15.172:0/1313403126 >> 172.21.15.172:0/1313403126 conn(0x7f90280ff620 msgr2=0x7f9028101a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:31.735 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.936+0000 7f902f5d8700 1 -- 172.21.15.172:0/1313403126 shutdown_connections 2023-11-28T01:07:31.735 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.936+0000 7f902f5d8700 1 -- 172.21.15.172:0/1313403126 wait complete. 2023-11-28T01:07:31.735 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.936+0000 7f902f5d8700 1 Processor -- start 2023-11-28T01:07:31.735 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.936+0000 7f902f5d8700 1 -- start start 2023-11-28T01:07:31.735 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f902f5d8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f902819b140 0x7f902819b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:31.735 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f902f5d8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f902819baa0 con 0x7f902819b140 2023-11-28T01:07:31.735 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f902d374700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f902819b140 0x7f902819b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:31.735 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f902d374700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f902819b140 0x7f902819b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49268/0 (socket says 172.21.15.172:49268) 2023-11-28T01:07:31.735 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f902d374700 1 -- 172.21.15.172:0/1464547728 learned_addr learned my addr 172.21.15.172:0/1464547728 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:31.735 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f902d374700 1 -- 172.21.15.172:0/1464547728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9018009550 con 0x7f902819b140 2023-11-28T01:07:31.736 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f902d374700 1 --2- 172.21.15.172:0/1464547728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f902819b140 0x7f902819b560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9018000c00 tx=0x7f90180045d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:31.736 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f901e7fc700 1 -- 172.21.15.172:0/1464547728 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9018011970 con 0x7f902819b140 2023-11-28T01:07:31.736 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f901e7fc700 1 -- 172.21.15.172:0/1464547728 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f901801b470 con 0x7f902819b140 2023-11-28T01:07:31.736 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f901e7fc700 1 -- 172.21.15.172:0/1464547728 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f901801c450 con 0x7f902819b140 2023-11-28T01:07:31.736 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f902f5d8700 1 -- 172.21.15.172:0/1464547728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902819bca0 con 0x7f902819b140 2023-11-28T01:07:31.736 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.937+0000 7f902f5d8700 1 -- 172.21.15.172:0/1464547728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f902819e7c0 con 0x7f902819b140 2023-11-28T01:07:31.736 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.938+0000 7f901e7fc700 1 -- 172.21.15.172:0/1464547728 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 8) v1 ==== 45719+0+0 (secure 0 0 0) 0x7f9018022070 con 0x7f902819b140 2023-11-28T01:07:31.736 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.938+0000 7f902f5d8700 1 -- 172.21.15.172:0/1464547728 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9028194ce0 con 0x7f902819b140 2023-11-28T01:07:31.736 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.938+0000 7f901e7fc700 1 --2- 172.21.15.172:0/1464547728 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f9014038be0 0x7f901403b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:31.736 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.938+0000 7f901e7fc700 1 -- 172.21.15.172:0/1464547728 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f901804cc60 con 0x7f902819b140 2023-11-28T01:07:31.736 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.938+0000 7f902cb73700 1 --2- 172.21.15.172:0/1464547728 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f9014038be0 0x7f901403b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:31.737 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.939+0000 7f902cb73700 1 --2- 172.21.15.172:0/1464547728 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f9014038be0 0x7f901403b0a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9024006fd0 tx=0x7f9024006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:31.737 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:30.943+0000 7f901e7fc700 1 -- 172.21.15.172:0/1464547728 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f901801a840 con 0x7f902819b140 2023-11-28T01:07:31.737 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.133+0000 7f902f5d8700 1 -- 172.21.15.172:0/1464547728 --> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f902802d3b0 con 0x7f9014038be0 2023-11-28T01:07:31.737 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.134+0000 7f901e7fc700 1 -- 172.21.15.172:0/1464547728 <== mgr.14120 v2:172.21.15.172:6800/2738031896 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f902802d3b0 con 0x7f9014038be0 2023-11-28T01:07:31.737 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.135+0000 7f902f5d8700 1 -- 172.21.15.172:0/1464547728 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f9014038be0 msgr2=0x7f901403b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:31.738 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.135+0000 7f902f5d8700 1 --2- 172.21.15.172:0/1464547728 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f9014038be0 0x7f901403b0a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9024006fd0 tx=0x7f9024006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:31.738 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.135+0000 7f902f5d8700 1 -- 172.21.15.172:0/1464547728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f902819b140 msgr2=0x7f902819b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:31.738 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.135+0000 7f902f5d8700 1 --2- 172.21.15.172:0/1464547728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f902819b140 0x7f902819b560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f9018000c00 tx=0x7f90180045d0 comp rx=0 tx=0).stop 2023-11-28T01:07:31.738 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.135+0000 7f902f5d8700 1 -- 172.21.15.172:0/1464547728 shutdown_connections 2023-11-28T01:07:31.738 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.135+0000 7f902f5d8700 1 --2- 172.21.15.172:0/1464547728 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f9014038be0 0x7f901403b0a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:31.738 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.135+0000 7f902f5d8700 1 --2- 172.21.15.172:0/1464547728 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f902819b140 0x7f902819b560 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:31.739 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.135+0000 7f902f5d8700 1 -- 172.21.15.172:0/1464547728 >> 172.21.15.172:0/1464547728 conn(0x7f90280ff620 msgr2=0x7f9028191e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:31.739 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.136+0000 7f902f5d8700 1 -- 172.21.15.172:0/1464547728 shutdown_connections 2023-11-28T01:07:31.739 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:31.136+0000 7f902f5d8700 1 -- 172.21.15.172:0/1464547728 wait complete. 2023-11-28T01:07:31.739 INFO:teuthology.orchestra.run.smithi172.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-28T01:07:31.739 INFO:teuthology.orchestra.run.smithi172.stdout:Adding key to root@localhost authorized_keys... 2023-11-28T01:07:31.739 INFO:teuthology.orchestra.run.smithi172.stdout:Adding host smithi172... 2023-11-28T01:07:32.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:32 smithi172 ceph-mon[100627]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:07:34.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:33 smithi172 ceph-mon[100627]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi172", "addr": "172.21.15.172", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:07:34.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:33 smithi172 ceph-mon[100627]: Deploying cephadm binary to smithi172 2023-11-28T01:07:36.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout Added host 'smithi172' with addr '172.21.15.172' 2023-11-28T01:07:36.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.165+0000 7ffa44a3a700 1 Processor -- start 2023-11-28T01:07:36.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.165+0000 7ffa44a3a700 1 -- start start 2023-11-28T01:07:36.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.165+0000 7ffa44a3a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa40103e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:36.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.165+0000 7ffa44a3a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa40077e40 con 0x7ffa40103a50 2023-11-28T01:07:36.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.165+0000 7ffa3e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa40103e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:36.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.166+0000 7ffa3e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa40103e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49272/0 (socket says 172.21.15.172:49272) 2023-11-28T01:07:36.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.166+0000 7ffa3e59c700 1 -- 172.21.15.172:0/1994316716 learned_addr learned my addr 172.21.15.172:0/1994316716 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.166+0000 7ffa3e59c700 1 -- 172.21.15.172:0/1994316716 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa401043b0 con 0x7ffa40103a50 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.166+0000 7ffa3e59c700 1 --2- 172.21.15.172:0/1994316716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa40103e70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffa300098a0 tx=0x7ffa30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c276b6d2f5f2938 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.166+0000 7ffa3d59a700 1 -- 172.21.15.172:0/1994316716 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa30004160 con 0x7ffa40103a50 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.166+0000 7ffa3d59a700 1 -- 172.21.15.172:0/1994316716 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7ffa300036a0 con 0x7ffa40103a50 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.166+0000 7ffa3d59a700 1 -- 172.21.15.172:0/1994316716 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa300117c0 con 0x7ffa40103a50 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.167+0000 7ffa44a3a700 1 -- 172.21.15.172:0/1994316716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 msgr2=0x7ffa40103e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.167+0000 7ffa44a3a700 1 --2- 172.21.15.172:0/1994316716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa40103e70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffa300098a0 tx=0x7ffa30009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.167+0000 7ffa44a3a700 1 -- 172.21.15.172:0/1994316716 shutdown_connections 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.167+0000 7ffa44a3a700 1 --2- 172.21.15.172:0/1994316716 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa40103e70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.167+0000 7ffa44a3a700 1 -- 172.21.15.172:0/1994316716 >> 172.21.15.172:0/1994316716 conn(0x7ffa400ff5f0 msgr2=0x7ffa40101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.167+0000 7ffa44a3a700 1 -- 172.21.15.172:0/1994316716 shutdown_connections 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.167+0000 7ffa44a3a700 1 -- 172.21.15.172:0/1994316716 wait complete. 2023-11-28T01:07:36.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.167+0000 7ffa44a3a700 1 Processor -- start 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa44a3a700 1 -- start start 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa44a3a700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa4019b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa44a3a700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4019b540 con 0x7ffa40103a50 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa3e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa4019b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa3e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa4019b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49274/0 (socket says 172.21.15.172:49274) 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa3e59c700 1 -- 172.21.15.172:0/372902094 learned_addr learned my addr 172.21.15.172:0/372902094 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa3e59c700 1 -- 172.21.15.172:0/372902094 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa30009550 con 0x7ffa40103a50 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa3e59c700 1 --2- 172.21.15.172:0/372902094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa4019b000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009870 tx=0x7ffa30011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa2f7fe700 1 -- 172.21.15.172:0/372902094 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa30004180 con 0x7ffa40103a50 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa2f7fe700 1 -- 172.21.15.172:0/372902094 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7ffa300042e0 con 0x7ffa40103a50 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa44a3a700 1 -- 172.21.15.172:0/372902094 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4019b740 con 0x7ffa40103a50 2023-11-28T01:07:36.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa2f7fe700 1 -- 172.21.15.172:0/372902094 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ffa3001bea0 con 0x7ffa40103a50 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.168+0000 7ffa44a3a700 1 -- 172.21.15.172:0/372902094 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4019bbe0 con 0x7ffa40103a50 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.169+0000 7ffa2f7fe700 1 -- 172.21.15.172:0/372902094 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 8) v1 ==== 45719+0+0 (secure 0 0 0) 0x7ffa3001b440 con 0x7ffa40103a50 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.169+0000 7ffa44a3a700 1 -- 172.21.15.172:0/372902094 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa40194d30 con 0x7ffa40103a50 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.169+0000 7ffa2f7fe700 1 --2- 172.21.15.172:0/372902094 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7ffa28038be0 0x7ffa2803b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.169+0000 7ffa2f7fe700 1 -- 172.21.15.172:0/372902094 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ffa3004cbc0 con 0x7ffa40103a50 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.170+0000 7ffa3dd9b700 1 --2- 172.21.15.172:0/372902094 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7ffa28038be0 0x7ffa2803b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.170+0000 7ffa3dd9b700 1 --2- 172.21.15.172:0/372902094 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7ffa28038be0 0x7ffa2803b0a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa34006fd0 tx=0x7ffa34006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.174+0000 7ffa2f7fe700 1 -- 172.21.15.172:0/372902094 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7ffa30018350 con 0x7ffa40103a50 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:32.367+0000 7ffa44a3a700 1 -- 172.21.15.172:0/372902094 --> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi172", "addr": "172.21.15.172", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa4002d3b0 con 0x7ffa28038be0 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:36.062+0000 7ffa2f7fe700 1 -- 172.21.15.172:0/372902094 <== mgr.14120 v2:172.21.15.172:6800/2738031896 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ffa4002d3b0 con 0x7ffa28038be0 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:36.065+0000 7ffa44a3a700 1 -- 172.21.15.172:0/372902094 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7ffa28038be0 msgr2=0x7ffa2803b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:36.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:36.065+0000 7ffa44a3a700 1 --2- 172.21.15.172:0/372902094 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7ffa28038be0 0x7ffa2803b0a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ffa34006fd0 tx=0x7ffa34006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:36.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:36.065+0000 7ffa44a3a700 1 -- 172.21.15.172:0/372902094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 msgr2=0x7ffa4019b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:36.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:36.065+0000 7ffa44a3a700 1 --2- 172.21.15.172:0/372902094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa4019b000 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ffa30009870 tx=0x7ffa30011d10 comp rx=0 tx=0).stop 2023-11-28T01:07:36.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:36.065+0000 7ffa44a3a700 1 -- 172.21.15.172:0/372902094 shutdown_connections 2023-11-28T01:07:36.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:36.065+0000 7ffa44a3a700 1 --2- 172.21.15.172:0/372902094 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7ffa28038be0 0x7ffa2803b0a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:36.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:36.065+0000 7ffa44a3a700 1 --2- 172.21.15.172:0/372902094 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ffa40103a50 0x7ffa4019b000 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:36.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:36.065+0000 7ffa44a3a700 1 -- 172.21.15.172:0/372902094 >> 172.21.15.172:0/372902094 conn(0x7ffa400ff5f0 msgr2=0x7ffa40191e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:36.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:36.065+0000 7ffa44a3a700 1 -- 172.21.15.172:0/372902094 shutdown_connections 2023-11-28T01:07:36.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:36.065+0000 7ffa44a3a700 1 -- 172.21.15.172:0/372902094 wait complete. 2023-11-28T01:07:36.628 INFO:teuthology.orchestra.run.smithi172.stdout:Deploying unmanaged mon service... 2023-11-28T01:07:37.330 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:37 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:37.330 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:37 smithi172 ceph-mon[100627]: Added host smithi172 2023-11-28T01:07:37.330 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:37 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:07:38.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.093+0000 7f6057112700 1 Processor -- start 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.093+0000 7f6057112700 1 -- start start 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.094+0000 7f6057112700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f6050104460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.094+0000 7f6057112700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6050104a30 con 0x7f6050104040 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.094+0000 7f6054eae700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f6050104460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.094+0000 7f6054eae700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f6050104460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49308/0 (socket says 172.21.15.172:49308) 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.094+0000 7f6054eae700 1 -- 172.21.15.172:0/2071338265 learned_addr learned my addr 172.21.15.172:0/2071338265 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.094+0000 7f6054eae700 1 -- 172.21.15.172:0/2071338265 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6050105240 con 0x7f6050104040 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.094+0000 7f6054eae700 1 --2- 172.21.15.172:0/2071338265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f6050104460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f60440098a0 tx=0x7f6044009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919db5fd4bb9b4b0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.094+0000 7f604f7fe700 1 -- 172.21.15.172:0/2071338265 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6044004160 con 0x7f6050104040 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.094+0000 7f604f7fe700 1 -- 172.21.15.172:0/2071338265 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f60440036a0 con 0x7f6050104040 2023-11-28T01:07:38.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.094+0000 7f604f7fe700 1 -- 172.21.15.172:0/2071338265 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f60440117c0 con 0x7f6050104040 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.095+0000 7f6057112700 1 -- 172.21.15.172:0/2071338265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 msgr2=0x7f6050104460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.095+0000 7f6057112700 1 --2- 172.21.15.172:0/2071338265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f6050104460 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f60440098a0 tx=0x7f6044009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.095+0000 7f6057112700 1 -- 172.21.15.172:0/2071338265 shutdown_connections 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.095+0000 7f6057112700 1 --2- 172.21.15.172:0/2071338265 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f6050104460 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.095+0000 7f6057112700 1 -- 172.21.15.172:0/2071338265 >> 172.21.15.172:0/2071338265 conn(0x7f60500ff5c0 msgr2=0x7f6050101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.095+0000 7f6057112700 1 -- 172.21.15.172:0/2071338265 shutdown_connections 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.095+0000 7f6057112700 1 -- 172.21.15.172:0/2071338265 wait complete. 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.096+0000 7f6057112700 1 Processor -- start 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.096+0000 7f6057112700 1 -- start start 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.096+0000 7f6057112700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f60500782a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.096+0000 7f6057112700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60500787e0 con 0x7f6050104040 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.097+0000 7f6054eae700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f60500782a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:38.275 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.097+0000 7f6054eae700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f60500782a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49310/0 (socket says 172.21.15.172:49310) 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.097+0000 7f6054eae700 1 -- 172.21.15.172:0/3840919620 learned_addr learned my addr 172.21.15.172:0/3840919620 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.097+0000 7f6054eae700 1 -- 172.21.15.172:0/3840919620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6044009550 con 0x7f6050104040 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.097+0000 7f6054eae700 1 --2- 172.21.15.172:0/3840919620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f60500782a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6044009870 tx=0x7f6044011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.097+0000 7f604dffb700 1 -- 172.21.15.172:0/3840919620 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f60440040a0 con 0x7f6050104040 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.097+0000 7f604dffb700 1 -- 172.21.15.172:0/3840919620 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 976+0+0 (secure 0 0 0) 0x7f604401b440 con 0x7f6050104040 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.097+0000 7f604dffb700 1 -- 172.21.15.172:0/3840919620 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f60440193d0 con 0x7f6050104040 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.097+0000 7f6057112700 1 -- 172.21.15.172:0/3840919620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605019bb00 con 0x7f6050104040 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.097+0000 7f6057112700 1 -- 172.21.15.172:0/3840919620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605019bf40 con 0x7f6050104040 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.098+0000 7f604dffb700 1 -- 172.21.15.172:0/3840919620 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 8) v1 ==== 45719+0+0 (secure 0 0 0) 0x7f604401b5b0 con 0x7f6050104040 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.098+0000 7f604dffb700 1 --2- 172.21.15.172:0/3840919620 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f6040038be0 0x7f604003b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.098+0000 7f604dffb700 1 -- 172.21.15.172:0/3840919620 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f604404cb30 con 0x7f6050104040 2023-11-28T01:07:38.276 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.099+0000 7f604ffff700 1 --2- 172.21.15.172:0/3840919620 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f6040038be0 0x7f604003b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.099+0000 7f604ffff700 1 --2- 172.21.15.172:0/3840919620 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f6040038be0 0x7f604003b0a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f603c006fd0 tx=0x7f603c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.100+0000 7f6057112700 1 -- 172.21.15.172:0/3840919620 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6050105f90 con 0x7f6050104040 2023-11-28T01:07:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.106+0000 7f604dffb700 1 -- 172.21.15.172:0/3840919620 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f604401f080 con 0x7f6050104040 2023-11-28T01:07:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.329+0000 7f6057112700 1 -- 172.21.15.172:0/3840919620 --> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f60501008a0 con 0x7f6040038be0 2023-11-28T01:07:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.338+0000 7f604dffb700 1 -- 172.21.15.172:0/3840919620 <== mgr.14120 v2:172.21.15.172:6800/2738031896 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f60501008a0 con 0x7f6040038be0 2023-11-28T01:07:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.340+0000 7f6057112700 1 -- 172.21.15.172:0/3840919620 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f6040038be0 msgr2=0x7f604003b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.340+0000 7f6057112700 1 --2- 172.21.15.172:0/3840919620 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f6040038be0 0x7f604003b0a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f603c006fd0 tx=0x7f603c006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.340+0000 7f6057112700 1 -- 172.21.15.172:0/3840919620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 msgr2=0x7f60500782a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.340+0000 7f6057112700 1 --2- 172.21.15.172:0/3840919620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f60500782a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6044009870 tx=0x7f6044011ae0 comp rx=0 tx=0).stop 2023-11-28T01:07:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.340+0000 7f6057112700 1 -- 172.21.15.172:0/3840919620 shutdown_connections 2023-11-28T01:07:38.277 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.340+0000 7f6057112700 1 --2- 172.21.15.172:0/3840919620 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f6040038be0 0x7f604003b0a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:38.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.340+0000 7f6057112700 1 --2- 172.21.15.172:0/3840919620 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6050104040 0x7f60500782a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:38.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.340+0000 7f6057112700 1 -- 172.21.15.172:0/3840919620 >> 172.21.15.172:0/3840919620 conn(0x7f60500ff5c0 msgr2=0x7f6050100190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:38.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.340+0000 7f6057112700 1 -- 172.21.15.172:0/3840919620 shutdown_connections 2023-11-28T01:07:38.278 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:37.340+0000 7f6057112700 1 -- 172.21.15.172:0/3840919620 wait complete. 2023-11-28T01:07:38.278 INFO:teuthology.orchestra.run.smithi172.stdout:Deploying unmanaged mgr service... 2023-11-28T01:07:38.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:38 smithi172 ceph-mon[100627]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:07:38.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:38 smithi172 ceph-mon[100627]: Saving service mon spec with placement count:5 2023-11-28T01:07:38.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:38 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:38.525 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:38 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:39.626 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.743+0000 7f0b59fa7700 1 Processor -- start 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.743+0000 7f0b59fa7700 1 -- start start 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.743+0000 7f0b59fa7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54105550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.743+0000 7f0b59fa7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b54105b20 con 0x7f0b54105130 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.743+0000 7f0b537fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54105550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.743+0000 7f0b537fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54105550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49314/0 (socket says 172.21.15.172:49314) 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.743+0000 7f0b537fe700 1 -- 172.21.15.172:0/4222989501 learned_addr learned my addr 172.21.15.172:0/4222989501 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.743+0000 7f0b537fe700 1 -- 172.21.15.172:0/4222989501 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b54106330 con 0x7f0b54105130 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.744+0000 7f0b537fe700 1 --2- 172.21.15.172:0/4222989501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54105550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0b440098a0 tx=0x7f0b44009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9c1ff3677af74fc server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.744+0000 7f0b527fc700 1 -- 172.21.15.172:0/4222989501 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b44004160 con 0x7f0b54105130 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.744+0000 7f0b527fc700 1 -- 172.21.15.172:0/4222989501 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f0b440036a0 con 0x7f0b54105130 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.744+0000 7f0b527fc700 1 -- 172.21.15.172:0/4222989501 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b440117a0 con 0x7f0b54105130 2023-11-28T01:07:39.627 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.744+0000 7f0b59fa7700 1 -- 172.21.15.172:0/4222989501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 msgr2=0x7f0b54105550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.744+0000 7f0b59fa7700 1 --2- 172.21.15.172:0/4222989501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54105550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0b440098a0 tx=0x7f0b44009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.745+0000 7f0b59fa7700 1 -- 172.21.15.172:0/4222989501 shutdown_connections 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.745+0000 7f0b59fa7700 1 --2- 172.21.15.172:0/4222989501 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54105550 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.745+0000 7f0b59fa7700 1 -- 172.21.15.172:0/4222989501 >> 172.21.15.172:0/4222989501 conn(0x7f0b541006b0 msgr2=0x7f0b54102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.745+0000 7f0b59fa7700 1 -- 172.21.15.172:0/4222989501 shutdown_connections 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.745+0000 7f0b59fa7700 1 -- 172.21.15.172:0/4222989501 wait complete. 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.745+0000 7f0b59fa7700 1 Processor -- start 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.745+0000 7f0b59fa7700 1 -- start start 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b59fa7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b59fa7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b54199340 con 0x7f0b54105130 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b537fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b537fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49316/0 (socket says 172.21.15.172:49316) 2023-11-28T01:07:39.628 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b537fe700 1 -- 172.21.15.172:0/992317252 learned_addr learned my addr 172.21.15.172:0/992317252 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b537fe700 1 -- 172.21.15.172:0/992317252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b44009550 con 0x7f0b54105130 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b537fe700 1 --2- 172.21.15.172:0/992317252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54198e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b44000c00 tx=0x7f0b440119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b50ff9700 1 -- 172.21.15.172:0/992317252 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b440041b0 con 0x7f0b54105130 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b50ff9700 1 -- 172.21.15.172:0/992317252 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f0b4401b440 con 0x7f0b54105130 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b59fa7700 1 -- 172.21.15.172:0/992317252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b54199540 con 0x7f0b54105130 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b50ff9700 1 -- 172.21.15.172:0/992317252 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b440193f0 con 0x7f0b54105130 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.746+0000 7f0b59fa7700 1 -- 172.21.15.172:0/992317252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b541999e0 con 0x7f0b54105130 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.747+0000 7f0b50ff9700 1 -- 172.21.15.172:0/992317252 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 8) v1 ==== 45719+0+0 (secure 0 0 0) 0x7f0b44019670 con 0x7f0b54105130 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.747+0000 7f0b59fa7700 1 -- 172.21.15.172:0/992317252 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b54062c00 con 0x7f0b54105130 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.747+0000 7f0b50ff9700 1 --2- 172.21.15.172:0/992317252 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f0b3c038b40 0x7f0b3c03b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.747+0000 7f0b50ff9700 1 -- 172.21.15.172:0/992317252 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0b4404ca90 con 0x7f0b54105130 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.748+0000 7f0b52ffd700 1 --2- 172.21.15.172:0/992317252 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f0b3c038b40 0x7f0b3c03b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:39.629 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.748+0000 7f0b52ffd700 1 --2- 172.21.15.172:0/992317252 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f0b3c038b40 0x7f0b3c03b000 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0b48006fd0 tx=0x7f0b48006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.752+0000 7f0b50ff9700 1 -- 172.21.15.172:0/992317252 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f0b44022720 con 0x7f0b54105130 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.972+0000 7f0b59fa7700 1 -- 172.21.15.172:0/992317252 --> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f0b54109530 con 0x7f0b3c038b40 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.980+0000 7f0b50ff9700 1 -- 172.21.15.172:0/992317252 <== mgr.14120 v2:172.21.15.172:6800/2738031896 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0b54109530 con 0x7f0b3c038b40 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.982+0000 7f0b59fa7700 1 -- 172.21.15.172:0/992317252 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f0b3c038b40 msgr2=0x7f0b3c03b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.982+0000 7f0b59fa7700 1 --2- 172.21.15.172:0/992317252 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f0b3c038b40 0x7f0b3c03b000 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0b48006fd0 tx=0x7f0b48006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.982+0000 7f0b59fa7700 1 -- 172.21.15.172:0/992317252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 msgr2=0x7f0b54198e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.982+0000 7f0b59fa7700 1 --2- 172.21.15.172:0/992317252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54198e00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b44000c00 tx=0x7f0b440119a0 comp rx=0 tx=0).stop 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.982+0000 7f0b59fa7700 1 -- 172.21.15.172:0/992317252 shutdown_connections 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.982+0000 7f0b59fa7700 1 --2- 172.21.15.172:0/992317252 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f0b3c038b40 0x7f0b3c03b000 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.982+0000 7f0b59fa7700 1 --2- 172.21.15.172:0/992317252 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0b54105130 0x7f0b54198e00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.983+0000 7f0b59fa7700 1 -- 172.21.15.172:0/992317252 >> 172.21.15.172:0/992317252 conn(0x7f0b541006b0 msgr2=0x7f0b541012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.983+0000 7f0b59fa7700 1 -- 172.21.15.172:0/992317252 shutdown_connections 2023-11-28T01:07:39.630 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:38.983+0000 7f0b59fa7700 1 -- 172.21.15.172:0/992317252 wait complete. 2023-11-28T01:07:40.009 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:39 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:40.009 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:39 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:41.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:40 smithi172 ceph-mon[100627]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:07:41.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:40 smithi172 ceph-mon[100627]: Saving service mgr spec with placement count:2 2023-11-28T01:07:41.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:40 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:41.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:40 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/3285822837' entity='client.admin' 2023-11-28T01:07:41.081 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.322+0000 7f599a1c8700 1 Processor -- start 2023-11-28T01:07:41.082 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.322+0000 7f599a1c8700 1 -- start start 2023-11-28T01:07:41.082 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.322+0000 7f599a1c8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994103830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:41.082 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.322+0000 7f599a1c8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5994103e00 con 0x7f5994103410 2023-11-28T01:07:41.082 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.323+0000 7f59937fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994103830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:41.082 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.323+0000 7f59937fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994103830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49320/0 (socket says 172.21.15.172:49320) 2023-11-28T01:07:41.082 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.323+0000 7f59937fe700 1 -- 172.21.15.172:0/2244916555 learned_addr learned my addr 172.21.15.172:0/2244916555 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:41.082 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.323+0000 7f59937fe700 1 -- 172.21.15.172:0/2244916555 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59941063a0 con 0x7f5994103410 2023-11-28T01:07:41.082 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.323+0000 7f59937fe700 1 --2- 172.21.15.172:0/2244916555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994103830 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f59840098a0 tx=0x7f5984009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11753e820b0bbe7 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:41.083 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.323+0000 7f59927fc700 1 -- 172.21.15.172:0/2244916555 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5984004160 con 0x7f5994103410 2023-11-28T01:07:41.083 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.323+0000 7f59927fc700 1 -- 172.21.15.172:0/2244916555 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f59840036a0 con 0x7f5994103410 2023-11-28T01:07:41.083 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.323+0000 7f59927fc700 1 -- 172.21.15.172:0/2244916555 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59840117a0 con 0x7f5994103410 2023-11-28T01:07:41.083 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.324+0000 7f599a1c8700 1 -- 172.21.15.172:0/2244916555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 msgr2=0x7f5994103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:41.085 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.324+0000 7f599a1c8700 1 --2- 172.21.15.172:0/2244916555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994103830 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f59840098a0 tx=0x7f5984009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:41.086 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.324+0000 7f599a1c8700 1 -- 172.21.15.172:0/2244916555 shutdown_connections 2023-11-28T01:07:41.086 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.324+0000 7f599a1c8700 1 --2- 172.21.15.172:0/2244916555 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994103830 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:41.086 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.324+0000 7f599a1c8700 1 -- 172.21.15.172:0/2244916555 >> 172.21.15.172:0/2244916555 conn(0x7f59940fed00 msgr2=0x7f5994101180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:41.087 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.324+0000 7f599a1c8700 1 -- 172.21.15.172:0/2244916555 shutdown_connections 2023-11-28T01:07:41.087 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.324+0000 7f599a1c8700 1 -- 172.21.15.172:0/2244916555 wait complete. 2023-11-28T01:07:41.087 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.325+0000 7f599a1c8700 1 Processor -- start 2023-11-28T01:07:41.088 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.325+0000 7f599a1c8700 1 -- start start 2023-11-28T01:07:41.088 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.325+0000 7f599a1c8700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:41.088 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.325+0000 7f599a1c8700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5994199350 con 0x7f5994103410 2023-11-28T01:07:41.088 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.325+0000 7f59937fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:41.088 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.325+0000 7f59937fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49322/0 (socket says 172.21.15.172:49322) 2023-11-28T01:07:41.088 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.325+0000 7f59937fe700 1 -- 172.21.15.172:0/3285822837 learned_addr learned my addr 172.21.15.172:0/3285822837 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:41.088 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.326+0000 7f59937fe700 1 -- 172.21.15.172:0/3285822837 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5984009550 con 0x7f5994103410 2023-11-28T01:07:41.088 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.326+0000 7f59937fe700 1 --2- 172.21.15.172:0/3285822837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994198e10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5984000c00 tx=0x7f59840119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:41.088 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.326+0000 7f5990ff9700 1 -- 172.21.15.172:0/3285822837 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59840041b0 con 0x7f5994103410 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.326+0000 7f599a1c8700 1 -- 172.21.15.172:0/3285822837 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5994199550 con 0x7f5994103410 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.326+0000 7f5990ff9700 1 -- 172.21.15.172:0/3285822837 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f5984004310 con 0x7f5994103410 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.326+0000 7f5990ff9700 1 -- 172.21.15.172:0/3285822837 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f598401be80 con 0x7f5994103410 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.326+0000 7f599a1c8700 1 -- 172.21.15.172:0/3285822837 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5994199970 con 0x7f5994103410 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.327+0000 7f5990ff9700 1 -- 172.21.15.172:0/3285822837 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 8) v1 ==== 45719+0+0 (secure 0 0 0) 0x7f59840194f0 con 0x7f5994103410 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.327+0000 7f5990ff9700 1 --2- 172.21.15.172:0/3285822837 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f597c038b40 0x7f597c03b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.327+0000 7f5990ff9700 1 -- 172.21.15.172:0/3285822837 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f598404cf70 con 0x7f5994103410 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.327+0000 7f5992ffd700 1 --2- 172.21.15.172:0/3285822837 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f597c038b40 0x7f597c03b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.327+0000 7f599a1c8700 1 -- 172.21.15.172:0/3285822837 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5994062c00 con 0x7f5994103410 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.328+0000 7f5992ffd700 1 --2- 172.21.15.172:0/3285822837 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f597c038b40 0x7f597c03b000 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5988006fd0 tx=0x7f5988006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.335+0000 7f5990ff9700 1 -- 172.21.15.172:0/3285822837 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f598401a3d0 con 0x7f5994103410 2023-11-28T01:07:41.089 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.539+0000 7f599a1c8700 1 -- 172.21.15.172:0/3285822837 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f5994104190 con 0x7f5994103410 2023-11-28T01:07:41.090 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.541+0000 7f5990ff9700 1 -- 172.21.15.172:0/3285822837 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5984018350 con 0x7f5994103410 2023-11-28T01:07:41.090 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.544+0000 7f599a1c8700 1 -- 172.21.15.172:0/3285822837 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f597c038b40 msgr2=0x7f597c03b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:41.090 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.544+0000 7f599a1c8700 1 --2- 172.21.15.172:0/3285822837 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f597c038b40 0x7f597c03b000 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5988006fd0 tx=0x7f5988006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:41.090 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.544+0000 7f599a1c8700 1 -- 172.21.15.172:0/3285822837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 msgr2=0x7f5994198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:41.090 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.544+0000 7f599a1c8700 1 --2- 172.21.15.172:0/3285822837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994198e10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5984000c00 tx=0x7f59840119a0 comp rx=0 tx=0).stop 2023-11-28T01:07:41.090 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.544+0000 7f599a1c8700 1 -- 172.21.15.172:0/3285822837 shutdown_connections 2023-11-28T01:07:41.090 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.544+0000 7f599a1c8700 1 --2- 172.21.15.172:0/3285822837 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f597c038b40 0x7f597c03b000 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:41.090 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.544+0000 7f599a1c8700 1 --2- 172.21.15.172:0/3285822837 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5994103410 0x7f5994198e10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:41.090 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.544+0000 7f599a1c8700 1 -- 172.21.15.172:0/3285822837 >> 172.21.15.172:0/3285822837 conn(0x7f59940fed00 msgr2=0x7f59940ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:41.090 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.544+0000 7f599a1c8700 1 -- 172.21.15.172:0/3285822837 shutdown_connections 2023-11-28T01:07:41.090 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:40.544+0000 7f599a1c8700 1 -- 172.21.15.172:0/3285822837 wait complete. 2023-11-28T01:07:42.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:41 smithi172 ceph-mon[100627]: from='mgr.14120 172.21.15.172:0/2519987422' entity='mgr.y' 2023-11-28T01:07:42.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:41 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/4121082651' entity='client.admin' 2023-11-28T01:07:42.773 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.529+0000 7fa55ccc2700 1 Processor -- start 2023-11-28T01:07:42.773 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.529+0000 7fa55ccc2700 1 -- start start 2023-11-28T01:07:42.774 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.529+0000 7fa55ccc2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558105550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:42.774 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.529+0000 7fa55ccc2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa558105b20 con 0x7fa558105130 2023-11-28T01:07:42.774 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.529+0000 7fa55659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558105550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:42.774 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.529+0000 7fa55659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558105550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49326/0 (socket says 172.21.15.172:49326) 2023-11-28T01:07:42.774 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.529+0000 7fa55659c700 1 -- 172.21.15.172:0/2875596735 learned_addr learned my addr 172.21.15.172:0/2875596735 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:42.774 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.530+0000 7fa55659c700 1 -- 172.21.15.172:0/2875596735 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa558106330 con 0x7fa558105130 2023-11-28T01:07:42.775 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.530+0000 7fa55659c700 1 --2- 172.21.15.172:0/2875596735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558105550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa5480098a0 tx=0x7fa548009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96aad39277a785f0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:42.775 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.530+0000 7fa55559a700 1 -- 172.21.15.172:0/2875596735 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa548004160 con 0x7fa558105130 2023-11-28T01:07:42.775 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.530+0000 7fa55559a700 1 -- 172.21.15.172:0/2875596735 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7fa5480036a0 con 0x7fa558105130 2023-11-28T01:07:42.775 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.530+0000 7fa55559a700 1 -- 172.21.15.172:0/2875596735 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5480117a0 con 0x7fa558105130 2023-11-28T01:07:42.775 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.531+0000 7fa55ccc2700 1 -- 172.21.15.172:0/2875596735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 msgr2=0x7fa558105550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:42.775 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.531+0000 7fa55ccc2700 1 --2- 172.21.15.172:0/2875596735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558105550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa5480098a0 tx=0x7fa548009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:42.775 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.531+0000 7fa55ccc2700 1 -- 172.21.15.172:0/2875596735 shutdown_connections 2023-11-28T01:07:42.775 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.531+0000 7fa55ccc2700 1 --2- 172.21.15.172:0/2875596735 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558105550 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:42.775 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.531+0000 7fa55ccc2700 1 -- 172.21.15.172:0/2875596735 >> 172.21.15.172:0/2875596735 conn(0x7fa5581006b0 msgr2=0x7fa558102b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:42.775 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.531+0000 7fa55ccc2700 1 -- 172.21.15.172:0/2875596735 shutdown_connections 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.531+0000 7fa55ccc2700 1 -- 172.21.15.172:0/2875596735 wait complete. 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.532+0000 7fa55ccc2700 1 Processor -- start 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.532+0000 7fa55ccc2700 1 -- start start 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.532+0000 7fa55ccc2700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558198e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.532+0000 7fa55ccc2700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa558199340 con 0x7fa558105130 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.533+0000 7fa55659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558198e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.533+0000 7fa55659c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558198e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49328/0 (socket says 172.21.15.172:49328) 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.533+0000 7fa55659c700 1 -- 172.21.15.172:0/4121082651 learned_addr learned my addr 172.21.15.172:0/4121082651 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.533+0000 7fa55659c700 1 -- 172.21.15.172:0/4121082651 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa548009550 con 0x7fa558105130 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.533+0000 7fa55659c700 1 --2- 172.21.15.172:0/4121082651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558198e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa548009fb0 tx=0x7fa548011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.533+0000 7fa5477fe700 1 -- 172.21.15.172:0/4121082651 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa548003880 con 0x7fa558105130 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.533+0000 7fa5477fe700 1 -- 172.21.15.172:0/4121082651 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7fa548004600 con 0x7fa558105130 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.533+0000 7fa5477fe700 1 -- 172.21.15.172:0/4121082651 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa54801bd70 con 0x7fa558105130 2023-11-28T01:07:42.776 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.533+0000 7fa55ccc2700 1 -- 172.21.15.172:0/4121082651 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa558199540 con 0x7fa558105130 2023-11-28T01:07:42.777 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.533+0000 7fa55ccc2700 1 -- 172.21.15.172:0/4121082651 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5581999e0 con 0x7fa558105130 2023-11-28T01:07:42.777 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.534+0000 7fa5477fe700 1 -- 172.21.15.172:0/4121082651 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 8) v1 ==== 45719+0+0 (secure 0 0 0) 0x7fa5480193d0 con 0x7fa558105130 2023-11-28T01:07:42.777 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.535+0000 7fa5477fe700 1 --2- 172.21.15.172:0/4121082651 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fa540038b40 0x7fa54003b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:42.777 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.535+0000 7fa55ccc2700 1 -- 172.21.15.172:0/4121082651 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa558062c00 con 0x7fa558105130 2023-11-28T01:07:42.777 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.535+0000 7fa5477fe700 1 -- 172.21.15.172:0/4121082651 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa54804cca0 con 0x7fa558105130 2023-11-28T01:07:42.777 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.535+0000 7fa555d9b700 1 --2- 172.21.15.172:0/4121082651 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fa540038b40 0x7fa54003b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:42.777 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.535+0000 7fa555d9b700 1 --2- 172.21.15.172:0/4121082651 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fa540038b40 0x7fa54003b000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa54c006fd0 tx=0x7fa54c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:42.778 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.539+0000 7fa5477fe700 1 -- 172.21.15.172:0/4121082651 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7fa548019720 con 0x7fa558105130 2023-11-28T01:07:42.778 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.744+0000 7fa55ccc2700 1 -- 172.21.15.172:0/4121082651 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fa558109530 con 0x7fa558105130 2023-11-28T01:07:42.778 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.746+0000 7fa5477fe700 1 -- 172.21.15.172:0/4121082651 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fa54800b3e0 con 0x7fa558105130 2023-11-28T01:07:42.778 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.749+0000 7fa55ccc2700 1 -- 172.21.15.172:0/4121082651 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fa540038b40 msgr2=0x7fa54003b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:42.778 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.749+0000 7fa55ccc2700 1 --2- 172.21.15.172:0/4121082651 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fa540038b40 0x7fa54003b000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa54c006fd0 tx=0x7fa54c006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:42.778 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.749+0000 7fa55ccc2700 1 -- 172.21.15.172:0/4121082651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 msgr2=0x7fa558198e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:42.778 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.749+0000 7fa55ccc2700 1 --2- 172.21.15.172:0/4121082651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558198e00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa548009fb0 tx=0x7fa548011cf0 comp rx=0 tx=0).stop 2023-11-28T01:07:42.778 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.749+0000 7fa55ccc2700 1 -- 172.21.15.172:0/4121082651 shutdown_connections 2023-11-28T01:07:42.778 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.749+0000 7fa55ccc2700 1 --2- 172.21.15.172:0/4121082651 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7fa540038b40 0x7fa54003b000 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:42.778 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.749+0000 7fa55ccc2700 1 --2- 172.21.15.172:0/4121082651 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fa558105130 0x7fa558198e00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:42.778 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.749+0000 7fa55ccc2700 1 -- 172.21.15.172:0/4121082651 >> 172.21.15.172:0/4121082651 conn(0x7fa5581006b0 msgr2=0x7fa5581012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:42.779 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.749+0000 7fa55ccc2700 1 -- 172.21.15.172:0/4121082651 shutdown_connections 2023-11-28T01:07:42.779 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:41.749+0000 7fa55ccc2700 1 -- 172.21.15.172:0/4121082651 wait complete. 2023-11-28T01:07:42.779 INFO:teuthology.orchestra.run.smithi172.stdout:Enabling the dashboard module... 2023-11-28T01:07:44.149 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:43 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/3627909088' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-28T01:07:44.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:43 smithi172 conmon[101049]: ignoring --setuser ceph since I am not root 2023-11-28T01:07:44.149 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:43 smithi172 conmon[101049]: ignoring --setgroup ceph since I am not root 2023-11-28T01:07:44.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:44 smithi172 conmon[101049]: 2023-11-28T01:07:44.064+0000 7fb9d57e4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T01:07:44.150 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:44 smithi172 conmon[101049]: 2023-11-28T01:07:44.148+0000 7fb9d57e4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T01:07:44.439 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.209+0000 7faf0eb47700 1 Processor -- start 2023-11-28T01:07:44.440 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.209+0000 7faf0eb47700 1 -- start start 2023-11-28T01:07:44.440 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.209+0000 7faf0eb47700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf081044a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:44.440 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.209+0000 7faf0eb47700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf08104a70 con 0x7faf08104080 2023-11-28T01:07:44.440 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.210+0000 7faf0c8e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf081044a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:44.441 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.210+0000 7faf0c8e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf081044a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49332/0 (socket says 172.21.15.172:49332) 2023-11-28T01:07:44.441 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.210+0000 7faf0c8e3700 1 -- 172.21.15.172:0/2382704161 learned_addr learned my addr 172.21.15.172:0/2382704161 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:44.441 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.210+0000 7faf0c8e3700 1 -- 172.21.15.172:0/2382704161 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf08105280 con 0x7faf08104080 2023-11-28T01:07:44.441 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.210+0000 7faf0c8e3700 1 --2- 172.21.15.172:0/2382704161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf081044a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faefc0098a0 tx=0x7faefc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba46019a2b11526c server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:44.442 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.210+0000 7faf077fe700 1 -- 172.21.15.172:0/2382704161 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faefc004160 con 0x7faf08104080 2023-11-28T01:07:44.442 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.210+0000 7faf077fe700 1 -- 172.21.15.172:0/2382704161 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7faefc0036a0 con 0x7faf08104080 2023-11-28T01:07:44.442 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.210+0000 7faf077fe700 1 -- 172.21.15.172:0/2382704161 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faefc0117a0 con 0x7faf08104080 2023-11-28T01:07:44.442 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.211+0000 7faf0eb47700 1 -- 172.21.15.172:0/2382704161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 msgr2=0x7faf081044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:44.443 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.211+0000 7faf0eb47700 1 --2- 172.21.15.172:0/2382704161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf081044a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7faefc0098a0 tx=0x7faefc009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:44.443 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.211+0000 7faf0eb47700 1 -- 172.21.15.172:0/2382704161 shutdown_connections 2023-11-28T01:07:44.443 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.211+0000 7faf0eb47700 1 --2- 172.21.15.172:0/2382704161 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf081044a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:44.443 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.211+0000 7faf0eb47700 1 -- 172.21.15.172:0/2382704161 >> 172.21.15.172:0/2382704161 conn(0x7faf080ff640 msgr2=0x7faf08101a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:44.444 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.211+0000 7faf0eb47700 1 -- 172.21.15.172:0/2382704161 shutdown_connections 2023-11-28T01:07:44.444 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.211+0000 7faf0eb47700 1 -- 172.21.15.172:0/2382704161 wait complete. 2023-11-28T01:07:44.444 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.211+0000 7faf0eb47700 1 Processor -- start 2023-11-28T01:07:44.444 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.211+0000 7faf0eb47700 1 -- start start 2023-11-28T01:07:44.444 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.211+0000 7faf0eb47700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf08197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:44.445 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.211+0000 7faf0eb47700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf081982c0 con 0x7faf08104080 2023-11-28T01:07:44.445 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.212+0000 7faf0c8e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf08197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:44.445 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.212+0000 7faf0c8e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf08197d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49334/0 (socket says 172.21.15.172:49334) 2023-11-28T01:07:44.446 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.212+0000 7faf0c8e3700 1 -- 172.21.15.172:0/3627909088 learned_addr learned my addr 172.21.15.172:0/3627909088 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:44.446 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.212+0000 7faf0c8e3700 1 -- 172.21.15.172:0/3627909088 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faefc009550 con 0x7faf08104080 2023-11-28T01:07:44.446 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.212+0000 7faf0c8e3700 1 --2- 172.21.15.172:0/3627909088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf08197d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faefc000c00 tx=0x7faefc0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:44.446 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.212+0000 7faf05ffb700 1 -- 172.21.15.172:0/3627909088 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faefc0041b0 con 0x7faf08104080 2023-11-28T01:07:44.447 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.212+0000 7faf05ffb700 1 -- 172.21.15.172:0/3627909088 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7faefc01b440 con 0x7faf08104080 2023-11-28T01:07:44.447 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.212+0000 7faf0eb47700 1 -- 172.21.15.172:0/3627909088 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf081984c0 con 0x7faf08104080 2023-11-28T01:07:44.447 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.212+0000 7faf05ffb700 1 -- 172.21.15.172:0/3627909088 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faefc0193d0 con 0x7faf08104080 2023-11-28T01:07:44.448 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.212+0000 7faf0eb47700 1 -- 172.21.15.172:0/3627909088 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf08198960 con 0x7faf08104080 2023-11-28T01:07:44.448 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.213+0000 7faf0eb47700 1 -- 172.21.15.172:0/3627909088 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf08062c00 con 0x7faf08104080 2023-11-28T01:07:44.448 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.214+0000 7faf05ffb700 1 -- 172.21.15.172:0/3627909088 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 8) v1 ==== 45719+0+0 (secure 0 0 0) 0x7faefc01b5b0 con 0x7faf08104080 2023-11-28T01:07:44.448 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.214+0000 7faf05ffb700 1 --2- 172.21.15.172:0/3627909088 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faef8038b90 0x7faef803b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:44.449 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.214+0000 7faf05ffb700 1 -- 172.21.15.172:0/3627909088 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7faefc04cce0 con 0x7faf08104080 2023-11-28T01:07:44.449 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.214+0000 7faf07fff700 1 --2- 172.21.15.172:0/3627909088 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faef8038b90 0x7faef803b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:44.449 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.215+0000 7faf07fff700 1 --2- 172.21.15.172:0/3627909088 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faef8038b90 0x7faef803b050 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faef4006fd0 tx=0x7faef4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:44.449 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.218+0000 7faf05ffb700 1 -- 172.21.15.172:0/3627909088 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7faefc004310 con 0x7faf08104080 2023-11-28T01:07:44.450 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.465+0000 7faf0eb47700 1 -- 172.21.15.172:0/3627909088 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7faf081084b0 con 0x7faf08104080 2023-11-28T01:07:44.450 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.887+0000 7faf05ffb700 1 -- 172.21.15.172:0/3627909088 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7faefc02bb50 con 0x7faf08104080 2023-11-28T01:07:44.450 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.888+0000 7faf05ffb700 1 -- 172.21.15.172:0/3627909088 <== mon.0 v2:172.21.15.172:3300/0 8 ==== mgrmap(e 9) v1 ==== 45732+0+0 (secure 0 0 0) 0x7faefc04b790 con 0x7faf08104080 2023-11-28T01:07:44.450 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.890+0000 7faf0eb47700 1 -- 172.21.15.172:0/3627909088 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faef8038b90 msgr2=0x7faef803b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:44.451 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.890+0000 7faf0eb47700 1 --2- 172.21.15.172:0/3627909088 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faef8038b90 0x7faef803b050 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faef4006fd0 tx=0x7faef4006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:44.451 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.890+0000 7faf0eb47700 1 -- 172.21.15.172:0/3627909088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 msgr2=0x7faf08197d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:44.451 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.890+0000 7faf0eb47700 1 --2- 172.21.15.172:0/3627909088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf08197d80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faefc000c00 tx=0x7faefc0119a0 comp rx=0 tx=0).stop 2023-11-28T01:07:44.451 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.890+0000 7faf0eb47700 1 -- 172.21.15.172:0/3627909088 shutdown_connections 2023-11-28T01:07:44.452 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.890+0000 7faf0eb47700 1 --2- 172.21.15.172:0/3627909088 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faef8038b90 0x7faef803b050 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:44.452 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.890+0000 7faf0eb47700 1 --2- 172.21.15.172:0/3627909088 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faf08104080 0x7faf08197d80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:44.452 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.890+0000 7faf0eb47700 1 -- 172.21.15.172:0/3627909088 >> 172.21.15.172:0/3627909088 conn(0x7faf080ff640 msgr2=0x7faf08100260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:44.452 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.890+0000 7faf0eb47700 1 -- 172.21.15.172:0/3627909088 shutdown_connections 2023-11-28T01:07:44.452 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:43.890+0000 7faf0eb47700 1 -- 172.21.15.172:0/3627909088 wait complete. 2023-11-28T01:07:44.453 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:44 smithi172 conmon[101049]: 2023-11-28T01:07:44.240+0000 7fb9d57e4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T01:07:44.453 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:44 smithi172 conmon[101049]: 2023-11-28T01:07:44.335+0000 7fb9d57e4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T01:07:44.709 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:44 smithi172 conmon[101049]: 2023-11-28T01:07:44.531+0000 7fb9d57e4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T01:07:45.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:44 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/3627909088' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-28T01:07:45.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:44 smithi172 ceph-mon[100627]: mgrmap e9: y(active, since 19s) 2023-11-28T01:07:45.053 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:44 smithi172 conmon[101049]: 2023-11-28T01:07:44.923+0000 7fb9d57e4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T01:07:45.425 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:45 smithi172 conmon[101049]: 2023-11-28T01:07:45.061+0000 7fb9d57e4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T01:07:45.425 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:45 smithi172 conmon[101049]: 2023-11-28T01:07:45.152+0000 7fb9d57e4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T01:07:45.425 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:45 smithi172 conmon[101049]: 2023-11-28T01:07:45.249+0000 7fb9d57e4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T01:07:45.839 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout { 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "available": true, 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "active_name": "y", 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout } 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.988+0000 7f76b1ebd700 1 Processor -- start 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.988+0000 7f76b1ebd700 1 -- start start 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.988+0000 7f76b1ebd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac108050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.988+0000 7f76b1ebd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac078690 con 0x7f76ac105c60 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.989+0000 7f76ab7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac108050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.989+0000 7f76ab7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac108050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49348/0 (socket says 172.21.15.172:49348) 2023-11-28T01:07:45.840 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.989+0000 7f76ab7fe700 1 -- 172.21.15.172:0/2466954081 learned_addr learned my addr 172.21.15.172:0/2466954081 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.989+0000 7f76ab7fe700 1 -- 172.21.15.172:0/2466954081 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac0787d0 con 0x7f76ac105c60 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.989+0000 7f76ab7fe700 1 --2- 172.21.15.172:0/2466954081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac108050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f769c0098a0 tx=0x7f769c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bacdf852659ecc27 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.989+0000 7f76aa7fc700 1 -- 172.21.15.172:0/2466954081 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f769c004160 con 0x7f76ac105c60 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.989+0000 7f76aa7fc700 1 -- 172.21.15.172:0/2466954081 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f769c0036a0 con 0x7f76ac105c60 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.989+0000 7f76aa7fc700 1 -- 172.21.15.172:0/2466954081 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f769c0117a0 con 0x7f76ac105c60 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.990+0000 7f76b1ebd700 1 -- 172.21.15.172:0/2466954081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 msgr2=0x7f76ac108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.990+0000 7f76b1ebd700 1 --2- 172.21.15.172:0/2466954081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac108050 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f769c0098a0 tx=0x7f769c009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.990+0000 7f76b1ebd700 1 -- 172.21.15.172:0/2466954081 shutdown_connections 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.990+0000 7f76b1ebd700 1 --2- 172.21.15.172:0/2466954081 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac108050 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.990+0000 7f76b1ebd700 1 -- 172.21.15.172:0/2466954081 >> 172.21.15.172:0/2466954081 conn(0x7f76ac0ff590 msgr2=0x7f76ac101a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.990+0000 7f76b1ebd700 1 -- 172.21.15.172:0/2466954081 shutdown_connections 2023-11-28T01:07:45.841 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.990+0000 7f76b1ebd700 1 -- 172.21.15.172:0/2466954081 wait complete. 2023-11-28T01:07:45.842 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.990+0000 7f76b1ebd700 1 Processor -- start 2023-11-28T01:07:45.842 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.991+0000 7f76b1ebd700 1 -- start start 2023-11-28T01:07:45.842 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.991+0000 7f76b1ebd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:45.842 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.991+0000 7f76b1ebd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac1992a0 con 0x7f76ac105c60 2023-11-28T01:07:45.842 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.991+0000 7f76ab7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:45.842 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.991+0000 7f76ab7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49350/0 (socket says 172.21.15.172:49350) 2023-11-28T01:07:45.842 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.991+0000 7f76ab7fe700 1 -- 172.21.15.172:0/1890381041 learned_addr learned my addr 172.21.15.172:0/1890381041 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:45.842 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.991+0000 7f76ab7fe700 1 -- 172.21.15.172:0/1890381041 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c009550 con 0x7f76ac105c60 2023-11-28T01:07:45.842 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.991+0000 7f76ab7fe700 1 --2- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac198d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f769c000c00 tx=0x7f769c0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:45.842 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.991+0000 7f76a8ff9700 1 -- 172.21.15.172:0/1890381041 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f769c0041b0 con 0x7f76ac105c60 2023-11-28T01:07:45.843 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.991+0000 7f76a8ff9700 1 -- 172.21.15.172:0/1890381041 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f769c004310 con 0x7f76ac105c60 2023-11-28T01:07:45.843 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.991+0000 7f76b1ebd700 1 -- 172.21.15.172:0/1890381041 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ac1994a0 con 0x7f76ac105c60 2023-11-28T01:07:45.843 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.992+0000 7f76b1ebd700 1 -- 172.21.15.172:0/1890381041 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ac199940 con 0x7f76ac105c60 2023-11-28T01:07:45.843 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.992+0000 7f76a8ff9700 1 -- 172.21.15.172:0/1890381041 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f769c0193f0 con 0x7f76ac105c60 2023-11-28T01:07:45.843 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.992+0000 7f76a8ff9700 1 -- 172.21.15.172:0/1890381041 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 9) v1 ==== 45732+0+0 (secure 0 0 0) 0x7f769c0195e0 con 0x7f76ac105c60 2023-11-28T01:07:45.843 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.992+0000 7f76b1ebd700 1 -- 172.21.15.172:0/1890381041 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac1091e0 con 0x7f76ac105c60 2023-11-28T01:07:45.843 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.993+0000 7f76a8ff9700 1 --2- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f7694038c40 0x7f769403b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:45.843 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.993+0000 7f76a8ff9700 1 -- 172.21.15.172:0/1890381041 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f769c04cd00 con 0x7f76ac105c60 2023-11-28T01:07:45.843 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.993+0000 7f76aaffd700 1 -- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f7694038c40 msgr2=0x7f769403b100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/2738031896 2023-11-28T01:07:45.843 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.993+0000 7f76aaffd700 1 --2- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f7694038c40 0x7f769403b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T01:07:45.844 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:44.997+0000 7f76a8ff9700 1 -- 172.21.15.172:0/1890381041 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f769c022720 con 0x7f76ac105c60 2023-11-28T01:07:45.844 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.194+0000 7f76aaffd700 1 -- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f7694038c40 msgr2=0x7f769403b100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/2738031896 2023-11-28T01:07:45.844 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.194+0000 7f76aaffd700 1 --2- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f7694038c40 0x7f769403b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T01:07:45.844 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.249+0000 7f76b1ebd700 1 -- 172.21.15.172:0/1890381041 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f76ac19c2e0 con 0x7f76ac105c60 2023-11-28T01:07:45.844 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.249+0000 7f76a8ff9700 1 -- 172.21.15.172:0/1890381041 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f769c01f020 con 0x7f76ac105c60 2023-11-28T01:07:45.844 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.251+0000 7f76b1ebd700 1 -- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f7694038c40 msgr2=0x7f769403b100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T01:07:45.844 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.251+0000 7f76b1ebd700 1 --2- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f7694038c40 0x7f769403b100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:45.844 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.251+0000 7f76b1ebd700 1 -- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 msgr2=0x7f76ac198d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:45.845 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.251+0000 7f76b1ebd700 1 --2- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac198d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f769c000c00 tx=0x7f769c0119a0 comp rx=0 tx=0).stop 2023-11-28T01:07:45.845 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.251+0000 7f76b1ebd700 1 -- 172.21.15.172:0/1890381041 shutdown_connections 2023-11-28T01:07:45.845 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.251+0000 7f76b1ebd700 1 --2- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7f7694038c40 0x7f769403b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:45.845 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.251+0000 7f76b1ebd700 1 --2- 172.21.15.172:0/1890381041 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f76ac105c60 0x7f76ac198d60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:45.845 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.251+0000 7f76b1ebd700 1 -- 172.21.15.172:0/1890381041 >> 172.21.15.172:0/1890381041 conn(0x7f76ac0ff590 msgr2=0x7f76ac100130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:45.845 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.252+0000 7f76b1ebd700 1 -- 172.21.15.172:0/1890381041 shutdown_connections 2023-11-28T01:07:45.845 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:45.252+0000 7f76b1ebd700 1 -- 172.21.15.172:0/1890381041 wait complete. 2023-11-28T01:07:45.845 INFO:teuthology.orchestra.run.smithi172.stdout:Waiting for the mgr to restart... 2023-11-28T01:07:45.845 INFO:teuthology.orchestra.run.smithi172.stdout:Waiting for mgr epoch 9... 2023-11-28T01:07:46.026 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:45 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/1890381041' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-28T01:07:46.026 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:45 smithi172 conmon[101049]: 2023-11-28T01:07:45.696+0000 7fb9d57e4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T01:07:46.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:46 smithi172 conmon[101049]: 2023-11-28T01:07:46.395+0000 7fb9d57e4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T01:07:46.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:46 smithi172 conmon[101049]: 2023-11-28T01:07:46.478+0000 7fb9d57e4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T01:07:47.525 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:47 smithi172 conmon[101049]: 2023-11-28T01:07:47.256+0000 7fb9d57e4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T01:07:47.525 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:47 smithi172 conmon[101049]: 2023-11-28T01:07:47.349+0000 7fb9d57e4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T01:07:47.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:47 smithi172 conmon[101049]: 2023-11-28T01:07:47.524+0000 7fb9d57e4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T01:07:48.303 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:47 smithi172 conmon[101049]: 2023-11-28T01:07:47.944+0000 7fb9d57e4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T01:07:48.303 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:48 smithi172 conmon[101049]: 2023-11-28T01:07:48.032+0000 7fb9d57e4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T01:07:48.303 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:48 smithi172 conmon[101049]: 2023-11-28T01:07:48.180+0000 7fb9d57e4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T01:07:48.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:48 smithi172 conmon[101049]: 2023-11-28T01:07:48.447+0000 7fb9d57e4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T01:07:49.478 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:49 smithi172 conmon[101049]: 2023-11-28T01:07:49.147+0000 7fb9d57e4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T01:07:49.479 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:49 smithi172 conmon[101049]: 2023-11-28T01:07:49.279+0000 7fb9d57e4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T01:07:49.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:49 smithi172 conmon[101049]: 2023-11-28T01:07:49.476+0000 7fb9d57e4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T01:07:49.803 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:49 smithi172 conmon[101049]: 2023-11-28T01:07:49.592+0000 7fb9d57e4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T01:07:50.303 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:49 smithi172 conmon[101049]: 2023-11-28T01:07:49.813+0000 7fb9d57e4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T01:07:51.053 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:07:50 smithi172 conmon[101049]: 2023-11-28T01:07:50.682+0000 7fb9d57e4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T01:07:51.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: Active manager daemon y restarted 2023-11-28T01:07:51.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: Activating manager daemon y 2023-11-28T01:07:51.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: osdmap e3: 0 total, 0 up, 0 in 2023-11-28T01:07:51.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: mgrmap e10: y(active, starting, since 0.00392536s) 2023-11-28T01:07:51.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:07:51.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-11-28T01:07:51.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-28T01:07:51.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-28T01:07:51.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-28T01:07:51.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: Manager daemon y is now available 2023-11-28T01:07:51.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:07:51.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:50 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:07:52.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2023-11-28T01:07:52.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2023-11-28T01:07:52.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:51 smithi172 ceph-mon[100627]: mgrmap e11: y(active, since 1.00577s) 2023-11-28T01:07:52.267 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout { 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout "initialized": true 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout } 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.301+0000 7faae9b83700 1 Processor -- start 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.301+0000 7faae9b83700 1 -- start start 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.301+0000 7faae9b83700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae41059d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.301+0000 7faae9b83700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae4105fa0 con 0x7faae41055d0 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.302+0000 7faae37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae41059d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.302+0000 7faae37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae41059d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49358/0 (socket says 172.21.15.172:49358) 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.302+0000 7faae37fe700 1 -- 172.21.15.172:0/3832477503 learned_addr learned my addr 172.21.15.172:0/3832477503 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.302+0000 7faae37fe700 1 -- 172.21.15.172:0/3832477503 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae41060e0 con 0x7faae41055d0 2023-11-28T01:07:52.268 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.302+0000 7faae37fe700 1 --2- 172.21.15.172:0/3832477503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae41059d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faad40098a0 tx=0x7faad4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db78bf6a3ade45e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:52.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.302+0000 7faae27fc700 1 -- 172.21.15.172:0/3832477503 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faad4004160 con 0x7faae41055d0 2023-11-28T01:07:52.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.302+0000 7faae27fc700 1 -- 172.21.15.172:0/3832477503 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7faad40036a0 con 0x7faae41055d0 2023-11-28T01:07:52.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.302+0000 7faae27fc700 1 -- 172.21.15.172:0/3832477503 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faad40117a0 con 0x7faae41055d0 2023-11-28T01:07:52.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.303+0000 7faae9b83700 1 -- 172.21.15.172:0/3832477503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 msgr2=0x7faae41059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:52.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.303+0000 7faae9b83700 1 --2- 172.21.15.172:0/3832477503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae41059d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faad40098a0 tx=0x7faad4009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:52.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.303+0000 7faae9b83700 1 -- 172.21.15.172:0/3832477503 shutdown_connections 2023-11-28T01:07:52.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.303+0000 7faae9b83700 1 --2- 172.21.15.172:0/3832477503 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae41059d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:52.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.303+0000 7faae9b83700 1 -- 172.21.15.172:0/3832477503 >> 172.21.15.172:0/3832477503 conn(0x7faae40fed00 msgr2=0x7faae4101180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:52.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.303+0000 7faae9b83700 1 -- 172.21.15.172:0/3832477503 shutdown_connections 2023-11-28T01:07:52.269 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.303+0000 7faae9b83700 1 -- 172.21.15.172:0/3832477503 wait complete. 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.304+0000 7faae9b83700 1 Processor -- start 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.304+0000 7faae9b83700 1 -- start start 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae9b83700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae419b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae9b83700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae419b590 con 0x7faae41055d0 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae419b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae419b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49360/0 (socket says 172.21.15.172:49360) 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae37fe700 1 -- 172.21.15.172:0/1635945151 learned_addr learned my addr 172.21.15.172:0/1635945151 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae37fe700 1 -- 172.21.15.172:0/1635945151 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad4009550 con 0x7faae41055d0 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae37fe700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae419b050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faad4000c00 tx=0x7faad40119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae0ff9700 1 -- 172.21.15.172:0/1635945151 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faad40041b0 con 0x7faae41055d0 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae0ff9700 1 -- 172.21.15.172:0/1635945151 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7faad4004310 con 0x7faae41055d0 2023-11-28T01:07:52.270 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae9b83700 1 -- 172.21.15.172:0/1635945151 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae419b790 con 0x7faae41055d0 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae0ff9700 1 -- 172.21.15.172:0/1635945151 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faad401be80 con 0x7faae41055d0 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.305+0000 7faae9b83700 1 -- 172.21.15.172:0/1635945151 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae419bc30 con 0x7faae41055d0 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.306+0000 7faae0ff9700 1 -- 172.21.15.172:0/1635945151 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 9) v1 ==== 45732+0+0 (secure 0 0 0) 0x7faad40194f0 con 0x7faae41055d0 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.306+0000 7faae0ff9700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 0x7faacc03b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.306+0000 7faae9b83700 1 -- 172.21.15.172:0/1635945151 --> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7faae4103c90 con 0x7faacc038b90 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.306+0000 7faae0ff9700 1 -- 172.21.15.172:0/1635945151 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7faad404dc60 con 0x7faae41055d0 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.306+0000 7faae2ffd700 1 -- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 msgr2=0x7faacc03b050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/2738031896 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.306+0000 7faae2ffd700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 0x7faacc03b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.507+0000 7faae2ffd700 1 -- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 msgr2=0x7faacc03b050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/2738031896 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.507+0000 7faae2ffd700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 0x7faacc03b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.908+0000 7faae2ffd700 1 -- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 msgr2=0x7faacc03b050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/2738031896 2023-11-28T01:07:52.271 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:46.908+0000 7faae2ffd700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 0x7faacc03b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-11-28T01:07:52.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:47.709+0000 7faae2ffd700 1 -- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 msgr2=0x7faacc03b050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/2738031896 2023-11-28T01:07:52.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:47.709+0000 7faae2ffd700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 0x7faacc03b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-11-28T01:07:52.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:49.309+0000 7faae2ffd700 1 -- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 msgr2=0x7faacc03b050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.172:6800/2738031896 2023-11-28T01:07:52.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:49.309+0000 7faae2ffd700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 0x7faacc03b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-11-28T01:07:52.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:50.697+0000 7faae0ff9700 1 -- 172.21.15.172:0/1635945151 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mgrmap(e 10) v1 ==== 45499+0+0 (secure 0 0 0) 0x7faad401a750 con 0x7faae41055d0 2023-11-28T01:07:52.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:50.697+0000 7faae0ff9700 1 -- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 msgr2=0x7faacc03b050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T01:07:52.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:50.697+0000 7faae0ff9700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 0x7faacc03b050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:52.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.698+0000 7faae0ff9700 1 -- 172.21.15.172:0/1635945151 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mgrmap(e 11) v1 ==== 45626+0+0 (secure 0 0 0) 0x7faad403c360 con 0x7faae41055d0 2023-11-28T01:07:52.272 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.699+0000 7faae0ff9700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faacc03c450 0x7faacc03e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.699+0000 7faae0ff9700 1 -- 172.21.15.172:0/1635945151 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7faae4103c90 con 0x7faacc03c450 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.699+0000 7faae2ffd700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faacc03c450 0x7faacc03e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.699+0000 7faae2ffd700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faacc03c450 0x7faacc03e840 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7faad8003a10 tx=0x7faad80092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.700+0000 7faae0ff9700 1 -- 172.21.15.172:0/1635945151 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7faae4103c90 con 0x7faacc03c450 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.706+0000 7faae9b83700 1 -- 172.21.15.172:0/1635945151 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7faae4103e00 con 0x7faacc03c450 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae0ff9700 1 -- 172.21.15.172:0/1635945151 <== mgr.14152 v2:172.21.15.172:6800/3346003710 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7faae4103e00 con 0x7faacc03c450 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae9b83700 1 -- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faacc03c450 msgr2=0x7faacc03e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae9b83700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faacc03c450 0x7faacc03e840 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7faad8003a10 tx=0x7faad80092b0 comp rx=0 tx=0).stop 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae9b83700 1 -- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 msgr2=0x7faae419b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae9b83700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae419b050 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faad4000c00 tx=0x7faad40119a0 comp rx=0 tx=0).stop 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae9b83700 1 -- 172.21.15.172:0/1635945151 shutdown_connections 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae9b83700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faacc03c450 0x7faacc03e840 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:52.273 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae9b83700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faae41055d0 0x7faae419b050 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:52.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae9b83700 1 --2- 172.21.15.172:0/1635945151 >> [v2:172.21.15.172:6800/2738031896,v1:172.21.15.172:6801/2738031896] conn(0x7faacc038b90 0x7faacc03b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:52.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae9b83700 1 -- 172.21.15.172:0/1635945151 >> 172.21.15.172:0/1635945151 conn(0x7faae40fed00 msgr2=0x7faae4191850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:52.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae9b83700 1 -- 172.21.15.172:0/1635945151 shutdown_connections 2023-11-28T01:07:52.274 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:51.707+0000 7faae9b83700 1 -- 172.21.15.172:0/1635945151 wait complete. 2023-11-28T01:07:52.274 INFO:teuthology.orchestra.run.smithi172.stdout:mgr epoch 9 is available 2023-11-28T01:07:52.274 INFO:teuthology.orchestra.run.smithi172.stdout:Generating a dashboard self-signed certificate... 2023-11-28T01:07:53.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:52 smithi172 ceph-mon[100627]: [28/Nov/2023:01:07:51] ENGINE Bus STARTING 2023-11-28T01:07:53.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:52 smithi172 ceph-mon[100627]: [28/Nov/2023:01:07:51] ENGINE Serving on http://172.21.15.172:8765 2023-11-28T01:07:53.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:52 smithi172 ceph-mon[100627]: [28/Nov/2023:01:07:51] ENGINE Serving on https://172.21.15.172:7150 2023-11-28T01:07:53.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:52 smithi172 ceph-mon[100627]: [28/Nov/2023:01:07:51] ENGINE Bus STARTED 2023-11-28T01:07:53.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:52 smithi172 ceph-mon[100627]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-28T01:07:53.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:52 smithi172 ceph-mon[100627]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-28T01:07:53.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:52 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:07:53.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:52 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:07:53.496 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-11-28T01:07:53.496 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.721+0000 7f4dcb213700 1 Processor -- start 2023-11-28T01:07:53.496 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.721+0000 7f4dcb213700 1 -- start start 2023-11-28T01:07:53.496 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.722+0000 7f4dcb213700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4104440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:53.496 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.722+0000 7f4dcb213700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc4104a10 con 0x7f4dc4104020 2023-11-28T01:07:53.496 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.722+0000 7f4dc8faf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4104440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:53.497 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.722+0000 7f4dc8faf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4104440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49414/0 (socket says 172.21.15.172:49414) 2023-11-28T01:07:53.497 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.722+0000 7f4dc8faf700 1 -- 172.21.15.172:0/3326778538 learned_addr learned my addr 172.21.15.172:0/3326778538 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:53.497 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.722+0000 7f4dc8faf700 1 -- 172.21.15.172:0/3326778538 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc4105220 con 0x7f4dc4104020 2023-11-28T01:07:53.497 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.722+0000 7f4dc8faf700 1 --2- 172.21.15.172:0/3326778538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4104440 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4db800bb40 tx=0x7f4db800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40afc487a4616e9e server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:53.497 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.722+0000 7f4dc37fe700 1 -- 172.21.15.172:0/3326778538 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4db800dbe0 con 0x7f4dc4104020 2023-11-28T01:07:53.497 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.722+0000 7f4dc37fe700 1 -- 172.21.15.172:0/3326778538 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f4db8004480 con 0x7f4dc4104020 2023-11-28T01:07:53.497 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.722+0000 7f4dc37fe700 1 -- 172.21.15.172:0/3326778538 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4db8003c40 con 0x7f4dc4104020 2023-11-28T01:07:53.497 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.723+0000 7f4dcb213700 1 -- 172.21.15.172:0/3326778538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 msgr2=0x7f4dc4104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:53.497 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.723+0000 7f4dcb213700 1 --2- 172.21.15.172:0/3326778538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4104440 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4db800bb40 tx=0x7f4db800be50 comp rx=0 tx=0).stop 2023-11-28T01:07:53.497 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.723+0000 7f4dcb213700 1 -- 172.21.15.172:0/3326778538 shutdown_connections 2023-11-28T01:07:53.497 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.723+0000 7f4dcb213700 1 --2- 172.21.15.172:0/3326778538 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4104440 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.723+0000 7f4dcb213700 1 -- 172.21.15.172:0/3326778538 >> 172.21.15.172:0/3326778538 conn(0x7f4dc40ff580 msgr2=0x7f4dc4101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.723+0000 7f4dcb213700 1 -- 172.21.15.172:0/3326778538 shutdown_connections 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.723+0000 7f4dcb213700 1 -- 172.21.15.172:0/3326778538 wait complete. 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.724+0000 7f4dcb213700 1 Processor -- start 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.724+0000 7f4dcb213700 1 -- start start 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.724+0000 7f4dcb213700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.724+0000 7f4dcb213700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc4197180 con 0x7f4dc4104020 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.724+0000 7f4dc8faf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.724+0000 7f4dc8faf700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49416/0 (socket says 172.21.15.172:49416) 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.724+0000 7f4dc8faf700 1 -- 172.21.15.172:0/1595433806 learned_addr learned my addr 172.21.15.172:0/1595433806 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.724+0000 7f4dc8faf700 1 -- 172.21.15.172:0/1595433806 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db800b7f0 con 0x7f4dc4104020 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.725+0000 7f4dc8faf700 1 --2- 172.21.15.172:0/1595433806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4196c40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4db800bfd0 tx=0x7f4db80041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:53.498 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.725+0000 7f4dc1ffb700 1 -- 172.21.15.172:0/1595433806 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4db801e070 con 0x7f4dc4104020 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.725+0000 7f4dc1ffb700 1 -- 172.21.15.172:0/1595433806 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f4db8010c00 con 0x7f4dc4104020 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.725+0000 7f4dcb213700 1 -- 172.21.15.172:0/1595433806 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc4197380 con 0x7f4dc4104020 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.725+0000 7f4dc1ffb700 1 -- 172.21.15.172:0/1595433806 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4db80229a0 con 0x7f4dc4104020 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.725+0000 7f4dcb213700 1 -- 172.21.15.172:0/1595433806 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc4197820 con 0x7f4dc4104020 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.726+0000 7f4dcb213700 1 -- 172.21.15.172:0/1595433806 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc41908b0 con 0x7f4dc4104020 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.726+0000 7f4dc1ffb700 1 -- 172.21.15.172:0/1595433806 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 11) v1 ==== 45626+0+0 (secure 0 0 0) 0x7f4db80107b0 con 0x7f4dc4104020 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.726+0000 7f4dc1ffb700 1 --2- 172.21.15.172:0/1595433806 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4db4038a80 0x7f4db403af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.726+0000 7f4dc1ffb700 1 -- 172.21.15.172:0/1595433806 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4db804f620 con 0x7f4dc4104020 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.726+0000 7f4dc3fff700 1 --2- 172.21.15.172:0/1595433806 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4db4038a80 0x7f4db403af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.727+0000 7f4dc3fff700 1 --2- 172.21.15.172:0/1595433806 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4db4038a80 0x7f4db403af40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4db0006fd0 tx=0x7f4db0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.731+0000 7f4dc1ffb700 1 -- 172.21.15.172:0/1595433806 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f4db8021ba0 con 0x7f4dc4104020 2023-11-28T01:07:53.499 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.924+0000 7f4dc1ffb700 1 -- 172.21.15.172:0/1595433806 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mgrmap(e 12) v1 ==== 45732+0+0 (secure 0 0 0) 0x7f4db802c960 con 0x7f4dc4104020 2023-11-28T01:07:53.500 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:52.940+0000 7f4dcb213700 1 -- 172.21.15.172:0/1595433806 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f4dc402d380 con 0x7f4db4038a80 2023-11-28T01:07:53.500 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.005+0000 7f4dc1ffb700 1 -- 172.21.15.172:0/1595433806 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f4dc402d380 con 0x7f4db4038a80 2023-11-28T01:07:53.500 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.007+0000 7f4dcb213700 1 -- 172.21.15.172:0/1595433806 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4db4038a80 msgr2=0x7f4db403af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:53.500 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.007+0000 7f4dcb213700 1 --2- 172.21.15.172:0/1595433806 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4db4038a80 0x7f4db403af40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4db0006fd0 tx=0x7f4db0006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:53.500 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.007+0000 7f4dcb213700 1 -- 172.21.15.172:0/1595433806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 msgr2=0x7f4dc4196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:53.500 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.007+0000 7f4dcb213700 1 --2- 172.21.15.172:0/1595433806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4196c40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4db800bfd0 tx=0x7f4db80041a0 comp rx=0 tx=0).stop 2023-11-28T01:07:53.500 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.008+0000 7f4dcb213700 1 -- 172.21.15.172:0/1595433806 shutdown_connections 2023-11-28T01:07:53.500 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.008+0000 7f4dcb213700 1 --2- 172.21.15.172:0/1595433806 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4db4038a80 0x7f4db403af40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:53.500 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.008+0000 7f4dcb213700 1 --2- 172.21.15.172:0/1595433806 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4dc4104020 0x7f4dc4196c40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:53.500 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.008+0000 7f4dcb213700 1 -- 172.21.15.172:0/1595433806 >> 172.21.15.172:0/1595433806 conn(0x7f4dc40ff580 msgr2=0x7f4dc418d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:53.500 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.008+0000 7f4dcb213700 1 -- 172.21.15.172:0/1595433806 shutdown_connections 2023-11-28T01:07:53.501 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.008+0000 7f4dcb213700 1 -- 172.21.15.172:0/1595433806 wait complete. 2023-11-28T01:07:53.501 INFO:teuthology.orchestra.run.smithi172.stdout:Creating initial admin user... 2023-11-28T01:07:54.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:53 smithi172 ceph-mon[100627]: mgrmap e12: y(active, since 2s) 2023-11-28T01:07:54.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:53 smithi172 ceph-mon[100627]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:07:54.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:53 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:07:54.132 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:53 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:07:54.954 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$w1bFcVUqNeaqKEQ8WHW/cuYrYwac2l0mlXAHKxkiHxXlrKOffmzge", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1701133674, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-28T01:07:54.955 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.969+0000 7f70e9f06700 1 Processor -- start 2023-11-28T01:07:54.955 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.969+0000 7f70e9f06700 1 -- start start 2023-11-28T01:07:54.955 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.969+0000 7f70e9f06700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e41055b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:54.955 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.969+0000 7f70e9f06700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e4105b80 con 0x7f70e4105190 2023-11-28T01:07:54.955 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.970+0000 7f70e37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e41055b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:54.955 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.970+0000 7f70e37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e41055b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49420/0 (socket says 172.21.15.172:49420) 2023-11-28T01:07:54.955 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.970+0000 7f70e37fe700 1 -- 172.21.15.172:0/3706852853 learned_addr learned my addr 172.21.15.172:0/3706852853 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:54.955 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.970+0000 7f70e37fe700 1 -- 172.21.15.172:0/3706852853 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e4106390 con 0x7f70e4105190 2023-11-28T01:07:54.955 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.970+0000 7f70e37fe700 1 --2- 172.21.15.172:0/3706852853 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e41055b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f70d80098a0 tx=0x7f70d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0f0431e97e9707 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:54.955 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.970+0000 7f70e27fc700 1 -- 172.21.15.172:0/3706852853 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70d8004160 con 0x7f70e4105190 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.970+0000 7f70e27fc700 1 -- 172.21.15.172:0/3706852853 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f70d80036a0 con 0x7f70e4105190 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.970+0000 7f70e27fc700 1 -- 172.21.15.172:0/3706852853 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70d80117a0 con 0x7f70e4105190 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.971+0000 7f70e9f06700 1 -- 172.21.15.172:0/3706852853 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 msgr2=0x7f70e41055b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.971+0000 7f70e9f06700 1 --2- 172.21.15.172:0/3706852853 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e41055b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f70d80098a0 tx=0x7f70d8009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.971+0000 7f70e9f06700 1 -- 172.21.15.172:0/3706852853 shutdown_connections 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.971+0000 7f70e9f06700 1 --2- 172.21.15.172:0/3706852853 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e41055b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.971+0000 7f70e9f06700 1 -- 172.21.15.172:0/3706852853 >> 172.21.15.172:0/3706852853 conn(0x7f70e4100790 msgr2=0x7f70e4102bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.971+0000 7f70e9f06700 1 -- 172.21.15.172:0/3706852853 shutdown_connections 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.971+0000 7f70e9f06700 1 -- 172.21.15.172:0/3706852853 wait complete. 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.971+0000 7f70e9f06700 1 Processor -- start 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.972+0000 7f70e9f06700 1 -- start start 2023-11-28T01:07:54.956 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.972+0000 7f70e9f06700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e407a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.972+0000 7f70e9f06700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e407a900 con 0x7f70e4105190 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.973+0000 7f70e37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e407a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.973+0000 7f70e37fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e407a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49422/0 (socket says 172.21.15.172:49422) 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.973+0000 7f70e37fe700 1 -- 172.21.15.172:0/19294287 learned_addr learned my addr 172.21.15.172:0/19294287 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.973+0000 7f70e37fe700 1 -- 172.21.15.172:0/19294287 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d8009550 con 0x7f70e4105190 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.973+0000 7f70e37fe700 1 --2- 172.21.15.172:0/19294287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e407a3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f70d8006b20 tx=0x7f70d8011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.974+0000 7f70e0ff9700 1 -- 172.21.15.172:0/19294287 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70d80042e0 con 0x7f70e4105190 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.974+0000 7f70e0ff9700 1 -- 172.21.15.172:0/19294287 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f70d8004440 con 0x7f70e4105190 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.974+0000 7f70e9f06700 1 -- 172.21.15.172:0/19294287 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e4078b40 con 0x7f70e4105190 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.974+0000 7f70e0ff9700 1 -- 172.21.15.172:0/19294287 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f70d801bdc0 con 0x7f70e4105190 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.974+0000 7f70e9f06700 1 -- 172.21.15.172:0/19294287 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e4078fe0 con 0x7f70e4105190 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.974+0000 7f70e0ff9700 1 -- 172.21.15.172:0/19294287 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 12) v1 ==== 45732+0+0 (secure 0 0 0) 0x7f70d80193f0 con 0x7f70e4105190 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.975+0000 7f70e0ff9700 1 --2- 172.21.15.172:0/19294287 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f70cc038be0 0x7f70cc03b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:54.957 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.975+0000 7f70e9f06700 1 -- 172.21.15.172:0/19294287 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e4107120 con 0x7f70e4105190 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.975+0000 7f70e0ff9700 1 -- 172.21.15.172:0/19294287 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f70d804cd50 con 0x7f70e4105190 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.975+0000 7f70e2ffd700 1 --2- 172.21.15.172:0/19294287 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f70cc038be0 0x7f70cc03b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.975+0000 7f70e2ffd700 1 --2- 172.21.15.172:0/19294287 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f70cc038be0 0x7f70cc03b0a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f70d4006fd0 tx=0x7f70d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:53.980+0000 7f70e0ff9700 1 -- 172.21.15.172:0/19294287 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f70d801a3d0 con 0x7f70e4105190 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.204+0000 7f70e9f06700 1 -- 172.21.15.172:0/19294287 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f70e4062a20 con 0x7f70cc038be0 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.426+0000 7f70e0ff9700 1 -- 172.21.15.172:0/19294287 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f70e4062a20 con 0x7f70cc038be0 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.429+0000 7f70e9f06700 1 -- 172.21.15.172:0/19294287 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f70cc038be0 msgr2=0x7f70cc03b0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.429+0000 7f70e9f06700 1 --2- 172.21.15.172:0/19294287 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f70cc038be0 0x7f70cc03b0a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f70d4006fd0 tx=0x7f70d4006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.429+0000 7f70e9f06700 1 -- 172.21.15.172:0/19294287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 msgr2=0x7f70e407a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.429+0000 7f70e9f06700 1 --2- 172.21.15.172:0/19294287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e407a3c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f70d8006b20 tx=0x7f70d8011cf0 comp rx=0 tx=0).stop 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.429+0000 7f70e9f06700 1 -- 172.21.15.172:0/19294287 shutdown_connections 2023-11-28T01:07:54.958 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.429+0000 7f70e9f06700 1 --2- 172.21.15.172:0/19294287 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f70cc038be0 0x7f70cc03b0a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:54.959 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.429+0000 7f70e9f06700 1 --2- 172.21.15.172:0/19294287 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f70e4105190 0x7f70e407a3c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:54.959 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.429+0000 7f70e9f06700 1 -- 172.21.15.172:0/19294287 >> 172.21.15.172:0/19294287 conn(0x7f70e4100790 msgr2=0x7f70e4101310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:54.959 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.429+0000 7f70e9f06700 1 -- 172.21.15.172:0/19294287 shutdown_connections 2023-11-28T01:07:54.959 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:54.429+0000 7f70e9f06700 1 -- 172.21.15.172:0/19294287 wait complete. 2023-11-28T01:07:54.959 INFO:teuthology.orchestra.run.smithi172.stdout:Fetching dashboard port number... 2023-11-28T01:07:55.756 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:55 smithi172 ceph-mon[100627]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:07:55.756 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:55 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:07:56.137 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stdout 8443 2023-11-28T01:07:56.137 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.372+0000 7f79e1f8c700 1 Processor -- start 2023-11-28T01:07:56.138 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.372+0000 7f79e1f8c700 1 -- start start 2023-11-28T01:07:56.138 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.372+0000 7f79e1f8c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc107fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:56.138 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.372+0000 7f79e1f8c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc078710 con 0x7f79dc105bf0 2023-11-28T01:07:56.138 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.373+0000 7f79db7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc107fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:56.138 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.373+0000 7f79db7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc107fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49426/0 (socket says 172.21.15.172:49426) 2023-11-28T01:07:56.138 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.373+0000 7f79db7fe700 1 -- 172.21.15.172:0/3973361539 learned_addr learned my addr 172.21.15.172:0/3973361539 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:56.138 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.373+0000 7f79db7fe700 1 -- 172.21.15.172:0/3973361539 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc108520 con 0x7f79dc105bf0 2023-11-28T01:07:56.138 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.373+0000 7f79db7fe700 1 --2- 172.21.15.172:0/3973361539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc107fe0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79d00098a0 tx=0x7f79d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1decde50deaf16 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:56.138 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.373+0000 7f79da7fc700 1 -- 172.21.15.172:0/3973361539 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79d0004160 con 0x7f79dc105bf0 2023-11-28T01:07:56.139 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.373+0000 7f79da7fc700 1 -- 172.21.15.172:0/3973361539 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f79d00036a0 con 0x7f79dc105bf0 2023-11-28T01:07:56.139 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.373+0000 7f79da7fc700 1 -- 172.21.15.172:0/3973361539 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79d00117a0 con 0x7f79dc105bf0 2023-11-28T01:07:56.139 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.374+0000 7f79e1f8c700 1 -- 172.21.15.172:0/3973361539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 msgr2=0x7f79dc107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:56.139 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.374+0000 7f79e1f8c700 1 --2- 172.21.15.172:0/3973361539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc107fe0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f79d00098a0 tx=0x7f79d0009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:56.139 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.374+0000 7f79e1f8c700 1 -- 172.21.15.172:0/3973361539 shutdown_connections 2023-11-28T01:07:56.139 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.374+0000 7f79e1f8c700 1 --2- 172.21.15.172:0/3973361539 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc107fe0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:56.139 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.374+0000 7f79e1f8c700 1 -- 172.21.15.172:0/3973361539 >> 172.21.15.172:0/3973361539 conn(0x7f79dc0ff5c0 msgr2=0x7f79dc101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:56.139 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.374+0000 7f79e1f8c700 1 -- 172.21.15.172:0/3973361539 shutdown_connections 2023-11-28T01:07:56.140 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.374+0000 7f79e1f8c700 1 -- 172.21.15.172:0/3973361539 wait complete. 2023-11-28T01:07:56.140 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.375+0000 7f79e1f8c700 1 Processor -- start 2023-11-28T01:07:56.140 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.375+0000 7f79e1f8c700 1 -- start start 2023-11-28T01:07:56.140 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.375+0000 7f79e1f8c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:56.140 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.375+0000 7f79e1f8c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc1992a0 con 0x7f79dc105bf0 2023-11-28T01:07:56.140 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.375+0000 7f79db7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:56.140 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.375+0000 7f79db7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49428/0 (socket says 172.21.15.172:49428) 2023-11-28T01:07:56.140 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.375+0000 7f79db7fe700 1 -- 172.21.15.172:0/1206433856 learned_addr learned my addr 172.21.15.172:0/1206433856 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:56.140 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.375+0000 7f79db7fe700 1 -- 172.21.15.172:0/1206433856 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d0009550 con 0x7f79dc105bf0 2023-11-28T01:07:56.140 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.375+0000 7f79db7fe700 1 --2- 172.21.15.172:0/1206433856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc198d60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79d0011c40 tx=0x7f79d0011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:56.140 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.376+0000 7f79d8ff9700 1 -- 172.21.15.172:0/1206433856 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79d00041b0 con 0x7f79dc105bf0 2023-11-28T01:07:56.141 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.376+0000 7f79d8ff9700 1 -- 172.21.15.172:0/1206433856 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f79d0004310 con 0x7f79dc105bf0 2023-11-28T01:07:56.141 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.376+0000 7f79e1f8c700 1 -- 172.21.15.172:0/1206433856 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc1994a0 con 0x7f79dc105bf0 2023-11-28T01:07:56.141 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.376+0000 7f79d8ff9700 1 -- 172.21.15.172:0/1206433856 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79d001be00 con 0x7f79dc105bf0 2023-11-28T01:07:56.141 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.376+0000 7f79e1f8c700 1 -- 172.21.15.172:0/1206433856 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc199940 con 0x7f79dc105bf0 2023-11-28T01:07:56.141 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.376+0000 7f79d8ff9700 1 -- 172.21.15.172:0/1206433856 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 12) v1 ==== 45732+0+0 (secure 0 0 0) 0x7f79d0019460 con 0x7f79dc105bf0 2023-11-28T01:07:56.141 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.377+0000 7f79e1f8c700 1 -- 172.21.15.172:0/1206433856 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc1091e0 con 0x7f79dc105bf0 2023-11-28T01:07:56.141 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.377+0000 7f79d8ff9700 1 --2- 172.21.15.172:0/1206433856 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f79c4038b90 0x7f79c403b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:56.141 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.377+0000 7f79d8ff9700 1 -- 172.21.15.172:0/1206433856 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f79d004d150 con 0x7f79dc105bf0 2023-11-28T01:07:56.141 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.377+0000 7f79daffd700 1 --2- 172.21.15.172:0/1206433856 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f79c4038b90 0x7f79c403b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:56.144 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.377+0000 7f79daffd700 1 --2- 172.21.15.172:0/1206433856 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f79c4038b90 0x7f79c403b050 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f79cc006fd0 tx=0x7f79cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:56.144 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.381+0000 7f79d8ff9700 1 -- 172.21.15.172:0/1206433856 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f79d0018db0 con 0x7f79dc105bf0 2023-11-28T01:07:56.144 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.570+0000 7f79e1f8c700 1 -- 172.21.15.172:0/1206433856 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f79dc1093f0 con 0x7f79dc105bf0 2023-11-28T01:07:56.144 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.570+0000 7f79d8ff9700 1 -- 172.21.15.172:0/1206433856 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f79d0018350 con 0x7f79dc105bf0 2023-11-28T01:07:56.144 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.572+0000 7f79e1f8c700 1 -- 172.21.15.172:0/1206433856 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f79c4038b90 msgr2=0x7f79c403b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:56.144 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.572+0000 7f79e1f8c700 1 --2- 172.21.15.172:0/1206433856 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f79c4038b90 0x7f79c403b050 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f79cc006fd0 tx=0x7f79cc006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:56.144 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.572+0000 7f79e1f8c700 1 -- 172.21.15.172:0/1206433856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 msgr2=0x7f79dc198d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:56.144 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.572+0000 7f79e1f8c700 1 --2- 172.21.15.172:0/1206433856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc198d60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f79d0011c40 tx=0x7f79d0011d20 comp rx=0 tx=0).stop 2023-11-28T01:07:56.145 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.572+0000 7f79e1f8c700 1 -- 172.21.15.172:0/1206433856 shutdown_connections 2023-11-28T01:07:56.145 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.572+0000 7f79e1f8c700 1 --2- 172.21.15.172:0/1206433856 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f79c4038b90 0x7f79c403b050 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:56.145 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.572+0000 7f79e1f8c700 1 --2- 172.21.15.172:0/1206433856 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f79dc105bf0 0x7f79dc198d60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:56.145 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.572+0000 7f79e1f8c700 1 -- 172.21.15.172:0/1206433856 >> 172.21.15.172:0/1206433856 conn(0x7f79dc0ff5c0 msgr2=0x7f79dc100140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:56.145 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.572+0000 7f79e1f8c700 1 -- 172.21.15.172:0/1206433856 shutdown_connections 2023-11-28T01:07:56.145 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:55.572+0000 7f79e1f8c700 1 -- 172.21.15.172:0/1206433856 wait complete. 2023-11-28T01:07:56.146 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-11-28T01:07:56.146 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout disabled 2023-11-28T01:07:56.151 INFO:teuthology.orchestra.run.smithi172.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-11-28T01:07:56.151 INFO:teuthology.orchestra.run.smithi172.stdout:systemctl: stdout inactive 2023-11-28T01:07:56.151 INFO:teuthology.orchestra.run.smithi172.stdout:firewalld.service is not enabled 2023-11-28T01:07:56.151 INFO:teuthology.orchestra.run.smithi172.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-28T01:07:56.152 INFO:teuthology.orchestra.run.smithi172.stdout:Ceph Dashboard is now available at: 2023-11-28T01:07:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:07:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: URL: https://smithi172.front.sepia.ceph.com:8443/ 2023-11-28T01:07:56.152 INFO:teuthology.orchestra.run.smithi172.stdout: User: admin 2023-11-28T01:07:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: Password: 006dcdqqj8 2023-11-28T01:07:56.153 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:07:56.153 INFO:teuthology.orchestra.run.smithi172.stdout:Saving cluster configuration to /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config directory 2023-11-28T01:07:56.153 INFO:teuthology.orchestra.run.smithi172.stdout:Enabling autotune for osd_memory_target 2023-11-28T01:07:56.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:56 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/1206433856' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-28T01:07:57.283 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.611+0000 7f2e900e6700 1 Processor -- start 2023-11-28T01:07:57.283 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.611+0000 7f2e900e6700 1 -- start start 2023-11-28T01:07:57.283 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.611+0000 7f2e900e6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88103880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:57.283 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.611+0000 7f2e900e6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e88103e50 con 0x7f2e88103460 2023-11-28T01:07:57.283 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.611+0000 7f2e8de82700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:57.284 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.611+0000 7f2e8de82700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88103880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49436/0 (socket says 172.21.15.172:49436) 2023-11-28T01:07:57.284 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.611+0000 7f2e8de82700 1 -- 172.21.15.172:0/2391428756 learned_addr learned my addr 172.21.15.172:0/2391428756 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:57.284 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.612+0000 7f2e8de82700 1 -- 172.21.15.172:0/2391428756 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e881063f0 con 0x7f2e88103460 2023-11-28T01:07:57.285 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.612+0000 7f2e8de82700 1 --2- 172.21.15.172:0/2391428756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88103880 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e840098a0 tx=0x7f2e84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9058b5bf1eaa780 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:57.285 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.612+0000 7f2e8ce80700 1 -- 172.21.15.172:0/2391428756 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e84004160 con 0x7f2e88103460 2023-11-28T01:07:57.285 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.612+0000 7f2e8ce80700 1 -- 172.21.15.172:0/2391428756 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f2e840036a0 con 0x7f2e88103460 2023-11-28T01:07:57.285 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.612+0000 7f2e8ce80700 1 -- 172.21.15.172:0/2391428756 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e840117a0 con 0x7f2e88103460 2023-11-28T01:07:57.285 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.612+0000 7f2e900e6700 1 -- 172.21.15.172:0/2391428756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 msgr2=0x7f2e88103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:57.286 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.612+0000 7f2e900e6700 1 --2- 172.21.15.172:0/2391428756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88103880 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2e840098a0 tx=0x7f2e84009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:57.286 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.613+0000 7f2e900e6700 1 -- 172.21.15.172:0/2391428756 shutdown_connections 2023-11-28T01:07:57.286 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.613+0000 7f2e900e6700 1 --2- 172.21.15.172:0/2391428756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88103880 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:57.286 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.613+0000 7f2e900e6700 1 -- 172.21.15.172:0/2391428756 >> 172.21.15.172:0/2391428756 conn(0x7f2e880fed70 msgr2=0x7f2e881011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:57.286 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.613+0000 7f2e900e6700 1 -- 172.21.15.172:0/2391428756 shutdown_connections 2023-11-28T01:07:57.286 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.613+0000 7f2e900e6700 1 -- 172.21.15.172:0/2391428756 wait complete. 2023-11-28T01:07:57.287 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.613+0000 7f2e900e6700 1 Processor -- start 2023-11-28T01:07:57.287 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.613+0000 7f2e900e6700 1 -- start start 2023-11-28T01:07:57.287 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.613+0000 7f2e900e6700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88198e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:57.287 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.613+0000 7f2e900e6700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e88199380 con 0x7f2e88103460 2023-11-28T01:07:57.287 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.613+0000 7f2e8de82700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88198e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:57.287 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.614+0000 7f2e8de82700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88198e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49438/0 (socket says 172.21.15.172:49438) 2023-11-28T01:07:57.287 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.614+0000 7f2e8de82700 1 -- 172.21.15.172:0/847419048 learned_addr learned my addr 172.21.15.172:0/847419048 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:57.287 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.614+0000 7f2e8de82700 1 -- 172.21.15.172:0/847419048 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e84009550 con 0x7f2e88103460 2023-11-28T01:07:57.288 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.614+0000 7f2e8de82700 1 --2- 172.21.15.172:0/847419048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88198e40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e84000c00 tx=0x7f2e840119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:57.288 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.614+0000 7f2e7effd700 1 -- 172.21.15.172:0/847419048 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e840041b0 con 0x7f2e88103460 2023-11-28T01:07:57.288 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.614+0000 7f2e7effd700 1 -- 172.21.15.172:0/847419048 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f2e84004310 con 0x7f2e88103460 2023-11-28T01:07:57.288 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.614+0000 7f2e900e6700 1 -- 172.21.15.172:0/847419048 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e88199580 con 0x7f2e88103460 2023-11-28T01:07:57.288 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.614+0000 7f2e7effd700 1 -- 172.21.15.172:0/847419048 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2e8401be60 con 0x7f2e88103460 2023-11-28T01:07:57.288 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.614+0000 7f2e900e6700 1 -- 172.21.15.172:0/847419048 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e88199a20 con 0x7f2e88103460 2023-11-28T01:07:57.288 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.615+0000 7f2e7effd700 1 -- 172.21.15.172:0/847419048 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 12) v1 ==== 45732+0+0 (secure 0 0 0) 0x7f2e8401b440 con 0x7f2e88103460 2023-11-28T01:07:57.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.615+0000 7f2e900e6700 1 -- 172.21.15.172:0/847419048 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e88062c00 con 0x7f2e88103460 2023-11-28T01:07:57.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.615+0000 7f2e7effd700 1 --2- 172.21.15.172:0/847419048 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f2e74038b90 0x7f2e7403b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:57.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.615+0000 7f2e7effd700 1 -- 172.21.15.172:0/847419048 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2e8404cfc0 con 0x7f2e88103460 2023-11-28T01:07:57.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.615+0000 7f2e8d681700 1 --2- 172.21.15.172:0/847419048 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f2e74038b90 0x7f2e7403b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:57.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.616+0000 7f2e8d681700 1 --2- 172.21.15.172:0/847419048 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f2e74038b90 0x7f2e7403b050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006fd0 tx=0x7f2e78006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:57.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.620+0000 7f2e7effd700 1 -- 172.21.15.172:0/847419048 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f2e840802f0 con 0x7f2e88103460 2023-11-28T01:07:57.289 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.808+0000 7f2e900e6700 1 -- 172.21.15.172:0/847419048 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f2e88109620 con 0x7f2e88103460 2023-11-28T01:07:57.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.809+0000 7f2e7effd700 1 -- 172.21.15.172:0/847419048 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f2e8400bd70 con 0x7f2e88103460 2023-11-28T01:07:57.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.811+0000 7f2e900e6700 1 -- 172.21.15.172:0/847419048 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f2e74038b90 msgr2=0x7f2e7403b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:57.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.811+0000 7f2e900e6700 1 --2- 172.21.15.172:0/847419048 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f2e74038b90 0x7f2e7403b050 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006fd0 tx=0x7f2e78006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:57.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.811+0000 7f2e900e6700 1 -- 172.21.15.172:0/847419048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 msgr2=0x7f2e88198e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:57.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.811+0000 7f2e900e6700 1 --2- 172.21.15.172:0/847419048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88198e40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e84000c00 tx=0x7f2e840119a0 comp rx=0 tx=0).stop 2023-11-28T01:07:57.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.811+0000 7f2e900e6700 1 -- 172.21.15.172:0/847419048 shutdown_connections 2023-11-28T01:07:57.290 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.811+0000 7f2e900e6700 1 --2- 172.21.15.172:0/847419048 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f2e74038b90 0x7f2e7403b050 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:57.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.811+0000 7f2e900e6700 1 --2- 172.21.15.172:0/847419048 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2e88103460 0x7f2e88198e40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:57.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.811+0000 7f2e900e6700 1 -- 172.21.15.172:0/847419048 >> 172.21.15.172:0/847419048 conn(0x7f2e880fed70 msgr2=0x7f2e880ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:57.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.811+0000 7f2e900e6700 1 -- 172.21.15.172:0/847419048 shutdown_connections 2023-11-28T01:07:57.291 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:56.811+0000 7f2e900e6700 1 -- 172.21.15.172:0/847419048 wait complete. 2023-11-28T01:07:58.620 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.708+0000 7f4f9736f700 1 Processor -- start 2023-11-28T01:07:58.621 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.708+0000 7f4f9736f700 1 -- start start 2023-11-28T01:07:58.621 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.708+0000 7f4f9736f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90103880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:58.621 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.708+0000 7f4f9736f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f90103e50 con 0x7f4f90103460 2023-11-28T01:07:58.621 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.709+0000 7f4f9510b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:58.621 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.709+0000 7f4f9510b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90103880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49442/0 (socket says 172.21.15.172:49442) 2023-11-28T01:07:58.621 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.709+0000 7f4f9510b700 1 -- 172.21.15.172:0/2031347844 learned_addr learned my addr 172.21.15.172:0/2031347844 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:58.621 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.709+0000 7f4f9510b700 1 -- 172.21.15.172:0/2031347844 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f901063f0 con 0x7f4f90103460 2023-11-28T01:07:58.621 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.709+0000 7f4f9510b700 1 --2- 172.21.15.172:0/2031347844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90103880 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f800098a0 tx=0x7f4f80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e019663fc992e1 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:58.621 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.709+0000 7f4f87fff700 1 -- 172.21.15.172:0/2031347844 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f80004160 con 0x7f4f90103460 2023-11-28T01:07:58.621 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.709+0000 7f4f87fff700 1 -- 172.21.15.172:0/2031347844 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f4f800036a0 con 0x7f4f90103460 2023-11-28T01:07:58.621 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.709+0000 7f4f87fff700 1 -- 172.21.15.172:0/2031347844 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f800117a0 con 0x7f4f90103460 2023-11-28T01:07:58.622 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.710+0000 7f4f9736f700 1 -- 172.21.15.172:0/2031347844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 msgr2=0x7f4f90103880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:58.622 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.710+0000 7f4f9736f700 1 --2- 172.21.15.172:0/2031347844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90103880 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f800098a0 tx=0x7f4f80009bb0 comp rx=0 tx=0).stop 2023-11-28T01:07:58.622 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.710+0000 7f4f9736f700 1 -- 172.21.15.172:0/2031347844 shutdown_connections 2023-11-28T01:07:58.622 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.710+0000 7f4f9736f700 1 --2- 172.21.15.172:0/2031347844 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90103880 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:58.622 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.710+0000 7f4f9736f700 1 -- 172.21.15.172:0/2031347844 >> 172.21.15.172:0/2031347844 conn(0x7f4f900fed70 msgr2=0x7f4f901011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:58.622 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.710+0000 7f4f9736f700 1 -- 172.21.15.172:0/2031347844 shutdown_connections 2023-11-28T01:07:58.622 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.710+0000 7f4f9736f700 1 -- 172.21.15.172:0/2031347844 wait complete. 2023-11-28T01:07:58.622 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.710+0000 7f4f9736f700 1 Processor -- start 2023-11-28T01:07:58.622 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f9736f700 1 -- start start 2023-11-28T01:07:58.622 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f9736f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90105c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:58.622 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f9736f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f90106160 con 0x7f4f90103460 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f9510b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90105c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f9510b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90105c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49444/0 (socket says 172.21.15.172:49444) 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f9510b700 1 -- 172.21.15.172:0/3483592498 learned_addr learned my addr 172.21.15.172:0/3483592498 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f9510b700 1 -- 172.21.15.172:0/3483592498 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f80009550 con 0x7f4f90103460 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f9510b700 1 --2- 172.21.15.172:0/3483592498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90105c20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f80006b20 tx=0x7f4f80011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f867fc700 1 -- 172.21.15.172:0/3483592498 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f800042e0 con 0x7f4f90103460 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f867fc700 1 -- 172.21.15.172:0/3483592498 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f4f80004440 con 0x7f4f90103460 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f9736f700 1 -- 172.21.15.172:0/3483592498 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f901043a0 con 0x7f4f90103460 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f867fc700 1 -- 172.21.15.172:0/3483592498 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4f800193f0 con 0x7f4f90103460 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.711+0000 7f4f9736f700 1 -- 172.21.15.172:0/3483592498 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f90104840 con 0x7f4f90103460 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.712+0000 7f4f9736f700 1 -- 172.21.15.172:0/3483592498 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f90062c00 con 0x7f4f90103460 2023-11-28T01:07:58.623 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.712+0000 7f4f867fc700 1 -- 172.21.15.172:0/3483592498 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7f4f8001b440 con 0x7f4f90103460 2023-11-28T01:07:58.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.713+0000 7f4f867fc700 1 --2- 172.21.15.172:0/3483592498 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4f7c038c80 0x7f4f7c03b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:07:58.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.713+0000 7f4f867fc700 1 -- 172.21.15.172:0/3483592498 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4f8004cea0 con 0x7f4f90103460 2023-11-28T01:07:58.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.713+0000 7f4f9490a700 1 --2- 172.21.15.172:0/3483592498 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4f7c038c80 0x7f4f7c03b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:07:58.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.713+0000 7f4f9490a700 1 --2- 172.21.15.172:0/3483592498 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4f7c038c80 0x7f4f7c03b140 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c006fd0 tx=0x7f4f8c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:07:58.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:57.717+0000 7f4f867fc700 1 -- 172.21.15.172:0/3483592498 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f4f8002cd20 con 0x7f4f90103460 2023-11-28T01:07:58.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.010+0000 7f4f9736f700 1 -- 172.21.15.172:0/3483592498 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f4f90104b80 con 0x7f4f90103460 2023-11-28T01:07:58.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.012+0000 7f4f867fc700 1 -- 172.21.15.172:0/3483592498 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f4f8001f030 con 0x7f4f90103460 2023-11-28T01:07:58.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.014+0000 7f4f9736f700 1 -- 172.21.15.172:0/3483592498 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4f7c038c80 msgr2=0x7f4f7c03b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:58.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.014+0000 7f4f9736f700 1 --2- 172.21.15.172:0/3483592498 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4f7c038c80 0x7f4f7c03b140 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c006fd0 tx=0x7f4f8c006e40 comp rx=0 tx=0).stop 2023-11-28T01:07:58.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.014+0000 7f4f9736f700 1 -- 172.21.15.172:0/3483592498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 msgr2=0x7f4f90105c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:07:58.624 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.014+0000 7f4f9736f700 1 --2- 172.21.15.172:0/3483592498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90105c20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4f80006b20 tx=0x7f4f80011cf0 comp rx=0 tx=0).stop 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.014+0000 7f4f9736f700 1 -- 172.21.15.172:0/3483592498 shutdown_connections 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.014+0000 7f4f9736f700 1 --2- 172.21.15.172:0/3483592498 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4f7c038c80 0x7f4f7c03b140 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.014+0000 7f4f9736f700 1 --2- 172.21.15.172:0/3483592498 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4f90103460 0x7f4f90105c20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.014+0000 7f4f9736f700 1 -- 172.21.15.172:0/3483592498 >> 172.21.15.172:0/3483592498 conn(0x7f4f900fed70 msgr2=0x7f4f900ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.015+0000 7f4f9736f700 1 -- 172.21.15.172:0/3483592498 shutdown_connections 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr 2023-11-28T01:07:58.015+0000 7f4f9736f700 1 -- 172.21.15.172:0/3483592498 wait complete. 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout: sudo /sbin/cephadm shell --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:07:58.625 INFO:teuthology.orchestra.run.smithi172.stdout:Or, if you are only running a single cluster on this host: 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout: sudo /sbin/cephadm shell 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout:Please consider enabling telemetry to help improve Ceph: 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout: ceph telemetry on 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout:For more information see: 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:07:58.626 INFO:teuthology.orchestra.run.smithi172.stdout:Bootstrap complete. 2023-11-28T01:07:58.661 INFO:tasks.cephadm:Fetching config... 2023-11-28T01:07:58.661 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:07:58.661 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-28T01:07:58.678 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-28T01:07:58.679 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:07:58.679 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-28T01:07:58.737 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-28T01:07:58.737 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:07:58.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/keyring of=/dev/stdout 2023-11-28T01:07:58.801 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:58 smithi172 ceph-mon[100627]: mgrmap e13: y(active, since 6s) 2023-11-28T01:07:58.801 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:07:58 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/3483592498' entity='client.admin' 2023-11-28T01:07:58.805 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-28T01:07:58.805 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:07:58.806 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-28T01:07:58.863 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-28T01:07:58.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1em39VpKJ+k4B3HCHf6AkpnGfMaeKjpQr3ijKMh8xqa2565AiVrnR0vUa6CXjJpwPS/EqODlNIaeJS8DX+8PADG3mgNFXZXD1uHvIZ15Wv9iL8M2ZgzxExwf00780dCQEkiTqBdqXAoXlkjXTsZk5a01+E/y3hAefwsiPNSixl+mc9VTOldRKOwf+83rEMmp4l5I+pZlMViajGbVwgX8zOneHJiclodaFKqW8KzB1mBDelDmVWawe5QXBkKjrxX7QA8/L05KkM00dhmEcgsIwDTupfBf66WrmrNDaZ/XQA4easCP6lAZ5lh5SSGlggSd8U4joehzu++8dmAgtErKuPsjiDCVGSYZBv29D8SuEyC7zdn90ISKAzXFL7lcVLTQeCgTbbsGelK3jc5N8zrBRQW6hs3n7MxUQsCK9mAa5AmXwTZaMgFVCRjPOsmb8o2D/aTx8WyDI8LefPy2rXWj4QMIClW1qUNLEJ/vXto+uZKSz3lz+0YBGAVO0E2hKy3M= ceph-31861c10-8d8a-11ee-95a2-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-28T01:07:58.941 INFO:teuthology.orchestra.run.smithi172.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1em39VpKJ+k4B3HCHf6AkpnGfMaeKjpQr3ijKMh8xqa2565AiVrnR0vUa6CXjJpwPS/EqODlNIaeJS8DX+8PADG3mgNFXZXD1uHvIZ15Wv9iL8M2ZgzxExwf00780dCQEkiTqBdqXAoXlkjXTsZk5a01+E/y3hAefwsiPNSixl+mc9VTOldRKOwf+83rEMmp4l5I+pZlMViajGbVwgX8zOneHJiclodaFKqW8KzB1mBDelDmVWawe5QXBkKjrxX7QA8/L05KkM00dhmEcgsIwDTupfBf66WrmrNDaZ/XQA4easCP6lAZ5lh5SSGlggSd8U4joehzu++8dmAgtErKuPsjiDCVGSYZBv29D8SuEyC7zdn90ISKAzXFL7lcVLTQeCgTbbsGelK3jc5N8zrBRQW6hs3n7MxUQsCK9mAa5AmXwTZaMgFVCRjPOsmb8o2D/aTx8WyDI8LefPy2rXWj4QMIClW1qUNLEJ/vXto+uZKSz3lz+0YBGAVO0E2hKy3M= ceph-31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:07:58.953 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1em39VpKJ+k4B3HCHf6AkpnGfMaeKjpQr3ijKMh8xqa2565AiVrnR0vUa6CXjJpwPS/EqODlNIaeJS8DX+8PADG3mgNFXZXD1uHvIZ15Wv9iL8M2ZgzxExwf00780dCQEkiTqBdqXAoXlkjXTsZk5a01+E/y3hAefwsiPNSixl+mc9VTOldRKOwf+83rEMmp4l5I+pZlMViajGbVwgX8zOneHJiclodaFKqW8KzB1mBDelDmVWawe5QXBkKjrxX7QA8/L05KkM00dhmEcgsIwDTupfBf66WrmrNDaZ/XQA4easCP6lAZ5lh5SSGlggSd8U4joehzu++8dmAgtErKuPsjiDCVGSYZBv29D8SuEyC7zdn90ISKAzXFL7lcVLTQeCgTbbsGelK3jc5N8zrBRQW6hs3n7MxUQsCK9mAa5AmXwTZaMgFVCRjPOsmb8o2D/aTx8WyDI8LefPy2rXWj4QMIClW1qUNLEJ/vXto+uZKSz3lz+0YBGAVO0E2hKy3M= ceph-31861c10-8d8a-11ee-95a2-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-28T01:07:58.998 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1em39VpKJ+k4B3HCHf6AkpnGfMaeKjpQr3ijKMh8xqa2565AiVrnR0vUa6CXjJpwPS/EqODlNIaeJS8DX+8PADG3mgNFXZXD1uHvIZ15Wv9iL8M2ZgzxExwf00780dCQEkiTqBdqXAoXlkjXTsZk5a01+E/y3hAefwsiPNSixl+mc9VTOldRKOwf+83rEMmp4l5I+pZlMViajGbVwgX8zOneHJiclodaFKqW8KzB1mBDelDmVWawe5QXBkKjrxX7QA8/L05KkM00dhmEcgsIwDTupfBf66WrmrNDaZ/XQA4easCP6lAZ5lh5SSGlggSd8U4joehzu++8dmAgtErKuPsjiDCVGSYZBv29D8SuEyC7zdn90ISKAzXFL7lcVLTQeCgTbbsGelK3jc5N8zrBRQW6hs3n7MxUQsCK9mAa5AmXwTZaMgFVCRjPOsmb8o2D/aTx8WyDI8LefPy2rXWj4QMIClW1qUNLEJ/vXto+uZKSz3lz+0YBGAVO0E2hKy3M= ceph-31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:07:59.010 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-28T01:07:59.348 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:08:01.082 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.078+0000 7fd335e5e700 1 -- 172.21.15.172:0/2371630705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd330100f30 msgr2=0x7fd330101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:01.082 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.078+0000 7fd335e5e700 1 --2- 172.21.15.172:0/2371630705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd330100f30 0x7fd330101350 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd320009910 tx=0x7fd320009c20 comp rx=0 tx=0).stop 2023-11-28T01:08:01.082 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.078+0000 7fd335e5e700 1 -- 172.21.15.172:0/2371630705 shutdown_connections 2023-11-28T01:08:01.082 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.078+0000 7fd335e5e700 1 --2- 172.21.15.172:0/2371630705 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd330100f30 0x7fd330101350 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:01.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.078+0000 7fd335e5e700 1 -- 172.21.15.172:0/2371630705 >> 172.21.15.172:0/2371630705 conn(0x7fd3300fc4f0 msgr2=0x7fd3300fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:01.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.078+0000 7fd335e5e700 1 -- 172.21.15.172:0/2371630705 shutdown_connections 2023-11-28T01:08:01.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.078+0000 7fd335e5e700 1 -- 172.21.15.172:0/2371630705 wait complete. 2023-11-28T01:08:01.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.078+0000 7fd335e5e700 1 Processor -- start 2023-11-28T01:08:01.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.078+0000 7fd335e5e700 1 -- start start 2023-11-28T01:08:01.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd335e5e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd330100f30 0x7fd330197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:01.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd335e5e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd330198040 con 0x7fd330100f30 2023-11-28T01:08:01.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd32f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd330100f30 0x7fd330197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:01.083 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd32f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd330100f30 0x7fd330197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49450/0 (socket says 172.21.15.172:49450) 2023-11-28T01:08:01.084 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd32f7fe700 1 -- 172.21.15.172:0/3532780196 learned_addr learned my addr 172.21.15.172:0/3532780196 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:08:01.084 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd32f7fe700 1 -- 172.21.15.172:0/3532780196 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3200095f0 con 0x7fd330100f30 2023-11-28T01:08:01.084 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd32f7fe700 1 --2- 172.21.15.172:0/3532780196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd330100f30 0x7fd330197b00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd320004800 tx=0x7fd320005600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:01.084 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd32cff9700 1 -- 172.21.15.172:0/3532780196 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd32000b960 con 0x7fd330100f30 2023-11-28T01:08:01.084 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd335e5e700 1 -- 172.21.15.172:0/3532780196 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd330198240 con 0x7fd330100f30 2023-11-28T01:08:01.084 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd32cff9700 1 -- 172.21.15.172:0/3532780196 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7fd32000f460 con 0x7fd330100f30 2023-11-28T01:08:01.084 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd32cff9700 1 -- 172.21.15.172:0/3532780196 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd32000fe50 con 0x7fd330100f30 2023-11-28T01:08:01.084 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.079+0000 7fd335e5e700 1 -- 172.21.15.172:0/3532780196 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3301986e0 con 0x7fd330100f30 2023-11-28T01:08:01.086 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.080+0000 7fd32cff9700 1 -- 172.21.15.172:0/3532780196 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7fd32002c440 con 0x7fd330100f30 2023-11-28T01:08:01.086 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.080+0000 7fd335e5e700 1 -- 172.21.15.172:0/3532780196 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd330062c00 con 0x7fd330100f30 2023-11-28T01:08:01.086 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.080+0000 7fd32cff9700 1 --2- 172.21.15.172:0/3532780196 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fd318038c30 0x7fd31803b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:01.087 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.080+0000 7fd32cff9700 1 -- 172.21.15.172:0/3532780196 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd32004d220 con 0x7fd330100f30 2023-11-28T01:08:01.087 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.081+0000 7fd32effd700 1 --2- 172.21.15.172:0/3532780196 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fd318038c30 0x7fd31803b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:01.087 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.081+0000 7fd32effd700 1 --2- 172.21.15.172:0/3532780196 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fd318038c30 0x7fd31803b0f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd324006fd0 tx=0x7fd324006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:01.087 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.085+0000 7fd32cff9700 1 -- 172.21.15.172:0/3532780196 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7fd320015920 con 0x7fd330100f30 2023-11-28T01:08:01.291 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.289+0000 7fd335e5e700 1 -- 172.21.15.172:0/3532780196 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fd330106060 con 0x7fd330100f30 2023-11-28T01:08:01.294 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.292+0000 7fd32cff9700 1 -- 172.21.15.172:0/3532780196 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fd320027030 con 0x7fd330100f30 2023-11-28T01:08:01.297 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.294+0000 7fd335e5e700 1 -- 172.21.15.172:0/3532780196 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fd318038c30 msgr2=0x7fd31803b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:01.297 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.294+0000 7fd335e5e700 1 --2- 172.21.15.172:0/3532780196 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fd318038c30 0x7fd31803b0f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd324006fd0 tx=0x7fd324006e40 comp rx=0 tx=0).stop 2023-11-28T01:08:01.297 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.294+0000 7fd335e5e700 1 -- 172.21.15.172:0/3532780196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd330100f30 msgr2=0x7fd330197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:01.298 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.294+0000 7fd335e5e700 1 --2- 172.21.15.172:0/3532780196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd330100f30 0x7fd330197b00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd320004800 tx=0x7fd320005600 comp rx=0 tx=0).stop 2023-11-28T01:08:01.298 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.294+0000 7fd335e5e700 1 -- 172.21.15.172:0/3532780196 shutdown_connections 2023-11-28T01:08:01.298 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.294+0000 7fd335e5e700 1 --2- 172.21.15.172:0/3532780196 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fd318038c30 0x7fd31803b0f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:01.299 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.294+0000 7fd335e5e700 1 --2- 172.21.15.172:0/3532780196 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fd330100f30 0x7fd330197b00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:01.299 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.294+0000 7fd335e5e700 1 -- 172.21.15.172:0/3532780196 >> 172.21.15.172:0/3532780196 conn(0x7fd3300fc4f0 msgr2=0x7fd3300fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:01.299 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.295+0000 7fd335e5e700 1 -- 172.21.15.172:0/3532780196 shutdown_connections 2023-11-28T01:08:01.299 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:01.295+0000 7fd335e5e700 1 -- 172.21.15.172:0/3532780196 wait complete. 2023-11-28T01:08:01.952 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-11-28T01:08:01.952 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-11-28T01:08:02.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:02 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/3532780196' entity='client.admin' 2023-11-28T01:08:02.719 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:08:03.987 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:03 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:03.987 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:03 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:03.988 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:03 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi172", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:08:03.988 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:03 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:03.988 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:03 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:03.988 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:03 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:04.171 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.169+0000 7f7d35bdd700 1 -- 172.21.15.172:0/73334304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d30100f10 msgr2=0x7f7d30101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:04.172 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.169+0000 7f7d35bdd700 1 --2- 172.21.15.172:0/73334304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d30100f10 0x7f7d30101350 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009910 tx=0x7f7d24009c20 comp rx=0 tx=0).stop 2023-11-28T01:08:04.172 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.169+0000 7f7d35bdd700 1 -- 172.21.15.172:0/73334304 shutdown_connections 2023-11-28T01:08:04.172 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.169+0000 7f7d35bdd700 1 --2- 172.21.15.172:0/73334304 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d30100f10 0x7f7d30101350 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:04.172 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.169+0000 7f7d35bdd700 1 -- 172.21.15.172:0/73334304 >> 172.21.15.172:0/73334304 conn(0x7f7d300fc520 msgr2=0x7f7d300fe940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:04.172 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.169+0000 7f7d35bdd700 1 -- 172.21.15.172:0/73334304 shutdown_connections 2023-11-28T01:08:04.172 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.169+0000 7f7d35bdd700 1 -- 172.21.15.172:0/73334304 wait complete. 2023-11-28T01:08:04.176 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.170+0000 7f7d35bdd700 1 Processor -- start 2023-11-28T01:08:04.176 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.170+0000 7f7d35bdd700 1 -- start start 2023-11-28T01:08:04.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.170+0000 7f7d35bdd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d30100f10 0x7f7d30197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:04.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.170+0000 7f7d35bdd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30198090 con 0x7f7d30100f10 2023-11-28T01:08:04.177 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.170+0000 7f7d2f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d30100f10 0x7f7d30197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:04.178 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.170+0000 7f7d2f7fe700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d30100f10 0x7f7d30197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49456/0 (socket says 172.21.15.172:49456) 2023-11-28T01:08:04.178 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.170+0000 7f7d2f7fe700 1 -- 172.21.15.172:0/3915708812 learned_addr learned my addr 172.21.15.172:0/3915708812 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:08:04.178 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.171+0000 7f7d2f7fe700 1 -- 172.21.15.172:0/3915708812 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d240095f0 con 0x7f7d30100f10 2023-11-28T01:08:04.178 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.171+0000 7f7d2f7fe700 1 --2- 172.21.15.172:0/3915708812 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d30100f10 0x7f7d30197b50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400b5c0 tx=0x7f7d24005600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:04.178 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.171+0000 7f7d2cff9700 1 -- 172.21.15.172:0/3915708812 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d2400b960 con 0x7f7d30100f10 2023-11-28T01:08:04.178 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.171+0000 7f7d2cff9700 1 -- 172.21.15.172:0/3915708812 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f7d2400f460 con 0x7f7d30100f10 2023-11-28T01:08:04.178 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.171+0000 7f7d35bdd700 1 -- 172.21.15.172:0/3915708812 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d301982f0 con 0x7f7d30100f10 2023-11-28T01:08:04.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.171+0000 7f7d2cff9700 1 -- 172.21.15.172:0/3915708812 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d2400fda0 con 0x7f7d30100f10 2023-11-28T01:08:04.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.171+0000 7f7d35bdd700 1 -- 172.21.15.172:0/3915708812 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3019afc0 con 0x7f7d30100f10 2023-11-28T01:08:04.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.172+0000 7f7d2cff9700 1 -- 172.21.15.172:0/3915708812 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7f7d2402c440 con 0x7f7d30100f10 2023-11-28T01:08:04.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.172+0000 7f7d2cff9700 1 --2- 172.21.15.172:0/3915708812 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f7d18038c30 0x7f7d1803b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:04.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.172+0000 7f7d2cff9700 1 -- 172.21.15.172:0/3915708812 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7d2404cd70 con 0x7f7d30100f10 2023-11-28T01:08:04.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.172+0000 7f7d35bdd700 1 -- 172.21.15.172:0/3915708812 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d30191930 con 0x7f7d30100f10 2023-11-28T01:08:04.180 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.172+0000 7f7d2effd700 1 --2- 172.21.15.172:0/3915708812 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f7d18038c30 0x7f7d1803b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:04.180 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.173+0000 7f7d2effd700 1 --2- 172.21.15.172:0/3915708812 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f7d18038c30 0x7f7d1803b0f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7d20006fd0 tx=0x7f7d20006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:04.180 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.177+0000 7f7d2cff9700 1 -- 172.21.15.172:0/3915708812 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f7d24027070 con 0x7f7d30100f10 2023-11-28T01:08:04.372 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.369+0000 7f7d35bdd700 1 -- 172.21.15.172:0/3915708812 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d301061d0 con 0x7f7d18038c30 2023-11-28T01:08:04.374 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.373+0000 7f7d2cff9700 1 -- 172.21.15.172:0/3915708812 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7d301061d0 con 0x7f7d18038c30 2023-11-28T01:08:04.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.375+0000 7f7d35bdd700 1 -- 172.21.15.172:0/3915708812 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f7d18038c30 msgr2=0x7f7d1803b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:04.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.375+0000 7f7d35bdd700 1 --2- 172.21.15.172:0/3915708812 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f7d18038c30 0x7f7d1803b0f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7d20006fd0 tx=0x7f7d20006e40 comp rx=0 tx=0).stop 2023-11-28T01:08:04.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.375+0000 7f7d35bdd700 1 -- 172.21.15.172:0/3915708812 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d30100f10 msgr2=0x7f7d30197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:04.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.375+0000 7f7d35bdd700 1 --2- 172.21.15.172:0/3915708812 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d30100f10 0x7f7d30197b50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400b5c0 tx=0x7f7d24005600 comp rx=0 tx=0).stop 2023-11-28T01:08:04.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.375+0000 7f7d35bdd700 1 -- 172.21.15.172:0/3915708812 shutdown_connections 2023-11-28T01:08:04.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.375+0000 7f7d35bdd700 1 --2- 172.21.15.172:0/3915708812 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f7d18038c30 0x7f7d1803b0f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:04.380 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.375+0000 7f7d35bdd700 1 --2- 172.21.15.172:0/3915708812 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f7d30100f10 0x7f7d30197b50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:04.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.375+0000 7f7d35bdd700 1 -- 172.21.15.172:0/3915708812 >> 172.21.15.172:0/3915708812 conn(0x7f7d300fc520 msgr2=0x7f7d300fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:04.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.375+0000 7f7d35bdd700 1 -- 172.21.15.172:0/3915708812 shutdown_connections 2023-11-28T01:08:04.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:04.375+0000 7f7d35bdd700 1 -- 172.21.15.172:0/3915708812 wait complete. 2023-11-28T01:08:05.064 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2023-11-28T01:08:05.064 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T01:08:05.064 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2023-11-28T01:08:05.086 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T01:08:05.087 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-28T01:08:05.145 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2023-11-28T01:08:05.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch host add smithi192 2023-11-28T01:08:05.503 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:08:05.713 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:05 smithi172 ceph-mon[100627]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:08:05.713 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:05.714 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:05.714 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:05.714 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:05.714 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:05 smithi172 ceph-mon[100627]: Updating smithi172:/etc/ceph/ceph.conf 2023-11-28T01:08:05.714 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:05 smithi172 ceph-mon[100627]: Updating smithi172:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.conf 2023-11-28T01:08:06.788 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:06 smithi172 ceph-mon[100627]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2023-11-28T01:08:06.789 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:06 smithi172 ceph-mon[100627]: Updating smithi172:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.client.admin.keyring 2023-11-28T01:08:06.789 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:06 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:06.789 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:06 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:06.789 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:06 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:06.998 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.994+0000 7faae23da700 1 -- 172.21.15.172:0/4235066190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faadc100040 msgr2=0x7faadc100420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:06.998 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.994+0000 7faae23da700 1 --2- 172.21.15.172:0/4235066190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faadc100040 0x7faadc100420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faacc009910 tx=0x7faacc009c20 comp rx=0 tx=0).stop 2023-11-28T01:08:06.998 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.994+0000 7faae23da700 1 -- 172.21.15.172:0/4235066190 shutdown_connections 2023-11-28T01:08:06.998 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.994+0000 7faae23da700 1 --2- 172.21.15.172:0/4235066190 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faadc100040 0x7faadc100420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:06.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.994+0000 7faae23da700 1 -- 172.21.15.172:0/4235066190 >> 172.21.15.172:0/4235066190 conn(0x7faadc076a40 msgr2=0x7faadc0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:06.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.995+0000 7faae23da700 1 -- 172.21.15.172:0/4235066190 shutdown_connections 2023-11-28T01:08:06.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.995+0000 7faae23da700 1 -- 172.21.15.172:0/4235066190 wait complete. 2023-11-28T01:08:06.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.995+0000 7faae23da700 1 Processor -- start 2023-11-28T01:08:06.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.995+0000 7faae23da700 1 -- start start 2023-11-28T01:08:06.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.995+0000 7faae23da700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faadc100040 0x7faadc196bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:06.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.995+0000 7faae23da700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc069640 con 0x7faadc100040 2023-11-28T01:08:06.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.996+0000 7faadbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faadc100040 0x7faadc196bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:06.999 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.996+0000 7faadbfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faadc100040 0x7faadc196bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49462/0 (socket says 172.21.15.172:49462) 2023-11-28T01:08:07.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.996+0000 7faadbfff700 1 -- 172.21.15.172:0/371325730 learned_addr learned my addr 172.21.15.172:0/371325730 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:08:07.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.996+0000 7faadbfff700 1 -- 172.21.15.172:0/371325730 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faacc0095f0 con 0x7faadc100040 2023-11-28T01:08:07.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.996+0000 7faadbfff700 1 --2- 172.21.15.172:0/371325730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faadc100040 0x7faadc196bc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faacc00f690 tx=0x7faacc00f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:07.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.996+0000 7faad97fa700 1 -- 172.21.15.172:0/371325730 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faacc01d070 con 0x7faadc100040 2023-11-28T01:08:07.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.996+0000 7faae23da700 1 -- 172.21.15.172:0/371325730 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc197160 con 0x7faadc100040 2023-11-28T01:08:07.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.996+0000 7faad97fa700 1 -- 172.21.15.172:0/371325730 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7faacc022470 con 0x7faadc100040 2023-11-28T01:08:07.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.996+0000 7faad97fa700 1 -- 172.21.15.172:0/371325730 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faacc017440 con 0x7faadc100040 2023-11-28T01:08:07.000 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.996+0000 7faae23da700 1 -- 172.21.15.172:0/371325730 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc1975e0 con 0x7faadc100040 2023-11-28T01:08:07.001 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.997+0000 7faad97fa700 1 -- 172.21.15.172:0/371325730 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7faacc017700 con 0x7faadc100040 2023-11-28T01:08:07.002 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.997+0000 7faad97fa700 1 --2- 172.21.15.172:0/371325730 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faac4038c30 0x7faac403b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:07.002 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.997+0000 7faae23da700 1 -- 172.21.15.172:0/371325730 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc0ff110 con 0x7faadc100040 2023-11-28T01:08:07.002 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.997+0000 7faad97fa700 1 -- 172.21.15.172:0/371325730 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faacc04cc90 con 0x7faadc100040 2023-11-28T01:08:07.002 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.997+0000 7faadb7fe700 1 --2- 172.21.15.172:0/371325730 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faac4038c30 0x7faac403b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:07.002 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:06.998+0000 7faadb7fe700 1 --2- 172.21.15.172:0/371325730 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faac4038c30 0x7faac403b0f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7faad0006fd0 tx=0x7faad0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:07.004 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:07.002+0000 7faad97fa700 1 -- 172.21.15.172:0/371325730 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7faacc027070 con 0x7faadc100040 2023-11-28T01:08:07.198 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:07.194+0000 7faae23da700 1 -- 172.21.15.172:0/371325730 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}) v1 -- 0x7faadc068cd0 con 0x7faac4038c30 2023-11-28T01:08:09.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:08 smithi172 ceph-mon[100627]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:08:09.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:08 smithi172 ceph-mon[100627]: Deploying cephadm binary to smithi192 2023-11-28T01:08:10.962 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:10.960+0000 7faad97fa700 1 -- 172.21.15.172:0/371325730 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7faadc068cd0 con 0x7faac4038c30 2023-11-28T01:08:10.963 INFO:teuthology.orchestra.run.smithi172.stdout:Added host 'smithi192' with addr '172.21.15.192' 2023-11-28T01:08:10.964 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:10.963+0000 7faae23da700 1 -- 172.21.15.172:0/371325730 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faac4038c30 msgr2=0x7faac403b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:10.964 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:10.963+0000 7faae23da700 1 --2- 172.21.15.172:0/371325730 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faac4038c30 0x7faac403b0f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7faad0006fd0 tx=0x7faad0006e40 comp rx=0 tx=0).stop 2023-11-28T01:08:10.964 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:10.963+0000 7faae23da700 1 -- 172.21.15.172:0/371325730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faadc100040 msgr2=0x7faadc196bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:10.965 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:10.963+0000 7faae23da700 1 --2- 172.21.15.172:0/371325730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faadc100040 0x7faadc196bc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faacc00f690 tx=0x7faacc00f770 comp rx=0 tx=0).stop 2023-11-28T01:08:10.965 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:10.963+0000 7faae23da700 1 -- 172.21.15.172:0/371325730 shutdown_connections 2023-11-28T01:08:10.965 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:10.963+0000 7faae23da700 1 --2- 172.21.15.172:0/371325730 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7faac4038c30 0x7faac403b0f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:10.965 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:10.963+0000 7faae23da700 1 --2- 172.21.15.172:0/371325730 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7faadc100040 0x7faadc196bc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:10.965 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:10.963+0000 7faae23da700 1 -- 172.21.15.172:0/371325730 >> 172.21.15.172:0/371325730 conn(0x7faadc076a40 msgr2=0x7faadc1041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:10.965 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:10.963+0000 7faae23da700 1 -- 172.21.15.172:0/371325730 shutdown_connections 2023-11-28T01:08:10.965 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:10.963+0000 7faae23da700 1 -- 172.21.15.172:0/371325730 wait complete. 2023-11-28T01:08:11.524 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch host ls --format=json 2023-11-28T01:08:11.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:11 smithi172 ceph-mon[100627]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:11.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:11 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:11.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:11 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:11.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:11 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:11.855 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:08:12.992 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:12 smithi172 ceph-mon[100627]: Added host smithi192 2023-11-28T01:08:12.992 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:12 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:12.992 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:12 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:13.178 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.176+0000 7f5a3f62d700 1 -- 172.21.15.172:0/910091670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a38105630 msgr2=0x7f5a38105a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:13.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.176+0000 7f5a3f62d700 1 --2- 172.21.15.172:0/910091670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a38105630 0x7f5a38105a10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a34009910 tx=0x7f5a34009c20 comp rx=0 tx=0).stop 2023-11-28T01:08:13.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.176+0000 7f5a3f62d700 1 -- 172.21.15.172:0/910091670 shutdown_connections 2023-11-28T01:08:13.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.176+0000 7f5a3f62d700 1 --2- 172.21.15.172:0/910091670 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a38105630 0x7f5a38105a10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:13.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.176+0000 7f5a3f62d700 1 -- 172.21.15.172:0/910091670 >> 172.21.15.172:0/910091670 conn(0x7f5a380fb030 msgr2=0x7f5a380fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:13.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.176+0000 7f5a3f62d700 1 -- 172.21.15.172:0/910091670 shutdown_connections 2023-11-28T01:08:13.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.176+0000 7f5a3f62d700 1 -- 172.21.15.172:0/910091670 wait complete. 2023-11-28T01:08:13.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.177+0000 7f5a3f62d700 1 Processor -- start 2023-11-28T01:08:13.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.177+0000 7f5a3f62d700 1 -- start start 2023-11-28T01:08:13.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.177+0000 7f5a3f62d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a38105630 0x7f5a380fb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:13.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.177+0000 7f5a3f62d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a380fd3a0 con 0x7f5a38105630 2023-11-28T01:08:13.179 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.177+0000 7f5a3d3c9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a38105630 0x7f5a380fb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:13.180 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.177+0000 7f5a3d3c9700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a38105630 0x7f5a380fb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49488/0 (socket says 172.21.15.172:49488) 2023-11-28T01:08:13.180 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.177+0000 7f5a3d3c9700 1 -- 172.21.15.172:0/3681853562 learned_addr learned my addr 172.21.15.172:0/3681853562 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:08:13.180 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.177+0000 7f5a3d3c9700 1 -- 172.21.15.172:0/3681853562 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a340095f0 con 0x7f5a38105630 2023-11-28T01:08:13.180 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.178+0000 7f5a3d3c9700 1 --2- 172.21.15.172:0/3681853562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a38105630 0x7f5a380fb4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400f690 tx=0x7f5a3400f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:13.181 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.178+0000 7f5a2a7fc700 1 -- 172.21.15.172:0/3681853562 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a3401d070 con 0x7f5a38105630 2023-11-28T01:08:13.181 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.178+0000 7f5a2a7fc700 1 -- 172.21.15.172:0/3681853562 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f5a34022470 con 0x7f5a38105630 2023-11-28T01:08:13.181 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.178+0000 7f5a3f62d700 1 -- 172.21.15.172:0/3681853562 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a380fba20 con 0x7f5a38105630 2023-11-28T01:08:13.181 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.178+0000 7f5a2a7fc700 1 -- 172.21.15.172:0/3681853562 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a34017440 con 0x7f5a38105630 2023-11-28T01:08:13.181 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.178+0000 7f5a3f62d700 1 -- 172.21.15.172:0/3681853562 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a380fbea0 con 0x7f5a38105630 2023-11-28T01:08:13.182 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.179+0000 7f5a2a7fc700 1 -- 172.21.15.172:0/3681853562 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7f5a34017700 con 0x7f5a38105630 2023-11-28T01:08:13.182 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.179+0000 7f5a3f62d700 1 -- 172.21.15.172:0/3681853562 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a38100ca0 con 0x7f5a38105630 2023-11-28T01:08:13.182 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.179+0000 7f5a2a7fc700 1 --2- 172.21.15.172:0/3681853562 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5a24038c30 0x7f5a2403b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:13.182 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.179+0000 7f5a2a7fc700 1 -- 172.21.15.172:0/3681853562 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5a3404cc30 con 0x7f5a38105630 2023-11-28T01:08:13.182 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.179+0000 7f5a3cbc8700 1 --2- 172.21.15.172:0/3681853562 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5a24038c30 0x7f5a2403b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:13.182 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.179+0000 7f5a3cbc8700 1 --2- 172.21.15.172:0/3681853562 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5a24038c30 0x7f5a2403b0f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c006fd0 tx=0x7f5a2c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:13.184 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.184+0000 7f5a2a7fc700 1 -- 172.21.15.172:0/3681853562 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f5a3402a7d0 con 0x7f5a38105630 2023-11-28T01:08:13.378 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.377+0000 7f5a3f62d700 1 -- 172.21.15.172:0/3681853562 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a380fc140 con 0x7f5a24038c30 2023-11-28T01:08:13.379 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.378+0000 7f5a2a7fc700 1 -- 172.21.15.172:0/3681853562 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f5a380fc140 con 0x7f5a24038c30 2023-11-28T01:08:13.379 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:08:13.379 INFO:teuthology.orchestra.run.smithi172.stdout:[{"addr": "172.21.15.172", "hostname": "smithi172", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2023-11-28T01:08:13.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.380+0000 7f5a3f62d700 1 -- 172.21.15.172:0/3681853562 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5a24038c30 msgr2=0x7f5a2403b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:13.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.380+0000 7f5a3f62d700 1 --2- 172.21.15.172:0/3681853562 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5a24038c30 0x7f5a2403b0f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c006fd0 tx=0x7f5a2c006e40 comp rx=0 tx=0).stop 2023-11-28T01:08:13.381 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.380+0000 7f5a3f62d700 1 -- 172.21.15.172:0/3681853562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a38105630 msgr2=0x7f5a380fb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:13.382 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.380+0000 7f5a3f62d700 1 --2- 172.21.15.172:0/3681853562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a38105630 0x7f5a380fb4e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400f690 tx=0x7f5a3400f770 comp rx=0 tx=0).stop 2023-11-28T01:08:13.382 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.380+0000 7f5a3f62d700 1 -- 172.21.15.172:0/3681853562 shutdown_connections 2023-11-28T01:08:13.382 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.380+0000 7f5a3f62d700 1 --2- 172.21.15.172:0/3681853562 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5a24038c30 0x7f5a2403b0f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:13.382 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.380+0000 7f5a3f62d700 1 --2- 172.21.15.172:0/3681853562 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5a38105630 0x7f5a380fb4e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:13.382 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.380+0000 7f5a3f62d700 1 -- 172.21.15.172:0/3681853562 >> 172.21.15.172:0/3681853562 conn(0x7f5a380fb030 msgr2=0x7f5a38104980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:13.382 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.380+0000 7f5a3f62d700 1 -- 172.21.15.172:0/3681853562 shutdown_connections 2023-11-28T01:08:13.382 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:13.380+0000 7f5a3f62d700 1 -- 172.21.15.172:0/3681853562 wait complete. 2023-11-28T01:08:13.941 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-28T01:08:13.942 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph osd crush tunables default 2023-11-28T01:08:14.268 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:08:14.307 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:13 smithi172 ceph-mon[100627]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:15.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:14 smithi172 ceph-mon[100627]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-28T01:08:15.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.608+0000 7f5db0d0c700 1 -- 172.21.15.172:0/1715889714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dac101140 msgr2=0x7f5dac101520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:15.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.608+0000 7f5db0d0c700 1 --2- 172.21.15.172:0/1715889714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dac101140 0x7f5dac101520 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5d94009910 tx=0x7f5d94009c20 comp rx=0 tx=0).stop 2023-11-28T01:08:15.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.609+0000 7f5db0d0c700 1 -- 172.21.15.172:0/1715889714 shutdown_connections 2023-11-28T01:08:15.610 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.609+0000 7f5db0d0c700 1 --2- 172.21.15.172:0/1715889714 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dac101140 0x7f5dac101520 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.609+0000 7f5db0d0c700 1 -- 172.21.15.172:0/1715889714 >> 172.21.15.172:0/1715889714 conn(0x7f5dac0fc9d0 msgr2=0x7f5dac0fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.609+0000 7f5db0d0c700 1 -- 172.21.15.172:0/1715889714 shutdown_connections 2023-11-28T01:08:15.611 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.609+0000 7f5db0d0c700 1 -- 172.21.15.172:0/1715889714 wait complete. 2023-11-28T01:08:15.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.610+0000 7f5db0d0c700 1 Processor -- start 2023-11-28T01:08:15.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.610+0000 7f5db0d0c700 1 -- start start 2023-11-28T01:08:15.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.610+0000 7f5db0d0c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dac101140 0x7f5dac19b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:15.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.610+0000 7f5db0d0c700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac19be70 con 0x7f5dac101140 2023-11-28T01:08:15.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.610+0000 7f5daa59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dac101140 0x7f5dac19b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:15.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.610+0000 7f5daa59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dac101140 0x7f5dac19b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49494/0 (socket says 172.21.15.172:49494) 2023-11-28T01:08:15.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.610+0000 7f5daa59c700 1 -- 172.21.15.172:0/71802052 learned_addr learned my addr 172.21.15.172:0/71802052 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:08:15.612 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.610+0000 7f5daa59c700 1 -- 172.21.15.172:0/71802052 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d940095f0 con 0x7f5dac101140 2023-11-28T01:08:15.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.610+0000 7f5daa59c700 1 --2- 172.21.15.172:0/71802052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dac101140 0x7f5dac19b930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400f690 tx=0x7f5d9400f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:15.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.611+0000 7f5da37fe700 1 -- 172.21.15.172:0/71802052 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d9401d070 con 0x7f5dac101140 2023-11-28T01:08:15.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.611+0000 7f5db0d0c700 1 -- 172.21.15.172:0/71802052 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dac19c070 con 0x7f5dac101140 2023-11-28T01:08:15.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.611+0000 7f5db0d0c700 1 -- 172.21.15.172:0/71802052 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dac19c490 con 0x7f5dac101140 2023-11-28T01:08:15.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.611+0000 7f5da37fe700 1 -- 172.21.15.172:0/71802052 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f5d94022470 con 0x7f5dac101140 2023-11-28T01:08:15.613 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.611+0000 7f5da37fe700 1 -- 172.21.15.172:0/71802052 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5d94017440 con 0x7f5dac101140 2023-11-28T01:08:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.611+0000 7f5da37fe700 1 -- 172.21.15.172:0/71802052 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7f5d940176b0 con 0x7f5dac101140 2023-11-28T01:08:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.611+0000 7f5db0d0c700 1 -- 172.21.15.172:0/71802052 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8c0050e0 con 0x7f5dac101140 2023-11-28T01:08:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.612+0000 7f5da37fe700 1 --2- 172.21.15.172:0/71802052 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5d98038c80 0x7f5d9803b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.612+0000 7f5da37fe700 1 -- 172.21.15.172:0/71802052 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5d9404cc70 con 0x7f5dac101140 2023-11-28T01:08:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.612+0000 7f5da9d9b700 1 --2- 172.21.15.172:0/71802052 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5d98038c80 0x7f5d9803b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:15.615 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.612+0000 7f5da9d9b700 1 --2- 172.21.15.172:0/71802052 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5d98038c80 0x7f5d9803b140 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c006fd0 tx=0x7f5d9c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:15.617 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.616+0000 7f5da37fe700 1 -- 172.21.15.172:0/71802052 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f5d9402adc0 con 0x7f5dac101140 2023-11-28T01:08:15.814 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:15.813+0000 7f5db0d0c700 1 -- 172.21.15.172:0/71802052 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5d8c005390 con 0x7f5dac101140 2023-11-28T01:08:16.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:15 smithi172 ceph-mon[100627]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:16.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:15 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:16.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:15 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:16.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:15 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:16.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:15 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:16.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:15 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:08:16.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:15 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:16.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:15 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:16.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:15 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/71802052' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-28T01:08:16.808 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:16.807+0000 7f5da37fe700 1 -- 172.21.15.172:0/71802052 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5d94027030 con 0x7f5dac101140 2023-11-28T01:08:16.811 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:16.809+0000 7f5db0d0c700 1 -- 172.21.15.172:0/71802052 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5d98038c80 msgr2=0x7f5d9803b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:16.811 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:16.809+0000 7f5db0d0c700 1 --2- 172.21.15.172:0/71802052 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5d98038c80 0x7f5d9803b140 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c006fd0 tx=0x7f5d9c006e40 comp rx=0 tx=0).stop 2023-11-28T01:08:16.811 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:16.809+0000 7f5db0d0c700 1 -- 172.21.15.172:0/71802052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dac101140 msgr2=0x7f5dac19b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:16.811 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:16.809+0000 7f5db0d0c700 1 --2- 172.21.15.172:0/71802052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dac101140 0x7f5dac19b930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400f690 tx=0x7f5d9400f770 comp rx=0 tx=0).stop 2023-11-28T01:08:16.811 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:16.809+0000 7f5db0d0c700 1 -- 172.21.15.172:0/71802052 shutdown_connections 2023-11-28T01:08:16.811 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:16.809+0000 7f5db0d0c700 1 --2- 172.21.15.172:0/71802052 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5d98038c80 0x7f5d9803b140 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:16.811 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:16.809+0000 7f5db0d0c700 1 --2- 172.21.15.172:0/71802052 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f5dac101140 0x7f5dac19b930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:16.811 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:16.810+0000 7f5db0d0c700 1 -- 172.21.15.172:0/71802052 >> 172.21.15.172:0/71802052 conn(0x7f5dac0fc9d0 msgr2=0x7f5dac107cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:16.811 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:16.810+0000 7f5db0d0c700 1 -- 172.21.15.172:0/71802052 shutdown_connections 2023-11-28T01:08:16.811 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:16.810+0000 7f5db0d0c700 1 -- 172.21.15.172:0/71802052 wait complete. 2023-11-28T01:08:16.812 INFO:teuthology.orchestra.run.smithi172.stderr:adjusted tunables profile to default 2023-11-28T01:08:17.302 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:16 smithi172 ceph-mon[100627]: Updating smithi192:/etc/ceph/ceph.conf 2023-11-28T01:08:17.302 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:17.302 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:17.302 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:17.302 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:16 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/71802052' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-28T01:08:17.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:16 smithi172 ceph-mon[100627]: osdmap e4: 0 total, 0 up, 0 in 2023-11-28T01:08:17.325 INFO:tasks.cephadm:Adding mon.a on smithi172 2023-11-28T01:08:17.325 INFO:tasks.cephadm:Adding mon.c on smithi172 2023-11-28T01:08:17.325 INFO:tasks.cephadm:Adding mon.b on smithi192 2023-11-28T01:08:17.325 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch apply mon '3;smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi192:172.21.15.192=b' 2023-11-28T01:08:17.706 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.conf 2023-11-28T01:08:18.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:17 smithi172 ceph-mon[100627]: Updating smithi192:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.conf 2023-11-28T01:08:18.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:17 smithi172 ceph-mon[100627]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2023-11-28T01:08:18.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:17 smithi172 ceph-mon[100627]: Updating smithi192:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.client.admin.keyring 2023-11-28T01:08:18.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:17 smithi172 ceph-mon[100627]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:19.206 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.204+0000 7fc7f17bd700 1 -- 172.21.15.192:0/1716322013 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7ec101150 msgr2=0x7fc7ec101530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:19.206 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.204+0000 7fc7f17bd700 1 --2- 172.21.15.192:0/1716322013 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7ec101150 0x7fc7ec101530 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009910 tx=0x7fc7e0009c20 comp rx=0 tx=0).stop 2023-11-28T01:08:19.206 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.204+0000 7fc7f17bd700 1 -- 172.21.15.192:0/1716322013 shutdown_connections 2023-11-28T01:08:19.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.204+0000 7fc7f17bd700 1 --2- 172.21.15.192:0/1716322013 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7ec101150 0x7fc7ec101530 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:19.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.204+0000 7fc7f17bd700 1 -- 172.21.15.192:0/1716322013 >> 172.21.15.192:0/1716322013 conn(0x7fc7ec0fca00 msgr2=0x7fc7ec0fee20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:19.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.204+0000 7fc7f17bd700 1 -- 172.21.15.192:0/1716322013 shutdown_connections 2023-11-28T01:08:19.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.204+0000 7fc7f17bd700 1 -- 172.21.15.192:0/1716322013 wait complete. 2023-11-28T01:08:19.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.204+0000 7fc7f17bd700 1 Processor -- start 2023-11-28T01:08:19.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.205+0000 7fc7f17bd700 1 -- start start 2023-11-28T01:08:19.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.205+0000 7fc7f17bd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7ec101150 0x7fc7ec195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:19.207 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.205+0000 7fc7f17bd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec196230 con 0x7fc7ec101150 2023-11-28T01:08:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.205+0000 7fc7eaffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7ec101150 0x7fc7ec195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.205+0000 7fc7eaffd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7ec101150 0x7fc7ec195cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.192:35274/0 (socket says 172.21.15.192:35274) 2023-11-28T01:08:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.205+0000 7fc7eaffd700 1 -- 172.21.15.192:0/717379266 learned_addr learned my addr 172.21.15.192:0/717379266 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:08:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.205+0000 7fc7eaffd700 1 -- 172.21.15.192:0/717379266 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e00095f0 con 0x7fc7ec101150 2023-11-28T01:08:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.205+0000 7fc7eaffd700 1 --2- 172.21.15.192:0/717379266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7ec101150 0x7fc7ec195cf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000f690 tx=0x7fc7e000f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.206+0000 7fc7d3fff700 1 -- 172.21.15.192:0/717379266 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7e001d070 con 0x7fc7ec101150 2023-11-28T01:08:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.206+0000 7fc7f17bd700 1 -- 172.21.15.192:0/717379266 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ec196430 con 0x7fc7ec101150 2023-11-28T01:08:19.209 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.206+0000 7fc7d3fff700 1 -- 172.21.15.192:0/717379266 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7fc7e0022470 con 0x7fc7ec101150 2023-11-28T01:08:19.210 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.206+0000 7fc7d3fff700 1 -- 172.21.15.192:0/717379266 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7e0017440 con 0x7fc7ec101150 2023-11-28T01:08:19.210 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.206+0000 7fc7f17bd700 1 -- 172.21.15.192:0/717379266 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ec196850 con 0x7fc7ec101150 2023-11-28T01:08:19.210 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.207+0000 7fc7d3fff700 1 -- 172.21.15.192:0/717379266 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7fc7e000fb90 con 0x7fc7ec101150 2023-11-28T01:08:19.210 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.207+0000 7fc7f17bd700 1 -- 172.21.15.192:0/717379266 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ec0ff320 con 0x7fc7ec101150 2023-11-28T01:08:19.210 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.207+0000 7fc7d3fff700 1 --2- 172.21.15.192:0/717379266 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc7d4038c30 0x7fc7d403b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:19.210 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.207+0000 7fc7d3fff700 1 -- 172.21.15.192:0/717379266 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc7e004cd80 con 0x7fc7ec101150 2023-11-28T01:08:19.210 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.207+0000 7fc7ea7fc700 1 --2- 172.21.15.192:0/717379266 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc7d4038c30 0x7fc7d403b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:19.210 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.207+0000 7fc7ea7fc700 1 --2- 172.21.15.192:0/717379266 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc7d4038c30 0x7fc7d403b0f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc006fd0 tx=0x7fc7dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:19.213 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.212+0000 7fc7d3fff700 1 -- 172.21.15.192:0/717379266 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7fc7e002a590 con 0x7fc7ec101150 2023-11-28T01:08:19.412 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.410+0000 7fc7f17bd700 1 -- 172.21.15.192:0/717379266 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi192:172.21.15.192=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7ec196fc0 con 0x7fc7d4038c30 2023-11-28T01:08:19.419 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.417+0000 7fc7d3fff700 1 -- 172.21.15.192:0/717379266 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc7ec196fc0 con 0x7fc7d4038c30 2023-11-28T01:08:19.419 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2023-11-28T01:08:19.420 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.419+0000 7fc7f17bd700 1 -- 172.21.15.192:0/717379266 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc7d4038c30 msgr2=0x7fc7d403b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.419+0000 7fc7f17bd700 1 --2- 172.21.15.192:0/717379266 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc7d4038c30 0x7fc7d403b0f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc006fd0 tx=0x7fc7dc006e40 comp rx=0 tx=0).stop 2023-11-28T01:08:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.419+0000 7fc7f17bd700 1 -- 172.21.15.192:0/717379266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7ec101150 msgr2=0x7fc7ec195cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.419+0000 7fc7f17bd700 1 --2- 172.21.15.192:0/717379266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7ec101150 0x7fc7ec195cf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc7e000f690 tx=0x7fc7e000f770 comp rx=0 tx=0).stop 2023-11-28T01:08:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.419+0000 7fc7f17bd700 1 -- 172.21.15.192:0/717379266 shutdown_connections 2023-11-28T01:08:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.419+0000 7fc7f17bd700 1 --2- 172.21.15.192:0/717379266 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc7d4038c30 0x7fc7d403b0f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.419+0000 7fc7f17bd700 1 --2- 172.21.15.192:0/717379266 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc7ec101150 0x7fc7ec195cf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.419+0000 7fc7f17bd700 1 -- 172.21.15.192:0/717379266 >> 172.21.15.192:0/717379266 conn(0x7fc7ec0fca00 msgr2=0x7fc7ec107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.419+0000 7fc7f17bd700 1 -- 172.21.15.192:0/717379266 shutdown_connections 2023-11-28T01:08:19.421 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:19.419+0000 7fc7f17bd700 1 -- 172.21.15.192:0/717379266 wait complete. 2023-11-28T01:08:20.088 DEBUG:teuthology.orchestra.run.smithi172:mon.c> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.c.service 2023-11-28T01:08:20.091 DEBUG:teuthology.orchestra.run.smithi192:mon.b> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.b.service 2023-11-28T01:08:20.093 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-28T01:08:20.093 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-28T01:08:20.124 INFO:journalctl@ceph.mon.b.smithi192.stdout:-- Logs begin at Tue 2023-11-28 00:47:11 UTC. -- 2023-11-28T01:08:20.128 INFO:journalctl@ceph.mon.c.smithi172.stdout:-- Logs begin at Tue 2023-11-28 00:47:41 UTC. -- 2023-11-28T01:08:20.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:19 smithi172 ceph-mon[100627]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:20.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:19 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:20.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:19 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:20.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:19 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:20.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:19 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:20.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:19 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:20.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:19 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:20.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:19 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:21.116 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.conf 2023-11-28T01:08:21.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:20 smithi172 ceph-mon[100627]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi192:172.21.15.192=b", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:08:21.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:20 smithi172 ceph-mon[100627]: Saving service mon spec with placement smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi192:172.21.15.192=b;count:3 2023-11-28T01:08:21.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:20 smithi172 ceph-mon[100627]: Deploying daemon mon.b on smithi192 2023-11-28T01:08:22.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:21 smithi172 ceph-mon[100627]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:23.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.128+0000 7f5613ab7700 1 -- 172.21.15.192:0/2928061428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f560c101140 msgr2=0x7f560c101520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:23.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.128+0000 7f5613ab7700 1 --2- 172.21.15.192:0/2928061428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f560c101140 0x7f560c101520 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f55fc009910 tx=0x7f55fc009c20 comp rx=0 tx=0).stop 2023-11-28T01:08:23.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.128+0000 7f5613ab7700 1 -- 172.21.15.192:0/2928061428 shutdown_connections 2023-11-28T01:08:23.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.128+0000 7f5613ab7700 1 --2- 172.21.15.192:0/2928061428 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f560c101140 0x7f560c101520 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:23.133 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.128+0000 7f5613ab7700 1 -- 172.21.15.192:0/2928061428 >> 172.21.15.192:0/2928061428 conn(0x7f560c0fc9d0 msgr2=0x7f560c0fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:23.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.128+0000 7f5613ab7700 1 -- 172.21.15.192:0/2928061428 shutdown_connections 2023-11-28T01:08:23.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.128+0000 7f5613ab7700 1 -- 172.21.15.192:0/2928061428 wait complete. 2023-11-28T01:08:23.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.128+0000 7f5613ab7700 1 Processor -- start 2023-11-28T01:08:23.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.128+0000 7f5613ab7700 1 -- start start 2023-11-28T01:08:23.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5613ab7700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f560c101140 0x7f560c19b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:23.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5613ab7700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f560c19bee0 con 0x7f560c101140 2023-11-28T01:08:23.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5611853700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f560c101140 0x7f560c19b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:23.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5611853700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f560c101140 0x7f560c19b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.192:35280/0 (socket says 172.21.15.192:35280) 2023-11-28T01:08:23.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5611853700 1 -- 172.21.15.192:0/380442089 learned_addr learned my addr 172.21.15.192:0/380442089 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:08:23.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5611853700 1 -- 172.21.15.192:0/380442089 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55fc0095f0 con 0x7f560c101140 2023-11-28T01:08:23.134 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5611853700 1 --2- 172.21.15.192:0/380442089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f560c101140 0x7f560c19b9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00f690 tx=0x7f55fc00f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:23.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5602ffd700 1 -- 172.21.15.192:0/380442089 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55fc01d070 con 0x7f560c101140 2023-11-28T01:08:23.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5613ab7700 1 -- 172.21.15.192:0/380442089 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f560c19c0e0 con 0x7f560c101140 2023-11-28T01:08:23.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5602ffd700 1 -- 172.21.15.192:0/380442089 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f55fc022470 con 0x7f560c101140 2023-11-28T01:08:23.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5602ffd700 1 -- 172.21.15.192:0/380442089 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55fc017440 con 0x7f560c101140 2023-11-28T01:08:23.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.129+0000 7f5613ab7700 1 -- 172.21.15.192:0/380442089 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f560c19c500 con 0x7f560c101140 2023-11-28T01:08:23.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.130+0000 7f5602ffd700 1 -- 172.21.15.192:0/380442089 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7f55fc00fb90 con 0x7f560c101140 2023-11-28T01:08:23.135 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.130+0000 7f5613ab7700 1 -- 172.21.15.192:0/380442089 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f560c102620 con 0x7f560c101140 2023-11-28T01:08:23.136 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.131+0000 7f5602ffd700 1 --2- 172.21.15.192:0/380442089 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f55f8038c30 0x7f55f803b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:23.136 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.131+0000 7f5602ffd700 1 -- 172.21.15.192:0/380442089 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f55fc04ccc0 con 0x7f560c101140 2023-11-28T01:08:23.136 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.131+0000 7f5611052700 1 --2- 172.21.15.192:0/380442089 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f55f8038c30 0x7f55f803b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:23.136 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.131+0000 7f5611052700 1 --2- 172.21.15.192:0/380442089 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f55f8038c30 0x7f55f803b0f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5608006fd0 tx=0x7f5608006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:23.138 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.136+0000 7f5602ffd700 1 -- 172.21.15.192:0/380442089 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f55fc02a590 con 0x7f560c101140 2023-11-28T01:08:23.435 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.433+0000 7f5613ab7700 1 -- 172.21.15.192:0/380442089 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f560c196fe0 con 0x7f560c101140 2023-11-28T01:08:23.435 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.433+0000 7f5602ffd700 1 -- 172.21.15.192:0/380442089 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+749 (secure 0 0 0) 0x7f55fc027020 con 0x7f560c101140 2023-11-28T01:08:23.436 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:08:23.436 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"31861c10-8d8a-11ee-95a2-87774f69a715","modified":"2023-11-28T01:06:55.519247Z","created":"2023-11-28T01:06:55.519247Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-28T01:08:23.438 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.436+0000 7f5613ab7700 1 -- 172.21.15.192:0/380442089 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f55f8038c30 msgr2=0x7f55f803b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:23.438 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.436+0000 7f5613ab7700 1 --2- 172.21.15.192:0/380442089 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f55f8038c30 0x7f55f803b0f0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5608006fd0 tx=0x7f5608006e40 comp rx=0 tx=0).stop 2023-11-28T01:08:23.438 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.436+0000 7f5613ab7700 1 -- 172.21.15.192:0/380442089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f560c101140 msgr2=0x7f560c19b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:23.438 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.436+0000 7f5613ab7700 1 --2- 172.21.15.192:0/380442089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f560c101140 0x7f560c19b9a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f55fc00f690 tx=0x7f55fc00f770 comp rx=0 tx=0).stop 2023-11-28T01:08:23.438 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.436+0000 7f5613ab7700 1 -- 172.21.15.192:0/380442089 shutdown_connections 2023-11-28T01:08:23.438 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.436+0000 7f5613ab7700 1 --2- 172.21.15.192:0/380442089 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f55f8038c30 0x7f55f803b0f0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:23.438 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.436+0000 7f5613ab7700 1 --2- 172.21.15.192:0/380442089 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f560c101140 0x7f560c19b9a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:23.438 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.436+0000 7f5613ab7700 1 -- 172.21.15.192:0/380442089 >> 172.21.15.192:0/380442089 conn(0x7f560c0fc9d0 msgr2=0x7f560c107cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:23.438 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.436+0000 7f5613ab7700 1 -- 172.21.15.192:0/380442089 shutdown_connections 2023-11-28T01:08:23.438 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:23.436+0000 7f5613ab7700 1 -- 172.21.15.192:0/380442089 wait complete. 2023-11-28T01:08:23.439 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2023-11-28T01:08:23.679 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:23 smithi192 ceph-mon[105371]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-11-28T01:08:25.158 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-28T01:08:25.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-28T01:08:25.501 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:08:28.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:08:28.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: mon.a calling monitor election 2023-11-28T01:08:28.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: mon.b calling monitor election 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: monmap e2: 2 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: fsmap 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: osdmap e4: 0 total, 0 up, 0 in 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: mgrmap e13: y(active, since 37s) 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: overall HEALTH_OK 2023-11-28T01:08:28.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:28.805 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:28.805 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:28.805 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:28.805 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:28.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:08:28.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: mon.a calling monitor election 2023-11-28T01:08:28.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: mon.b calling monitor election 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: monmap e2: 2 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: fsmap 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: osdmap e4: 0 total, 0 up, 0 in 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: mgrmap e13: y(active, since 37s) 2023-11-28T01:08:28.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: overall HEALTH_OK 2023-11-28T01:08:28.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:28.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:28.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:28.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:28.823 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:29.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:29 smithi172 ceph-mon[100627]: Deploying daemon mon.c on smithi172 2023-11-28T01:08:29.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:29 smithi172 ceph-mon[100627]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:29.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:29.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:29 smithi192 ceph-mon[105371]: Deploying daemon mon.c on smithi172 2023-11-28T01:08:29.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:29 smithi192 ceph-mon[105371]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:29.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:29.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.870+0000 7f51a4c31700 1 -- 172.21.15.192:0/2854147961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51880035c0 msgr2=0x7f5188005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:29.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.870+0000 7f51a4c31700 1 --2- 172.21.15.192:0/2854147961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51880035c0 0x7f5188005a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f51a00ffe40 tx=0x7f519400feb0 comp rx=0 tx=0).stop 2023-11-28T01:08:29.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.870+0000 7f51a4c31700 1 -- 172.21.15.192:0/2854147961 shutdown_connections 2023-11-28T01:08:29.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.870+0000 7f51a4c31700 1 --2- 172.21.15.192:0/2854147961 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51880035c0 0x7f5188005a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:29.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.870+0000 7f51a4c31700 1 -- 172.21.15.192:0/2854147961 >> 172.21.15.192:0/2854147961 conn(0x7f51a0076bb0 msgr2=0x7f51a00fde50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:29.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.870+0000 7f51a4c31700 1 -- 172.21.15.192:0/2854147961 shutdown_connections 2023-11-28T01:08:29.872 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.870+0000 7f51a4c31700 1 -- 172.21.15.192:0/2854147961 wait complete. 2023-11-28T01:08:29.873 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f51a4c31700 1 Processor -- start 2023-11-28T01:08:29.873 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f51a4c31700 1 -- start start 2023-11-28T01:08:29.874 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f51a4c31700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51880035c0 0x7f51a0101c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f51a4c31700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a0102150 0x7f51a01025d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f51a4c31700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a0103ad0 con 0x7f51880035c0 2023-11-28T01:08:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f51a4c31700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51a01a2080 con 0x7f51a0102150 2023-11-28T01:08:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f519dd9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a0102150 0x7f51a01025d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f519e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51880035c0 0x7f51a0101c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f519e59c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51880035c0 0x7f51a0101c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.192:35312/0 (socket says 172.21.15.192:35312) 2023-11-28T01:08:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f519e59c700 1 -- 172.21.15.192:0/327472976 learned_addr learned my addr 172.21.15.192:0/327472976 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:08:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f519dd9b700 1 -- 172.21.15.192:0/327472976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a0102150 msgr2=0x7f51a01025d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-11-28T01:08:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f519dd9b700 1 -- 172.21.15.192:0/327472976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a0102150 msgr2=0x7f51a01025d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T01:08:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f519dd9b700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a0102150 0x7f51a01025d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T01:08:29.875 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f519dd9b700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a0102150 0x7f51a01025d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T01:08:29.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f519e59c700 1 -- 172.21.15.192:0/327472976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a0102150 msgr2=0x7f51a01025d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T01:08:29.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f519e59c700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a0102150 0x7f51a01025d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:29.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.872+0000 7f519e59c700 1 -- 172.21.15.192:0/327472976 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519400f900 con 0x7f51880035c0 2023-11-28T01:08:29.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.873+0000 7f519e59c700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51880035c0 0x7f51a0101c10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5194001eb0 tx=0x7f5194005260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:29.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.873+0000 7f518f7fe700 1 -- 172.21.15.192:0/327472976 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f5194013960 con 0x7f51880035c0 2023-11-28T01:08:29.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.873+0000 7f518f7fe700 1 -- 172.21.15.192:0/327472976 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f5194004030 con 0x7f51880035c0 2023-11-28T01:08:29.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.873+0000 7f51a4c31700 1 -- 172.21.15.192:0/327472976 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51a01a21c0 con 0x7f51880035c0 2023-11-28T01:08:29.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.873+0000 7f518f7fe700 1 -- 172.21.15.192:0/327472976 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f51940039c0 con 0x7f51880035c0 2023-11-28T01:08:29.876 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.873+0000 7f51a4c31700 1 -- 172.21.15.192:0/327472976 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51a01a2690 con 0x7f51880035c0 2023-11-28T01:08:29.877 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.874+0000 7f518f7fe700 1 -- 172.21.15.192:0/327472976 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7f5194003c30 con 0x7f51880035c0 2023-11-28T01:08:29.877 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.874+0000 7f518f7fe700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5180038a70 0x7f518003af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:29.877 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.874+0000 7f518f7fe700 1 -- 172.21.15.192:0/327472976 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5194000e40 con 0x7f51880035c0 2023-11-28T01:08:29.877 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.874+0000 7f51a4c31700 1 -- 172.21.15.192:0/327472976 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51a0101650 con 0x7f51880035c0 2023-11-28T01:08:29.877 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.874+0000 7f519dd9b700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5180038a70 0x7f518003af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:29.877 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.875+0000 7f519dd9b700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5180038a70 0x7f518003af30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5190009850 tx=0x7f5190008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:29.881 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:29.880+0000 7f518f7fe700 1 -- 172.21.15.192:0/327472976 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f519401a980 con 0x7f51880035c0 2023-11-28T01:08:30.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.149+0000 7f51a4c31700 1 -- 172.21.15.192:0/327472976 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f51a0062c00 con 0x7f51880035c0 2023-11-28T01:08:30.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.151+0000 7f518f7fe700 1 -- 172.21.15.192:0/327472976 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7f519401a390 con 0x7f51880035c0 2023-11-28T01:08:30.152 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:08:30.152 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"31861c10-8d8a-11ee-95a2-87774f69a715","modified":"2023-11-28T01:08:23.439820Z","created":"2023-11-28T01:06:55.519247Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-11-28T01:08:30.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.152+0000 7f51a4c31700 1 -- 172.21.15.192:0/327472976 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5180038a70 msgr2=0x7f518003af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:30.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.152+0000 7f51a4c31700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5180038a70 0x7f518003af30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5190009850 tx=0x7f5190008040 comp rx=0 tx=0).stop 2023-11-28T01:08:30.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.152+0000 7f51a4c31700 1 -- 172.21.15.192:0/327472976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51880035c0 msgr2=0x7f51a0101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:30.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.152+0000 7f51a4c31700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51880035c0 0x7f51a0101c10 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5194001eb0 tx=0x7f5194005260 comp rx=0 tx=0).stop 2023-11-28T01:08:30.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.152+0000 7f51a4c31700 1 -- 172.21.15.192:0/327472976 shutdown_connections 2023-11-28T01:08:30.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.152+0000 7f51a4c31700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f5180038a70 0x7f518003af30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:30.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.152+0000 7f51a4c31700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f51880035c0 0x7f51a0101c10 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:30.154 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.152+0000 7f51a4c31700 1 --2- 172.21.15.192:0/327472976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51a0102150 0x7f51a01025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:30.155 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.152+0000 7f51a4c31700 1 -- 172.21.15.192:0/327472976 >> 172.21.15.192:0/327472976 conn(0x7f51a0076bb0 msgr2=0x7f51a00fc620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:30.155 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.152+0000 7f51a4c31700 1 -- 172.21.15.192:0/327472976 shutdown_connections 2023-11-28T01:08:30.155 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:30.152+0000 7f51a4c31700 1 -- 172.21.15.192:0/327472976 wait complete. 2023-11-28T01:08:30.155 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2023-11-28T01:08:30.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:30 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/327472976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-28T01:08:30.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:30 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/327472976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-28T01:08:31.553 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 systemd[1]: Starting Ceph mon.c for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:08:31.772 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-11-28T01:08:31.773 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-11-28T01:08:31.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:31 smithi192 ceph-mon[105371]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:32.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[100627]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:32.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: set uid:gid to 167:167 (ceph:ceph) 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: ceph version 18.0.0-7498-g369173db (369173db14b6995b2bd07c60ec5f63d01cf21631) squid (dev), process ceph-mon, pid 7 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: pidfile_write: ignore empty --pid-file 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: load: jerasure load: lrc 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: RocksDB version: 7.9.2 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Git sha 0 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Compile date 2023-11-22 22:33:43 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: DB SUMMARY 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: DB Session ID: FI3NEMGJT6ZAGJY1TFK5 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: CURRENT file: CURRENT 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: IDENTITY file: IDENTITY 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-11-28T01:08:32.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.error_if_exists: 0 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.create_if_missing: 0 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.paranoid_checks: 1 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.flush_verify_memtable_count: 1 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.env: 0x5606a9061a60 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.fs: PosixFileSystem 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.info_log: 0x5606a9c9d740 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_file_opening_threads: 16 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.statistics: (nil) 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.use_fsync: 0 2023-11-28T01:08:32.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_log_file_size: 0 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.log_file_time_to_roll: 0 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.keep_log_file_num: 1000 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.recycle_log_file_num: 0 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.allow_fallocate: 1 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.allow_mmap_reads: 0 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.allow_mmap_writes: 0 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.use_direct_reads: 0 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.create_missing_column_families: 0 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.db_log_dir: 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.wal_dir: 2023-11-28T01:08:32.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.table_cache_numshardbits: 6 2023-11-28T01:08:32.058 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.WAL_ttl_seconds: 0 2023-11-28T01:08:32.058 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.WAL_size_limit_MB: 0 2023-11-28T01:08:32.058 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-28T01:08:32.058 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-28T01:08:32.058 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.is_fd_close_on_exec: 1 2023-11-28T01:08:32.058 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.advise_random_on_open: 1 2023-11-28T01:08:32.058 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.db_write_buffer_size: 0 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.write_buffer_manager: 0x5606a9c514a0 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.use_adaptive_mutex: 0 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.rate_limiter: (nil) 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.wal_recovery_mode: 2 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.enable_thread_tracking: 0 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.enable_pipelined_write: 0 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.unordered_write: 0 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-28T01:08:32.059 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.row_cache: None 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.wal_filter: None 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.allow_ingest_behind: 0 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.two_write_queues: 0 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.manual_wal_flush: 0 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.wal_compression: 0 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.atomic_flush: 0 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.persist_stats_to_disk: 0 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.write_dbid_to_manifest: 0 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.log_readahead_size: 0 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.best_efforts_recovery: 0 2023-11-28T01:08:32.060 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 bash[118719]: a71d29d005226851d7d1217691dab8dbf5848cbb937e63365d8f7dc471672c4d 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.allow_data_in_errors: 0 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.db_host_id: __hostname__ 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.enforce_single_del_contracts: true 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_background_jobs: 2 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_background_compactions: -1 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_subcompactions: 1 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.delayed_write_rate : 16777216 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_total_wal_size: 0 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-28T01:08:32.061 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.stats_dump_period_sec: 600 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.stats_persist_period_sec: 600 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_open_files: -1 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bytes_per_sync: 0 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.wal_bytes_per_sync: 0 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.strict_bytes_per_sync: 0 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_readahead_size: 0 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_background_flushes: -1 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Compression algorithms supported: 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: kZSTD supported: 0 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: kXpressCompression supported: 0 2023-11-28T01:08:32.062 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: kLZ4HCCompression supported: 1 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: kZlibCompression supported: 1 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: kSnappyCompression supported: 1 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: kLZ4Compression supported: 1 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: kBZip2Compression supported: 0 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Fast CRC32 supported: Supported on x86 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: DMutex implementation: pthread_mutex_t 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.merge_operator: 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_filter: None 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_filter_factory: None 2023-11-28T01:08:32.063 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.sst_partitioner_factory: None 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.memtable_factory: SkipListFactory 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.table_factory: BlockBasedTable 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5606a9c9d860) 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout: cache_index_and_filter_blocks: 1 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout: pin_top_level_index_and_filter: 1 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_type: 0 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout: data_block_index_type: 0 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_shortening: 1 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout: data_block_hash_table_util_ratio: 0.750000 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout: checksum: 4 2023-11-28T01:08:32.064 INFO:journalctl@ceph.mon.c.smithi172.stdout: no_block_cache: 0 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache: 0x5606a9cf3090 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_name: BinnedLRUCache 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_options: 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: capacity : 536870912 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: num_shard_bits : 4 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: strict_capacity_limit : 0 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: high_pri_pool_ratio: 0.000 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_cache_compressed: (nil) 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: persistent_cache: (nil) 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_size: 4096 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_size_deviation: 10 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_restart_interval: 16 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: index_block_restart_interval: 1 2023-11-28T01:08:32.065 INFO:journalctl@ceph.mon.c.smithi172.stdout: metadata_block_size: 4096 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: partition_filters: 0 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: use_delta_encoding: 1 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: filter_policy: bloomfilter 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: whole_key_filtering: 1 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: verify_compression: 0 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: read_amp_bytes_per_bit: 0 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: format_version: 5 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: enable_index_compression: 1 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: block_align: 0 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: max_auto_readahead_size: 262144 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: prepopulate_block_cache: 0 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: initial_auto_readahead_size: 8192 2023-11-28T01:08:32.066 INFO:journalctl@ceph.mon.c.smithi172.stdout: num_file_reads_for_auto_readahead: 2 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.write_buffer_size: 33554432 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_write_buffer_number: 2 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compression: NoCompression 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bottommost_compression: Disabled 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.prefix_extractor: nullptr 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.num_levels: 7 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-28T01:08:32.067 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compression_opts.window_bits: -14 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compression_opts.level: 32767 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compression_opts.strategy: 0 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-11-28T01:08:32.068 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compression_opts.enabled: false 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.target_file_size_base: 67108864 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.target_file_size_multiplier: 1 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-28T01:08:32.069 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-28T01:08:32.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-28T01:08:32.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-28T01:08:32.070 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-28T01:08:32.071 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-28T01:08:32.071 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-11-28T01:08:32.071 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.arena_block_size: 1048576 2023-11-28T01:08:32.071 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-28T01:08:32.071 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-28T01:08:32.071 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.disable_auto_compactions: 0 2023-11-28T01:08:32.071 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-28T01:08:32.071 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-28T01:08:32.071 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.table_properties_collectors: 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.inplace_update_support: 0 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.inplace_update_num_locks: 10000 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-28T01:08:32.072 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.memtable_huge_page_size: 0 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.bloom_locality: 0 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.max_successive_merges: 0 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.optimize_filters_for_hits: 0 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.paranoid_file_checks: 0 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.force_consistency_checks: 1 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.report_bg_io_stats: 0 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.ttl: 2592000 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.periodic_compaction_seconds: 0 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.enable_blob_files: false 2023-11-28T01:08:32.073 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.min_blob_size: 0 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.blob_file_size: 268435456 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.blob_compression_type: NoCompression 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.enable_blob_garbage_collection: false 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.blob_file_starting_level: 0 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 491186fa-c784-4c36-a673-2020cc848cd1 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1701133711712719, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-11-28T01:08:32.074 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1701133711713270, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1701133711, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "491186fa-c784-4c36-a673-2020cc848cd1", "db_session_id": "FI3NEMGJT6ZAGJY1TFK5", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: EVENT_LOG_v1 {"time_micros": 1701133711713362, "job": 1, "event": "recovery_finished"} 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5606a9d8c000 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: DB pointer 0x5606a9d7a000 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** DB Stats ** 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-28T01:08:32.075 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** Compaction Stats [default] ** 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-28T01:08:32.076 INFO:journalctl@ceph.mon.c.smithi172.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** Compaction Stats [default] ** 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-28T01:08:32.077 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout: AddFile(Keys): cumulative 0, interval 0 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout: Cumulative compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout: Interval compaction: 0.00 GB write, 0.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout: Block cache BinnedLRUCache@0x5606a9cf3090#7 capacity: 512.00 MB usage: 0.98 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout: Block cache entry stats(count,size,portion): DataBlock(1,0.77 KB,0.000146031%) FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout: ** File Read Latency Histogram By Level [default] ** 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: using public_addrv [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: starting mon.c rank -1 at public addrs [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] at bind addrs [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:08:32.078 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(???) e0 preinit fsid 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 systemd[1]: Started Ceph mon.c for 31861c10-8d8a-11ee-95a2-87774f69a715. 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).mds e1 new map 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).mds e1 print_map 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout: e1 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout: legacy client fscid: -1 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout: 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout: No filesystems configured 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-11-28T01:08:32.079 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-11-28T01:08:32.080 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-11-28T01:08:32.080 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-11-28T01:08:32.080 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-11-28T01:08:32.080 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-28T01:08:32.080 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-28T01:08:32.080 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-28T01:08:32.080 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-11-28T01:08:32.080 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: Updating smithi192:/etc/ceph/ceph.conf 2023-11-28T01:08:32.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:32.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:32.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:32.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/71802052' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-28T01:08:32.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: osdmap e4: 0 total, 0 up, 0 in 2023-11-28T01:08:32.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: Updating smithi192:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.conf 2023-11-28T01:08:32.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2023-11-28T01:08:32.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: Updating smithi192:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.client.admin.keyring 2023-11-28T01:08:32.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:32.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:32.081 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi192:172.21.15.192=b", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: Saving service mon spec with placement smithi172:172.21.15.172=a;smithi172:[v2:172.21.15.172:3301,v1:172.21.15.172:6790]=c;smithi192:172.21.15.192=b;count:3 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: Deploying daemon mon.b on smithi192 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:08:32.082 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.a calling monitor election 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.b calling monitor election 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: monmap e2: 2 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2023-11-28T01:08:32.083 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: fsmap 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: osdmap e4: 0 total, 0 up, 0 in 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mgrmap e13: y(active, since 37s) 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: overall HEALTH_OK 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: Deploying daemon mon.c on smithi172 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/327472976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:32.084 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:31 smithi172 ceph-mon[118784]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-11-28T01:08:32.116 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:08:37.053 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.050+0000 7f16b33d4700 1 -- 172.21.15.192:0/2096696249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16980057d0 msgr2=0x7f1698007c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:37.053 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.050+0000 7f16b33d4700 1 --2- 172.21.15.192:0/2096696249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16980057d0 0x7f1698007c60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f16ac0ff5a0 tx=0x7f16a800ce70 comp rx=0 tx=0).stop 2023-11-28T01:08:37.053 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.050+0000 7f16b33d4700 1 -- 172.21.15.192:0/2096696249 shutdown_connections 2023-11-28T01:08:37.053 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.050+0000 7f16b33d4700 1 --2- 172.21.15.192:0/2096696249 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16980057d0 0x7f1698007c60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:37.053 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.050+0000 7f16b33d4700 1 -- 172.21.15.192:0/2096696249 >> 172.21.15.192:0/2096696249 conn(0x7f16ac0765f0 msgr2=0x7f16ac076a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:37.053 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.051+0000 7f16b33d4700 1 -- 172.21.15.192:0/2096696249 shutdown_connections 2023-11-28T01:08:37.054 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.051+0000 7f16b33d4700 1 -- 172.21.15.192:0/2096696249 wait complete. 2023-11-28T01:08:37.055 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b33d4700 1 Processor -- start 2023-11-28T01:08:37.056 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b33d4700 1 -- start start 2023-11-28T01:08:37.056 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b33d4700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16980057d0 0x7f16ac1999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:37.056 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b33d4700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ac199ef0 0x7f16ac193af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:37.056 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b33d4700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac194030 0x7f16ac1944c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:37.056 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b33d4700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac19a4f0 con 0x7f16980057d0 2023-11-28T01:08:37.056 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b33d4700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac1a07d0 con 0x7f16ac199ef0 2023-11-28T01:08:37.056 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b33d4700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac1a0910 con 0x7f16ac194030 2023-11-28T01:08:37.056 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b1170700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16980057d0 0x7f16ac1999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:37.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b096f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ac199ef0 0x7f16ac193af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:37.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b1170700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16980057d0 0x7f16ac1999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.192:35334/0 (socket says 172.21.15.192:35334) 2023-11-28T01:08:37.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b1170700 1 -- 172.21.15.192:0/412935566 learned_addr learned my addr 172.21.15.192:0/412935566 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:08:37.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.052+0000 7f16b1971700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac194030 0x7f16ac1944c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:37.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b096f700 1 -- 172.21.15.192:0/412935566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ac199ef0 msgr2=0x7f16ac193af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-11-28T01:08:37.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b1971700 1 -- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac194030 msgr2=0x7f16ac1944c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-11-28T01:08:37.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b1971700 1 -- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac194030 msgr2=0x7f16ac1944c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T01:08:37.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b096f700 1 -- 172.21.15.192:0/412935566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ac199ef0 msgr2=0x7f16ac193af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-11-28T01:08:37.057 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b1971700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac194030 0x7f16ac1944c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T01:08:37.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b096f700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ac199ef0 0x7f16ac193af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-11-28T01:08:37.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b1971700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac194030 0x7f16ac1944c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T01:08:37.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b096f700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ac199ef0 0x7f16ac193af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-11-28T01:08:37.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b1170700 1 -- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac194030 msgr2=0x7f16ac1944c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T01:08:37.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b1170700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac194030 0x7f16ac1944c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:37.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b1170700 1 -- 172.21.15.192:0/412935566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ac199ef0 msgr2=0x7f16ac193af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-28T01:08:37.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b1170700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ac199ef0 0x7f16ac193af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:37.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b1170700 1 -- 172.21.15.192:0/412935566 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a800c8c0 con 0x7f16980057d0 2023-11-28T01:08:37.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b1170700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16980057d0 0x7f16ac1999b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f16a8004a70 tx=0x7f16a8004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:37.058 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16a27fc700 1 -- 172.21.15.192:0/412935566 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a8005330 con 0x7f16980057d0 2023-11-28T01:08:37.059 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16a27fc700 1 -- 172.21.15.192:0/412935566 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f16a80238b0 con 0x7f16980057d0 2023-11-28T01:08:37.059 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16a27fc700 1 -- 172.21.15.192:0/412935566 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a80197e0 con 0x7f16980057d0 2023-11-28T01:08:37.059 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b33d4700 1 -- 172.21.15.192:0/412935566 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac1a0b90 con 0x7f16980057d0 2023-11-28T01:08:37.059 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.053+0000 7f16b33d4700 1 -- 172.21.15.192:0/412935566 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac1a1090 con 0x7f16980057d0 2023-11-28T01:08:37.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.054+0000 7f16a27fc700 1 -- 172.21.15.192:0/412935566 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7f16a80199e0 con 0x7f16980057d0 2023-11-28T01:08:37.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.054+0000 7f16a27fc700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1690039110 0x7f169003b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:37.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.055+0000 7f16a27fc700 1 -- 172.21.15.192:0/412935566 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f16a804dfd0 con 0x7f16980057d0 2023-11-28T01:08:37.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.055+0000 7f16b33d4700 1 -- 172.21.15.192:0/412935566 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac100db0 con 0x7f16980057d0 2023-11-28T01:08:37.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.055+0000 7f16b096f700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1690039110 0x7f169003b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:37.060 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.055+0000 7f16b096f700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1690039110 0x7f169003b5d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f169c009850 tx=0x7f169c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:37.061 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.060+0000 7f16a27fc700 1 -- 172.21.15.192:0/412935566 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f16a8027070 con 0x7f16980057d0 2023-11-28T01:08:37.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:08:37.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: mon.a calling monitor election 2023-11-28T01:08:37.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: mon.b calling monitor election 2023-11-28T01:08:37.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:37.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:37.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: mon.c calling monitor election 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: monmap e3: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: fsmap 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: osdmap e4: 0 total, 0 up, 0 in 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: mgrmap e13: y(active, since 46s) 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: overall HEALTH_OK 2023-11-28T01:08:37.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: mon.a calling monitor election 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: mon.b calling monitor election 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: mon.c calling monitor election 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:37.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-11-28T01:08:37.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: monmap e3: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-11-28T01:08:37.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: fsmap 2023-11-28T01:08:37.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: osdmap e4: 0 total, 0 up, 0 in 2023-11-28T01:08:37.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: mgrmap e13: y(active, since 46s) 2023-11-28T01:08:37.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: overall HEALTH_OK 2023-11-28T01:08:37.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:37.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:37.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:08:37.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: mon.a calling monitor election 2023-11-28T01:08:37.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: mon.b calling monitor election 2023-11-28T01:08:37.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:08:37.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:37.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: mon.c calling monitor election 2023-11-28T01:08:37.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:37.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:37.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2023-11-28T01:08:37.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: monmap e3: 3 mons at {a=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0]} removed_ranks: {} disallowed_leaders: {} 2023-11-28T01:08:37.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: fsmap 2023-11-28T01:08:37.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: osdmap e4: 0 total, 0 up, 0 in 2023-11-28T01:08:37.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: mgrmap e13: y(active, since 46s) 2023-11-28T01:08:37.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: overall HEALTH_OK 2023-11-28T01:08:37.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:37.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:37.348 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.346+0000 7f16b33d4700 1 -- 172.21.15.192:0/412935566 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16ac195010 con 0x7f16980057d0 2023-11-28T01:08:37.348 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.346+0000 7f16a27fc700 1 -- 172.21.15.192:0/412935566 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1289 (secure 0 0 0) 0x7f16a802c450 con 0x7f16980057d0 2023-11-28T01:08:37.349 INFO:teuthology.orchestra.run.smithi192.stdout: 2023-11-28T01:08:37.349 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":3,"fsid":"31861c10-8d8a-11ee-95a2-87774f69a715","modified":"2023-11-28T01:08:31.738852Z","created":"2023-11-28T01:06:55.519247Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3301","nonce":0},{"type":"v1","addr":"172.21.15.172:6790","nonce":0}]},"addr":"172.21.15.172:6790/0","public_addr":"172.21.15.172:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 -- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1690039110 msgr2=0x7f169003b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1690039110 0x7f169003b5d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f169c009850 tx=0x7f169c008040 comp rx=0 tx=0).stop 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 -- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16980057d0 msgr2=0x7f16ac1999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16980057d0 0x7f16ac1999b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f16a8004a70 tx=0x7f16a8004b50 comp rx=0 tx=0).stop 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 -- 172.21.15.192:0/412935566 shutdown_connections 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1690039110 0x7f169003b5d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f16ac194030 0x7f16ac1944c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16980057d0 0x7f16ac1999b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 --2- 172.21.15.192:0/412935566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16ac199ef0 0x7f16ac193af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 -- 172.21.15.192:0/412935566 >> 172.21.15.192:0/412935566 conn(0x7f16ac0765f0 msgr2=0x7f16ac102280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 -- 172.21.15.192:0/412935566 shutdown_connections 2023-11-28T01:08:37.351 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:37.349+0000 7f16b33d4700 1 -- 172.21.15.192:0/412935566 wait complete. 2023-11-28T01:08:37.352 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 3 2023-11-28T01:08:38.106 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-11-28T01:08:38.106 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph config generate-minimal-conf 2023-11-28T01:08:38.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[118784]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:38.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:38.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:38.133 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:38.133 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/412935566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-28T01:08:38.133 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:38.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[100627]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:38.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:38.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:38.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:38.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/412935566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-28T01:08:38.133 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:37 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:38.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:37 smithi192 ceph-mon[105371]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:38.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:37 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:38.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:37 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:38.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:37 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:38.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:37 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/412935566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-28T01:08:38.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:37 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:08:38.447 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:08:39.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: Updating smithi172:/etc/ceph/ceph.conf 2023-11-28T01:08:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: Updating smithi192:/etc/ceph/ceph.conf 2023-11-28T01:08:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: Updating smithi172:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.conf 2023-11-28T01:08:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: Updating smithi192:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.conf 2023-11-28T01:08:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:39.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-28T01:08:39.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:39.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:39.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:39.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: Updating smithi172:/etc/ceph/ceph.conf 2023-11-28T01:08:39.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: Updating smithi192:/etc/ceph/ceph.conf 2023-11-28T01:08:39.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: Updating smithi172:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.conf 2023-11-28T01:08:39.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: Updating smithi192:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.conf 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:39.307 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-28T01:08:39.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:39.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:39.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:39.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: Updating smithi172:/etc/ceph/ceph.conf 2023-11-28T01:08:39.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: Updating smithi192:/etc/ceph/ceph.conf 2023-11-28T01:08:39.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: Updating smithi172:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.conf 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: Updating smithi192:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/config/ceph.conf 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:39.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:39.325 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-28T01:08:39.325 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:40.404 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.398+0000 7f35f2704700 1 -- 172.21.15.172:0/2586855239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35ec100060 msgr2=0x7f35ec100440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:40.404 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.398+0000 7f35f2704700 1 --2- 172.21.15.172:0/2586855239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35ec100060 0x7f35ec100440 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009910 tx=0x7f35dc009c20 comp rx=0 tx=0).stop 2023-11-28T01:08:40.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.398+0000 7f35f2704700 1 -- 172.21.15.172:0/2586855239 shutdown_connections 2023-11-28T01:08:40.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.398+0000 7f35f2704700 1 --2- 172.21.15.172:0/2586855239 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35ec100060 0x7f35ec100440 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:40.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.398+0000 7f35f2704700 1 -- 172.21.15.172:0/2586855239 >> 172.21.15.172:0/2586855239 conn(0x7f35ec0fb930 msgr2=0x7f35ec0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:40.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.398+0000 7f35f2704700 1 -- 172.21.15.172:0/2586855239 shutdown_connections 2023-11-28T01:08:40.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.398+0000 7f35f2704700 1 -- 172.21.15.172:0/2586855239 wait complete. 2023-11-28T01:08:40.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.398+0000 7f35f2704700 1 Processor -- start 2023-11-28T01:08:40.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.398+0000 7f35f2704700 1 -- start start 2023-11-28T01:08:40.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35f2704700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35ec100060 0x7f35ec196dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:40.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35f2704700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec197300 0x7f35ec197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:40.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35f2704700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35ec19fe30 0x7f35ec1a2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:40.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35f2704700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec1a28c0 con 0x7f35ec100060 2023-11-28T01:08:40.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35f2704700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec1a2a30 con 0x7f35ec197300 2023-11-28T01:08:40.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35f2704700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec1a2bf0 con 0x7f35ec19fe30 2023-11-28T01:08:40.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35ebfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35ec100060 0x7f35ec196dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:40.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35ebfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35ec100060 0x7f35ec196dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49524/0 (socket says 172.21.15.172:49524) 2023-11-28T01:08:40.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35ebfff700 1 -- 172.21.15.172:0/3919789110 learned_addr learned my addr 172.21.15.172:0/3919789110 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:08:40.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35f0ca1700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35ec19fe30 0x7f35ec1a2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:40.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35ebfff700 1 -- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35ec19fe30 msgr2=0x7f35ec1a2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:40.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35ebfff700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35ec19fe30 0x7f35ec1a2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:40.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35ebfff700 1 -- 172.21.15.172:0/3919789110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec197300 msgr2=0x7f35ec197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:40.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35ebfff700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec197300 0x7f35ec197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:40.406 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35ebfff700 1 -- 172.21.15.172:0/3919789110 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc0095f0 con 0x7f35ec100060 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35ebfff700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35ec100060 0x7f35ec196dc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f35dc006010 tx=0x7f35dc00bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35e97fa700 1 -- 172.21.15.172:0/3919789110 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc01d070 con 0x7f35ec100060 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35e97fa700 1 -- 172.21.15.172:0/3919789110 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f35dc003ad0 con 0x7f35ec100060 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35f2704700 1 -- 172.21.15.172:0/3919789110 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ec1a2e70 con 0x7f35ec100060 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35e97fa700 1 -- 172.21.15.172:0/3919789110 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc00fa50 con 0x7f35ec100060 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.399+0000 7f35f2704700 1 -- 172.21.15.172:0/3919789110 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ec1a3330 con 0x7f35ec100060 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.400+0000 7f35e97fa700 1 -- 172.21.15.172:0/3919789110 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7f35dc00fbb0 con 0x7f35ec100060 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.400+0000 7f35f2704700 1 -- 172.21.15.172:0/3919789110 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35ec101540 con 0x7f35ec100060 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.400+0000 7f35e97fa700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f35d4039110 0x7f35d403b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.400+0000 7f35e97fa700 1 -- 172.21.15.172:0/3919789110 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f35dc04c850 con 0x7f35ec100060 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.401+0000 7f35eb7fe700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f35d4039110 0x7f35d403b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:40.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.401+0000 7f35eb7fe700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f35d4039110 0x7f35d403b5d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f35e00098e0 tx=0x7f35e0006dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:40.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.406+0000 7f35e97fa700 1 -- 172.21.15.172:0/3919789110 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f35dc015350 con 0x7f35ec100060 2023-11-28T01:08:40.605 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.603+0000 7f35f2704700 1 -- 172.21.15.172:0/3919789110 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f35ec069940 con 0x7f35ec100060 2023-11-28T01:08:40.605 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.603+0000 7f35e97fa700 1 -- 172.21.15.172:0/3919789110 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f35dc026070 con 0x7f35ec100060 2023-11-28T01:08:40.606 INFO:teuthology.orchestra.run.smithi172.stdout:# minimal ceph.conf for 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:08:40.606 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2023-11-28T01:08:40.606 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:08:40.606 INFO:teuthology.orchestra.run.smithi172.stdout: mon_host = [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] 2023-11-28T01:08:40.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.605+0000 7f35f2704700 1 -- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f35d4039110 msgr2=0x7f35d403b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:40.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.605+0000 7f35f2704700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f35d4039110 0x7f35d403b5d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f35e00098e0 tx=0x7f35e0006dd0 comp rx=0 tx=0).stop 2023-11-28T01:08:40.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.605+0000 7f35f2704700 1 -- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35ec100060 msgr2=0x7f35ec196dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:40.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.605+0000 7f35f2704700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35ec100060 0x7f35ec196dc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f35dc006010 tx=0x7f35dc00bac0 comp rx=0 tx=0).stop 2023-11-28T01:08:40.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.606+0000 7f35f2704700 1 -- 172.21.15.172:0/3919789110 shutdown_connections 2023-11-28T01:08:40.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.606+0000 7f35f2704700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f35d4039110 0x7f35d403b5d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:40.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.606+0000 7f35f2704700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f35ec19fe30 0x7f35ec1a2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:40.608 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.606+0000 7f35f2704700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f35ec100060 0x7f35ec196dc0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:40.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.606+0000 7f35f2704700 1 --2- 172.21.15.172:0/3919789110 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec197300 0x7f35ec197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:40.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.606+0000 7f35f2704700 1 -- 172.21.15.172:0/3919789110 >> 172.21.15.172:0/3919789110 conn(0x7f35ec0fb930 msgr2=0x7f35ec101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:40.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.606+0000 7f35f2704700 1 -- 172.21.15.172:0/3919789110 shutdown_connections 2023-11-28T01:08:40.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:40.606+0000 7f35f2704700 1 -- 172.21.15.172:0/3919789110 wait complete. 2023-11-28T01:08:40.773 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:40 smithi172 ceph-mon[100627]: Reconfiguring mon.a (unknown last config time)... 2023-11-28T01:08:40.774 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:40 smithi172 ceph-mon[100627]: Reconfiguring daemon mon.a on smithi172 2023-11-28T01:08:40.774 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:40 smithi172 ceph-mon[118784]: Reconfiguring mon.a (unknown last config time)... 2023-11-28T01:08:40.774 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:40 smithi172 ceph-mon[118784]: Reconfiguring daemon mon.a on smithi172 2023-11-28T01:08:40.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:40 smithi192 ceph-mon[105371]: Reconfiguring mon.a (unknown last config time)... 2023-11-28T01:08:40.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:40 smithi192 ceph-mon[105371]: Reconfiguring daemon mon.a on smithi172 2023-11-28T01:08:41.195 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-11-28T01:08:41.195 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:08:41.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.conf 2023-11-28T01:08:41.264 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:08:41.264 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-28T01:08:41.330 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T01:08:41.330 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2023-11-28T01:08:41.366 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T01:08:41.366 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-28T01:08:41.438 INFO:tasks.cephadm:Adding mgr.y on smithi172 2023-11-28T01:08:41.438 INFO:tasks.cephadm:Adding mgr.x on smithi192 2023-11-28T01:08:41.439 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch apply mgr '2;smithi172=y;smithi192=x' 2023-11-28T01:08:41.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/3919789110' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:41.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:41.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:41.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[100627]: Reconfiguring mon.c (monmap changed)... 2023-11-28T01:08:41.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:41.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-28T01:08:41.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:41.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[100627]: Reconfiguring daemon mon.c on smithi172 2023-11-28T01:08:41.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[100627]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:41.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:41.805 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/3919789110' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:41.805 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:41.805 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:41.805 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[118784]: Reconfiguring mon.c (monmap changed)... 2023-11-28T01:08:41.805 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:41.805 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-28T01:08:41.805 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:41.806 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[118784]: Reconfiguring daemon mon.c on smithi172 2023-11-28T01:08:41.806 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[118784]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:41.806 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:41 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:41.807 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:08:41.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:41 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/3919789110' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:41.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:41 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:41.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:41 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:41.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:41 smithi192 ceph-mon[105371]: Reconfiguring mon.c (monmap changed)... 2023-11-28T01:08:41.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:41 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:41.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:41 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-28T01:08:41.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:41 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:41.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:41 smithi192 ceph-mon[105371]: Reconfiguring daemon mon.c on smithi172 2023-11-28T01:08:41.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:41 smithi192 ceph-mon[105371]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:41.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:41 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:43.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:43 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:43.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:43 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:43.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:43 smithi192 ceph-mon[105371]: Reconfiguring mon.b (monmap changed)... 2023-11-28T01:08:43.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:43 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:43.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:43 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-28T01:08:43.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:43 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:43.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:43 smithi192 ceph-mon[105371]: Reconfiguring daemon mon.b on smithi192 2023-11-28T01:08:43.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:43 smithi192 ceph-mon[105371]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:43.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:43.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:43.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[100627]: Reconfiguring mon.b (monmap changed)... 2023-11-28T01:08:43.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:43.555 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-28T01:08:43.555 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:43.555 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[100627]: Reconfiguring daemon mon.b on smithi192 2023-11-28T01:08:43.555 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[100627]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:43.556 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:43.556 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:43.556 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[118784]: Reconfiguring mon.b (monmap changed)... 2023-11-28T01:08:43.556 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-11-28T01:08:43.556 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-11-28T01:08:43.557 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:43.557 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[118784]: Reconfiguring daemon mon.b on smithi192 2023-11-28T01:08:43.557 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:43 smithi172 ceph-mon[118784]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:43.774 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.738+0000 7f6a509dd700 1 -- 172.21.15.192:0/3517826552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a4c0fff90 msgr2=0x7f6a4c1003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:43.774 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.738+0000 7f6a509dd700 1 --2- 172.21.15.192:0/3517826552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a4c0fff90 0x7f6a4c1003b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6a40009910 tx=0x7f6a40009c20 comp rx=0 tx=0).stop 2023-11-28T01:08:43.774 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.738+0000 7f6a509dd700 1 -- 172.21.15.192:0/3517826552 shutdown_connections 2023-11-28T01:08:43.774 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.738+0000 7f6a509dd700 1 --2- 172.21.15.192:0/3517826552 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a4c0fff90 0x7f6a4c1003b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:43.774 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.738+0000 7f6a509dd700 1 -- 172.21.15.192:0/3517826552 >> 172.21.15.192:0/3517826552 conn(0x7f6a4c0fb530 msgr2=0x7f6a4c0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:43.774 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.738+0000 7f6a509dd700 1 -- 172.21.15.192:0/3517826552 shutdown_connections 2023-11-28T01:08:43.774 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.738+0000 7f6a509dd700 1 -- 172.21.15.192:0/3517826552 wait complete. 2023-11-28T01:08:43.774 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.738+0000 7f6a509dd700 1 Processor -- start 2023-11-28T01:08:43.774 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.738+0000 7f6a509dd700 1 -- start start 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a509dd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a4c0fff90 0x7f6a4c069fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a509dd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a4c198910 0x7f6a4c19ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a509dd700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a4c19b240 0x7f6a4c19d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a509dd700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c19de70 con 0x7f6a4c19b240 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a509dd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c19dfe0 con 0x7f6a4c0fff90 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a509dd700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a4c19e1a0 con 0x7f6a4c198910 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a4a59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a4c0fff90 0x7f6a4c069fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a4a59c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a4c0fff90 0x7f6a4c069fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:49148/0 (socket says 172.21.15.192:49148) 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a4a59c700 1 -- 172.21.15.192:0/4114842764 learned_addr learned my addr 172.21.15.192:0/4114842764 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a4ad9d700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a4c19b240 0x7f6a4c19d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a49d9b700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a4c198910 0x7f6a4c19ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:43.775 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a4a59c700 1 -- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a4c198910 msgr2=0x7f6a4c19ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a4a59c700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a4c198910 0x7f6a4c19ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a4a59c700 1 -- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a4c19b240 msgr2=0x7f6a4c19d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a4a59c700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a4c19b240 0x7f6a4c19d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a4a59c700 1 -- 172.21.15.192:0/4114842764 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400095f0 con 0x7f6a4c0fff90 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a4a59c700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a4c0fff90 0x7f6a4c069fd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6a40005950 tx=0x7f6a40005430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a3b7fe700 1 -- 172.21.15.192:0/4114842764 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a4001d070 con 0x7f6a4c0fff90 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a3b7fe700 1 -- 172.21.15.192:0/4114842764 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f6a4000bba0 con 0x7f6a4c0fff90 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.739+0000 7f6a509dd700 1 -- 172.21.15.192:0/4114842764 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a4c19e420 con 0x7f6a4c0fff90 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.740+0000 7f6a3b7fe700 1 -- 172.21.15.192:0/4114842764 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a40022a80 con 0x7f6a4c0fff90 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.740+0000 7f6a509dd700 1 -- 172.21.15.192:0/4114842764 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4c19e990 con 0x7f6a4c0fff90 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.740+0000 7f6a3b7fe700 1 -- 172.21.15.192:0/4114842764 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7f6a4000f460 con 0x7f6a4c0fff90 2023-11-28T01:08:43.776 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.740+0000 7f6a3b7fe700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6a34039110 0x7f6a3403b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:43.777 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.740+0000 7f6a3b7fe700 1 -- 172.21.15.192:0/4114842764 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6a4004cfe0 con 0x7f6a4c0fff90 2023-11-28T01:08:43.777 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.740+0000 7f6a509dd700 1 -- 172.21.15.192:0/4114842764 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4c18e670 con 0x7f6a4c0fff90 2023-11-28T01:08:43.777 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.741+0000 7f6a49d9b700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6a34039110 0x7f6a3403b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:43.777 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.741+0000 7f6a49d9b700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6a34039110 0x7f6a3403b5d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009ad0 tx=0x7f6a3c009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:43.777 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.746+0000 7f6a3b7fe700 1 -- 172.21.15.192:0/4114842764 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f6a40027030 con 0x7f6a4c0fff90 2023-11-28T01:08:43.948 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.947+0000 7f6a509dd700 1 -- 172.21.15.192:0/4114842764 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi172=y;smithi192=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a4c061a10 con 0x7f6a34039110 2023-11-28T01:08:43.954 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.953+0000 7f6a3b7fe700 1 -- 172.21.15.192:0/4114842764 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6a4c061a10 con 0x7f6a34039110 2023-11-28T01:08:43.954 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mgr update... 2023-11-28T01:08:43.956 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.955+0000 7f6a509dd700 1 -- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6a34039110 msgr2=0x7f6a3403b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:43.956 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.955+0000 7f6a509dd700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6a34039110 0x7f6a3403b5d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c009ad0 tx=0x7f6a3c009210 comp rx=0 tx=0).stop 2023-11-28T01:08:43.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.955+0000 7f6a509dd700 1 -- 172.21.15.192:0/4114842764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a4c0fff90 msgr2=0x7f6a4c069fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:43.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.955+0000 7f6a509dd700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a4c0fff90 0x7f6a4c069fd0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6a40005950 tx=0x7f6a40005430 comp rx=0 tx=0).stop 2023-11-28T01:08:43.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.955+0000 7f6a509dd700 1 -- 172.21.15.192:0/4114842764 shutdown_connections 2023-11-28T01:08:43.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.955+0000 7f6a509dd700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6a34039110 0x7f6a3403b5d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:43.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.955+0000 7f6a509dd700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6a4c19b240 0x7f6a4c19d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:43.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.955+0000 7f6a509dd700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a4c0fff90 0x7f6a4c069fd0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:43.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.956+0000 7f6a509dd700 1 --2- 172.21.15.192:0/4114842764 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6a4c198910 0x7f6a4c19ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:43.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.956+0000 7f6a509dd700 1 -- 172.21.15.192:0/4114842764 >> 172.21.15.192:0/4114842764 conn(0x7f6a4c0fb530 msgr2=0x7f6a4c0fc1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:43.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.956+0000 7f6a509dd700 1 -- 172.21.15.192:0/4114842764 shutdown_connections 2023-11-28T01:08:43.957 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:08:43.956+0000 7f6a509dd700 1 -- 172.21.15.192:0/4114842764 wait complete. 2023-11-28T01:08:44.467 DEBUG:teuthology.orchestra.run.smithi192:mgr.x> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mgr.x.service 2023-11-28T01:08:44.469 INFO:tasks.cephadm:Deploying OSDs... 2023-11-28T01:08:44.469 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:08:44.470 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T01:08:44.491 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T01:08:44.491 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2023-11-28T01:08:44.500 INFO:journalctl@ceph.mgr.x.smithi192.stdout:-- Logs begin at Tue 2023-11-28 00:47:11 UTC. -- 2023-11-28T01:08:44.551 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T01:08:44.551 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T01:08:44.551 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 385619 Links: 1 2023-11-28T01:08:44.551 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T01:08:44.551 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T01:08:44.552 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-11-28 01:08:31.101986571 +0000 2023-11-28T01:08:44.552 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-11-28 00:57:21.965256626 +0000 2023-11-28T01:08:44.552 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-11-28 00:57:21.965256626 +0000 2023-11-28T01:08:44.552 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-11-28T01:08:44.552 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T01:08:44.622 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-11-28T01:08:44.623 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-11-28T01:08:44.623 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000277373 s, 1.8 MB/s 2023-11-28T01:08:44.624 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T01:08:44.684 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2023-11-28T01:08:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T01:08:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T01:08:44.743 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 386935 Links: 1 2023-11-28T01:08:44.743 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T01:08:44.743 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T01:08:44.743 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-11-28 01:08:31.101986571 +0000 2023-11-28T01:08:44.743 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-11-28 00:57:22.478244284 +0000 2023-11-28T01:08:44.743 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-11-28 00:57:22.478244284 +0000 2023-11-28T01:08:44.743 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-11-28T01:08:44.744 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T01:08:44.810 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-11-28T01:08:44.810 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-11-28T01:08:44.810 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000253591 s, 2.0 MB/s 2023-11-28T01:08:44.811 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T01:08:44.871 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2023-11-28T01:08:44.929 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T01:08:44.929 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T01:08:44.929 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 385833 Links: 1 2023-11-28T01:08:44.929 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T01:08:44.930 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T01:08:44.930 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-11-28 01:08:31.101986571 +0000 2023-11-28T01:08:44.930 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-11-28 00:57:22.989231990 +0000 2023-11-28T01:08:44.931 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-11-28 00:57:22.989231990 +0000 2023-11-28T01:08:44.931 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-11-28T01:08:44.932 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T01:08:44.995 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-11-28T01:08:44.995 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-11-28T01:08:44.995 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000228205 s, 2.2 MB/s 2023-11-28T01:08:44.996 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T01:08:45.055 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2023-11-28T01:08:45.074 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi172=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:08:45.074 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: Saving service mgr spec with placement smithi172=y;smithi192=x;count:2 2023-11-28T01:08:45.075 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.075 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.075 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.075 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:45.076 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:45.076 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:45.076 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.076 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-28T01:08:45.076 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-28T01:08:45.077 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-28T01:08:45.077 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:45.114 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T01:08:45.114 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T01:08:45.114 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 385915 Links: 1 2023-11-28T01:08:45.115 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T01:08:45.115 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T01:08:45.115 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2023-11-28 01:08:31.101986571 +0000 2023-11-28T01:08:45.115 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2023-11-28 00:57:23.485220057 +0000 2023-11-28T01:08:45.115 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2023-11-28 00:57:23.485220057 +0000 2023-11-28T01:08:45.115 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2023-11-28T01:08:45.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T01:08:45.180 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2023-11-28T01:08:45.181 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2023-11-28T01:08:45.181 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000200138 s, 2.6 MB/s 2023-11-28T01:08:45.182 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T01:08:45.240 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T01:08:45.240 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2023-11-28T01:08:45.258 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-11-28T01:08:45.259 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2023-11-28T01:08:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi172=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:08:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: Saving service mgr spec with placement smithi172=y;smithi192=x;count:2 2023-11-28T01:08:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:45.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-28T01:08:45.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-28T01:08:45.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-28T01:08:45.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:45.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi172=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:08:45.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: Saving service mgr spec with placement smithi172=y;smithi192=x;count:2 2023-11-28T01:08:45.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:45.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:45.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:45.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:45.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-28T01:08:45.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-11-28T01:08:45.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-28T01:08:45.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:45.316 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-11-28T01:08:45.316 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T01:08:45.316 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 378662 Links: 1 2023-11-28T01:08:45.316 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T01:08:45.316 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T01:08:45.316 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-11-28 01:08:22.752559811 +0000 2023-11-28T01:08:45.316 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-11-28 00:57:21.971376868 +0000 2023-11-28T01:08:45.316 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-11-28 00:57:21.971376868 +0000 2023-11-28T01:08:45.317 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-11-28T01:08:45.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-11-28T01:08:45.384 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-11-28T01:08:45.385 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-11-28T01:08:45.385 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000265576 s, 1.9 MB/s 2023-11-28T01:08:45.386 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-11-28T01:08:45.444 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2023-11-28T01:08:45.502 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-11-28T01:08:45.502 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T01:08:45.502 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 378690 Links: 1 2023-11-28T01:08:45.502 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T01:08:45.502 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T01:08:45.503 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-11-28 01:08:22.752559811 +0000 2023-11-28T01:08:45.503 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-11-28 00:57:22.497366399 +0000 2023-11-28T01:08:45.503 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-11-28 00:57:22.497366399 +0000 2023-11-28T01:08:45.503 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-11-28T01:08:45.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-11-28T01:08:45.569 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-11-28T01:08:45.569 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-11-28T01:08:45.570 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000217093 s, 2.4 MB/s 2023-11-28T01:08:45.571 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-11-28T01:08:45.628 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2023-11-28T01:08:45.686 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-11-28T01:08:45.687 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T01:08:45.687 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 392303 Links: 1 2023-11-28T01:08:45.687 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T01:08:45.687 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T01:08:45.687 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-11-28 01:08:45.677108372 +0000 2023-11-28T01:08:45.687 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-11-28 00:57:23.013356129 +0000 2023-11-28T01:08:45.687 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-11-28 00:57:23.013356129 +0000 2023-11-28T01:08:45.687 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-11-28T01:08:45.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-11-28T01:08:45.755 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-11-28T01:08:45.756 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-11-28T01:08:45.756 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000255785 s, 2.0 MB/s 2023-11-28T01:08:45.757 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-11-28T01:08:45.814 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2023-11-28T01:08:45.873 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-11-28T01:08:45.874 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-11-28T01:08:45.874 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 6h/6d Inode: 386980 Links: 1 2023-11-28T01:08:45.874 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-11-28T01:08:45.874 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2023-11-28T01:08:45.874 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2023-11-28 01:08:45.801105930 +0000 2023-11-28T01:08:45.874 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2023-11-28 00:57:23.525345939 +0000 2023-11-28T01:08:45.874 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2023-11-28 00:57:23.525345939 +0000 2023-11-28T01:08:45.874 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: - 2023-11-28T01:08:45.875 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-11-28T01:08:45.944 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2023-11-28T01:08:45.944 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2023-11-28T01:08:45.944 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000195882 s, 2.6 MB/s 2023-11-28T01:08:45.945 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-11-28T01:08:46.005 INFO:tasks.cephadm:Deploying osd.0 on smithi172 with /dev/vg_nvme/lv_4... 2023-11-28T01:08:46.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-28T01:08:46.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:45 smithi172 ceph-mon[100627]: Deploying daemon mgr.x on smithi192 2023-11-28T01:08:46.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:45 smithi172 ceph-mon[100627]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:46.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:45 smithi172 ceph-mon[118784]: Deploying daemon mgr.x on smithi192 2023-11-28T01:08:46.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:45 smithi172 ceph-mon[118784]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:46.312 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:45 smithi192 ceph-mon[105371]: Deploying daemon mgr.x on smithi192 2023-11-28T01:08:46.312 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:45 smithi192 ceph-mon[105371]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:46.312 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:45 smithi192 systemd[1]: Starting Ceph mgr.x for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:08:46.349 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:08:46.733 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:46 smithi192 bash[108549]: c7290151eecb8ded907e4a8322566ee391fd680d4440d5b2ce34965f3d404a43 2023-11-28T01:08:46.733 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:46 smithi192 ceph-mgr[108615]: -- 172.21.15.192:0/3234297383 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558e4b3ba1a0 con 0x558e4b392800 2023-11-28T01:08:46.733 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:46 smithi192 systemd[1]: Started Ceph mgr.x for 31861c10-8d8a-11ee-95a2-87774f69a715. 2023-11-28T01:08:46.734 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:46 smithi192 conmon[108591]: 2023-11-28T01:08:46.555+0000 7f1290263200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-11-28T01:08:46.734 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:46 smithi192 conmon[108591]: 2023-11-28T01:08:46.635+0000 7f1290263200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-11-28T01:08:46.734 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:46 smithi192 conmon[108591]: 2023-11-28T01:08:46.732+0000 7f1290263200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-11-28T01:08:47.020 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:46 smithi192 conmon[108591]: 2023-11-28T01:08:46.830+0000 7f1290263200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-11-28T01:08:47.020 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:47 smithi192 conmon[108591]: 2023-11-28T01:08:47.018+0000 7f1290263200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-11-28T01:08:47.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:47.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[100627]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:47.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:47.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:47 smithi172 ceph-mon[118784]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:47.564 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:47 smithi192 conmon[108591]: 2023-11-28T01:08:47.425+0000 7f1290263200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-11-28T01:08:47.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:47 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:47 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:47 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:47 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:47.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:47 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:47.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:47 smithi192 ceph-mon[105371]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:47.821 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:47 smithi192 conmon[108591]: 2023-11-28T01:08:47.563+0000 7f1290263200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-11-28T01:08:47.821 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:47 smithi192 conmon[108591]: 2023-11-28T01:08:47.660+0000 7f1290263200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-11-28T01:08:47.821 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:47 smithi192 conmon[108591]: 2023-11-28T01:08:47.757+0000 7f1290263200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-11-28T01:08:48.057 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-28T01:08:48.057 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-28T01:08:48.057 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2023-11-28T01:08:48.057 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2023-11-28T01:08:48.057 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130275 s, 805 MB/s 2023-11-28T01:08:48.057 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2023-11-28T01:08:48.570 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:48 smithi192 conmon[108591]: 2023-11-28T01:08:48.175+0000 7f1290263200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-11-28T01:08:48.851 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:48.851 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:48.851 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:48.851 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:48.851 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:48.851 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[100627]: Reconfiguring mgr.y (unknown last config time)... 2023-11-28T01:08:48.851 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-28T01:08:48.852 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-28T01:08:48.852 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:48.852 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[100627]: Reconfiguring daemon mgr.y on smithi172 2023-11-28T01:08:48.852 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:48.852 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:48.852 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:48.852 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:48.852 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:48.852 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[118784]: Reconfiguring mgr.y (unknown last config time)... 2023-11-28T01:08:48.853 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-28T01:08:48.853 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-28T01:08:48.853 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:48.853 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:48 smithi172 ceph-mon[118784]: Reconfiguring daemon mgr.y on smithi172 2023-11-28T01:08:48.875 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_4 2023-11-28T01:08:48.921 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:48 smithi192 conmon[108591]: 2023-11-28T01:08:48.839+0000 7f1290263200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-11-28T01:08:48.921 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:48 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:48.921 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:48 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:48.921 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:48 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:48.921 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:48 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:48.921 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:48 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:48.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:48 smithi192 ceph-mon[105371]: Reconfiguring mgr.y (unknown last config time)... 2023-11-28T01:08:48.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:48 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-11-28T01:08:48.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:48 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2023-11-28T01:08:48.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:48 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:48.922 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:48 smithi192 ceph-mon[105371]: Reconfiguring daemon mgr.y on smithi172 2023-11-28T01:08:49.320 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:48 smithi192 conmon[108591]: 2023-11-28T01:08:48.920+0000 7f1290263200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-11-28T01:08:49.403 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:08:49.930 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[100627]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:49.930 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:49.930 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:49.931 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:49.931 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:49.931 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:49.931 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:49.932 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[118784]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:49.932 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:49.932 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:49.932 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:49.932 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:49.933 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:49.933 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:49 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:49.960 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:49 smithi192 conmon[108591]: 2023-11-28T01:08:49.694+0000 7f1290263200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-11-28T01:08:49.960 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:49 smithi192 conmon[108591]: 2023-11-28T01:08:49.785+0000 7f1290263200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-11-28T01:08:49.960 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:49 smithi192 ceph-mon[105371]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:49.960 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:49 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:49.960 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:49 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:49.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:49 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:08:49.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:49 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:49.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:49 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:08:49.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:49 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:50.320 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:49 smithi192 conmon[108591]: 2023-11-28T01:08:49.959+0000 7f1290263200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-11-28T01:08:50.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.715+0000 7f0758ba3700 1 -- 172.21.15.172:0/2372088624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0754102820 msgr2=0x7f0754102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:50.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.715+0000 7f0758ba3700 1 --2- 172.21.15.172:0/2372088624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0754102820 0x7f0754102c80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f07480098f0 tx=0x7f0748009c00 comp rx=0 tx=0).stop 2023-11-28T01:08:50.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.716+0000 7f0758ba3700 1 -- 172.21.15.172:0/2372088624 shutdown_connections 2023-11-28T01:08:50.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.716+0000 7f0758ba3700 1 --2- 172.21.15.172:0/2372088624 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07541031c0 0x7f0754109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:50.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.716+0000 7f0758ba3700 1 --2- 172.21.15.172:0/2372088624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0754101620 0x7f0754101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:50.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.716+0000 7f0758ba3700 1 --2- 172.21.15.172:0/2372088624 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0754102820 0x7f0754102c80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:50.718 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.716+0000 7f0758ba3700 1 -- 172.21.15.172:0/2372088624 >> 172.21.15.172:0/2372088624 conn(0x7f07540fcbe0 msgr2=0x7f07540ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:08:50.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.716+0000 7f0758ba3700 1 -- 172.21.15.172:0/2372088624 shutdown_connections 2023-11-28T01:08:50.719 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.716+0000 7f0758ba3700 1 -- 172.21.15.172:0/2372088624 wait complete. 2023-11-28T01:08:50.722 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.716+0000 7f0758ba3700 1 Processor -- start 2023-11-28T01:08:50.722 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.716+0000 7f0758ba3700 1 -- start start 2023-11-28T01:08:50.722 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f0758ba3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0754101620 0x7f0754196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:50.722 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f0758ba3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0754102820 0x7f0754197060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:50.722 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f0758ba3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07541031c0 0x7f075419e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:50.722 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f0758ba3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075419e660 con 0x7f0754101620 2023-11-28T01:08:50.723 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f0758ba3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075419e7a0 con 0x7f07541031c0 2023-11-28T01:08:50.723 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f0758ba3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075419e930 con 0x7f0754102820 2023-11-28T01:08:50.723 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f075259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0754101620 0x7f0754196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:50.723 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f0751d9b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0754102820 0x7f0754197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:50.723 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f075259c700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0754101620 0x7f0754196b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49536/0 (socket says 172.21.15.172:49536) 2023-11-28T01:08:50.724 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f075259c700 1 -- 172.21.15.172:0/3903803275 learned_addr learned my addr 172.21.15.172:0/3903803275 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:08:50.724 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f0752d9d700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07541031c0 0x7f075419e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:50.724 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f075259c700 1 -- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0754102820 msgr2=0x7f0754197060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:50.724 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f075259c700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0754102820 0x7f0754197060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:50.726 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f075259c700 1 -- 172.21.15.172:0/3903803275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07541031c0 msgr2=0x7f075419e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:08:50.726 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f075259c700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07541031c0 0x7f075419e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:08:50.728 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f075259c700 1 -- 172.21.15.172:0/3903803275 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c009d00 con 0x7f0754101620 2023-11-28T01:08:50.728 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f0751d9b700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0754102820 0x7f0754197060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T01:08:50.728 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f075259c700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0754101620 0x7f0754196b20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f074400e980 tx=0x7f074400ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:50.728 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f07437fe700 1 -- 172.21.15.172:0/3903803275 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0744009770 con 0x7f0754101620 2023-11-28T01:08:50.728 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f07437fe700 1 -- 172.21.15.172:0/3903803275 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f074400c810 con 0x7f0754101620 2023-11-28T01:08:50.729 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f0758ba3700 1 -- 172.21.15.172:0/3903803275 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0748009520 con 0x7f0754101620 2023-11-28T01:08:50.729 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f07437fe700 1 -- 172.21.15.172:0/3903803275 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0744003a60 con 0x7f0754101620 2023-11-28T01:08:50.729 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.717+0000 7f0758ba3700 1 -- 172.21.15.172:0/3903803275 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075419ef30 con 0x7f0754101620 2023-11-28T01:08:50.729 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.718+0000 7f0758ba3700 1 -- 172.21.15.172:0/3903803275 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0754108ba0 con 0x7f0754101620 2023-11-28T01:08:50.729 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.719+0000 7f07437fe700 1 -- 172.21.15.172:0/3903803275 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 13) v1 ==== 45776+0+0 (secure 0 0 0) 0x7f0744003be0 con 0x7f0754101620 2023-11-28T01:08:50.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.719+0000 7f07437fe700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f073c038e70 0x7f073c03b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:08:50.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.719+0000 7f07437fe700 1 -- 172.21.15.172:0/3903803275 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0744014070 con 0x7f0754101620 2023-11-28T01:08:50.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.719+0000 7f0751d9b700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f073c038e70 0x7f073c03b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:08:50.730 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.719+0000 7f0751d9b700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f073c038e70 0x7f073c03b330 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f07480098c0 tx=0x7f0748019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:08:50.731 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.725+0000 7f07437fe700 1 -- 172.21.15.172:0/3903803275 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f074400c980 con 0x7f0754101620 2023-11-28T01:08:50.820 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:50 smithi192 conmon[108591]: 2023-11-28T01:08:50.372+0000 7f1290263200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-11-28T01:08:50.821 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:50 smithi192 conmon[108591]: 2023-11-28T01:08:50.459+0000 7f1290263200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-11-28T01:08:50.821 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:50 smithi192 conmon[108591]: 2023-11-28T01:08:50.606+0000 7f1290263200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-11-28T01:08:50.928 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:50.927+0000 7f0758ba3700 1 -- 172.21.15.172:0/3903803275 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f075410c5e0 con 0x7f073c038e70 2023-11-28T01:08:51.320 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:50 smithi192 conmon[108591]: 2023-11-28T01:08:50.868+0000 7f1290263200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-11-28T01:08:51.821 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:51 smithi192 conmon[108591]: 2023-11-28T01:08:51.564+0000 7f1290263200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-11-28T01:08:51.821 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:51 smithi192 conmon[108591]: 2023-11-28T01:08:51.694+0000 7f1290263200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-11-28T01:08:51.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:51 smithi192 ceph-mon[105371]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:51.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:51.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:51 smithi192 ceph-mon[105371]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:08:51.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:08:51.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:08:51.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:52.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[100627]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:52.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:52.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[100627]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:08:52.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:08:52.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:08:52.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:52.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[118784]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:52.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:08:52.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[118784]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:08:52.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:08:52.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:08:52.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:08:52.220 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:51 smithi192 conmon[108591]: 2023-11-28T01:08:51.887+0000 7f1290263200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-11-28T01:08:52.220 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:52 smithi192 conmon[108591]: 2023-11-28T01:08:52.001+0000 7f1290263200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-11-28T01:08:52.570 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:52 smithi192 conmon[108591]: 2023-11-28T01:08:52.218+0000 7f1290263200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-11-28T01:08:53.570 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:08:53 smithi192 conmon[108591]: 2023-11-28T01:08:53.076+0000 7f1290263200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-11-28T01:08:53.901 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:08:53.900+0000 7f07437fe700 1 -- 172.21.15.172:0/3903803275 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mgrmap(e 14) v1 ==== 90933+0+0 (secure 0 0 0) 0x7f074401dcf0 con 0x7f0754101620 2023-11-28T01:08:54.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:54.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: Standby manager daemon x started 2023-11-28T01:08:54.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-11-28T01:08:54.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-28T01:08:54.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/1535720228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8f0a47b-64f5-4ccd-9db0-e3fb0fd30997"}]: dispatch 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8f0a47b-64f5-4ccd-9db0-e3fb0fd30997"}]: dispatch 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8f0a47b-64f5-4ccd-9db0-e3fb0fd30997"}]': finished 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: osdmap e5: 1 total, 0 up, 1 in 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/196306713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: Standby manager daemon x started 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-28T01:08:54.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-11-28T01:08:54.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-28T01:08:54.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/1535720228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8f0a47b-64f5-4ccd-9db0-e3fb0fd30997"}]: dispatch 2023-11-28T01:08:54.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8f0a47b-64f5-4ccd-9db0-e3fb0fd30997"}]: dispatch 2023-11-28T01:08:54.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8f0a47b-64f5-4ccd-9db0-e3fb0fd30997"}]': finished 2023-11-28T01:08:54.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: osdmap e5: 1 total, 0 up, 1 in 2023-11-28T01:08:54.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:08:54.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:53 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/196306713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:08:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: Standby manager daemon x started 2023-11-28T01:08:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2023-11-28T01:08:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-11-28T01:08:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2023-11-28T01:08:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: from='mgr.? 172.21.15.192:0/1910083235' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-11-28T01:08:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/1535720228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8f0a47b-64f5-4ccd-9db0-e3fb0fd30997"}]: dispatch 2023-11-28T01:08:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e8f0a47b-64f5-4ccd-9db0-e3fb0fd30997"}]: dispatch 2023-11-28T01:08:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e8f0a47b-64f5-4ccd-9db0-e3fb0fd30997"}]': finished 2023-11-28T01:08:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: osdmap e5: 1 total, 0 up, 1 in 2023-11-28T01:08:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:08:54.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:53 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/196306713' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:08:55.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:54 smithi172 ceph-mon[100627]: mgrmap e14: y(active, since 63s), standbys: x 2023-11-28T01:08:55.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:54 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-28T01:08:55.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:54 smithi172 ceph-mon[118784]: mgrmap e14: y(active, since 63s), standbys: x 2023-11-28T01:08:55.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:54 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-28T01:08:55.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:54 smithi192 ceph-mon[105371]: mgrmap e14: y(active, since 63s), standbys: x 2023-11-28T01:08:55.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:54 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-11-28T01:08:56.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:55 smithi172 ceph-mon[100627]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:56.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:55 smithi172 ceph-mon[118784]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:56.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:55 smithi192 ceph-mon[105371]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:58.296 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:57 smithi172 ceph-mon[100627]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:58.297 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:57 smithi172 ceph-mon[118784]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:08:58.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:57 smithi192 ceph-mon[105371]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:00.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:59 smithi172 ceph-mon[100627]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:00.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:59 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-28T01:09:00.152 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:08:59 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:00.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:59 smithi172 ceph-mon[118784]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:00.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:59 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-28T01:09:00.152 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:08:59 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:00.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:59 smithi192 ceph-mon[105371]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:00.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:59 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-11-28T01:09:00.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:08:59 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:01.153 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:00 smithi172 ceph-mon[100627]: Deploying daemon osd.0 on smithi172 2023-11-28T01:09:01.153 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:00 smithi172 ceph-mon[118784]: Deploying daemon osd.0 on smithi172 2023-11-28T01:09:01.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:00 smithi192 ceph-mon[105371]: Deploying daemon osd.0 on smithi172 2023-11-28T01:09:02.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:01 smithi172 ceph-mon[100627]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:02.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:01 smithi172 ceph-mon[118784]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:02.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:01 smithi192 ceph-mon[105371]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:04.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:03 smithi172 ceph-mon[100627]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:04.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:03 smithi172 ceph-mon[118784]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:04.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:03 smithi192 ceph-mon[105371]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:05.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:05.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:05.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:05.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:05 smithi172 ceph-mon[100627]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:05.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:05.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:05.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:05.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:05 smithi172 ceph-mon[118784]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:05.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:05.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:05.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:05.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:05 smithi192 ceph-mon[105371]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:07.132 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.131+0000 7f07437fe700 1 -- 172.21.15.172:0/3903803275 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f075410c5e0 con 0x7f073c038e70 2023-11-28T01:09:07.133 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 0 on host 'smithi172' 2023-11-28T01:09:07.135 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.133+0000 7f0758ba3700 1 -- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f073c038e70 msgr2=0x7f073c03b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:07.135 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.133+0000 7f0758ba3700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f073c038e70 0x7f073c03b330 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f07480098c0 tx=0x7f0748019040 comp rx=0 tx=0).stop 2023-11-28T01:09:07.136 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.133+0000 7f0758ba3700 1 -- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0754101620 msgr2=0x7f0754196b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:07.136 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.133+0000 7f0758ba3700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0754101620 0x7f0754196b20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f074400e980 tx=0x7f074400ec90 comp rx=0 tx=0).stop 2023-11-28T01:09:07.136 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.133+0000 7f0758ba3700 1 -- 172.21.15.172:0/3903803275 shutdown_connections 2023-11-28T01:09:07.136 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.133+0000 7f0758ba3700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f073c038e70 0x7f073c03b330 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:07.136 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.133+0000 7f0758ba3700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07541031c0 0x7f075419e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:07.136 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.133+0000 7f0758ba3700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f0754101620 0x7f0754196b20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:07.136 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.133+0000 7f0758ba3700 1 --2- 172.21.15.172:0/3903803275 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f0754102820 0x7f0754197060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:07.136 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.133+0000 7f0758ba3700 1 -- 172.21.15.172:0/3903803275 >> 172.21.15.172:0/3903803275 conn(0x7f07540fcbe0 msgr2=0x7f0754105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:09:07.136 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.134+0000 7f0758ba3700 1 -- 172.21.15.172:0/3903803275 shutdown_connections 2023-11-28T01:09:07.136 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:07.134+0000 7f0758ba3700 1 -- 172.21.15.172:0/3903803275 wait complete. 2023-11-28T01:09:07.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:07.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:07.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:07.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:07.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:07.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[100627]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:07.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:07.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:07.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:07.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:07.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:07.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:07 smithi172 ceph-mon[118784]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:07.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:07 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:07.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:07 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:07.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:07 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:07.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:07 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:07.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:07 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:07.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:07 smithi192 ceph-mon[105371]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:07.722 DEBUG:teuthology.orchestra.run.smithi172:osd.0> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.0.service 2023-11-28T01:09:07.724 INFO:tasks.cephadm:Deploying osd.1 on smithi172 with /dev/vg_nvme/lv_3... 2023-11-28T01:09:07.724 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-28T01:09:07.755 INFO:journalctl@ceph.osd.0.smithi172.stdout:-- Logs begin at Tue 2023-11-28 00:47:41 UTC. -- 2023-11-28T01:09:08.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:08 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:08.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:08 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:08.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:08 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:08.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:08 smithi172 ceph-mon[100627]: from='osd.0 [v2:172.21.15.172:6802/2204873896,v1:172.21.15.172:6803/2204873896]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-28T01:09:08.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:08 smithi172 ceph-mon[100627]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-28T01:09:08.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:08 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:08.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:08 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:08.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:08 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:08.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:08 smithi172 ceph-mon[118784]: from='osd.0 [v2:172.21.15.172:6802/2204873896,v1:172.21.15.172:6803/2204873896]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-28T01:09:08.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:08 smithi172 ceph-mon[118784]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-28T01:09:08.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:08 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:08 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:08 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:08 smithi192 ceph-mon[105371]: from='osd.0 [v2:172.21.15.172:6802/2204873896,v1:172.21.15.172:6803/2204873896]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-28T01:09:08.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:08 smithi192 ceph-mon[105371]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-11-28T01:09:08.712 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:09:09.402 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:09:09 smithi172 conmon[125062]: 2023-11-28T01:09:09.146+0000 7f416c26a700 -1 osd.0 0 waiting for initial osdmap 2023-11-28T01:09:09.402 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:09:09 smithi172 conmon[125062]: 2023-11-28T01:09:09.148+0000 7f4166060700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T01:09:09.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[118784]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-28T01:09:09.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[118784]: from='osd.0 [v2:172.21.15.172:6802/2204873896,v1:172.21.15.172:6803/2204873896]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:09.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[118784]: osdmap e6: 1 total, 0 up, 1 in 2023-11-28T01:09:09.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:09.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[118784]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:09.403 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[118784]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:09.403 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[100627]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-28T01:09:09.403 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[100627]: from='osd.0 [v2:172.21.15.172:6802/2204873896,v1:172.21.15.172:6803/2204873896]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:09.403 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[100627]: osdmap e6: 1 total, 0 up, 1 in 2023-11-28T01:09:09.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:09.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[100627]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:09.404 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:09 smithi172 ceph-mon[100627]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:09.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:09 smithi192 ceph-mon[105371]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-11-28T01:09:09.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:09 smithi192 ceph-mon[105371]: from='osd.0 [v2:172.21.15.172:6802/2204873896,v1:172.21.15.172:6803/2204873896]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:09.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:09 smithi192 ceph-mon[105371]: osdmap e6: 1 total, 0 up, 1 in 2023-11-28T01:09:09.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:09 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:09.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:09 smithi192 ceph-mon[105371]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:09.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:09 smithi192 ceph-mon[105371]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:10.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:10 smithi172 ceph-mon[100627]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:09:10.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:10 smithi172 ceph-mon[100627]: osdmap e7: 1 total, 0 up, 1 in 2023-11-28T01:09:10.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:10 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:10.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:10 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:10.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:10 smithi172 ceph-mon[118784]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:09:10.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:10 smithi172 ceph-mon[118784]: osdmap e7: 1 total, 0 up, 1 in 2023-11-28T01:09:10.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:10 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:10.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:10 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:10.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:10 smithi192 ceph-mon[105371]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:09:10.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:10 smithi192 ceph-mon[105371]: osdmap e7: 1 total, 0 up, 1 in 2023-11-28T01:09:10.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:10 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:10.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:10 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:10.715 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-28T01:09:10.715 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-28T01:09:10.715 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2023-11-28T01:09:10.715 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2023-11-28T01:09:10.715 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01255 s, 836 MB/s 2023-11-28T01:09:10.715 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2023-11-28T01:09:11.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[100627]: purged_snaps scrub starts 2023-11-28T01:09:11.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[100627]: purged_snaps scrub ok 2023-11-28T01:09:11.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:11.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[100627]: osd.0 [v2:172.21.15.172:6802/2204873896,v1:172.21.15.172:6803/2204873896] boot 2023-11-28T01:09:11.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[100627]: osdmap e8: 1 total, 1 up, 1 in 2023-11-28T01:09:11.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:11.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[100627]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:11.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[118784]: purged_snaps scrub starts 2023-11-28T01:09:11.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[118784]: purged_snaps scrub ok 2023-11-28T01:09:11.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:11.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[118784]: osd.0 [v2:172.21.15.172:6802/2204873896,v1:172.21.15.172:6803/2204873896] boot 2023-11-28T01:09:11.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[118784]: osdmap e8: 1 total, 1 up, 1 in 2023-11-28T01:09:11.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:11.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:11 smithi172 ceph-mon[118784]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:11.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:11 smithi192 ceph-mon[105371]: purged_snaps scrub starts 2023-11-28T01:09:11.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:11 smithi192 ceph-mon[105371]: purged_snaps scrub ok 2023-11-28T01:09:11.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:11 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:11.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:11 smithi192 ceph-mon[105371]: osd.0 [v2:172.21.15.172:6802/2204873896,v1:172.21.15.172:6803/2204873896] boot 2023-11-28T01:09:11.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:11 smithi192 ceph-mon[105371]: osdmap e8: 1 total, 1 up, 1 in 2023-11-28T01:09:11.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:11 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-11-28T01:09:11.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:11 smithi192 ceph-mon[105371]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-28T01:09:11.713 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_3 2023-11-28T01:09:12.046 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:09:12.459 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[100627]: osdmap e9: 1 total, 1 up, 1 in 2023-11-28T01:09:12.459 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[100627]: Detected new or changed devices on smithi172 2023-11-28T01:09:12.459 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:12.459 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:12.459 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:12.459 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[100627]: Adjusting osd_memory_target on smithi172 to 16144M 2023-11-28T01:09:12.459 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:12.460 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:12.460 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:12.460 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:12.460 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[118784]: osdmap e9: 1 total, 1 up, 1 in 2023-11-28T01:09:12.460 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[118784]: Detected new or changed devices on smithi172 2023-11-28T01:09:12.460 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:12.460 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:12.460 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:12.460 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[118784]: Adjusting osd_memory_target on smithi172 to 16144M 2023-11-28T01:09:12.461 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:12.461 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:12.461 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:12.461 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:12 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:12.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:12 smithi192 ceph-mon[105371]: osdmap e9: 1 total, 1 up, 1 in 2023-11-28T01:09:12.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:12 smithi192 ceph-mon[105371]: Detected new or changed devices on smithi172 2023-11-28T01:09:12.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:12 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:12.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:12 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:12.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:12 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:12.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:12 smithi192 ceph-mon[105371]: Adjusting osd_memory_target on smithi172 to 16144M 2023-11-28T01:09:12.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:12 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:12.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:12 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:12.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:12 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:12.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:12 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:13.404 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.402+0000 7f1695c56700 1 -- 172.21.15.172:0/3123446572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1690104a70 msgr2=0x7f1690106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:13.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.402+0000 7f1695c56700 1 --2- 172.21.15.172:0/3123446572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1690104a70 0x7f1690106e60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1680009990 tx=0x7f1680009ca0 comp rx=0 tx=0).stop 2023-11-28T01:09:13.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.402+0000 7f1695c56700 1 -- 172.21.15.172:0/3123446572 shutdown_connections 2023-11-28T01:09:13.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.402+0000 7f1695c56700 1 --2- 172.21.15.172:0/3123446572 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1690102140 0x7f1690104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:13.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.402+0000 7f1695c56700 1 --2- 172.21.15.172:0/3123446572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16901073a0 0x7f1690109790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:13.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.402+0000 7f1695c56700 1 --2- 172.21.15.172:0/3123446572 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1690104a70 0x7f1690106e60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:13.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.402+0000 7f1695c56700 1 -- 172.21.15.172:0/3123446572 >> 172.21.15.172:0/3123446572 conn(0x7f16900fbad0 msgr2=0x7f16900fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:09:13.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.402+0000 7f1695c56700 1 -- 172.21.15.172:0/3123446572 shutdown_connections 2023-11-28T01:09:13.405 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.402+0000 7f1695c56700 1 -- 172.21.15.172:0/3123446572 wait complete. 2023-11-28T01:09:13.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f1695c56700 1 Processor -- start 2023-11-28T01:09:13.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f1695c56700 1 -- start start 2023-11-28T01:09:13.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f1695c56700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1690102140 0x7f1690196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:13.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f1695c56700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1690104a70 0x7f1690197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:13.407 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f1695c56700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16901073a0 0x7f169019e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f1695c56700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169019e5f0 con 0x7f16901073a0 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f1695c56700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169019e730 con 0x7f1690102140 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f1695c56700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169019e8c0 con 0x7f1690104a70 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f168effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1690104a70 0x7f1690197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f168ffff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16901073a0 0x7f169019e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f168effd700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1690104a70 0x7f1690197040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:34040/0 (socket says 172.21.15.172:34040) 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f168effd700 1 -- 172.21.15.172:0/2947659171 learned_addr learned my addr 172.21.15.172:0/2947659171 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.403+0000 7f168f7fe700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1690102140 0x7f1690196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f168effd700 1 -- 172.21.15.172:0/2947659171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1690102140 msgr2=0x7f1690196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f168effd700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1690102140 0x7f1690196b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f168effd700 1 -- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16901073a0 msgr2=0x7f169019e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:13.408 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f168effd700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16901073a0 0x7f169019e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:13.409 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f168effd700 1 -- 172.21.15.172:0/2947659171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16800095f0 con 0x7f1690104a70 2023-11-28T01:09:13.409 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f168ffff700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16901073a0 0x7f169019e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T01:09:13.409 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f168effd700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1690104a70 0x7f1690197040 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1680004360 tx=0x7f16800039f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:09:13.409 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f168cff9700 1 -- 172.21.15.172:0/2947659171 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f168001d070 con 0x7f1690104a70 2023-11-28T01:09:13.409 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f168cff9700 1 -- 172.21.15.172:0/2947659171 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f168000bd10 con 0x7f1690104a70 2023-11-28T01:09:13.409 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f1695c56700 1 -- 172.21.15.172:0/2947659171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16900fff30 con 0x7f1690104a70 2023-11-28T01:09:13.409 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f168cff9700 1 -- 172.21.15.172:0/2947659171 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1680005020 con 0x7f1690104a70 2023-11-28T01:09:13.409 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.404+0000 7f1695c56700 1 -- 172.21.15.172:0/2947659171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1690100370 con 0x7f1690104a70 2023-11-28T01:09:13.409 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.405+0000 7f1695c56700 1 -- 172.21.15.172:0/2947659171 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16900fd650 con 0x7f1690104a70 2023-11-28T01:09:13.409 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.405+0000 7f168cff9700 1 -- 172.21.15.172:0/2947659171 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 14) v1 ==== 90933+0+0 (secure 0 0 0) 0x7f168000b8c0 con 0x7f1690104a70 2023-11-28T01:09:13.410 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.406+0000 7f168cff9700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f167c06d2f0 0x7f167c06f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:13.410 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.406+0000 7f168cff9700 1 -- 172.21.15.172:0/2947659171 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f168008cac0 con 0x7f1690104a70 2023-11-28T01:09:13.410 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.406+0000 7f168f7fe700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f167c06d2f0 0x7f167c06f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:13.411 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.406+0000 7f168f7fe700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f167c06d2f0 0x7f167c06f7b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1678009c60 tx=0x7f1678009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:09:13.412 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.410+0000 7f168cff9700 1 -- 172.21.15.172:0/2947659171 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f1680057930 con 0x7f1690104a70 2023-11-28T01:09:13.609 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:13.606+0000 7f1695c56700 1 -- 172.21.15.172:0/2947659171 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f169010c430 con 0x7f167c06d2f0 2023-11-28T01:09:14.183 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:13 smithi172 ceph-mon[100627]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:14.183 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:09:14.183 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:09:14.183 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:14.183 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:13 smithi172 ceph-mon[118784]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:14.183 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:09:14.183 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:09:14.183 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:14.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:13 smithi192 ceph-mon[105371]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:14.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:09:14.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:09:14.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:15.168 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:14 smithi172 ceph-mon[100627]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:09:15.168 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:14 smithi172 ceph-mon[118784]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:09:15.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:14 smithi192 ceph-mon[105371]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:09:16.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:15 smithi172 ceph-mon[100627]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:16.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:15 smithi172 ceph-mon[118784]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:16.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:15 smithi192 ceph-mon[105371]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:17.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:16 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/1876421065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ef0c783-7288-49f0-b832-a3398aaa19a8"}]: dispatch 2023-11-28T01:09:17.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:16 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/1876421065' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ef0c783-7288-49f0-b832-a3398aaa19a8"}]': finished 2023-11-28T01:09:17.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:16 smithi172 ceph-mon[100627]: osdmap e10: 2 total, 1 up, 2 in 2023-11-28T01:09:17.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:17.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:16 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2329830236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:09:17.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:16 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/1876421065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ef0c783-7288-49f0-b832-a3398aaa19a8"}]: dispatch 2023-11-28T01:09:17.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:16 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/1876421065' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ef0c783-7288-49f0-b832-a3398aaa19a8"}]': finished 2023-11-28T01:09:17.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:16 smithi172 ceph-mon[118784]: osdmap e10: 2 total, 1 up, 2 in 2023-11-28T01:09:17.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:16 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:17.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:16 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2329830236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:09:17.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:16 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/1876421065' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ef0c783-7288-49f0-b832-a3398aaa19a8"}]: dispatch 2023-11-28T01:09:17.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:16 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/1876421065' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ef0c783-7288-49f0-b832-a3398aaa19a8"}]': finished 2023-11-28T01:09:17.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:16 smithi192 ceph-mon[105371]: osdmap e10: 2 total, 1 up, 2 in 2023-11-28T01:09:17.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:16 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:17.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:16 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2329830236' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:09:18.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:17 smithi172 ceph-mon[118784]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:18.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:17 smithi172 ceph-mon[100627]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:18.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:17 smithi192 ceph-mon[105371]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:20.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:19 smithi172 ceph-mon[100627]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:20.286 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:19 smithi172 ceph-mon[118784]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:20.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:19 smithi192 ceph-mon[105371]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:22.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:21 smithi172 ceph-mon[100627]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:22.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:21 smithi172 ceph-mon[118784]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:22.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:21 smithi192 ceph-mon[105371]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:23.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:22 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-28T01:09:23.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:22 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:23.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:22 smithi172 ceph-mon[100627]: Deploying daemon osd.1 on smithi172 2023-11-28T01:09:23.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:22 smithi172 ceph-mon[100627]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:23.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:22 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-28T01:09:23.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:22 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:23.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:22 smithi172 ceph-mon[118784]: Deploying daemon osd.1 on smithi172 2023-11-28T01:09:23.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:22 smithi172 ceph-mon[118784]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:23.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:22 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-11-28T01:09:23.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:22 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:23.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:22 smithi192 ceph-mon[105371]: Deploying daemon osd.1 on smithi172 2023-11-28T01:09:23.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:22 smithi192 ceph-mon[105371]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:26.041 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:25 smithi172 ceph-mon[100627]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:26.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:25 smithi172 ceph-mon[118784]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:26.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:25 smithi192 ceph-mon[105371]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:28.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:27 smithi172 ceph-mon[118784]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:28.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:27 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:28.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:27 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:28.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:27 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:28.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:27 smithi172 ceph-mon[100627]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:28.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:27 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:28.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:27 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:28.056 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:27 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:28.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:27 smithi192 ceph-mon[105371]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:28.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:27 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:28.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:27 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:28.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:27 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:30.008 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.005+0000 7f168cff9700 1 -- 172.21.15.172:0/2947659171 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f169010c430 con 0x7f167c06d2f0 2023-11-28T01:09:30.008 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 1 on host 'smithi172' 2023-11-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 -- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f167c06d2f0 msgr2=0x7f167c06f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f167c06d2f0 0x7f167c06f7b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1678009c60 tx=0x7f1678009260 comp rx=0 tx=0).stop 2023-11-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 -- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1690104a70 msgr2=0x7f1690197040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1690104a70 0x7f1690197040 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1680004360 tx=0x7f16800039f0 comp rx=0 tx=0).stop 2023-11-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 -- 172.21.15.172:0/2947659171 shutdown_connections 2023-11-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f167c06d2f0 0x7f167c06f7b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f16901073a0 0x7f169019e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:30.010 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1690102140 0x7f1690196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 --2- 172.21.15.172:0/2947659171 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1690104a70 0x7f1690197040 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 -- 172.21.15.172:0/2947659171 >> 172.21.15.172:0/2947659171 conn(0x7f16900fbad0 msgr2=0x7f1690102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 -- 172.21.15.172:0/2947659171 shutdown_connections 2023-11-28T01:09:30.011 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:30.008+0000 7f1695c56700 1 -- 172.21.15.172:0/2947659171 wait complete. 2023-11-28T01:09:30.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[100627]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:30.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:30.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:30.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:30.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:30.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:30.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[118784]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:30.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:30.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:30.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:30.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:30.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:30.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:29 smithi192 ceph-mon[105371]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:30.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:30.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:30.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:30.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:30.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:30.607 DEBUG:teuthology.orchestra.run.smithi172:osd.1> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.1.service 2023-11-28T01:09:30.609 INFO:tasks.cephadm:Deploying osd.2 on smithi172 with /dev/vg_nvme/lv_2... 2023-11-28T01:09:30.609 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-28T01:09:30.640 INFO:journalctl@ceph.osd.1.smithi172.stdout:-- Logs begin at Tue 2023-11-28 00:47:41 UTC. -- 2023-11-28T01:09:31.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:31.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:31.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:31.164 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[100627]: from='osd.1 [v2:172.21.15.172:6810/2715954141,v1:172.21.15.172:6811/2715954141]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-28T01:09:31.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[100627]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-28T01:09:31.165 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[100627]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:31.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:31.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:31.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:31.165 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[118784]: from='osd.1 [v2:172.21.15.172:6810/2715954141,v1:172.21.15.172:6811/2715954141]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-28T01:09:31.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[118784]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-28T01:09:31.166 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:30 smithi172 ceph-mon[118784]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:31.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:30 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:31.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:30 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:31.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:30 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:31.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:30 smithi192 ceph-mon[105371]: from='osd.1 [v2:172.21.15.172:6810/2715954141,v1:172.21.15.172:6811/2715954141]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-28T01:09:31.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:30 smithi192 ceph-mon[105371]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-11-28T01:09:31.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:30 smithi192 ceph-mon[105371]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:31.612 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:09:32.138 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:31 smithi172 ceph-mon[100627]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-28T01:09:32.139 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:31 smithi172 ceph-mon[100627]: osdmap e11: 2 total, 1 up, 2 in 2023-11-28T01:09:32.139 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:31 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:32.139 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:31 smithi172 ceph-mon[100627]: from='osd.1 [v2:172.21.15.172:6810/2715954141,v1:172.21.15.172:6811/2715954141]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:32.139 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:31 smithi172 ceph-mon[100627]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:32.139 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:31 smithi172 ceph-mon[118784]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-28T01:09:32.139 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:31 smithi172 ceph-mon[118784]: osdmap e11: 2 total, 1 up, 2 in 2023-11-28T01:09:32.139 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:32.139 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:31 smithi172 ceph-mon[118784]: from='osd.1 [v2:172.21.15.172:6810/2715954141,v1:172.21.15.172:6811/2715954141]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:32.139 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:31 smithi172 ceph-mon[118784]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:32.140 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:09:31 smithi172 conmon[134833]: 2023-11-28T01:09:31.983+0000 7fca47aa8700 -1 osd.1 0 waiting for initial osdmap 2023-11-28T01:09:32.140 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:09:31 smithi172 conmon[134833]: 2023-11-28T01:09:31.986+0000 7fca3f89a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T01:09:32.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:31 smithi192 ceph-mon[105371]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-11-28T01:09:32.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:31 smithi192 ceph-mon[105371]: osdmap e11: 2 total, 1 up, 2 in 2023-11-28T01:09:32.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:31 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:32.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:31 smithi192 ceph-mon[105371]: from='osd.1 [v2:172.21.15.172:6810/2715954141,v1:172.21.15.172:6811/2715954141]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:32.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:31 smithi192 ceph-mon[105371]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:33.207 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[100627]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:09:33.207 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[100627]: osdmap e12: 2 total, 1 up, 2 in 2023-11-28T01:09:33.207 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:33.207 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:33.207 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[100627]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:33.208 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:33.208 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[118784]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:09:33.208 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[118784]: osdmap e12: 2 total, 1 up, 2 in 2023-11-28T01:09:33.208 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:33.209 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:33.209 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[118784]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:33.209 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:32 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:33.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:32 smithi192 ceph-mon[105371]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:09:33.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:32 smithi192 ceph-mon[105371]: osdmap e12: 2 total, 1 up, 2 in 2023-11-28T01:09:33.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:32 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:33.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:32 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:33.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:32 smithi192 ceph-mon[105371]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-11-28T01:09:33.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:32 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:33.573 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-28T01:09:33.573 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-28T01:09:33.573 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2023-11-28T01:09:33.573 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2023-11-28T01:09:33.573 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111381 s, 941 MB/s 2023-11-28T01:09:33.573 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2023-11-28T01:09:34.152 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_2 2023-11-28T01:09:34.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: purged_snaps scrub starts 2023-11-28T01:09:34.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: purged_snaps scrub ok 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: osd.1 [v2:172.21.15.172:6810/2715954141,v1:172.21.15.172:6811/2715954141] boot 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: osdmap e13: 2 total, 2 up, 2 in 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: Detected new or changed devices on smithi172 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: Adjusting osd_memory_target on smithi172 to 8072M 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: purged_snaps scrub starts 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: purged_snaps scrub ok 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: osd.1 [v2:172.21.15.172:6810/2715954141,v1:172.21.15.172:6811/2715954141] boot 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: osdmap e13: 2 total, 2 up, 2 in 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: Detected new or changed devices on smithi172 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: Adjusting osd_memory_target on smithi172 to 8072M 2023-11-28T01:09:34.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:34.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:34.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:33 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: purged_snaps scrub starts 2023-11-28T01:09:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: purged_snaps scrub ok 2023-11-28T01:09:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: osd.1 [v2:172.21.15.172:6810/2715954141,v1:172.21.15.172:6811/2715954141] boot 2023-11-28T01:09:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: osdmap e13: 2 total, 2 up, 2 in 2023-11-28T01:09:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-11-28T01:09:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: Detected new or changed devices on smithi172 2023-11-28T01:09:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:34.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: Adjusting osd_memory_target on smithi172 to 8072M 2023-11-28T01:09:34.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:34.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:34.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:33 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:34.493 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:09:35.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:35 smithi172 ceph-mon[100627]: osdmap e14: 2 total, 2 up, 2 in 2023-11-28T01:09:35.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:35 smithi172 ceph-mon[100627]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:35.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:35 smithi172 ceph-mon[118784]: osdmap e14: 2 total, 2 up, 2 in 2023-11-28T01:09:35.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:35 smithi172 ceph-mon[118784]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:35.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:35 smithi192 ceph-mon[105371]: osdmap e14: 2 total, 2 up, 2 in 2023-11-28T01:09:35.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:35 smithi192 ceph-mon[105371]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:35.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 -- 172.21.15.172:0/1590839569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f288c101b20 msgr2=0x7f288c101f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:35.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 --2- 172.21.15.172:0/1590839569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f288c101b20 0x7f288c101f40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2888009990 tx=0x7f2888009ca0 comp rx=0 tx=0).stop 2023-11-28T01:09:35.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 -- 172.21.15.172:0/1590839569 shutdown_connections 2023-11-28T01:09:35.889 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 --2- 172.21.15.172:0/1590839569 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f288c101b20 0x7f288c101f40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:35.890 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 --2- 172.21.15.172:0/1590839569 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f288c101140 0x7f288c1089e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:35.890 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 --2- 172.21.15.172:0/1590839569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f288c100780 0x7f288c100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:35.890 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 -- 172.21.15.172:0/1590839569 >> 172.21.15.172:0/1590839569 conn(0x7f288c0fb230 msgr2=0x7f288c0fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:09:35.890 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 -- 172.21.15.172:0/1590839569 shutdown_connections 2023-11-28T01:09:35.890 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 -- 172.21.15.172:0/1590839569 wait complete. 2023-11-28T01:09:35.890 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 Processor -- start 2023-11-28T01:09:35.890 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 -- start start 2023-11-28T01:09:35.892 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f288c100780 0x7f288c1044d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:35.892 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f288c101140 0x7f288c102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:35.892 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f288c101b20 0x7f288c1030b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288c103850 con 0x7f288c101140 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288c1039c0 con 0x7f288c101b20 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.888+0000 7f289467e700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f288c1a0b40 con 0x7f288c100780 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f2891c19700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f288c101140 0x7f288c102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f2891c19700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f288c101140 0x7f288c102b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49676/0 (socket says 172.21.15.172:49676) 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f2891c19700 1 -- 172.21.15.172:0/3632256290 learned_addr learned my addr 172.21.15.172:0/3632256290 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f289241a700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f288c100780 0x7f288c1044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f2892c1b700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f288c101b20 0x7f288c1030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f2891c19700 1 -- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f288c100780 msgr2=0x7f288c1044d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f2891c19700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f288c100780 0x7f288c1044d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f2891c19700 1 -- 172.21.15.172:0/3632256290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f288c101b20 msgr2=0x7f288c1030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f2891c19700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f288c101b20 0x7f288c1030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:35.893 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f2891c19700 1 -- 172.21.15.172:0/3632256290 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2884009c40 con 0x7f288c101140 2023-11-28T01:09:35.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f289241a700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f288c100780 0x7f288c1044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T01:09:35.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f2891c19700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f288c101140 0x7f288c102b70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f287c00d950 tx=0x7f287c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:09:35.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f28837fe700 1 -- 172.21.15.172:0/3632256290 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c009940 con 0x7f288c101140 2023-11-28T01:09:35.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f28837fe700 1 -- 172.21.15.172:0/3632256290 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f287c010460 con 0x7f288c101140 2023-11-28T01:09:35.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f289467e700 1 -- 172.21.15.172:0/3632256290 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28880095f0 con 0x7f288c101140 2023-11-28T01:09:35.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f28837fe700 1 -- 172.21.15.172:0/3632256290 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c00f620 con 0x7f288c101140 2023-11-28T01:09:35.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.889+0000 7f289467e700 1 -- 172.21.15.172:0/3632256290 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288c1a1140 con 0x7f288c101140 2023-11-28T01:09:35.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.890+0000 7f289467e700 1 -- 172.21.15.172:0/3632256290 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f288c04f290 con 0x7f288c101140 2023-11-28T01:09:35.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.891+0000 7f28837fe700 1 -- 172.21.15.172:0/3632256290 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 14) v1 ==== 90933+0+0 (secure 0 0 0) 0x7f287c009aa0 con 0x7f288c101140 2023-11-28T01:09:35.894 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.891+0000 7f28837fe700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f287806d2f0 0x7f287806f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:35.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.892+0000 7f28837fe700 1 -- 172.21.15.172:0/3632256290 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2199+0+0 (secure 0 0 0) 0x7f287c08b930 con 0x7f288c101140 2023-11-28T01:09:35.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.892+0000 7f289241a700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f287806d2f0 0x7f287806f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:35.895 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.892+0000 7f289241a700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f287806d2f0 0x7f287806f7b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f288800b2e0 tx=0x7f2888005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:09:35.897 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:35.896+0000 7f28837fe700 1 -- 172.21.15.172:0/3632256290 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f287c056680 con 0x7f288c101140 2023-11-28T01:09:36.097 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:36.096+0000 7f289467e700 1 -- 172.21.15.172:0/3632256290 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f288c0ffd80 con 0x7f287806d2f0 2023-11-28T01:09:36.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:09:36.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:09:36.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:36 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:36.553 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:09:36.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:09:36.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:36 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:36.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:09:36.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:09:36.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:36 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:37.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:37 smithi172 ceph-mon[100627]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:09:37.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:37 smithi172 ceph-mon[100627]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:37.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:37 smithi172 ceph-mon[118784]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:09:37.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:37 smithi172 ceph-mon[118784]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:37.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:37 smithi192 ceph-mon[105371]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:09:37.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:37 smithi192 ceph-mon[105371]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:38 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/4265806531' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e762414e-fc5a-4aad-83a8-2f6c636bfde9"}]: dispatch 2023-11-28T01:09:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:38 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/4265806531' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e762414e-fc5a-4aad-83a8-2f6c636bfde9"}]': finished 2023-11-28T01:09:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:38 smithi172 ceph-mon[100627]: osdmap e15: 3 total, 2 up, 3 in 2023-11-28T01:09:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:38 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:39.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:38 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/4265806531' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e762414e-fc5a-4aad-83a8-2f6c636bfde9"}]: dispatch 2023-11-28T01:09:39.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:38 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/4265806531' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e762414e-fc5a-4aad-83a8-2f6c636bfde9"}]': finished 2023-11-28T01:09:39.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:38 smithi172 ceph-mon[118784]: osdmap e15: 3 total, 2 up, 3 in 2023-11-28T01:09:39.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:38 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:39.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:38 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/4265806531' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e762414e-fc5a-4aad-83a8-2f6c636bfde9"}]: dispatch 2023-11-28T01:09:39.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:38 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/4265806531' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e762414e-fc5a-4aad-83a8-2f6c636bfde9"}]': finished 2023-11-28T01:09:39.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:38 smithi192 ceph-mon[105371]: osdmap e15: 3 total, 2 up, 3 in 2023-11-28T01:09:39.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:38 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:40.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:39 smithi172 ceph-mon[100627]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:40.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:39 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/3847078583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:09:40.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:39 smithi172 ceph-mon[118784]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:40.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:39 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/3847078583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:09:40.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:39 smithi192 ceph-mon[105371]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:40.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:39 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/3847078583' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:09:42.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:41 smithi172 ceph-mon[100627]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:42.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:41 smithi172 ceph-mon[118784]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:42.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:41 smithi192 ceph-mon[105371]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:44.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:43 smithi172 ceph-mon[100627]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:44.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:43 smithi172 ceph-mon[118784]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:44.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:43 smithi192 ceph-mon[105371]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:44.918 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-28T01:09:44.918 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:45.270 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-28T01:09:45.271 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:45.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-11-28T01:09:45.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:46.259 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:45 smithi172 ceph-mon[100627]: Deploying daemon osd.2 on smithi172 2023-11-28T01:09:46.259 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:45 smithi172 ceph-mon[100627]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:46.259 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:45 smithi172 ceph-mon[118784]: Deploying daemon osd.2 on smithi172 2023-11-28T01:09:46.259 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:45 smithi172 ceph-mon[118784]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:46.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:45 smithi192 ceph-mon[105371]: Deploying daemon osd.2 on smithi172 2023-11-28T01:09:46.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:45 smithi192 ceph-mon[105371]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:48.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:47 smithi172 ceph-mon[100627]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:48.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:47 smithi172 ceph-mon[118784]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:48.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:47 smithi192 ceph-mon[105371]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:50.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:49 smithi172 ceph-mon[100627]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:50.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:49 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:50.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:49 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:50.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:49 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:50.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:49 smithi172 ceph-mon[118784]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:50.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:49 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:50.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:49 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:50.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:49 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:50.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:49 smithi192 ceph-mon[105371]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:50.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:49 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:50.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:49 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:50.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:49 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:52.240 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[100627]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:52.240 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:52.240 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:52.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:52.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:52.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:52.241 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[118784]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:52.241 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:52.241 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:52.241 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:52.241 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:52.241 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:52.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:51 smithi192 ceph-mon[105371]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:52.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:52.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:52.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:52.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:52.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:52.845 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.843+0000 7f28837fe700 1 -- 172.21.15.172:0/3632256290 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f288c0ffd80 con 0x7f287806d2f0 2023-11-28T01:09:52.845 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 2 on host 'smithi172' 2023-11-28T01:09:52.847 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.846+0000 7f289467e700 1 -- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f287806d2f0 msgr2=0x7f287806f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:52.847 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.846+0000 7f289467e700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f287806d2f0 0x7f287806f7b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f288800b2e0 tx=0x7f2888005fd0 comp rx=0 tx=0).stop 2023-11-28T01:09:52.847 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.846+0000 7f289467e700 1 -- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f288c101140 msgr2=0x7f288c102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:52.848 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.846+0000 7f289467e700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f288c101140 0x7f288c102b70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f287c00d950 tx=0x7f287c00dc60 comp rx=0 tx=0).stop 2023-11-28T01:09:52.850 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.848+0000 7f289467e700 1 -- 172.21.15.172:0/3632256290 shutdown_connections 2023-11-28T01:09:52.850 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.848+0000 7f289467e700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f287806d2f0 0x7f287806f7b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:52.850 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.848+0000 7f289467e700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f288c100780 0x7f288c1044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:52.850 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.848+0000 7f289467e700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f288c101b20 0x7f288c1030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:52.850 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.848+0000 7f289467e700 1 --2- 172.21.15.172:0/3632256290 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f288c101140 0x7f288c102b70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:52.850 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.848+0000 7f289467e700 1 -- 172.21.15.172:0/3632256290 >> 172.21.15.172:0/3632256290 conn(0x7f288c0fb230 msgr2=0x7f288c0ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:09:52.850 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.849+0000 7f289467e700 1 -- 172.21.15.172:0/3632256290 shutdown_connections 2023-11-28T01:09:52.850 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:52.849+0000 7f289467e700 1 -- 172.21.15.172:0/3632256290 wait complete. 2023-11-28T01:09:53.380 DEBUG:teuthology.orchestra.run.smithi172:osd.2> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.2.service 2023-11-28T01:09:53.382 INFO:tasks.cephadm:Deploying osd.3 on smithi172 with /dev/vg_nvme/lv_1... 2023-11-28T01:09:53.383 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-28T01:09:53.413 INFO:journalctl@ceph.osd.2.smithi172.stdout:-- Logs begin at Tue 2023-11-28 00:47:41 UTC. -- 2023-11-28T01:09:53.898 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:09:53.978 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[100627]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:53.978 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:53.978 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:53.978 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:53.978 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[100627]: from='osd.2 [v2:172.21.15.172:6818/3771992877,v1:172.21.15.172:6819/3771992877]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-28T01:09:53.978 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[100627]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-28T01:09:53.979 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[118784]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:53.979 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:53.979 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:53.979 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:53.979 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[118784]: from='osd.2 [v2:172.21.15.172:6818/3771992877,v1:172.21.15.172:6819/3771992877]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-28T01:09:53.979 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:53 smithi172 ceph-mon[118784]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-28T01:09:54.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:53 smithi192 ceph-mon[105371]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:54.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:53 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:54.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:53 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:54.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:53 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:09:54.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:53 smithi192 ceph-mon[105371]: from='osd.2 [v2:172.21.15.172:6818/3771992877,v1:172.21.15.172:6819/3771992877]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-28T01:09:54.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:53 smithi192 ceph-mon[105371]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-11-28T01:09:54.952 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:09:54 smithi172 conmon[143893]: 2023-11-28T01:09:54.824+0000 7f2144169700 -1 osd.2 0 waiting for initial osdmap 2023-11-28T01:09:54.952 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:09:54 smithi172 conmon[143893]: 2023-11-28T01:09:54.829+0000 7f213e760700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T01:09:54.953 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:54 smithi172 ceph-mon[100627]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-28T01:09:54.953 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:54 smithi172 ceph-mon[100627]: osdmap e16: 3 total, 2 up, 3 in 2023-11-28T01:09:54.953 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:54 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:54.953 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:54 smithi172 ceph-mon[100627]: from='osd.2 [v2:172.21.15.172:6818/3771992877,v1:172.21.15.172:6819/3771992877]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:54.953 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:54 smithi172 ceph-mon[100627]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:54.953 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:54 smithi172 ceph-mon[118784]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-28T01:09:54.953 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:54 smithi172 ceph-mon[118784]: osdmap e16: 3 total, 2 up, 3 in 2023-11-28T01:09:54.954 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:54 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:54.954 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:54 smithi172 ceph-mon[118784]: from='osd.2 [v2:172.21.15.172:6818/3771992877,v1:172.21.15.172:6819/3771992877]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:54.954 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:54 smithi172 ceph-mon[118784]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:55.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:54 smithi192 ceph-mon[105371]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-11-28T01:09:55.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:54 smithi192 ceph-mon[105371]: osdmap e16: 3 total, 2 up, 3 in 2023-11-28T01:09:55.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:54 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:55.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:54 smithi192 ceph-mon[105371]: from='osd.2 [v2:172.21.15.172:6818/3771992877,v1:172.21.15.172:6819/3771992877]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:55.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:54 smithi192 ceph-mon[105371]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:09:55.840 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[118784]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:55.841 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[118784]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:09:55.841 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[118784]: osdmap e17: 3 total, 2 up, 3 in 2023-11-28T01:09:55.841 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:55.841 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:55.841 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:56.285 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-28T01:09:56.286 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-28T01:09:56.286 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2023-11-28T01:09:56.286 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2023-11-28T01:09:56.286 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112475 s, 932 MB/s 2023-11-28T01:09:56.286 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2023-11-28T01:09:56.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[100627]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:56.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[100627]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:09:56.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[100627]: osdmap e17: 3 total, 2 up, 3 in 2023-11-28T01:09:56.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:56.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:56.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:55 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:56.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:55 smithi192 ceph-mon[105371]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:56.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:55 smithi192 ceph-mon[105371]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:09:56.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:55 smithi192 ceph-mon[105371]: osdmap e17: 3 total, 2 up, 3 in 2023-11-28T01:09:56.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:55 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:56.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:55 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:56.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:55 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:57.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_1 2023-11-28T01:09:57.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: purged_snaps scrub starts 2023-11-28T01:09:57.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: purged_snaps scrub ok 2023-11-28T01:09:57.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: osd.2 [v2:172.21.15.172:6818/3771992877,v1:172.21.15.172:6819/3771992877] boot 2023-11-28T01:09:57.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: osdmap e18: 3 total, 3 up, 3 in 2023-11-28T01:09:57.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:57.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.305 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:57.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:57.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:57.307 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.307 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:57.307 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:57.307 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: purged_snaps scrub starts 2023-11-28T01:09:57.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: purged_snaps scrub ok 2023-11-28T01:09:57.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: osd.2 [v2:172.21.15.172:6818/3771992877,v1:172.21.15.172:6819/3771992877] boot 2023-11-28T01:09:57.308 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: osdmap e18: 3 total, 3 up, 3 in 2023-11-28T01:09:57.309 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:57.309 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.309 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.309 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:57.310 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:57.310 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:57.310 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.310 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:57.311 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:57.311 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:57 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: purged_snaps scrub starts 2023-11-28T01:09:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: purged_snaps scrub ok 2023-11-28T01:09:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: osd.2 [v2:172.21.15.172:6818/3771992877,v1:172.21.15.172:6819/3771992877] boot 2023-11-28T01:09:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: osdmap e18: 3 total, 3 up, 3 in 2023-11-28T01:09:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-11-28T01:09:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:09:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:57.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:09:57.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:57 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:09:57.383 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:09:58.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[118784]: Detected new or changed devices on smithi172 2023-11-28T01:09:58.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[118784]: Adjusting osd_memory_target on smithi172 to 5381M 2023-11-28T01:09:58.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[118784]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:58.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-28T01:09:58.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-28T01:09:58.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[118784]: osdmap e19: 3 total, 3 up, 3 in 2023-11-28T01:09:58.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-28T01:09:58.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[100627]: Detected new or changed devices on smithi172 2023-11-28T01:09:58.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[100627]: Adjusting osd_memory_target on smithi172 to 5381M 2023-11-28T01:09:58.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[100627]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:58.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-28T01:09:58.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-28T01:09:58.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[100627]: osdmap e19: 3 total, 3 up, 3 in 2023-11-28T01:09:58.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:58 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-28T01:09:58.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:58 smithi192 ceph-mon[105371]: Detected new or changed devices on smithi172 2023-11-28T01:09:58.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:58 smithi192 ceph-mon[105371]: Adjusting osd_memory_target on smithi172 to 5381M 2023-11-28T01:09:58.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:58 smithi192 ceph-mon[105371]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-11-28T01:09:58.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:58 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-11-28T01:09:58.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:58 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-11-28T01:09:58.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:58 smithi192 ceph-mon[105371]: osdmap e19: 3 total, 3 up, 3 in 2023-11-28T01:09:58.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:58 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-11-28T01:09:58.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.756+0000 7f838b7e5700 1 -- 172.21.15.172:0/4044480644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8384100510 msgr2=0x7f8384100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:58.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.756+0000 7f838b7e5700 1 --2- 172.21.15.172:0/4044480644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8384100510 0x7f8384100930 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8378009990 tx=0x7f8378009ca0 comp rx=0 tx=0).stop 2023-11-28T01:09:58.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 -- 172.21.15.172:0/4044480644 shutdown_connections 2023-11-28T01:09:58.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 --2- 172.21.15.172:0/4044480644 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f8384100510 0x7f8384100930 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:58.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 --2- 172.21.15.172:0/4044480644 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f83841020b0 0x7f8384108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:58.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 --2- 172.21.15.172:0/4044480644 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8384101710 0x7f8384101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:58.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 -- 172.21.15.172:0/4044480644 >> 172.21.15.172:0/4044480644 conn(0x7f83840fbad0 msgr2=0x7f83840fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:09:58.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 -- 172.21.15.172:0/4044480644 shutdown_connections 2023-11-28T01:09:58.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 -- 172.21.15.172:0/4044480644 wait complete. 2023-11-28T01:09:58.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 Processor -- start 2023-11-28T01:09:58.761 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 -- start start 2023-11-28T01:09:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8384100510 0x7f838406a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8384101710 0x7f838419b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83841020b0 0x7f838419de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838419e5b0 con 0x7f8384100510 2023-11-28T01:09:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838419e720 con 0x7f8384101710 2023-11-28T01:09:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.757+0000 7f838b7e5700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838419e8e0 con 0x7f83841020b0 2023-11-28T01:09:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f8389581700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8384100510 0x7f838406a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f8389581700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8384100510 0x7f838406a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49772/0 (socket says 172.21.15.172:49772) 2023-11-28T01:09:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f8389581700 1 -- 172.21.15.172:0/2359599611 learned_addr learned my addr 172.21.15.172:0/2359599611 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:09:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f8388d80700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8384101710 0x7f838419b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:58.762 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f8389d82700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83841020b0 0x7f838419de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:58.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f8389581700 1 -- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83841020b0 msgr2=0x7f838419de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:58.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f8389581700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83841020b0 0x7f838419de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:58.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f8389581700 1 -- 172.21.15.172:0/2359599611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8384101710 msgr2=0x7f838419b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:09:58.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f8389581700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8384101710 0x7f838419b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:09:58.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f8389581700 1 -- 172.21.15.172:0/2359599611 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c009c40 con 0x7f8384100510 2023-11-28T01:09:58.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f8389581700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8384100510 0x7f838406a310 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8378000c00 tx=0x7f83780038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:09:58.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f83767fc700 1 -- 172.21.15.172:0/2359599611 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837801d070 con 0x7f8384100510 2023-11-28T01:09:58.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f838b7e5700 1 -- 172.21.15.172:0/2359599611 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83780095f0 con 0x7f8384100510 2023-11-28T01:09:58.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f83767fc700 1 -- 172.21.15.172:0/2359599611 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f837800bd10 con 0x7f8384100510 2023-11-28T01:09:58.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f838b7e5700 1 -- 172.21.15.172:0/2359599611 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83841958f0 con 0x7f8384100510 2023-11-28T01:09:58.763 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.758+0000 7f83767fc700 1 -- 172.21.15.172:0/2359599611 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8378022cc0 con 0x7f8384100510 2023-11-28T01:09:58.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.759+0000 7f838b7e5700 1 -- 172.21.15.172:0/2359599611 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8384107a90 con 0x7f8384100510 2023-11-28T01:09:58.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.760+0000 7f83767fc700 1 -- 172.21.15.172:0/2359599611 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 14) v1 ==== 90933+0+0 (secure 0 0 0) 0x7f837800b8c0 con 0x7f8384100510 2023-11-28T01:09:58.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.760+0000 7f83767fc700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f837006d2f0 0x7f837006f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:09:58.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.760+0000 7f83767fc700 1 -- 172.21.15.172:0/2359599611 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3017+0+0 (secure 0 0 0) 0x7f837808d2e0 con 0x7f8384100510 2023-11-28T01:09:58.764 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.760+0000 7f8388d80700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f837006d2f0 0x7f837006f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:09:58.765 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.761+0000 7f8388d80700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f837006d2f0 0x7f837006f7b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8380005950 tx=0x7f838000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:09:58.768 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.765+0000 7f83767fc700 1 -- 172.21.15.172:0/2359599611 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f8378057c70 con 0x7f8384100510 2023-11-28T01:09:58.980 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:09:58.977+0000 7f838b7e5700 1 -- 172.21.15.172:0/2359599611 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f838410b2d0 con 0x7f837006d2f0 2023-11-28T01:09:59.040 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:09:58 smithi172 sudo[150534]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-28T01:09:59.040 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:09:58 smithi172 sudo[150534]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:09:59.040 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:09:58 smithi172 sudo[150534]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:09:59.040 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:09:58 smithi172 sudo[150534]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:09:59.041 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-28T01:09:59.042 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[100627]: osdmap e20: 3 total, 3 up, 3 in 2023-11-28T01:09:59.042 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[100627]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:09:59.042 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:09:59.042 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:09:59.042 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:59.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-28T01:09:59.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[118784]: osdmap e20: 3 total, 3 up, 3 in 2023-11-28T01:09:59.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[118784]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:09:59.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:09:59.042 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:09:59.043 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:09:59 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:59.303 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150600]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-28T01:09:59.303 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150600]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:09:59.303 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150600]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:09:59.303 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150600]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:09:59.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:59 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-11-28T01:09:59.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:59 smithi192 ceph-mon[105371]: osdmap e20: 3 total, 3 up, 3 in 2023-11-28T01:09:59.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:59 smithi192 ceph-mon[105371]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:09:59.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:59 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:09:59.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:59 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:09:59.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:09:59 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:09:59.591 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150690]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-28T01:09:59.592 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150690]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:09:59.592 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150690]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:09:59.592 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150690]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:09:59.867 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150775]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-28T01:09:59.867 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150775]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:09:59.867 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150775]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:09:59.867 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150775]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:10:00.086 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:00.084+0000 7f83767fc700 1 -- 172.21.15.172:0/2359599611 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7f8378058040 con 0x7f8384100510 2023-11-28T01:10:00.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 ceph-mon[100627]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:10:00.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 ceph-mon[100627]: osdmap e21: 3 total, 3 up, 3 in 2023-11-28T01:10:00.131 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:00.132 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 ceph-mon[100627]: overall HEALTH_OK 2023-11-28T01:10:00.132 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150805]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-28T01:10:00.132 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150805]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:10:00.132 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150805]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:10:00.132 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:09:59 smithi172 sudo[150805]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:10:00.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:00 smithi172 ceph-mon[118784]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:10:00.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:00 smithi172 ceph-mon[118784]: osdmap e21: 3 total, 3 up, 3 in 2023-11-28T01:10:00.132 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:00 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:00.133 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:00 smithi172 ceph-mon[118784]: overall HEALTH_OK 2023-11-28T01:10:00.308 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:00 smithi192 ceph-mon[105371]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:10:00.308 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:00 smithi192 ceph-mon[105371]: osdmap e21: 3 total, 3 up, 3 in 2023-11-28T01:10:00.308 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:00 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:00.308 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:00 smithi192 ceph-mon[105371]: overall HEALTH_OK 2023-11-28T01:10:00.391 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150844]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-28T01:10:00.392 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150844]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:10:00.392 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150844]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:10:00.392 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150841]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-28T01:10:00.392 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150841]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:10:00.392 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150841]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:10:00.392 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150841]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:10:00.654 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150844]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:10:01.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150904]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-28T01:10:01.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150904]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:10:01.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150904]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:10:01.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:00 smithi172 sudo[150904]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:10:01.151 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:00 smithi192 sudo[109494]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-28T01:10:01.151 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:00 smithi192 sudo[109494]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:10:01.151 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:00 smithi192 sudo[109494]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:10:01.151 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:00 smithi192 sudo[109494]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:10:01.432 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[100627]: mgrmap e15: y(active, since 2m), standbys: x 2023-11-28T01:10:01.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[100627]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-28T01:10:01.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[100627]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-28T01:10:01.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:10:01.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:10:01.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:10:01.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:10:01.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:10:01.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:10:01.433 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[100627]: pgmap v76: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:01.434 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[118784]: mgrmap e15: y(active, since 2m), standbys: x 2023-11-28T01:10:01.434 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[118784]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-28T01:10:01.434 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[118784]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-28T01:10:01.434 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:10:01.434 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:10:01.434 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:10:01.434 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:10:01.434 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:10:01.434 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:10:01.434 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 ceph-mon[118784]: pgmap v76: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:01.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 sudo[109497]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-28T01:10:01.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 sudo[109497]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:10:01.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 sudo[109497]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:10:01.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 sudo[109497]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:10:01.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 ceph-mon[105371]: mgrmap e15: y(active, since 2m), standbys: x 2023-11-28T01:10:01.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 ceph-mon[105371]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-28T01:10:01.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 ceph-mon[105371]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-28T01:10:01.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:10:01.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:10:01.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:10:01.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:10:01.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:10:01.573 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:10:01.573 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:01 smithi192 ceph-mon[105371]: pgmap v76: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:01.689 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 sudo[150977]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-11-28T01:10:01.689 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 sudo[150977]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:10:01.689 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 sudo[150977]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:10:01.689 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 sudo[150977]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:10:02.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 sudo[150981]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-11-28T01:10:02.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 sudo[150981]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-11-28T01:10:02.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 sudo[150981]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-11-28T01:10:02.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:01 smithi172 sudo[150981]: pam_unix(sudo:session): session closed for user root 2023-11-28T01:10:02.553 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-28T01:10:02.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-28T01:10:02.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:02.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-28T01:10:02.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:10:02.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:10:02.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:10:02.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/3363111586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c57f1e20-cf1c-4a89-82ab-274033efe6b6"}]: dispatch 2023-11-28T01:10:02.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-28T01:10:02.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/3363111586' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c57f1e20-cf1c-4a89-82ab-274033efe6b6"}]': finished 2023-11-28T01:10:02.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: osdmap e22: 4 total, 3 up, 4 in 2023-11-28T01:10:02.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:02.556 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-28T01:10:02.556 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-28T01:10:02.556 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:02.556 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-28T01:10:02.556 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:10:02.557 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:10:02.557 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:10:02.557 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/3363111586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c57f1e20-cf1c-4a89-82ab-274033efe6b6"}]: dispatch 2023-11-28T01:10:02.557 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-28T01:10:02.557 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/3363111586' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c57f1e20-cf1c-4a89-82ab-274033efe6b6"}]': finished 2023-11-28T01:10:02.557 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: osdmap e22: 4 total, 3 up, 4 in 2023-11-28T01:10:02.558 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:02.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-28T01:10:02.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-28T01:10:02.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:02.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-11-28T01:10:02.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-11-28T01:10:02.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-11-28T01:10:02.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-11-28T01:10:02.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/3363111586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c57f1e20-cf1c-4a89-82ab-274033efe6b6"}]: dispatch 2023-11-28T01:10:02.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-11-28T01:10:02.572 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/3363111586' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c57f1e20-cf1c-4a89-82ab-274033efe6b6"}]': finished 2023-11-28T01:10:02.573 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: osdmap e22: 4 total, 3 up, 4 in 2023-11-28T01:10:02.573 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:03.553 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:03 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/912369010' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:10:03.553 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:03 smithi172 ceph-mon[118784]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:03.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:03 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/912369010' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:10:03.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:03 smithi172 ceph-mon[100627]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:03.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:03 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/912369010' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:10:03.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:03 smithi192 ceph-mon[105371]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:05.778 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:05 smithi172 ceph-mon[118784]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.8 KiB/s rd, 230 KiB/s wr, 15 op/s 2023-11-28T01:10:06.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:05 smithi172 ceph-mon[100627]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.8 KiB/s rd, 230 KiB/s wr, 15 op/s 2023-11-28T01:10:06.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:05 smithi192 ceph-mon[105371]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.8 KiB/s rd, 230 KiB/s wr, 15 op/s 2023-11-28T01:10:08.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:07 smithi172 ceph-mon[100627]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-11-28T01:10:08.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:07 smithi172 ceph-mon[118784]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-11-28T01:10:08.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:07 smithi192 ceph-mon[105371]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-11-28T01:10:09.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:08 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-28T01:10:09.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:08 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:09.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:08 smithi172 ceph-mon[118784]: Deploying daemon osd.3 on smithi172 2023-11-28T01:10:09.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:08 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-28T01:10:09.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:08 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:09.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:08 smithi172 ceph-mon[100627]: Deploying daemon osd.3 on smithi172 2023-11-28T01:10:09.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:08 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-11-28T01:10:09.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:08 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:09.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:08 smithi192 ceph-mon[105371]: Deploying daemon osd.3 on smithi172 2023-11-28T01:10:10.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:09 smithi172 ceph-mon[100627]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-11-28T01:10:10.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:09 smithi172 ceph-mon[118784]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-11-28T01:10:10.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:09 smithi192 ceph-mon[105371]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-11-28T01:10:12.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:11 smithi172 ceph-mon[100627]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-11-28T01:10:17.002 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:11 smithi172 ceph-mon[118784]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-11-28T01:10:17.003 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:13 smithi172 ceph-mon[100627]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-11-28T01:10:17.003 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.003 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.003 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:17.003 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:13 smithi172 ceph-mon[118784]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-11-28T01:10:17.003 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.004 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.004 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:17.004 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.129+0000 7f83767fc700 1 -- 172.21.15.172:0/2359599611 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f838410b2d0 con 0x7f837006d2f0 2023-11-28T01:10:17.004 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 3 on host 'smithi172' 2023-11-28T01:10:17.005 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.131+0000 7f838b7e5700 1 -- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f837006d2f0 msgr2=0x7f837006f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:10:17.005 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.131+0000 7f838b7e5700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f837006d2f0 0x7f837006f7b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8380005950 tx=0x7f838000b410 comp rx=0 tx=0).stop 2023-11-28T01:10:17.005 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.131+0000 7f838b7e5700 1 -- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8384100510 msgr2=0x7f838406a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:10:17.005 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.131+0000 7f838b7e5700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8384100510 0x7f838406a310 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8378000c00 tx=0x7f83780038b0 comp rx=0 tx=0).stop 2023-11-28T01:10:17.005 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.131+0000 7f838b7e5700 1 -- 172.21.15.172:0/2359599611 shutdown_connections 2023-11-28T01:10:17.006 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.131+0000 7f838b7e5700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f837006d2f0 0x7f837006f7b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:17.006 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.131+0000 7f838b7e5700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f83841020b0 0x7f838419de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:17.006 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.131+0000 7f838b7e5700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f8384100510 0x7f838406a310 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:17.006 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.131+0000 7f838b7e5700 1 --2- 172.21.15.172:0/2359599611 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8384101710 0x7f838419b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:17.006 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.131+0000 7f838b7e5700 1 -- 172.21.15.172:0/2359599611 >> 172.21.15.172:0/2359599611 conn(0x7f83840fbad0 msgr2=0x7f8384104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:10:17.006 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.132+0000 7f838b7e5700 1 -- 172.21.15.172:0/2359599611 shutdown_connections 2023-11-28T01:10:17.006 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:10:16.132+0000 7f838b7e5700 1 -- 172.21.15.172:0/2359599611 wait complete. 2023-11-28T01:10:17.006 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:15 smithi172 ceph-mon[100627]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-11-28T01:10:17.006 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:15 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.006 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:15 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.007 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:15 smithi172 ceph-mon[118784]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-11-28T01:10:17.007 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:15 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.007 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:15 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.008 DEBUG:teuthology.orchestra.run.smithi172:osd.3> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.3.service 2023-11-28T01:10:17.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:17.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:17.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[118784]: from='osd.3 [v2:172.21.15.172:6826/659880403,v1:172.21.15.172:6827/659880403]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-28T01:10:17.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[118784]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-28T01:10:17.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.009 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:17.010 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:17.010 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:17.010 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.010 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[100627]: from='osd.3 [v2:172.21.15.172:6826/659880403,v1:172.21.15.172:6827/659880403]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-28T01:10:17.010 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[100627]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-28T01:10:17.010 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.010 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.010 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:17.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:11 smithi192 ceph-mon[105371]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-11-28T01:10:17.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:13 smithi192 ceph-mon[105371]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.4 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-11-28T01:10:17.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:17.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:15 smithi192 ceph-mon[105371]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-11-28T01:10:17.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:15 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.011 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:15 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.012 INFO:tasks.cephadm:Deploying osd.4 on smithi192 with /dev/vg_nvme/lv_4... 2023-11-28T01:10:17.012 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-11-28T01:10:17.046 INFO:journalctl@ceph.osd.3.smithi172.stdout:-- Logs begin at Tue 2023-11-28 00:47:41 UTC. -- 2023-11-28T01:10:17.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:16 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:17.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:16 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:17.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:16 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:16 smithi192 ceph-mon[105371]: from='osd.3 [v2:172.21.15.172:6826/659880403,v1:172.21.15.172:6827/659880403]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-28T01:10:17.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:16 smithi192 ceph-mon[105371]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-11-28T01:10:17.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:16 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:16 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:17.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:16 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:17.361 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:10:17.977 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[100627]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:17.977 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[100627]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-28T01:10:17.977 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[100627]: osdmap e23: 4 total, 3 up, 4 in 2023-11-28T01:10:17.977 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:17.977 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[100627]: from='osd.3 [v2:172.21.15.172:6826/659880403,v1:172.21.15.172:6827/659880403]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:10:17.977 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[100627]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:10:17.977 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[118784]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:17.978 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[118784]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-28T01:10:17.978 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[118784]: osdmap e23: 4 total, 3 up, 4 in 2023-11-28T01:10:17.978 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:17.978 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[118784]: from='osd.3 [v2:172.21.15.172:6826/659880403,v1:172.21.15.172:6827/659880403]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:10:17.978 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:17 smithi172 ceph-mon[118784]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:10:18.303 INFO:journalctl@ceph.osd.3.smithi172.stdout:Nov 28 01:10:18 smithi172 conmon[153551]: 2023-11-28T01:10:18.028+0000 7ff6c31df700 -1 osd.3 0 waiting for initial osdmap 2023-11-28T01:10:18.304 INFO:journalctl@ceph.osd.3.smithi172.stdout:Nov 28 01:10:18 smithi172 conmon[153551]: 2023-11-28T01:10:18.031+0000 7ff6bc7d4700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T01:10:18.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:17 smithi192 ceph-mon[105371]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:18.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:17 smithi192 ceph-mon[105371]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-11-28T01:10:18.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:17 smithi192 ceph-mon[105371]: osdmap e23: 4 total, 3 up, 4 in 2023-11-28T01:10:18.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:17 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:18.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:17 smithi192 ceph-mon[105371]: from='osd.3 [v2:172.21.15.172:6826/659880403,v1:172.21.15.172:6827/659880403]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:10:18.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:17 smithi192 ceph-mon[105371]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2023-11-28T01:10:19.180 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-11-28T01:10:19.180 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-11-28T01:10:19.180 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-11-28T01:10:19.180 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-11-28T01:10:19.180 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109044 s, 962 MB/s 2023-11-28T01:10:19.180 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-11-28T01:10:19.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:19 smithi172 ceph-mon[100627]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:10:19.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:19 smithi172 ceph-mon[100627]: osdmap e24: 4 total, 3 up, 4 in 2023-11-28T01:10:19.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:19 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:19.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:19 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:19.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:19 smithi172 ceph-mon[100627]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:19.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:19 smithi172 ceph-mon[118784]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:10:19.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:19 smithi172 ceph-mon[118784]: osdmap e24: 4 total, 3 up, 4 in 2023-11-28T01:10:19.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:19 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:19.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:19 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:19.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:19 smithi172 ceph-mon[118784]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:19.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:19 smithi192 ceph-mon[105371]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2023-11-28T01:10:19.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:19 smithi192 ceph-mon[105371]: osdmap e24: 4 total, 3 up, 4 in 2023-11-28T01:10:19.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:19 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:19.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:19 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:19.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:19 smithi192 ceph-mon[105371]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-11-28T01:10:19.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2023-11-28T01:10:20.138 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:10:20.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: purged_snaps scrub starts 2023-11-28T01:10:20.382 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: purged_snaps scrub ok 2023-11-28T01:10:20.382 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: osd.3 [v2:172.21.15.172:6826/659880403,v1:172.21.15.172:6827/659880403] boot 2023-11-28T01:10:20.382 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: osdmap e25: 4 total, 4 up, 4 in 2023-11-28T01:10:20.383 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:20.383 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: Detected new or changed devices on smithi172 2023-11-28T01:10:20.383 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:20.383 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:20.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: Adjusting osd_memory_target on smithi172 to 4036M 2023-11-28T01:10:20.384 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:20.385 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:20.385 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:20.385 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:20.385 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: purged_snaps scrub starts 2023-11-28T01:10:20.385 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: purged_snaps scrub ok 2023-11-28T01:10:20.386 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: osd.3 [v2:172.21.15.172:6826/659880403,v1:172.21.15.172:6827/659880403] boot 2023-11-28T01:10:20.386 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: osdmap e25: 4 total, 4 up, 4 in 2023-11-28T01:10:20.386 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:20.387 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: Detected new or changed devices on smithi172 2023-11-28T01:10:20.387 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:20.387 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:20.387 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.388 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.388 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.388 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.388 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: Adjusting osd_memory_target on smithi172 to 4036M 2023-11-28T01:10:20.388 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:20.389 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:20.389 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:20.389 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:20 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:20.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: purged_snaps scrub starts 2023-11-28T01:10:20.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: purged_snaps scrub ok 2023-11-28T01:10:20.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: osd.3 [v2:172.21.15.172:6826/659880403,v1:172.21.15.172:6827/659880403] boot 2023-11-28T01:10:20.391 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: osdmap e25: 4 total, 4 up, 4 in 2023-11-28T01:10:20.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-11-28T01:10:20.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: Detected new or changed devices on smithi172 2023-11-28T01:10:20.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:20.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:20.392 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:20.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: Adjusting osd_memory_target on smithi172 to 4036M 2023-11-28T01:10:20.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:20.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:20.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:20.394 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:20 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:21.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:21 smithi172 ceph-mon[100627]: osdmap e26: 4 total, 4 up, 4 in 2023-11-28T01:10:21.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:21 smithi172 ceph-mon[100627]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:21.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:21 smithi172 ceph-mon[118784]: osdmap e26: 4 total, 4 up, 4 in 2023-11-28T01:10:21.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:21 smithi172 ceph-mon[118784]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:21.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:21 smithi192 ceph-mon[105371]: osdmap e26: 4 total, 4 up, 4 in 2023-11-28T01:10:21.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:21 smithi192 ceph-mon[105371]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:21.611 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.609+0000 7f4621208700 1 -- 172.21.15.192:0/2902635023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f461c1031c0 msgr2=0x7f461c109a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:10:21.611 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.609+0000 7f4621208700 1 --2- 172.21.15.192:0/2902635023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f461c1031c0 0x7f461c109a70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f461400b0d0 tx=0x7f461400b3e0 comp rx=0 tx=0).stop 2023-11-28T01:10:21.611 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.609+0000 7f4621208700 1 -- 172.21.15.192:0/2902635023 shutdown_connections 2023-11-28T01:10:21.611 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.609+0000 7f4621208700 1 --2- 172.21.15.192:0/2902635023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f461c1031c0 0x7f461c109a70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:21.611 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.609+0000 7f4621208700 1 --2- 172.21.15.192:0/2902635023 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c101620 0x7f461c101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:21.611 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.609+0000 7f4621208700 1 --2- 172.21.15.192:0/2902635023 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c102820 0x7f461c102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:21.612 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.609+0000 7f4621208700 1 -- 172.21.15.192:0/2902635023 >> 172.21.15.192:0/2902635023 conn(0x7f461c0fcbe0 msgr2=0x7f461c0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:10:21.612 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.609+0000 7f4621208700 1 -- 172.21.15.192:0/2902635023 shutdown_connections 2023-11-28T01:10:21.612 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.609+0000 7f4621208700 1 -- 172.21.15.192:0/2902635023 wait complete. 2023-11-28T01:10:21.614 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f4621208700 1 Processor -- start 2023-11-28T01:10:21.614 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f4621208700 1 -- start start 2023-11-28T01:10:21.614 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f4621208700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c101620 0x7f461c075320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:10:21.614 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f4621208700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c102820 0x7f461c0739c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:10:21.615 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f4621208700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f461c1031c0 0x7f461c073f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:10:21.615 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f4621208700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c19a900 con 0x7f461c102820 2023-11-28T01:10:21.615 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f4621208700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f461c074440 con 0x7f461c1031c0 2023-11-28T01:10:21.615 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f4621208700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461c074600 con 0x7f461c101620 2023-11-28T01:10:21.615 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f461b59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f461c1031c0 0x7f461c073f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:10:21.615 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f461b59e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f461c1031c0 0x7f461c073f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:49184/0 (socket says 172.21.15.192:49184) 2023-11-28T01:10:21.615 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f461b59e700 1 -- 172.21.15.192:0/1505898895 learned_addr learned my addr 172.21.15.192:0/1505898895 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:10:21.616 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f461ad9d700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c101620 0x7f461c075320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:10:21.616 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f461a59c700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c102820 0x7f461c0739c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:10:21.616 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f461b59e700 1 -- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c101620 msgr2=0x7f461c075320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:10:21.616 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f461b59e700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c101620 0x7f461c075320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:21.616 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f461b59e700 1 -- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c102820 msgr2=0x7f461c0739c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:10:21.616 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f461b59e700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c102820 0x7f461c0739c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:21.616 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.610+0000 7f461b59e700 1 -- 172.21.15.192:0/1505898895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4614009d00 con 0x7f461c1031c0 2023-11-28T01:10:21.616 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.611+0000 7f461b59e700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f461c1031c0 0x7f461c073f00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4614007fd0 tx=0x7f46140090d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:10:21.617 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.611+0000 7f4603fff700 1 -- 172.21.15.192:0/1505898895 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461400e040 con 0x7f461c1031c0 2023-11-28T01:10:21.617 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.611+0000 7f4603fff700 1 -- 172.21.15.192:0/1505898895 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f4614003c00 con 0x7f461c1031c0 2023-11-28T01:10:21.617 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.611+0000 7f4603fff700 1 -- 172.21.15.192:0/1505898895 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461401a650 con 0x7f461c1031c0 2023-11-28T01:10:21.617 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.611+0000 7f4621208700 1 -- 172.21.15.192:0/1505898895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f461c1a52b0 con 0x7f461c1031c0 2023-11-28T01:10:21.617 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.611+0000 7f4621208700 1 -- 172.21.15.192:0/1505898895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461c1a5610 con 0x7f461c1031c0 2023-11-28T01:10:21.617 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.612+0000 7f4621208700 1 -- 172.21.15.192:0/1505898895 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f461c1089e0 con 0x7f461c1031c0 2023-11-28T01:10:21.617 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.612+0000 7f4603fff700 1 -- 172.21.15.192:0/1505898895 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7f4614023410 con 0x7f461c1031c0 2023-11-28T01:10:21.618 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.613+0000 7f4603fff700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f460406d380 0x7f460406f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:10:21.618 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.613+0000 7f4603fff700 1 -- 172.21.15.192:0/1505898895 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3449+0+0 (secure 0 0 0) 0x7f461408ce70 con 0x7f461c1031c0 2023-11-28T01:10:21.618 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.613+0000 7f461ad9d700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f460406d380 0x7f460406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:10:21.618 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.613+0000 7f461ad9d700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f460406d380 0x7f460406f840 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f461c102680 tx=0x7f460c00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:10:21.619 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.617+0000 7f4603fff700 1 -- 172.21.15.192:0/1505898895 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f46140575f0 con 0x7f461c1031c0 2023-11-28T01:10:21.816 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:21.814+0000 7f4621208700 1 -- 172.21.15.192:0/1505898895 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f461c10c4b0 con 0x7f460406d380 2023-11-28T01:10:22.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:22 smithi192 ceph-mon[105371]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:10:22.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:22 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:10:22.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:22 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:10:22.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:22 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:22.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:22 smithi172 ceph-mon[100627]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:10:22.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:22 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:10:22.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:22 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:10:22.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:22 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:22.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:22 smithi172 ceph-mon[118784]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:10:22.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:22 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:10:22.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:22 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:10:22.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:22 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:24.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:23 smithi172 ceph-mon[100627]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:24.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:23 smithi172 ceph-mon[118784]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:24.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:23 smithi192 ceph-mon[105371]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:25.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/393055559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2097369-4ab9-4e3a-8f60-ea410df7ec59"}]: dispatch 2023-11-28T01:10:25.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[100627]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2097369-4ab9-4e3a-8f60-ea410df7ec59"}]: dispatch 2023-11-28T01:10:25.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[100627]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2097369-4ab9-4e3a-8f60-ea410df7ec59"}]': finished 2023-11-28T01:10:25.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[100627]: osdmap e27: 5 total, 4 up, 5 in 2023-11-28T01:10:25.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:25.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/2701661634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:10:25.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/393055559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2097369-4ab9-4e3a-8f60-ea410df7ec59"}]: dispatch 2023-11-28T01:10:25.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[118784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2097369-4ab9-4e3a-8f60-ea410df7ec59"}]: dispatch 2023-11-28T01:10:25.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[118784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2097369-4ab9-4e3a-8f60-ea410df7ec59"}]': finished 2023-11-28T01:10:25.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[118784]: osdmap e27: 5 total, 4 up, 5 in 2023-11-28T01:10:25.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:25.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:24 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/2701661634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:10:25.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:24 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/393055559' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2097369-4ab9-4e3a-8f60-ea410df7ec59"}]: dispatch 2023-11-28T01:10:25.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:24 smithi192 ceph-mon[105371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d2097369-4ab9-4e3a-8f60-ea410df7ec59"}]: dispatch 2023-11-28T01:10:25.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:24 smithi192 ceph-mon[105371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d2097369-4ab9-4e3a-8f60-ea410df7ec59"}]': finished 2023-11-28T01:10:25.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:24 smithi192 ceph-mon[105371]: osdmap e27: 5 total, 4 up, 5 in 2023-11-28T01:10:25.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:24 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:25.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:24 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/2701661634' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:10:26.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:25 smithi172 ceph-mon[100627]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:26.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:25 smithi172 ceph-mon[118784]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:26.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:25 smithi192 ceph-mon[105371]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:28.231 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:27 smithi192 ceph-mon[105371]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:28.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:27 smithi172 ceph-mon[100627]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:28.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:27 smithi172 ceph-mon[118784]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:30.277 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:29 smithi192 ceph-mon[105371]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:30.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:29 smithi172 ceph-mon[100627]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:30.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:29 smithi172 ceph-mon[118784]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:31.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:30 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-28T01:10:31.273 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:30 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:31.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:30 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-28T01:10:31.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:30 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:31.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:30 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-11-28T01:10:31.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:30 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:32.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:31 smithi172 ceph-mon[100627]: Deploying daemon osd.4 on smithi192 2023-11-28T01:10:32.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:31 smithi172 ceph-mon[100627]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:32.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:31 smithi172 ceph-mon[118784]: Deploying daemon osd.4 on smithi192 2023-11-28T01:10:32.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:31 smithi172 ceph-mon[118784]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:32.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:31 smithi192 ceph-mon[105371]: Deploying daemon osd.4 on smithi192 2023-11-28T01:10:32.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:31 smithi192 ceph-mon[105371]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:34.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:33 smithi172 ceph-mon[100627]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:34.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:33 smithi172 ceph-mon[118784]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:33 smithi192 ceph-mon[105371]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:36.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:35 smithi192 ceph-mon[105371]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:36.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:35 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:36.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:35 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:36.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:35 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:36.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:35 smithi172 ceph-mon[100627]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:36.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:35 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:36.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:35 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:36.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:35 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:36.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:35 smithi172 ceph-mon[118784]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:36.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:35 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:36.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:35 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:36.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:35 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:38.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:37 smithi192 ceph-mon[105371]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:38.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:37 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:38.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:37 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:38.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:37 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:38.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:37 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:38.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:37 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:38.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[100627]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:38.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:38.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:38.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:38.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:38.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:38.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[118784]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:38.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:38.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:38.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:38.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:38.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:37 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:38.376 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.374+0000 7f4603fff700 1 -- 172.21.15.192:0/1505898895 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f461c10c4b0 con 0x7f460406d380 2023-11-28T01:10:38.376 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 4 on host 'smithi192' 2023-11-28T01:10:38.378 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 -- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f460406d380 msgr2=0x7f460406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:10:38.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f460406d380 0x7f460406f840 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f461c102680 tx=0x7f460c00a3b0 comp rx=0 tx=0).stop 2023-11-28T01:10:38.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 -- 172.21.15.192:0/1505898895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f461c1031c0 msgr2=0x7f461c073f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:10:38.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f461c1031c0 0x7f461c073f00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4614007fd0 tx=0x7f46140090d0 comp rx=0 tx=0).stop 2023-11-28T01:10:38.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 -- 172.21.15.192:0/1505898895 shutdown_connections 2023-11-28T01:10:38.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f460406d380 0x7f460406f840 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:38.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f461c101620 0x7f461c075320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:38.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f461c1031c0 0x7f461c073f00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:38.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 --2- 172.21.15.192:0/1505898895 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f461c102820 0x7f461c0739c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:38.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 -- 172.21.15.192:0/1505898895 >> 172.21.15.192:0/1505898895 conn(0x7f461c0fcbe0 msgr2=0x7f461c105b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:10:38.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 -- 172.21.15.192:0/1505898895 shutdown_connections 2023-11-28T01:10:38.379 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:38.377+0000 7f4621208700 1 -- 172.21.15.192:0/1505898895 wait complete. 2023-11-28T01:10:38.932 DEBUG:teuthology.orchestra.run.smithi192:osd.4> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.4.service 2023-11-28T01:10:38.935 INFO:tasks.cephadm:Deploying osd.5 on smithi192 with /dev/vg_nvme/lv_3... 2023-11-28T01:10:38.935 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-11-28T01:10:38.965 INFO:journalctl@ceph.osd.4.smithi192.stdout:-- Logs begin at Tue 2023-11-28 00:47:11 UTC. -- 2023-11-28T01:10:39.216 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:38 smithi192 ceph-mon[105371]: from='osd.4 [v2:172.21.15.192:6800/1069750856,v1:172.21.15.192:6801/1069750856]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-28T01:10:39.216 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:38 smithi192 ceph-mon[105371]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-28T01:10:39.216 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:38 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:39.216 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:38 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:39.216 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:38 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:39.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:38 smithi172 ceph-mon[100627]: from='osd.4 [v2:172.21.15.192:6800/1069750856,v1:172.21.15.192:6801/1069750856]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-28T01:10:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:38 smithi172 ceph-mon[100627]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-28T01:10:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:38 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:38 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:39.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:38 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:39.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:38 smithi172 ceph-mon[118784]: from='osd.4 [v2:172.21.15.192:6800/1069750856,v1:172.21.15.192:6801/1069750856]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-28T01:10:39.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:38 smithi172 ceph-mon[118784]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-11-28T01:10:39.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:38 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:39.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:38 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:39.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:38 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:40.000 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:10:40.219 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:10:39 smithi192 conmon[113405]: 2023-11-28T01:10:39.963+0000 7f0b5715f700 -1 osd.4 0 waiting for initial osdmap 2023-11-28T01:10:40.219 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:10:39 smithi192 conmon[113405]: 2023-11-28T01:10:39.967+0000 7f0b52f59700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T01:10:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:39 smithi192 ceph-mon[105371]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:39 smithi192 ceph-mon[105371]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-28T01:10:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:39 smithi192 ceph-mon[105371]: osdmap e28: 5 total, 4 up, 5 in 2023-11-28T01:10:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:39 smithi192 ceph-mon[105371]: from='osd.4 [v2:172.21.15.192:6800/1069750856,v1:172.21.15.192:6801/1069750856]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:10:40.220 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:39 smithi192 ceph-mon[105371]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:10:40.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[100627]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:40.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[100627]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-28T01:10:40.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[100627]: osdmap e28: 5 total, 4 up, 5 in 2023-11-28T01:10:40.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:40.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[100627]: from='osd.4 [v2:172.21.15.192:6800/1069750856,v1:172.21.15.192:6801/1069750856]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:10:40.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[100627]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:10:40.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[118784]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:40.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[118784]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-11-28T01:10:40.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[118784]: osdmap e28: 5 total, 4 up, 5 in 2023-11-28T01:10:40.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:40.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[118784]: from='osd.4 [v2:172.21.15.192:6800/1069750856,v1:172.21.15.192:6801/1069750856]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:10:40.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:39 smithi172 ceph-mon[118784]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:10:41.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:40 smithi192 ceph-mon[105371]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:10:41.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:40 smithi192 ceph-mon[105371]: osdmap e29: 5 total, 4 up, 5 in 2023-11-28T01:10:41.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:40 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:41.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:40 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:41.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:40 smithi172 ceph-mon[100627]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:10:41.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:40 smithi172 ceph-mon[100627]: osdmap e29: 5 total, 4 up, 5 in 2023-11-28T01:10:41.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:40 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:41.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:40 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:41.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:40 smithi172 ceph-mon[118784]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:10:41.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:40 smithi172 ceph-mon[118784]: osdmap e29: 5 total, 4 up, 5 in 2023-11-28T01:10:41.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:40 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:41.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:40 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:42.291 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-11-28T01:10:42.291 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-11-28T01:10:42.291 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-11-28T01:10:42.291 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-11-28T01:10:42.291 INFO:teuthology.orchestra.run.smithi192.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111167 s, 943 MB/s 2023-11-28T01:10:42.291 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-11-28T01:10:42.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[118784]: purged_snaps scrub starts 2023-11-28T01:10:42.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[118784]: purged_snaps scrub ok 2023-11-28T01:10:42.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[118784]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:42.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[118784]: osd.4 [v2:172.21.15.192:6800/1069750856,v1:172.21.15.192:6801/1069750856] boot 2023-11-28T01:10:42.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[118784]: osdmap e30: 5 total, 5 up, 5 in 2023-11-28T01:10:42.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:42.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[100627]: purged_snaps scrub starts 2023-11-28T01:10:42.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[100627]: purged_snaps scrub ok 2023-11-28T01:10:42.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[100627]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:42.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[100627]: osd.4 [v2:172.21.15.192:6800/1069750856,v1:172.21.15.192:6801/1069750856] boot 2023-11-28T01:10:42.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[100627]: osdmap e30: 5 total, 5 up, 5 in 2023-11-28T01:10:42.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:41 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:42.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:41 smithi192 ceph-mon[105371]: purged_snaps scrub starts 2023-11-28T01:10:42.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:41 smithi192 ceph-mon[105371]: purged_snaps scrub ok 2023-11-28T01:10:42.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:41 smithi192 ceph-mon[105371]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-11-28T01:10:42.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:41 smithi192 ceph-mon[105371]: osd.4 [v2:172.21.15.192:6800/1069750856,v1:172.21.15.192:6801/1069750856] boot 2023-11-28T01:10:42.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:41 smithi192 ceph-mon[105371]: osdmap e30: 5 total, 5 up, 5 in 2023-11-28T01:10:42.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:41 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-11-28T01:10:42.862 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2023-11-28T01:10:43.200 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:10:43.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: osdmap e31: 5 total, 5 up, 5 in 2023-11-28T01:10:43.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: Detected new or changed devices on smithi192 2023-11-28T01:10:43.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:43.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: Adjusting osd_memory_target on smithi192 to 17168M 2023-11-28T01:10:43.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:43.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:43.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:43.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[100627]: osdmap e32: 5 total, 5 up, 5 in 2023-11-28T01:10:43.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: osdmap e31: 5 total, 5 up, 5 in 2023-11-28T01:10:43.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: Detected new or changed devices on smithi192 2023-11-28T01:10:43.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:43.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: Adjusting osd_memory_target on smithi192 to 17168M 2023-11-28T01:10:43.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:43.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:43.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:43.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:42 smithi172 ceph-mon[118784]: osdmap e32: 5 total, 5 up, 5 in 2023-11-28T01:10:43.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: osdmap e31: 5 total, 5 up, 5 in 2023-11-28T01:10:43.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: Detected new or changed devices on smithi192 2023-11-28T01:10:43.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:10:43.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: Adjusting osd_memory_target on smithi192 to 17168M 2023-11-28T01:10:43.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:43.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:10:43.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:43.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:43.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:42 smithi192 ceph-mon[105371]: osdmap e32: 5 total, 5 up, 5 in 2023-11-28T01:10:44.553 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.549+0000 7f54bec53700 1 -- 172.21.15.192:0/2017916675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54b8104a50 msgr2=0x7f54b8106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:10:44.553 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.549+0000 7f54bec53700 1 --2- 172.21.15.192:0/2017916675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54b8104a50 0x7f54b8106e40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f54a4009990 tx=0x7f54a4009ca0 comp rx=0 tx=0).stop 2023-11-28T01:10:44.553 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.549+0000 7f54bec53700 1 -- 172.21.15.192:0/2017916675 shutdown_connections 2023-11-28T01:10:44.553 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.549+0000 7f54bec53700 1 --2- 172.21.15.192:0/2017916675 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54b8107380 0x7f54b8109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:44.554 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.549+0000 7f54bec53700 1 --2- 172.21.15.192:0/2017916675 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54b8102120 0x7f54b8104510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:44.554 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.549+0000 7f54bec53700 1 --2- 172.21.15.192:0/2017916675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54b8104a50 0x7f54b8106e40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:44.554 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.549+0000 7f54bec53700 1 -- 172.21.15.192:0/2017916675 >> 172.21.15.192:0/2017916675 conn(0x7f54b80fbad0 msgr2=0x7f54b80fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:10:44.554 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.549+0000 7f54bec53700 1 -- 172.21.15.192:0/2017916675 shutdown_connections 2023-11-28T01:10:44.554 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.549+0000 7f54bec53700 1 -- 172.21.15.192:0/2017916675 wait complete. 2023-11-28T01:10:44.555 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.549+0000 7f54bec53700 1 Processor -- start 2023-11-28T01:10:44.555 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.549+0000 7f54bec53700 1 -- start start 2023-11-28T01:10:44.555 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bec53700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54b8102120 0x7f54b8196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:10:44.555 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bec53700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54b8104a50 0x7f54b8197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:10:44.555 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bec53700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54b8107380 0x7f54b819e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:10:44.555 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bec53700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b819e650 con 0x7f54b8104a50 2023-11-28T01:10:44.556 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bec53700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54b819e790 con 0x7f54b8102120 2023-11-28T01:10:44.556 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bec53700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54b819e920 con 0x7f54b8107380 2023-11-28T01:10:44.556 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bc9ef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54b8102120 0x7f54b8196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:10:44.556 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bc9ef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54b8102120 0x7f54b8196b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:49260/0 (socket says 172.21.15.192:49260) 2023-11-28T01:10:44.556 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bc9ef700 1 -- 172.21.15.192:0/2382180390 learned_addr learned my addr 172.21.15.192:0/2382180390 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:10:44.557 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54b7fff700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54b8104a50 0x7f54b8197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:10:44.557 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bd1f0700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54b8107380 0x7f54b819e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:10:44.557 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bc9ef700 1 -- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54b8107380 msgr2=0x7f54b819e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:10:44.557 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bc9ef700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54b8107380 0x7f54b819e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:44.557 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bc9ef700 1 -- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54b8104a50 msgr2=0x7f54b8197050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:10:44.557 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bc9ef700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54b8104a50 0x7f54b8197050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:10:44.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bc9ef700 1 -- 172.21.15.192:0/2382180390 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a40095f0 con 0x7f54b8102120 2023-11-28T01:10:44.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bc9ef700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54b8102120 0x7f54b8196b10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00d950 tx=0x7f54ac00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:10:44.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54b5ffb700 1 -- 172.21.15.192:0/2382180390 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac00f890 con 0x7f54b8102120 2023-11-28T01:10:44.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54b5ffb700 1 -- 172.21.15.192:0/2382180390 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f54ac00fe40 con 0x7f54b8102120 2023-11-28T01:10:44.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54bec53700 1 -- 172.21.15.192:0/2382180390 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b819ec00 con 0x7f54b8102120 2023-11-28T01:10:44.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.550+0000 7f54b5ffb700 1 -- 172.21.15.192:0/2382180390 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54ac00e3b0 con 0x7f54b8102120 2023-11-28T01:10:44.558 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.551+0000 7f54bec53700 1 -- 172.21.15.192:0/2382180390 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b819f100 con 0x7f54b8102120 2023-11-28T01:10:44.560 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.552+0000 7f54bec53700 1 -- 172.21.15.192:0/2382180390 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54b80fd650 con 0x7f54b8102120 2023-11-28T01:10:44.560 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.552+0000 7f54b5ffb700 1 -- 172.21.15.192:0/2382180390 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7f54ac00f9f0 con 0x7f54b8102120 2023-11-28T01:10:44.560 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.552+0000 7f54b5ffb700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f54a806d380 0x7f54a806f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:10:44.560 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.552+0000 7f54b5ffb700 1 -- 172.21.15.192:0/2382180390 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 3991+0+0 (secure 0 0 0) 0x7f54ac08bfb0 con 0x7f54b8102120 2023-11-28T01:10:44.561 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.553+0000 7f54b7fff700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f54a806d380 0x7f54a806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:10:44.561 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.553+0000 7f54b7fff700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f54a806d380 0x7f54a806f840 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f54a400b5c0 tx=0x7f54a400bfb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:10:44.561 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.557+0000 7f54b5ffb700 1 -- 172.21.15.192:0/2382180390 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f54ac0bf2f0 con 0x7f54b8102120 2023-11-28T01:10:44.757 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:10:44.755+0000 7f54bec53700 1 -- 172.21.15.192:0/2382180390 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f54b810c5b0 con 0x7f54a806d380 2023-11-28T01:10:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[100627]: osdmap e33: 5 total, 5 up, 5 in 2023-11-28T01:10:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[100627]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-28T01:10:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[100627]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:10:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:10:45.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:10:45.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:45.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[118784]: osdmap e33: 5 total, 5 up, 5 in 2023-11-28T01:10:45.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[118784]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-28T01:10:45.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[118784]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:10:45.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:10:45.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:10:45.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:44 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:45.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:44 smithi192 ceph-mon[105371]: osdmap e33: 5 total, 5 up, 5 in 2023-11-28T01:10:45.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:44 smithi192 ceph-mon[105371]: pgmap v110: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-11-28T01:10:45.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:44 smithi192 ceph-mon[105371]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:10:45.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:10:45.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:10:45.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:44 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:46.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:45 smithi172 ceph-mon[100627]: osdmap e34: 5 total, 5 up, 5 in 2023-11-28T01:10:46.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:45 smithi172 ceph-mon[118784]: osdmap e34: 5 total, 5 up, 5 in 2023-11-28T01:10:46.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:45 smithi192 ceph-mon[105371]: osdmap e34: 5 total, 5 up, 5 in 2023-11-28T01:10:47.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:46 smithi172 ceph-mon[100627]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 121 KiB/s, 0 objects/s recovering 2023-11-28T01:10:47.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:46 smithi172 ceph-mon[118784]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 121 KiB/s, 0 objects/s recovering 2023-11-28T01:10:47.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:46 smithi192 ceph-mon[105371]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 121 KiB/s, 0 objects/s recovering 2023-11-28T01:10:48.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/3064787037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a3cb271-985d-412a-affd-6b6bceff3184"}]: dispatch 2023-11-28T01:10:48.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[100627]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a3cb271-985d-412a-affd-6b6bceff3184"}]: dispatch 2023-11-28T01:10:48.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[100627]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a3cb271-985d-412a-affd-6b6bceff3184"}]': finished 2023-11-28T01:10:48.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[100627]: osdmap e35: 6 total, 5 up, 6 in 2023-11-28T01:10:48.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:10:48.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/2981661000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:10:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/3064787037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a3cb271-985d-412a-affd-6b6bceff3184"}]: dispatch 2023-11-28T01:10:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[118784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a3cb271-985d-412a-affd-6b6bceff3184"}]: dispatch 2023-11-28T01:10:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[118784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a3cb271-985d-412a-affd-6b6bceff3184"}]': finished 2023-11-28T01:10:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[118784]: osdmap e35: 6 total, 5 up, 6 in 2023-11-28T01:10:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:10:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:47 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/2981661000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:10:48.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:47 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/3064787037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a3cb271-985d-412a-affd-6b6bceff3184"}]: dispatch 2023-11-28T01:10:48.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:47 smithi192 ceph-mon[105371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a3cb271-985d-412a-affd-6b6bceff3184"}]: dispatch 2023-11-28T01:10:48.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:47 smithi192 ceph-mon[105371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a3cb271-985d-412a-affd-6b6bceff3184"}]': finished 2023-11-28T01:10:48.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:47 smithi192 ceph-mon[105371]: osdmap e35: 6 total, 5 up, 6 in 2023-11-28T01:10:48.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:47 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:10:48.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:47 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/2981661000' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:10:49.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:49 smithi172 ceph-mon[100627]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2023-11-28T01:10:49.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:49 smithi172 ceph-mon[118784]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2023-11-28T01:10:49.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:49 smithi192 ceph-mon[105371]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2023-11-28T01:10:52.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:51 smithi172 ceph-mon[100627]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-28T01:10:52.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:51 smithi172 ceph-mon[118784]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-28T01:10:52.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:51 smithi192 ceph-mon[105371]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-28T01:10:54.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:53 smithi172 ceph-mon[100627]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:54.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:53 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-28T01:10:54.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:53 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:54.304 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:53 smithi192 ceph-mon[105371]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:54.304 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:53 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-28T01:10:54.304 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:53 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:54.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:53 smithi172 ceph-mon[118784]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:54.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:53 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-11-28T01:10:54.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:53 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:10:55.020 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:54 smithi192 ceph-mon[105371]: Deploying daemon osd.5 on smithi192 2023-11-28T01:10:55.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:54 smithi172 ceph-mon[100627]: Deploying daemon osd.5 on smithi192 2023-11-28T01:10:55.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:54 smithi172 ceph-mon[118784]: Deploying daemon osd.5 on smithi192 2023-11-28T01:10:56.193 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:55 smithi192 ceph-mon[105371]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:56.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:55 smithi172 ceph-mon[100627]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:56.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:55 smithi172 ceph-mon[118784]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:58.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:57 smithi172 ceph-mon[100627]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:58.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:57 smithi172 ceph-mon[118784]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:58.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:57 smithi192 ceph-mon[105371]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:59.570 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:59 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:59.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:59 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:59.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:59 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:59.571 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:10:59 smithi192 ceph-mon[105371]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:59.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:59 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:59.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:59 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:59.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:59 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:59.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:10:59 smithi172 ceph-mon[100627]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:10:59.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:59 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:59.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:59 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:10:59.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:59 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:10:59.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:10:59 smithi172 ceph-mon[118784]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:11:01.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:01 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:01.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:01 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:01.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:01 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:01.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:01 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:01.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:01 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:01.424 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:01 smithi192 ceph-mon[105371]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:11:01.465 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.464+0000 7f54b5ffb700 1 -- 172.21.15.192:0/2382180390 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f54b810c5b0 con 0x7f54a806d380 2023-11-28T01:11:01.466 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 5 on host 'smithi192' 2023-11-28T01:11:01.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 -- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f54a806d380 msgr2=0x7f54a806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:01.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f54a806d380 0x7f54a806f840 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f54a400b5c0 tx=0x7f54a400bfb0 comp rx=0 tx=0).stop 2023-11-28T01:11:01.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 -- 172.21.15.192:0/2382180390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54b8102120 msgr2=0x7f54b8196b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:01.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54b8102120 0x7f54b8196b10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00d950 tx=0x7f54ac00dc60 comp rx=0 tx=0).stop 2023-11-28T01:11:01.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 -- 172.21.15.192:0/2382180390 shutdown_connections 2023-11-28T01:11:01.468 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f54a806d380 0x7f54a806f840 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:01.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f54b8107380 0x7f54b819e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:01.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54b8102120 0x7f54b8196b10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:01.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 --2- 172.21.15.192:0/2382180390 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f54b8104a50 0x7f54b8197050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:01.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 -- 172.21.15.192:0/2382180390 >> 172.21.15.192:0/2382180390 conn(0x7f54b80fbad0 msgr2=0x7f54b8102b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:01.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 -- 172.21.15.192:0/2382180390 shutdown_connections 2023-11-28T01:11:01.469 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:01.466+0000 7f54bec53700 1 -- 172.21.15.192:0/2382180390 wait complete. 2023-11-28T01:11:01.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:01.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:01.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:01.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:01.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:01.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[100627]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:11:01.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:01.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:01.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:01.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:01.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:01.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:01 smithi172 ceph-mon[118784]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:11:02.089 DEBUG:teuthology.orchestra.run.smithi192:osd.5> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.5.service 2023-11-28T01:11:02.092 INFO:tasks.cephadm:Deploying osd.6 on smithi192 with /dev/vg_nvme/lv_2... 2023-11-28T01:11:02.092 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-11-28T01:11:02.122 INFO:journalctl@ceph.osd.5.smithi192.stdout:-- Logs begin at Tue 2023-11-28 00:47:11 UTC. -- 2023-11-28T01:11:02.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:02.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:02.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:02.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:02 smithi192 ceph-mon[105371]: from='osd.5 [v2:172.21.15.192:6808/1165321246,v1:172.21.15.192:6809/1165321246]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-28T01:11:02.685 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:02 smithi192 ceph-mon[105371]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-28T01:11:02.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:02.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:02.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:02.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:02 smithi172 ceph-mon[100627]: from='osd.5 [v2:172.21.15.192:6808/1165321246,v1:172.21.15.192:6809/1165321246]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-28T01:11:02.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:02 smithi172 ceph-mon[100627]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-28T01:11:02.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:02.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:02.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:02.805 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:02 smithi172 ceph-mon[118784]: from='osd.5 [v2:172.21.15.192:6808/1165321246,v1:172.21.15.192:6809/1165321246]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-28T01:11:02.805 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:02 smithi172 ceph-mon[118784]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-11-28T01:11:03.091 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:11:03.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[100627]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-28T01:11:03.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[100627]: from='osd.5 [v2:172.21.15.192:6808/1165321246,v1:172.21.15.192:6809/1165321246]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:03.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[100627]: osdmap e36: 6 total, 5 up, 6 in 2023-11-28T01:11:03.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:03.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[100627]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:03.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[100627]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:11:03.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[118784]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-28T01:11:03.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[118784]: from='osd.5 [v2:172.21.15.192:6808/1165321246,v1:172.21.15.192:6809/1165321246]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:03.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[118784]: osdmap e36: 6 total, 5 up, 6 in 2023-11-28T01:11:03.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:03.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[118784]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:03.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:03 smithi172 ceph-mon[118784]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:11:03.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:03 smithi192 ceph-mon[105371]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-11-28T01:11:03.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:03 smithi192 ceph-mon[105371]: from='osd.5 [v2:172.21.15.192:6808/1165321246,v1:172.21.15.192:6809/1165321246]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:03.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:03 smithi192 ceph-mon[105371]: osdmap e36: 6 total, 5 up, 6 in 2023-11-28T01:11:03.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:03 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:03.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:03 smithi192 ceph-mon[105371]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:03.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:03 smithi192 ceph-mon[105371]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-11-28T01:11:03.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:11:03 smithi192 conmon[122308]: 2023-11-28T01:11:03.466+0000 7fd961398700 -1 osd.5 0 waiting for initial osdmap 2023-11-28T01:11:03.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:11:03 smithi192 conmon[122308]: 2023-11-28T01:11:03.470+0000 7fd95c190700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T01:11:04.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:04 smithi192 ceph-mon[105371]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:11:04.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:04 smithi192 ceph-mon[105371]: osdmap e37: 6 total, 5 up, 6 in 2023-11-28T01:11:04.694 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:04 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:04.695 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:04 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:04.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:04 smithi172 ceph-mon[100627]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:11:04.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:04 smithi172 ceph-mon[100627]: osdmap e37: 6 total, 5 up, 6 in 2023-11-28T01:11:04.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:04 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:04.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:04 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:04.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:04 smithi172 ceph-mon[118784]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:11:04.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:04 smithi172 ceph-mon[118784]: osdmap e37: 6 total, 5 up, 6 in 2023-11-28T01:11:04.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:04 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:04.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:04 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:05.634 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-11-28T01:11:05.638 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-11-28T01:11:05.638 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-11-28T01:11:05.638 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-11-28T01:11:05.638 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111106 s, 944 MB/s 2023-11-28T01:11:05.638 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-11-28T01:11:05.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: purged_snaps scrub starts 2023-11-28T01:11:05.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: purged_snaps scrub ok 2023-11-28T01:11:05.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: osd.5 [v2:172.21.15.192:6808/1165321246,v1:172.21.15.192:6809/1165321246] boot 2023-11-28T01:11:05.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: osdmap e38: 6 total, 6 up, 6 in 2023-11-28T01:11:05.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:05.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:05.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:05.805 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:05.805 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:05.805 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:05.805 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:05.805 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:05.805 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:05.805 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:05.806 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: purged_snaps scrub starts 2023-11-28T01:11:05.806 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: purged_snaps scrub ok 2023-11-28T01:11:05.806 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: osd.5 [v2:172.21.15.192:6808/1165321246,v1:172.21.15.192:6809/1165321246] boot 2023-11-28T01:11:05.806 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: osdmap e38: 6 total, 6 up, 6 in 2023-11-28T01:11:05.806 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:05.806 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:05.807 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:05.807 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:05.807 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:05.807 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:05.807 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:05.807 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:05.807 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:05.807 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:05.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: purged_snaps scrub starts 2023-11-28T01:11:05.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: purged_snaps scrub ok 2023-11-28T01:11:05.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: osd.5 [v2:172.21.15.192:6808/1165321246,v1:172.21.15.192:6809/1165321246] boot 2023-11-28T01:11:05.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: osdmap e38: 6 total, 6 up, 6 in 2023-11-28T01:11:05.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-11-28T01:11:05.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:05.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:05.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:05.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:05.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:05.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:05.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:05.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:05.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:06.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2023-11-28T01:11:06.484 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:06 smithi192 ceph-mon[105371]: Detected new or changed devices on smithi192 2023-11-28T01:11:06.484 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:06 smithi192 ceph-mon[105371]: Adjusting osd_memory_target on smithi192 to 8584M 2023-11-28T01:11:06.484 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:06 smithi192 ceph-mon[105371]: osdmap e39: 6 total, 6 up, 6 in 2023-11-28T01:11:06.484 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:06 smithi192 ceph-mon[105371]: osdmap e40: 6 total, 6 up, 6 in 2023-11-28T01:11:06.553 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:11:06.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:06 smithi172 ceph-mon[100627]: Detected new or changed devices on smithi192 2023-11-28T01:11:06.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:06 smithi172 ceph-mon[100627]: Adjusting osd_memory_target on smithi192 to 8584M 2023-11-28T01:11:06.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:06 smithi172 ceph-mon[100627]: osdmap e39: 6 total, 6 up, 6 in 2023-11-28T01:11:06.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:06 smithi172 ceph-mon[100627]: osdmap e40: 6 total, 6 up, 6 in 2023-11-28T01:11:06.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:06 smithi172 ceph-mon[118784]: Detected new or changed devices on smithi192 2023-11-28T01:11:06.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:06 smithi172 ceph-mon[118784]: Adjusting osd_memory_target on smithi192 to 8584M 2023-11-28T01:11:06.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:06 smithi172 ceph-mon[118784]: osdmap e39: 6 total, 6 up, 6 in 2023-11-28T01:11:06.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:06 smithi172 ceph-mon[118784]: osdmap e40: 6 total, 6 up, 6 in 2023-11-28T01:11:07.786 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:07 smithi192 ceph-mon[105371]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:07.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:07 smithi172 ceph-mon[100627]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:07.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:07 smithi172 ceph-mon[118784]: pgmap v128: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:07.973 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.970+0000 7f4e95538700 1 -- 172.21.15.192:0/4030625852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e90102820 msgr2=0x7f4e90102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.970+0000 7f4e95538700 1 --2- 172.21.15.192:0/4030625852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e90102820 0x7f4e90102c80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4e800099c0 tx=0x7f4e80009cd0 comp rx=0 tx=0).stop 2023-11-28T01:11:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.970+0000 7f4e95538700 1 -- 172.21.15.192:0/4030625852 shutdown_connections 2023-11-28T01:11:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.970+0000 7f4e95538700 1 --2- 172.21.15.192:0/4030625852 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4e901031c0 0x7f4e90109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.970+0000 7f4e95538700 1 --2- 172.21.15.192:0/4030625852 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e90101620 0x7f4e90101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.970+0000 7f4e95538700 1 --2- 172.21.15.192:0/4030625852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e90102820 0x7f4e90102c80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.970+0000 7f4e95538700 1 -- 172.21.15.192:0/4030625852 >> 172.21.15.192:0/4030625852 conn(0x7f4e900fcbe0 msgr2=0x7f4e900ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.970+0000 7f4e95538700 1 -- 172.21.15.192:0/4030625852 shutdown_connections 2023-11-28T01:11:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.970+0000 7f4e95538700 1 -- 172.21.15.192:0/4030625852 wait complete. 2023-11-28T01:11:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e95538700 1 Processor -- start 2023-11-28T01:11:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e95538700 1 -- start start 2023-11-28T01:11:07.974 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e95538700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e90101620 0x7f4e901969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e95538700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e90102820 0x7f4e90196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e95538700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4e901031c0 0x7f4e9019df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e95538700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9019e520 con 0x7f4e90102820 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e95538700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9019e660 con 0x7f4e90101620 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e95538700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9019e7f0 con 0x7f4e901031c0 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e8effd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e90101620 0x7f4e901969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e8effd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e90101620 0x7f4e901969e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:49354/0 (socket says 172.21.15.192:49354) 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e8effd700 1 -- 172.21.15.192:0/3180050587 learned_addr learned my addr 172.21.15.192:0/3180050587 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e8f7fe700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4e901031c0 0x7f4e9019df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.971+0000 7f4e8e7fc700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e90102820 0x7f4e90196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e8effd700 1 -- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4e901031c0 msgr2=0x7f4e9019df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:07.975 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e8effd700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4e901031c0 0x7f4e9019df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:07.976 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e8effd700 1 -- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e90102820 msgr2=0x7f4e90196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:07.976 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e8effd700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e90102820 0x7f4e90196f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:07.976 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e8effd700 1 -- 172.21.15.192:0/3180050587 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e800095f0 con 0x7f4e90101620 2023-11-28T01:11:07.976 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e8e7fc700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e90102820 0x7f4e90196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T01:11:07.976 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e8effd700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e90101620 0x7f4e901969e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400c990 tx=0x7f4e8400cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:07.976 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e77fff700 1 -- 172.21.15.192:0/3180050587 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e84007b00 con 0x7f4e90101620 2023-11-28T01:11:07.976 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e95538700 1 -- 172.21.15.192:0/3180050587 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9019ead0 con 0x7f4e90101620 2023-11-28T01:11:07.976 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e77fff700 1 -- 172.21.15.192:0/3180050587 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f4e84007c60 con 0x7f4e90101620 2023-11-28T01:11:07.976 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e77fff700 1 -- 172.21.15.192:0/3180050587 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e84018400 con 0x7f4e90101620 2023-11-28T01:11:07.976 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.972+0000 7f4e95538700 1 -- 172.21.15.192:0/3180050587 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e90073800 con 0x7f4e90101620 2023-11-28T01:11:07.977 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.973+0000 7f4e95538700 1 -- 172.21.15.192:0/3180050587 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e90108b50 con 0x7f4e90101620 2023-11-28T01:11:07.977 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.973+0000 7f4e77fff700 1 -- 172.21.15.192:0/3180050587 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7f4e8400f450 con 0x7f4e90101620 2023-11-28T01:11:07.977 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.974+0000 7f4e77fff700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4e78071790 0x7f4e78073c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:07.977 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.974+0000 7f4e77fff700 1 -- 172.21.15.192:0/3180050587 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4423+0+0 (secure 0 0 0) 0x7f4e8408c430 con 0x7f4e90101620 2023-11-28T01:11:07.977 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.974+0000 7f4e8e7fc700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4e78071790 0x7f4e78073c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:07.978 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.974+0000 7f4e8e7fc700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4e78071790 0x7f4e78073c50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000c010 tx=0x7f4e80005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:07.980 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:07.978+0000 7f4e77fff700 1 -- 172.21.15.192:0/3180050587 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f4e84056810 con 0x7f4e90101620 2023-11-28T01:11:08.179 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:08.178+0000 7f4e95538700 1 -- 172.21.15.192:0/3180050587 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e9010c4c0 con 0x7f4e78071790 2023-11-28T01:11:08.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:08 smithi192 ceph-mon[105371]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-11-28T01:11:08.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:08 smithi192 ceph-mon[105371]: osdmap e41: 6 total, 6 up, 6 in 2023-11-28T01:11:08.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:08 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:11:08.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:08 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:11:08.744 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:08 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:08.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:08 smithi172 ceph-mon[100627]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-11-28T01:11:08.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:08 smithi172 ceph-mon[100627]: osdmap e41: 6 total, 6 up, 6 in 2023-11-28T01:11:08.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:08 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:11:08.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:08 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:11:08.803 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:08 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:08.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:08 smithi172 ceph-mon[118784]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2023-11-28T01:11:08.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:08 smithi172 ceph-mon[118784]: osdmap e41: 6 total, 6 up, 6 in 2023-11-28T01:11:08.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:08 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:11:08.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:08 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:11:08.804 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:08 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:09.776 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:09 smithi192 ceph-mon[105371]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:11:09.777 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:09 smithi192 ceph-mon[105371]: osdmap e42: 6 total, 6 up, 6 in 2023-11-28T01:11:09.777 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:09 smithi192 ceph-mon[105371]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:09.803 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:09 smithi172 ceph-mon[118784]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:11:09.803 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:09 smithi172 ceph-mon[118784]: osdmap e42: 6 total, 6 up, 6 in 2023-11-28T01:11:09.803 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:09 smithi172 ceph-mon[118784]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:09.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:09 smithi172 ceph-mon[100627]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:11:09.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:09 smithi172 ceph-mon[100627]: osdmap e42: 6 total, 6 up, 6 in 2023-11-28T01:11:09.804 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:09 smithi172 ceph-mon[100627]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:11.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:10 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/3704861032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5883ed0-fa9b-4bc7-90a6-3f5e8700d8eb"}]: dispatch 2023-11-28T01:11:11.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:10 smithi172 ceph-mon[100627]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5883ed0-fa9b-4bc7-90a6-3f5e8700d8eb"}]: dispatch 2023-11-28T01:11:11.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:10 smithi172 ceph-mon[100627]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5883ed0-fa9b-4bc7-90a6-3f5e8700d8eb"}]': finished 2023-11-28T01:11:11.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:10 smithi172 ceph-mon[100627]: osdmap e43: 7 total, 6 up, 7 in 2023-11-28T01:11:11.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:10 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:11.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:10 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/3704861032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5883ed0-fa9b-4bc7-90a6-3f5e8700d8eb"}]: dispatch 2023-11-28T01:11:11.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:10 smithi172 ceph-mon[118784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5883ed0-fa9b-4bc7-90a6-3f5e8700d8eb"}]: dispatch 2023-11-28T01:11:11.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:10 smithi172 ceph-mon[118784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5883ed0-fa9b-4bc7-90a6-3f5e8700d8eb"}]': finished 2023-11-28T01:11:11.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:10 smithi172 ceph-mon[118784]: osdmap e43: 7 total, 6 up, 7 in 2023-11-28T01:11:11.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:10 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:11.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:10 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/3704861032' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5883ed0-fa9b-4bc7-90a6-3f5e8700d8eb"}]: dispatch 2023-11-28T01:11:11.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:10 smithi192 ceph-mon[105371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a5883ed0-fa9b-4bc7-90a6-3f5e8700d8eb"}]: dispatch 2023-11-28T01:11:11.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:10 smithi192 ceph-mon[105371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a5883ed0-fa9b-4bc7-90a6-3f5e8700d8eb"}]': finished 2023-11-28T01:11:11.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:10 smithi192 ceph-mon[105371]: osdmap e43: 7 total, 6 up, 7 in 2023-11-28T01:11:11.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:10 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:12.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:11 smithi172 ceph-mon[100627]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 135 KiB/s, 0 objects/s recovering 2023-11-28T01:11:12.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:11 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/3752011915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:11:12.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:11 smithi172 ceph-mon[118784]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 135 KiB/s, 0 objects/s recovering 2023-11-28T01:11:12.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:11 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/3752011915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:11:12.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:11 smithi192 ceph-mon[105371]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 135 KiB/s, 0 objects/s recovering 2023-11-28T01:11:12.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:11 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/3752011915' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:11:13.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:12 smithi172 ceph-mon[100627]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-11-28T01:11:13.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:12 smithi172 ceph-mon[100627]: Cluster is now healthy 2023-11-28T01:11:13.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:12 smithi172 ceph-mon[118784]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-11-28T01:11:13.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:12 smithi172 ceph-mon[118784]: Cluster is now healthy 2023-11-28T01:11:13.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:12 smithi192 ceph-mon[105371]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2023-11-28T01:11:13.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:12 smithi192 ceph-mon[105371]: Cluster is now healthy 2023-11-28T01:11:14.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:13 smithi172 ceph-mon[100627]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-28T01:11:14.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:13 smithi172 ceph-mon[118784]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-28T01:11:14.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:13 smithi192 ceph-mon[105371]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-28T01:11:16.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:15 smithi192 ceph-mon[105371]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-11-28T01:11:16.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:15 smithi172 ceph-mon[100627]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-11-28T01:11:16.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:15 smithi172 ceph-mon[118784]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-11-28T01:11:17.241 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:16 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-28T01:11:17.241 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:16 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:17.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-28T01:11:17.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:16 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:17.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:16 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-11-28T01:11:17.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:16 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:18.275 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:17 smithi192 ceph-mon[105371]: Deploying daemon osd.6 on smithi192 2023-11-28T01:11:18.276 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:17 smithi192 ceph-mon[105371]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-11-28T01:11:18.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:17 smithi172 ceph-mon[100627]: Deploying daemon osd.6 on smithi192 2023-11-28T01:11:18.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:17 smithi172 ceph-mon[100627]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-11-28T01:11:18.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:17 smithi172 ceph-mon[118784]: Deploying daemon osd.6 on smithi192 2023-11-28T01:11:18.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:17 smithi172 ceph-mon[118784]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-11-28T01:11:20.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:19 smithi172 ceph-mon[100627]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-28T01:11:20.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:19 smithi172 ceph-mon[118784]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-28T01:11:20.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:19 smithi192 ceph-mon[105371]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-28T01:11:22.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:21 smithi192 ceph-mon[105371]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:22.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:21 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:22.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:21 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:22.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:21 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:22.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:21 smithi172 ceph-mon[100627]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:22.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:21 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:22.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:21 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:22.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:21 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:22.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:21 smithi172 ceph-mon[118784]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:22.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:21 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:22.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:21 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:22.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:21 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:24.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[100627]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:24.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:24.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:24.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:24.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:24.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:24.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[118784]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:24.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:24.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:24.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:24.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:24.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:23 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:24.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:23 smithi192 ceph-mon[105371]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:24.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:23 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:24.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:23 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:24.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:23 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:24.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:23 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:24.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:23 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:24.740 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.737+0000 7f4e77fff700 1 -- 172.21.15.192:0/3180050587 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4e9010c4c0 con 0x7f4e78071790 2023-11-28T01:11:24.741 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 6 on host 'smithi192' 2023-11-28T01:11:24.742 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 -- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4e78071790 msgr2=0x7f4e78073c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:24.742 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4e78071790 0x7f4e78073c50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4e8000c010 tx=0x7f4e80005fb0 comp rx=0 tx=0).stop 2023-11-28T01:11:24.742 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 -- 172.21.15.192:0/3180050587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e90101620 msgr2=0x7f4e901969e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:24.742 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e90101620 0x7f4e901969e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4e8400c990 tx=0x7f4e8400cca0 comp rx=0 tx=0).stop 2023-11-28T01:11:24.742 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 -- 172.21.15.192:0/3180050587 shutdown_connections 2023-11-28T01:11:24.742 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f4e78071790 0x7f4e78073c50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:24.742 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f4e901031c0 0x7f4e9019df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:24.742 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e90101620 0x7f4e901969e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:24.742 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 --2- 172.21.15.192:0/3180050587 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f4e90102820 0x7f4e90196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:24.742 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 -- 172.21.15.192:0/3180050587 >> 172.21.15.192:0/3180050587 conn(0x7f4e900fcbe0 msgr2=0x7f4e90105c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:24.743 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 -- 172.21.15.192:0/3180050587 shutdown_connections 2023-11-28T01:11:24.743 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:24.740+0000 7f4e95538700 1 -- 172.21.15.192:0/3180050587 wait complete. 2023-11-28T01:11:25.410 DEBUG:teuthology.orchestra.run.smithi192:osd.6> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.6.service 2023-11-28T01:11:25.413 INFO:tasks.cephadm:Deploying osd.7 on smithi192 with /dev/vg_nvme/lv_1... 2023-11-28T01:11:25.413 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-11-28T01:11:25.442 INFO:journalctl@ceph.osd.6.smithi192.stdout:-- Logs begin at Tue 2023-11-28 00:47:11 UTC. -- 2023-11-28T01:11:25.975 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:11:26.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:26.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:26.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[118784]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:26.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:26.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[118784]: from='osd.6 [v2:172.21.15.192:6816/1409239444,v1:172.21.15.192:6817/1409239444]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-28T01:11:26.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[118784]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-28T01:11:26.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:26.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:26.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[100627]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:26.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:26.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[100627]: from='osd.6 [v2:172.21.15.192:6816/1409239444,v1:172.21.15.192:6817/1409239444]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-28T01:11:26.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:25 smithi172 ceph-mon[100627]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-28T01:11:26.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:25 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:26.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:25 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:26.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:25 smithi192 ceph-mon[105371]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:26.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:25 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:26.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:25 smithi192 ceph-mon[105371]: from='osd.6 [v2:172.21.15.192:6816/1409239444,v1:172.21.15.192:6817/1409239444]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-28T01:11:26.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:25 smithi192 ceph-mon[105371]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-11-28T01:11:27.020 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:26 smithi192 ceph-mon[105371]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-28T01:11:27.020 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:26 smithi192 ceph-mon[105371]: from='osd.6 [v2:172.21.15.192:6816/1409239444,v1:172.21.15.192:6817/1409239444]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:27.020 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:26 smithi192 ceph-mon[105371]: osdmap e44: 7 total, 6 up, 7 in 2023-11-28T01:11:27.020 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:26 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:27.020 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:26 smithi192 ceph-mon[105371]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:27.020 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:11:26 smithi192 conmon[131202]: 2023-11-28T01:11:26.743+0000 7f4a57c28700 -1 osd.6 0 waiting for initial osdmap 2023-11-28T01:11:27.020 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:11:26 smithi192 conmon[131202]: 2023-11-28T01:11:26.751+0000 7f4a52a20700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T01:11:27.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:26 smithi172 ceph-mon[100627]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-28T01:11:27.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:26 smithi172 ceph-mon[100627]: from='osd.6 [v2:172.21.15.192:6816/1409239444,v1:172.21.15.192:6817/1409239444]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:27.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:26 smithi172 ceph-mon[100627]: osdmap e44: 7 total, 6 up, 7 in 2023-11-28T01:11:27.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:26 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:27.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:26 smithi172 ceph-mon[100627]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:27.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:26 smithi172 ceph-mon[118784]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-11-28T01:11:27.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:26 smithi172 ceph-mon[118784]: from='osd.6 [v2:172.21.15.192:6816/1409239444,v1:172.21.15.192:6817/1409239444]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:27.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:26 smithi172 ceph-mon[118784]: osdmap e44: 7 total, 6 up, 7 in 2023-11-28T01:11:27.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:26 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:27.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:26 smithi172 ceph-mon[118784]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:28.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:27 smithi172 ceph-mon[100627]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:11:28.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:27 smithi172 ceph-mon[100627]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:28.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:27 smithi172 ceph-mon[100627]: osdmap e45: 7 total, 6 up, 7 in 2023-11-28T01:11:28.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:27 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:28.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:27 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:28.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:27 smithi172 ceph-mon[118784]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:11:28.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:27 smithi172 ceph-mon[118784]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:28.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:27 smithi172 ceph-mon[118784]: osdmap e45: 7 total, 6 up, 7 in 2023-11-28T01:11:28.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:27 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:28.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:27 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:28.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:27 smithi192 ceph-mon[105371]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:11:28.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:27 smithi192 ceph-mon[105371]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:28.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:27 smithi192 ceph-mon[105371]: osdmap e45: 7 total, 6 up, 7 in 2023-11-28T01:11:28.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:27 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:28.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:27 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:28.269 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-11-28T01:11:28.269 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-11-28T01:11:28.270 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2023-11-28T01:11:28.270 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2023-11-28T01:11:28.270 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112845 s, 929 MB/s 2023-11-28T01:11:28.270 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2023-11-28T01:11:29.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[100627]: purged_snaps scrub starts 2023-11-28T01:11:29.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[100627]: purged_snaps scrub ok 2023-11-28T01:11:29.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[100627]: osd.6 [v2:172.21.15.192:6816/1409239444,v1:172.21.15.192:6817/1409239444] boot 2023-11-28T01:11:29.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[100627]: osdmap e46: 7 total, 7 up, 7 in 2023-11-28T01:11:29.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:29.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[100627]: osdmap e47: 7 total, 7 up, 7 in 2023-11-28T01:11:29.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[118784]: purged_snaps scrub starts 2023-11-28T01:11:29.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[118784]: purged_snaps scrub ok 2023-11-28T01:11:29.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[118784]: osd.6 [v2:172.21.15.192:6816/1409239444,v1:172.21.15.192:6817/1409239444] boot 2023-11-28T01:11:29.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[118784]: osdmap e46: 7 total, 7 up, 7 in 2023-11-28T01:11:29.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:29.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:28 smithi172 ceph-mon[118784]: osdmap e47: 7 total, 7 up, 7 in 2023-11-28T01:11:29.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:28 smithi192 ceph-mon[105371]: purged_snaps scrub starts 2023-11-28T01:11:29.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:28 smithi192 ceph-mon[105371]: purged_snaps scrub ok 2023-11-28T01:11:29.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:28 smithi192 ceph-mon[105371]: osd.6 [v2:172.21.15.192:6816/1409239444,v1:172.21.15.192:6817/1409239444] boot 2023-11-28T01:11:29.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:28 smithi192 ceph-mon[105371]: osdmap e46: 7 total, 7 up, 7 in 2023-11-28T01:11:29.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:28 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-11-28T01:11:29.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:28 smithi192 ceph-mon[105371]: osdmap e47: 7 total, 7 up, 7 in 2023-11-28T01:11:29.194 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2023-11-28T01:11:29.543 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:11:30.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:29 smithi192 ceph-mon[105371]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:30.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:29 smithi192 ceph-mon[105371]: osdmap e48: 7 total, 7 up, 7 in 2023-11-28T01:11:30.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:30.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:30.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:30.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:30.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:30.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:29 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[100627]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:30.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[100627]: osdmap e48: 7 total, 7 up, 7 in 2023-11-28T01:11:30.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:30.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:30.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:30.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:30.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:30.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[118784]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-11-28T01:11:30.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[118784]: osdmap e48: 7 total, 7 up, 7 in 2023-11-28T01:11:30.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:30.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:30.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:30.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:30.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:30.305 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:29 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:30.980 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.976+0000 7f2020090700 1 -- 172.21.15.192:0/2674570269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2018100510 msgr2=0x7f2018100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:30.980 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.976+0000 7f2020090700 1 --2- 172.21.15.192:0/2674570269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2018100510 0x7f2018100930 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f20080099c0 tx=0x7f2008009cd0 comp rx=0 tx=0).stop 2023-11-28T01:11:30.980 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.976+0000 7f2020090700 1 -- 172.21.15.192:0/2674570269 shutdown_connections 2023-11-28T01:11:30.980 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.976+0000 7f2020090700 1 --2- 172.21.15.192:0/2674570269 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f20181020b0 0x7f2018108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:30.980 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.976+0000 7f2020090700 1 --2- 172.21.15.192:0/2674570269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2018100510 0x7f2018100930 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:30.980 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.976+0000 7f2020090700 1 --2- 172.21.15.192:0/2674570269 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f2018101710 0x7f2018101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:30.980 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.976+0000 7f2020090700 1 -- 172.21.15.192:0/2674570269 >> 172.21.15.192:0/2674570269 conn(0x7f20180fbad0 msgr2=0x7f20180fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.976+0000 7f2020090700 1 -- 172.21.15.192:0/2674570269 shutdown_connections 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.976+0000 7f2020090700 1 -- 172.21.15.192:0/2674570269 wait complete. 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f2020090700 1 Processor -- start 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f2020090700 1 -- start start 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f2020090700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2018100510 0x7f2018195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f2020090700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2018101710 0x7f2018195fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f2020090700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20181020b0 0x7f201819d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f2020090700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201819d5e0 con 0x7f20181020b0 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f2020090700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201819d720 con 0x7f2018100510 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f2020090700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f201819d8b0 con 0x7f2018101710 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f201d62b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2018101710 0x7f2018195fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f201d62b700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2018101710 0x7f2018195fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.192:59390/0 (socket says 172.21.15.192:59390) 2023-11-28T01:11:30.981 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f201d62b700 1 -- 172.21.15.192:0/889740756 learned_addr learned my addr 172.21.15.192:0/889740756 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:11:30.982 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f201de2c700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2018100510 0x7f2018195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:30.982 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f201e62d700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20181020b0 0x7f201819d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:30.982 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f201de2c700 1 -- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2018101710 msgr2=0x7f2018195fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:30.982 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f201de2c700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2018101710 0x7f2018195fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:30.982 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f201de2c700 1 -- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20181020b0 msgr2=0x7f201819d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:30.982 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.977+0000 7f201de2c700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20181020b0 0x7f201819d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:30.982 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.978+0000 7f201de2c700 1 -- 172.21.15.192:0/889740756 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20080095f0 con 0x7f2018100510 2023-11-28T01:11:30.982 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.978+0000 7f201de2c700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2018100510 0x7f2018195aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2008009990 tx=0x7f200800f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:30.982 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.978+0000 7f200effd700 1 -- 172.21.15.192:0/889740756 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200801d070 con 0x7f2018100510 2023-11-28T01:11:30.982 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.978+0000 7f200effd700 1 -- 172.21.15.192:0/889740756 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f2008022470 con 0x7f2018100510 2023-11-28T01:11:30.983 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.978+0000 7f2020090700 1 -- 172.21.15.192:0/889740756 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201819db30 con 0x7f2018100510 2023-11-28T01:11:30.983 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.978+0000 7f200effd700 1 -- 172.21.15.192:0/889740756 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20080173f0 con 0x7f2018100510 2023-11-28T01:11:30.983 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.978+0000 7f2020090700 1 -- 172.21.15.192:0/889740756 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201819dfa0 con 0x7f2018100510 2023-11-28T01:11:30.984 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.979+0000 7f2020090700 1 -- 172.21.15.192:0/889740756 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2018107800 con 0x7f2018100510 2023-11-28T01:11:30.984 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.979+0000 7f200effd700 1 -- 172.21.15.192:0/889740756 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7f20080227e0 con 0x7f2018100510 2023-11-28T01:11:30.984 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.980+0000 7f200effd700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f200406d380 0x7f200406f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:30.984 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.980+0000 7f200effd700 1 -- 172.21.15.192:0/889740756 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4855+0+0 (secure 0 0 0) 0x7f200808d8c0 con 0x7f2018100510 2023-11-28T01:11:30.984 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.980+0000 7f201d62b700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f200406d380 0x7f200406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:30.984 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.981+0000 7f201d62b700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f200406d380 0x7f200406f840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2014009ad0 tx=0x7f2014009290 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:30.986 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:30.984+0000 7f200effd700 1 -- 172.21.15.192:0/889740756 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f2008057a40 con 0x7f2018100510 2023-11-28T01:11:31.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:30 smithi192 ceph-mon[105371]: Detected new or changed devices on smithi192 2023-11-28T01:11:31.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:30 smithi192 ceph-mon[105371]: Adjusting osd_memory_target on smithi192 to 5722M 2023-11-28T01:11:31.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:30 smithi192 ceph-mon[105371]: osdmap e49: 7 total, 7 up, 7 in 2023-11-28T01:11:31.184 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:31.182+0000 7f2020090700 1 -- 172.21.15.192:0/889740756 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f201810b4a0 con 0x7f200406d380 2023-11-28T01:11:31.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:30 smithi172 ceph-mon[100627]: Detected new or changed devices on smithi192 2023-11-28T01:11:31.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:30 smithi172 ceph-mon[100627]: Adjusting osd_memory_target on smithi192 to 5722M 2023-11-28T01:11:31.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:30 smithi172 ceph-mon[100627]: osdmap e49: 7 total, 7 up, 7 in 2023-11-28T01:11:31.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:30 smithi172 ceph-mon[118784]: Detected new or changed devices on smithi192 2023-11-28T01:11:31.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:30 smithi172 ceph-mon[118784]: Adjusting osd_memory_target on smithi192 to 5722M 2023-11-28T01:11:31.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:30 smithi172 ceph-mon[118784]: osdmap e49: 7 total, 7 up, 7 in 2023-11-28T01:11:32.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:31 smithi192 ceph-mon[105371]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:32.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:31 smithi192 ceph-mon[105371]: osdmap e50: 7 total, 7 up, 7 in 2023-11-28T01:11:32.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:31 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:11:32.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:31 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:11:32.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:31 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:32.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:31 smithi172 ceph-mon[100627]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:32.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:31 smithi172 ceph-mon[100627]: osdmap e50: 7 total, 7 up, 7 in 2023-11-28T01:11:32.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:31 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:11:32.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:31 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:11:32.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:31 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:32.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:31 smithi172 ceph-mon[118784]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:32.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:31 smithi172 ceph-mon[118784]: osdmap e50: 7 total, 7 up, 7 in 2023-11-28T01:11:32.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-11-28T01:11:32.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-11-28T01:11:32.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:31 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:33.306 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:32 smithi172 ceph-mon[100627]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:11:33.306 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:32 smithi172 ceph-mon[118784]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:11:33.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:32 smithi192 ceph-mon[105371]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:11:34.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[100627]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:34.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/54286016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8092d059-facb-4b45-93cc-096da7c79f69"}]: dispatch 2023-11-28T01:11:34.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[100627]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8092d059-facb-4b45-93cc-096da7c79f69"}]: dispatch 2023-11-28T01:11:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[100627]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8092d059-facb-4b45-93cc-096da7c79f69"}]': finished 2023-11-28T01:11:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[100627]: osdmap e51: 8 total, 7 up, 8 in 2023-11-28T01:11:34.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:34.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[118784]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:34.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/54286016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8092d059-facb-4b45-93cc-096da7c79f69"}]: dispatch 2023-11-28T01:11:34.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[118784]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8092d059-facb-4b45-93cc-096da7c79f69"}]: dispatch 2023-11-28T01:11:34.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[118784]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8092d059-facb-4b45-93cc-096da7c79f69"}]': finished 2023-11-28T01:11:34.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[118784]: osdmap e51: 8 total, 7 up, 8 in 2023-11-28T01:11:34.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:33 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:33 smithi192 ceph-mon[105371]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:33 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/54286016' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8092d059-facb-4b45-93cc-096da7c79f69"}]: dispatch 2023-11-28T01:11:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:33 smithi192 ceph-mon[105371]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8092d059-facb-4b45-93cc-096da7c79f69"}]: dispatch 2023-11-28T01:11:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:33 smithi192 ceph-mon[105371]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8092d059-facb-4b45-93cc-096da7c79f69"}]': finished 2023-11-28T01:11:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:33 smithi192 ceph-mon[105371]: osdmap e51: 8 total, 7 up, 8 in 2023-11-28T01:11:34.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:33 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:35.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:34 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/87535817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:11:35.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:34 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/87535817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:11:35.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:34 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/87535817' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-11-28T01:11:36.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:35 smithi172 ceph-mon[100627]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2023-11-28T01:11:36.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:35 smithi172 ceph-mon[118784]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2023-11-28T01:11:36.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:35 smithi192 ceph-mon[105371]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2023-11-28T01:11:38.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:37 smithi192 ceph-mon[105371]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-28T01:11:38.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:37 smithi172 ceph-mon[100627]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-28T01:11:38.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:37 smithi172 ceph-mon[118784]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2023-11-28T01:11:40.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:39 smithi192 ceph-mon[105371]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-28T01:11:40.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-28T01:11:40.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:39 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:40.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:39 smithi172 ceph-mon[100627]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-28T01:11:40.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-28T01:11:40.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:39 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:40.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:39 smithi172 ceph-mon[118784]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-11-28T01:11:40.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-11-28T01:11:40.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:39 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:41.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:40 smithi172 ceph-mon[100627]: Deploying daemon osd.7 on smithi192 2023-11-28T01:11:41.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:40 smithi172 ceph-mon[118784]: Deploying daemon osd.7 on smithi192 2023-11-28T01:11:41.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:40 smithi192 ceph-mon[105371]: Deploying daemon osd.7 on smithi192 2023-11-28T01:11:42.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:41 smithi172 ceph-mon[100627]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-28T01:11:42.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:41 smithi172 ceph-mon[118784]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-28T01:11:42.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:41 smithi192 ceph-mon[105371]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-28T01:11:44.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:43 smithi172 ceph-mon[100627]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-28T01:11:44.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:43 smithi172 ceph-mon[118784]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-28T01:11:44.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:43 smithi192 ceph-mon[105371]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-11-28T01:11:46.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:45 smithi192 ceph-mon[105371]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-11-28T01:11:46.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:45 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:46.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:45 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:46.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:45 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:46.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:45 smithi172 ceph-mon[100627]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-11-28T01:11:46.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:45 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:46.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:45 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:46.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:45 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:46.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:45 smithi172 ceph-mon[118784]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-11-28T01:11:46.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:45 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:46.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:45 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:46.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:45 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:48.150 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 7 on host 'smithi192' 2023-11-28T01:11:48.150 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.147+0000 7f200effd700 1 -- 172.21.15.192:0/889740756 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f201810b4a0 con 0x7f200406d380 2023-11-28T01:11:48.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 -- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f200406d380 msgr2=0x7f200406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:48.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f200406d380 0x7f200406f840 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2014009ad0 tx=0x7f2014009290 comp rx=0 tx=0).stop 2023-11-28T01:11:48.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 -- 172.21.15.192:0/889740756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2018100510 msgr2=0x7f2018195aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:48.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2018100510 0x7f2018195aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2008009990 tx=0x7f200800f690 comp rx=0 tx=0).stop 2023-11-28T01:11:48.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 -- 172.21.15.192:0/889740756 shutdown_connections 2023-11-28T01:11:48.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f200406d380 0x7f200406f840 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:48.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f20181020b0 0x7f201819d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:48.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2018100510 0x7f2018195aa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:48.152 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 --2- 172.21.15.192:0/889740756 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f2018101710 0x7f2018195fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:48.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 -- 172.21.15.192:0/889740756 >> 172.21.15.192:0/889740756 conn(0x7f20180fbad0 msgr2=0x7f2018104940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:48.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 -- 172.21.15.192:0/889740756 shutdown_connections 2023-11-28T01:11:48.153 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:11:48.150+0000 7f2020090700 1 -- 172.21.15.192:0/889740756 wait complete. 2023-11-28T01:11:48.175 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:47 smithi192 ceph-mon[105371]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:48.175 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:47 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:48.175 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:47 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:48.175 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:47 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:48.175 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:47 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:48.175 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:47 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:48.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[100627]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:48.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:48.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:48.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:48.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:48.304 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[118784]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:48.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:47 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:48.755 DEBUG:teuthology.orchestra.run.smithi192:osd.7> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.7.service 2023-11-28T01:11:48.757 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2023-11-28T01:11:48.757 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph osd stat -f json 2023-11-28T01:11:48.788 INFO:journalctl@ceph.osd.7.smithi192.stdout:-- Logs begin at Tue 2023-11-28 00:47:11 UTC. -- 2023-11-28T01:11:49.002 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:48 smithi192 ceph-mon[105371]: from='osd.7 [v2:172.21.15.192:6824/4284606111,v1:172.21.15.192:6825/4284606111]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-28T01:11:49.002 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:48 smithi192 ceph-mon[105371]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-28T01:11:49.002 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:48 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:49.003 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:48 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:49.003 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:48 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:49.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:48 smithi172 ceph-mon[118784]: from='osd.7 [v2:172.21.15.192:6824/4284606111,v1:172.21.15.192:6825/4284606111]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-28T01:11:49.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:48 smithi172 ceph-mon[118784]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-28T01:11:49.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:48 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:49.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:48 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:49.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:48 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:49.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:48 smithi172 ceph-mon[100627]: from='osd.7 [v2:172.21.15.192:6824/4284606111,v1:172.21.15.192:6825/4284606111]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-28T01:11:49.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:48 smithi172 ceph-mon[100627]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-11-28T01:11:49.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:48 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:49.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:48 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:49.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:48 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:11:49.109 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:11:49.954 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[118784]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:49.954 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[118784]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-28T01:11:49.954 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[118784]: from='osd.7 [v2:172.21.15.192:6824/4284606111,v1:172.21.15.192:6825/4284606111]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:49.954 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[118784]: osdmap e52: 8 total, 7 up, 8 in 2023-11-28T01:11:49.954 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:49.955 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[118784]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:50.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:11:49 smithi192 conmon[140126]: 2023-11-28T01:11:49.952+0000 7fe22edae700 -1 osd.7 0 waiting for initial osdmap 2023-11-28T01:11:50.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:11:49 smithi192 conmon[140126]: 2023-11-28T01:11:49.959+0000 7fe2293a5700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-11-28T01:11:50.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:49 smithi192 ceph-mon[105371]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:50.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:49 smithi192 ceph-mon[105371]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-28T01:11:50.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:49 smithi192 ceph-mon[105371]: from='osd.7 [v2:172.21.15.192:6824/4284606111,v1:172.21.15.192:6825/4284606111]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:50.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:49 smithi192 ceph-mon[105371]: osdmap e52: 8 total, 7 up, 8 in 2023-11-28T01:11:50.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:49 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:50.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:49 smithi192 ceph-mon[105371]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:50.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[100627]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:50.284 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[100627]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-11-28T01:11:50.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[100627]: from='osd.7 [v2:172.21.15.192:6824/4284606111,v1:172.21.15.192:6825/4284606111]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:50.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[100627]: osdmap e52: 8 total, 7 up, 8 in 2023-11-28T01:11:50.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:50.285 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:49 smithi172 ceph-mon[100627]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2023-11-28T01:11:50.461 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.457+0000 7fc347b75700 1 -- 172.21.15.172:0/154952860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc340101aa0 msgr2=0x7fc340101ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:50.461 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.457+0000 7fc347b75700 1 --2- 172.21.15.172:0/154952860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc340101aa0 0x7fc340101ec0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc334009990 tx=0x7fc334009ca0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.461 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.457+0000 7fc347b75700 1 -- 172.21.15.172:0/154952860 shutdown_connections 2023-11-28T01:11:50.461 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.457+0000 7fc347b75700 1 --2- 172.21.15.172:0/154952860 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc340101aa0 0x7fc340101ec0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.461 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.457+0000 7fc347b75700 1 --2- 172.21.15.172:0/154952860 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc340101180 0x7fc340108a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.461 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.457+0000 7fc347b75700 1 --2- 172.21.15.172:0/154952860 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3401007e0 0x7fc340100c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.461 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.457+0000 7fc347b75700 1 -- 172.21.15.172:0/154952860 >> 172.21.15.172:0/154952860 conn(0x7fc3400fb230 msgr2=0x7fc3400fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:50.461 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc347b75700 1 -- 172.21.15.172:0/154952860 shutdown_connections 2023-11-28T01:11:50.462 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc347b75700 1 -- 172.21.15.172:0/154952860 wait complete. 2023-11-28T01:11:50.462 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc347b75700 1 Processor -- start 2023-11-28T01:11:50.462 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc347b75700 1 -- start start 2023-11-28T01:11:50.462 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc347b75700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3401007e0 0x7fc340196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:50.462 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc347b75700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc340101180 0x7fc340197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:50.462 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc347b75700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc340101aa0 0x7fc34019e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:50.462 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc347b75700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34019e630 con 0x7fc3401007e0 2023-11-28T01:11:50.462 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc347b75700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34019e770 con 0x7fc340101aa0 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc347b75700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34019e900 con 0x7fc340101180 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc345110700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc340101180 0x7fc340197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc345110700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc340101180 0x7fc340197030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:34384/0 (socket says 172.21.15.172:34384) 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.458+0000 7fc345110700 1 -- 172.21.15.172:0/2039543729 learned_addr learned my addr 172.21.15.172:0/2039543729 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc346112700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc340101aa0 0x7fc34019e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc345911700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3401007e0 0x7fc340196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc345110700 1 -- 172.21.15.172:0/2039543729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc340101aa0 msgr2=0x7fc34019e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc345110700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc340101aa0 0x7fc34019e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc345110700 1 -- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3401007e0 msgr2=0x7fc340196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc345110700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3401007e0 0x7fc340196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc345110700 1 -- 172.21.15.172:0/2039543729 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3340095f0 con 0x7fc340101180 2023-11-28T01:11:50.463 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc345911700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3401007e0 0x7fc340196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T01:11:50.464 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc345110700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc340101180 0x7fc340197030 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc33c00d9b0 tx=0x7fc33c00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:50.464 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc332ffd700 1 -- 172.21.15.172:0/2039543729 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c004220 con 0x7fc340101180 2023-11-28T01:11:50.464 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc332ffd700 1 -- 172.21.15.172:0/2039543729 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7fc33c007640 con 0x7fc340101180 2023-11-28T01:11:50.464 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc347b75700 1 -- 172.21.15.172:0/2039543729 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc34019ebe0 con 0x7fc340101180 2023-11-28T01:11:50.464 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc332ffd700 1 -- 172.21.15.172:0/2039543729 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33c0053a0 con 0x7fc340101180 2023-11-28T01:11:50.464 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.459+0000 7fc347b75700 1 -- 172.21.15.172:0/2039543729 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34019f0e0 con 0x7fc340101180 2023-11-28T01:11:50.465 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.460+0000 7fc347b75700 1 -- 172.21.15.172:0/2039543729 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34004f290 con 0x7fc340101180 2023-11-28T01:11:50.465 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.461+0000 7fc332ffd700 1 -- 172.21.15.172:0/2039543729 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7fc33c005500 con 0x7fc340101180 2023-11-28T01:11:50.465 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.461+0000 7fc332ffd700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc32c06d380 0x7fc32c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:50.465 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.461+0000 7fc332ffd700 1 -- 172.21.15.172:0/2039543729 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5007+0+0 (secure 0 0 0) 0x7fc33c08c470 con 0x7fc340101180 2023-11-28T01:11:50.465 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.461+0000 7fc345911700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc32c06d380 0x7fc32c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:50.465 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.461+0000 7fc345911700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc32c06d380 0x7fc32c06f840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc33400b5c0 tx=0x7fc334005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:50.467 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.466+0000 7fc332ffd700 1 -- 172.21.15.172:0/2039543729 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7fc33c056560 con 0x7fc340101180 2023-11-28T01:11:50.656 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.655+0000 7fc347b75700 1 -- 172.21.15.172:0/2039543729 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fc34019f270 con 0x7fc340101180 2023-11-28T01:11:50.657 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.656+0000 7fc332ffd700 1 -- 172.21.15.172:0/2039543729 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v53) v1 ==== 74+0+130 (secure 0 0 0) 0x7fc33c0160a0 con 0x7fc340101180 2023-11-28T01:11:50.657 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:11:50.659 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 -- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc32c06d380 msgr2=0x7fc32c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:50.659 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc32c06d380 0x7fc32c06f840 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc33400b5c0 tx=0x7fc334005fb0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.659 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 -- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc340101180 msgr2=0x7fc340197030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:50.659 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc340101180 0x7fc340197030 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc33c00d9b0 tx=0x7fc33c00dcc0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.659 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 -- 172.21.15.172:0/2039543729 shutdown_connections 2023-11-28T01:11:50.659 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fc32c06d380 0x7fc32c06f840 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.659 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc340101aa0 0x7fc34019e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.659 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fc3401007e0 0x7fc340196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.660 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 --2- 172.21.15.172:0/2039543729 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fc340101180 0x7fc340197030 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:50.660 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 -- 172.21.15.172:0/2039543729 >> 172.21.15.172:0/2039543729 conn(0x7fc3400fb230 msgr2=0x7fc3400ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:50.660 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 -- 172.21.15.172:0/2039543729 shutdown_connections 2023-11-28T01:11:50.660 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:50.658+0000 7fc347b75700 1 -- 172.21.15.172:0/2039543729 wait complete. 2023-11-28T01:11:51.194 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":53,"num_osds":8,"num_up_osds":7,"osd_up_since":1701133887,"num_in_osds":8,"osd_in_since":1701133893,"num_remapped_pgs":0} 2023-11-28T01:11:51.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:50 smithi192 ceph-mon[105371]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:11:51.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:50 smithi192 ceph-mon[105371]: osdmap e53: 8 total, 7 up, 8 in 2023-11-28T01:11:51.269 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:50 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:51.270 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:50 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:51.270 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:50 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2039543729' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-28T01:11:51.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:50 smithi172 ceph-mon[100627]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:11:51.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:50 smithi172 ceph-mon[100627]: osdmap e53: 8 total, 7 up, 8 in 2023-11-28T01:11:51.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:50 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:51.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:50 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:51.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:50 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2039543729' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-28T01:11:51.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:50 smithi172 ceph-mon[118784]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2023-11-28T01:11:51.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:50 smithi172 ceph-mon[118784]: osdmap e53: 8 total, 7 up, 8 in 2023-11-28T01:11:51.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:50 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:51.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:50 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:51.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:50 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2039543729' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-28T01:11:52.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: purged_snaps scrub starts 2023-11-28T01:11:52.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: purged_snaps scrub ok 2023-11-28T01:11:52.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:52.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: osd.7 [v2:172.21.15.192:6824/4284606111,v1:172.21.15.192:6825/4284606111] boot 2023-11-28T01:11:52.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: osdmap e54: 8 total, 8 up, 8 in 2023-11-28T01:11:52.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:52.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:52.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:52.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:51 smithi192 ceph-mon[105371]: osdmap e55: 8 total, 8 up, 8 in 2023-11-28T01:11:52.195 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph osd stat -f json 2023-11-28T01:11:52.237 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: purged_snaps scrub starts 2023-11-28T01:11:52.237 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: purged_snaps scrub ok 2023-11-28T01:11:52.237 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:52.238 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: osd.7 [v2:172.21.15.192:6824/4284606111,v1:172.21.15.192:6825/4284606111] boot 2023-11-28T01:11:52.238 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: osdmap e54: 8 total, 8 up, 8 in 2023-11-28T01:11:52.238 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:52.239 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.239 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.239 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.240 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.240 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.240 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.240 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.240 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:52.241 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:52.241 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.241 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[118784]: osdmap e55: 8 total, 8 up, 8 in 2023-11-28T01:11:52.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: purged_snaps scrub starts 2023-11-28T01:11:52.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: purged_snaps scrub ok 2023-11-28T01:11:52.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2023-11-28T01:11:52.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: osd.7 [v2:172.21.15.192:6824/4284606111,v1:172.21.15.192:6825/4284606111] boot 2023-11-28T01:11:52.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: osdmap e54: 8 total, 8 up, 8 in 2023-11-28T01:11:52.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-11-28T01:11:52.241 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.242 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.242 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.242 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.242 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.242 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-11-28T01:11:52.242 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.242 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:11:52.242 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:11:52.242 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:11:52.242 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:51 smithi172 ceph-mon[100627]: osdmap e55: 8 total, 8 up, 8 in 2023-11-28T01:11:52.550 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:11:53.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:52 smithi172 ceph-mon[100627]: Detected new or changed devices on smithi192 2023-11-28T01:11:53.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:52 smithi172 ceph-mon[100627]: Adjusting osd_memory_target on smithi192 to 4292M 2023-11-28T01:11:53.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:52 smithi172 ceph-mon[118784]: Detected new or changed devices on smithi192 2023-11-28T01:11:53.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:52 smithi172 ceph-mon[118784]: Adjusting osd_memory_target on smithi192 to 4292M 2023-11-28T01:11:53.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:52 smithi192 ceph-mon[105371]: Detected new or changed devices on smithi192 2023-11-28T01:11:53.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:52 smithi192 ceph-mon[105371]: Adjusting osd_memory_target on smithi192 to 4292M 2023-11-28T01:11:53.929 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.926+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/3823671398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4ac102810 msgr2=0x7ff4ac102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:53.929 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.926+0000 7ff4b2a3d700 1 --2- 172.21.15.172:0/3823671398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4ac102810 0x7ff4ac102c70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff494009990 tx=0x7ff494009ca0 comp rx=0 tx=0).stop 2023-11-28T01:11:53.929 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.926+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/3823671398 shutdown_connections 2023-11-28T01:11:53.929 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.926+0000 7ff4b2a3d700 1 --2- 172.21.15.172:0/3823671398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4ac1031b0 0x7ff4ac109a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:53.929 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.926+0000 7ff4b2a3d700 1 --2- 172.21.15.172:0/3823671398 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4ac101610 0x7ff4ac101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:53.929 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.926+0000 7ff4b2a3d700 1 --2- 172.21.15.172:0/3823671398 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4ac102810 0x7ff4ac102c70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:53.929 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.926+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/3823671398 >> 172.21.15.172:0/3823671398 conn(0x7ff4ac0fcbb0 msgr2=0x7ff4ac0feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.926+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/3823671398 shutdown_connections 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.926+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/3823671398 wait complete. 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4b2a3d700 1 Processor -- start 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4b2a3d700 1 -- start start 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4b2a3d700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4ac101610 0x7ff4ac196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4b2a3d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4ac102810 0x7ff4ac197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4b2a3d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4ac1031b0 0x7ff4ac19e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4b2a3d700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac19e5f0 con 0x7ff4ac101610 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4b2a3d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac19e730 con 0x7ff4ac102810 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4b2a3d700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac19e8c0 con 0x7ff4ac1031b0 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4abfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4ac101610 0x7ff4ac196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4abfff700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4ac101610 0x7ff4ac196b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3300/0 says I am v2:172.21.15.172:49982/0 (socket says 172.21.15.172:49982) 2023-11-28T01:11:53.930 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4abfff700 1 -- 172.21.15.172:0/686428760 learned_addr learned my addr 172.21.15.172:0/686428760 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4b0fda700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4ac1031b0 0x7ff4ac19e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.927+0000 7ff4ab7fe700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4ac102810 0x7ff4ac197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4abfff700 1 -- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4ac1031b0 msgr2=0x7ff4ac19e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4abfff700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4ac1031b0 0x7ff4ac19e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4abfff700 1 -- 172.21.15.172:0/686428760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4ac102810 msgr2=0x7ff4ac197040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4abfff700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4ac102810 0x7ff4ac197040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4abfff700 1 -- 172.21.15.172:0/686428760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c009520 con 0x7ff4ac101610 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4ab7fe700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4ac102810 0x7ff4ac197040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4abfff700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4ac101610 0x7ff4ac196b00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff49c00ea80 tx=0x7ff49c00ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4a97fa700 1 -- 172.21.15.172:0/686428760 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c00cbd0 con 0x7ff4ac101610 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4a97fa700 1 -- 172.21.15.172:0/686428760 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7ff49c00cd30 con 0x7ff4ac101610 2023-11-28T01:11:53.931 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4a97fa700 1 -- 172.21.15.172:0/686428760 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c018480 con 0x7ff4ac101610 2023-11-28T01:11:53.932 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/686428760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4940095f0 con 0x7ff4ac101610 2023-11-28T01:11:53.932 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.928+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/686428760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ac19ef10 con 0x7ff4ac101610 2023-11-28T01:11:53.932 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.929+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/686428760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ac108b80 con 0x7ff4ac101610 2023-11-28T01:11:53.933 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.929+0000 7ff4a97fa700 1 -- 172.21.15.172:0/686428760 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7ff49c010430 con 0x7ff4ac101610 2023-11-28T01:11:53.933 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.930+0000 7ff4a97fa700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7ff49806d380 0x7ff49806f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:53.933 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.930+0000 7ff4a97fa700 1 -- 172.21.15.172:0/686428760 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5316+0+0 (secure 0 0 0) 0x7ff49c014070 con 0x7ff4ac101610 2023-11-28T01:11:53.933 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.930+0000 7ff4ab7fe700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7ff49806d380 0x7ff49806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:53.933 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.930+0000 7ff4ab7fe700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7ff49806d380 0x7ff49806f840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff494000c00 tx=0x7ff494005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:53.935 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:53.934+0000 7ff4a97fa700 1 -- 172.21.15.172:0/686428760 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7ff49c056bb0 con 0x7ff4ac101610 2023-11-28T01:11:54.128 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.127+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/686428760 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7ff4ac10c440 con 0x7ff4ac101610 2023-11-28T01:11:54.128 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.127+0000 7ff4a97fa700 1 -- 172.21.15.172:0/686428760 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v57) v1 ==== 74+0+130 (secure 0 0 0) 0x7ff49c05a1d0 con 0x7ff4ac101610 2023-11-28T01:11:54.129 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:11:54.131 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.129+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7ff49806d380 msgr2=0x7ff49806f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:54.131 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.129+0000 7ff4b2a3d700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7ff49806d380 0x7ff49806f840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff494000c00 tx=0x7ff494005fb0 comp rx=0 tx=0).stop 2023-11-28T01:11:54.131 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.129+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4ac101610 msgr2=0x7ff4ac196b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:54.131 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.129+0000 7ff4b2a3d700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4ac101610 0x7ff4ac196b00 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff49c00ea80 tx=0x7ff49c00ed90 comp rx=0 tx=0).stop 2023-11-28T01:11:54.131 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.130+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/686428760 shutdown_connections 2023-11-28T01:11:54.131 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.130+0000 7ff4b2a3d700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7ff49806d380 0x7ff49806f840 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:54.131 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.130+0000 7ff4b2a3d700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7ff4ac1031b0 0x7ff4ac19e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:54.131 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.130+0000 7ff4b2a3d700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7ff4ac101610 0x7ff4ac196b00 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:54.131 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.130+0000 7ff4b2a3d700 1 --2- 172.21.15.172:0/686428760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4ac102810 0x7ff4ac197040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:54.131 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.130+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/686428760 >> 172.21.15.172:0/686428760 conn(0x7ff4ac0fcbb0 msgr2=0x7ff4ac105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:54.132 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.130+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/686428760 shutdown_connections 2023-11-28T01:11:54.132 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:54.130+0000 7ff4b2a3d700 1 -- 172.21.15.172:0/686428760 wait complete. 2023-11-28T01:11:54.289 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:53 smithi172 ceph-mon[100627]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:11:54.289 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:53 smithi172 ceph-mon[100627]: osdmap e56: 8 total, 8 up, 8 in 2023-11-28T01:11:54.290 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:53 smithi172 ceph-mon[118784]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:11:54.290 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:53 smithi172 ceph-mon[118784]: osdmap e56: 8 total, 8 up, 8 in 2023-11-28T01:11:54.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:53 smithi192 ceph-mon[105371]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:11:54.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:53 smithi192 ceph-mon[105371]: osdmap e56: 8 total, 8 up, 8 in 2023-11-28T01:11:54.744 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":57,"num_osds":8,"num_up_osds":8,"osd_up_since":1701133910,"num_in_osds":8,"osd_in_since":1701133893,"num_remapped_pgs":0} 2023-11-28T01:11:54.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph osd dump --format=json 2023-11-28T01:11:55.125 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:11:55.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:54 smithi172 ceph-mon[100627]: osdmap e57: 8 total, 8 up, 8 in 2023-11-28T01:11:55.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:54 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/686428760' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-28T01:11:55.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:54 smithi172 ceph-mon[118784]: osdmap e57: 8 total, 8 up, 8 in 2023-11-28T01:11:55.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:54 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/686428760' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-28T01:11:55.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:54 smithi192 ceph-mon[105371]: osdmap e57: 8 total, 8 up, 8 in 2023-11-28T01:11:55.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:54 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/686428760' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-11-28T01:11:55.984 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:55 smithi172 ceph-mon[118784]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:11:55.984 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:55 smithi172 ceph-mon[118784]: osdmap e58: 8 total, 8 up, 8 in 2023-11-28T01:11:55.984 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:55 smithi172 ceph-mon[100627]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:11:55.984 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:55 smithi172 ceph-mon[100627]: osdmap e58: 8 total, 8 up, 8 in 2023-11-28T01:11:56.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:55 smithi192 ceph-mon[105371]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:11:56.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:55 smithi192 ceph-mon[105371]: osdmap e58: 8 total, 8 up, 8 in 2023-11-28T01:11:56.417 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.415+0000 7f1f38b58700 1 -- 172.21.15.172:0/3457578913 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f34100530 msgr2=0x7f1f341009b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:56.417 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.415+0000 7f1f38b58700 1 --2- 172.21.15.172:0/3457578913 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f34100530 0x7f1f341009b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009990 tx=0x7f1f24009ca0 comp rx=0 tx=0).stop 2023-11-28T01:11:56.418 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.416+0000 7f1f38b58700 1 -- 172.21.15.172:0/3457578913 shutdown_connections 2023-11-28T01:11:56.418 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.416+0000 7f1f38b58700 1 --2- 172.21.15.172:0/3457578913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f34100ef0 0x7f1f3410d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:56.418 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.416+0000 7f1f38b58700 1 --2- 172.21.15.172:0/3457578913 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f34106560 0x7f1f34106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:56.418 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.416+0000 7f1f38b58700 1 --2- 172.21.15.172:0/3457578913 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f34100530 0x7f1f341009b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:56.418 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.416+0000 7f1f38b58700 1 -- 172.21.15.172:0/3457578913 >> 172.21.15.172:0/3457578913 conn(0x7f1f34076570 msgr2=0x7f1f34076980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:56.418 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.416+0000 7f1f38b58700 1 -- 172.21.15.172:0/3457578913 shutdown_connections 2023-11-28T01:11:56.418 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.416+0000 7f1f38b58700 1 -- 172.21.15.172:0/3457578913 wait complete. 2023-11-28T01:11:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f38b58700 1 Processor -- start 2023-11-28T01:11:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f38b58700 1 -- start start 2023-11-28T01:11:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f38b58700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f34100530 0x7f1f34197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f38b58700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f34100ef0 0x7f1f34198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f38b58700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f34106560 0x7f1f341a2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f38b58700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34198b30 con 0x7f1f34100ef0 2023-11-28T01:11:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f38b58700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f341a32f0 con 0x7f1f34100530 2023-11-28T01:11:56.420 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f38b58700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f341a3430 con 0x7f1f34106560 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f32d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f34106560 0x7f1f341a2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f31d9b700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f34100ef0 0x7f1f34198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f32d9d700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f34106560 0x7f1f341a2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:34436/0 (socket says 172.21.15.172:34436) 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f32d9d700 1 -- 172.21.15.172:0/2803114270 learned_addr learned my addr 172.21.15.172:0/2803114270 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f3259c700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f34100530 0x7f1f34197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f31d9b700 1 -- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f34106560 msgr2=0x7f1f341a2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f31d9b700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f34106560 0x7f1f341a2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f31d9b700 1 -- 172.21.15.172:0/2803114270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f34100530 msgr2=0x7f1f34197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f31d9b700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f34100530 0x7f1f34197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f31d9b700 1 -- 172.21.15.172:0/2803114270 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f28009520 con 0x7f1f34100ef0 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.417+0000 7f1f32d9d700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f34106560 0x7f1f341a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.418+0000 7f1f31d9b700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f34100ef0 0x7f1f34198270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1f24000c00 tx=0x7f1f24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:56.421 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.418+0000 7f1f237fe700 1 -- 172.21.15.172:0/2803114270 <== mon.0 v2:172.21.15.172:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2401d070 con 0x7f1f34100ef0 2023-11-28T01:11:56.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.418+0000 7f1f237fe700 1 -- 172.21.15.172:0/2803114270 <== mon.0 v2:172.21.15.172:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f1f24021410 con 0x7f1f34100ef0 2023-11-28T01:11:56.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.418+0000 7f1f38b58700 1 -- 172.21.15.172:0/2803114270 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f240095f0 con 0x7f1f34100ef0 2023-11-28T01:11:56.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.418+0000 7f1f237fe700 1 -- 172.21.15.172:0/2803114270 <== mon.0 v2:172.21.15.172:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f240173f0 con 0x7f1f34100ef0 2023-11-28T01:11:56.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.418+0000 7f1f38b58700 1 -- 172.21.15.172:0/2803114270 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f341a3990 con 0x7f1f34100ef0 2023-11-28T01:11:56.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.419+0000 7f1f38b58700 1 -- 172.21.15.172:0/2803114270 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f34101bd0 con 0x7f1f34100ef0 2023-11-28T01:11:56.422 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.419+0000 7f1f237fe700 1 -- 172.21.15.172:0/2803114270 <== mon.0 v2:172.21.15.172:3300/0 4 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7f1f2400fb10 con 0x7f1f34100ef0 2023-11-28T01:11:56.423 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.420+0000 7f1f237fe700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1f1c06d380 0x7f1f1c06f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:56.423 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.420+0000 7f1f237fe700 1 -- 172.21.15.172:0/2803114270 <== mon.0 v2:172.21.15.172:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f1f2408d690 con 0x7f1f34100ef0 2023-11-28T01:11:56.423 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.420+0000 7f1f3259c700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1f1c06d380 0x7f1f1c06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:56.423 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.420+0000 7f1f3259c700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1f1c06d380 0x7f1f1c06f840 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1f28009d70 tx=0x7f1f28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:56.425 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.424+0000 7f1f237fe700 1 -- 172.21.15.172:0/2803114270 <== mon.0 v2:172.21.15.172:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f1f24057790 con 0x7f1f34100ef0 2023-11-28T01:11:56.618 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.617+0000 7f1f38b58700 1 -- 172.21.15.172:0/2803114270 --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f34069510 con 0x7f1f34100ef0 2023-11-28T01:11:56.618 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.617+0000 7f1f237fe700 1 -- 172.21.15.172:0/2803114270 <== mon.0 v2:172.21.15.172:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+13590 (secure 0 0 0) 0x7f1f2405adb0 con 0x7f1f34100ef0 2023-11-28T01:11:56.619 INFO:teuthology.orchestra.run.smithi172.stdout: 2023-11-28T01:11:56.619 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":58,"fsid":"31861c10-8d8a-11ee-95a2-87774f69a715","created":"2023-11-28T01:06:58.085390+0000","modified":"2023-11-28T01:11:54.977401+0000","last_up_change":"2023-11-28T01:11:50.952025+0000","last_in_change":"2023-11-28T01:11:33.629706+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-11-28T01:09:57.022343+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e8f0a47b-64f5-4ccd-9db0-e3fb0fd30997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":2204873896},{"type":"v1","addr":"172.21.15.172:6803","nonce":2204873896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":2204873896},{"type":"v1","addr":"172.21.15.172:6805","nonce":2204873896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":2204873896},{"type":"v1","addr":"172.21.15.172:6809","nonce":2204873896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":2204873896},{"type":"v1","addr":"172.21.15.172:6807","nonce":2204873896}]},"public_addr":"172.21.15.172:6803/2204873896","cluster_addr":"172.21.15.172:6805/2204873896","heartbeat_back_addr":"172.21.15.172:6809/2204873896","heartbeat_front_addr":"172.21.15.172:6807/2204873896","state":["exists","up"]},{"osd":1,"uuid":"3ef0c783-7288-49f0-b832-a3398aaa19a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":2715954141},{"type":"v1","addr":"172.21.15.172:6811","nonce":2715954141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":2715954141},{"type":"v1","addr":"172.21.15.172:6813","nonce":2715954141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":2715954141},{"type":"v1","addr":"172.21.15.172:6817","nonce":2715954141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":2715954141},{"type":"v1","addr":"172.21.15.172:6815","nonce":2715954141}]},"public_addr":"172.21.15.172:6811/2715954141","cluster_addr":"172.21.15.172:6813/2715954141","heartbeat_back_addr":"172.21.15.172:6817/2715954141","heartbeat_front_addr":"172.21.15.172:6815/2715954141","state":["exists","up"]},{"osd":2,"uuid":"e762414e-fc5a-4aad-83a8-2f6c636bfde9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":3771992877},{"type":"v1","addr":"172.21.15.172:6819","nonce":3771992877}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":3771992877},{"type":"v1","addr":"172.21.15.172:6821","nonce":3771992877}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3771992877},{"type":"v1","addr":"172.21.15.172:6825","nonce":3771992877}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":3771992877},{"type":"v1","addr":"172.21.15.172:6823","nonce":3771992877}]},"public_addr":"172.21.15.172:6819/3771992877","cluster_addr":"172.21.15.172:6821/3771992877","heartbeat_back_addr":"172.21.15.172:6825/3771992877","heartbeat_front_addr":"172.21.15.172:6823/3771992877","state":["exists","up"]},{"osd":3,"uuid":"c57f1e20-cf1c-4a89-82ab-274033efe6b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":659880403},{"type":"v1","addr":"172.21.15.172:6827","nonce":659880403}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":659880403},{"type":"v1","addr":"172.21.15.172:6829","nonce":659880403}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6832","nonce":659880403},{"type":"v1","addr":"172.21.15.172:6833","nonce":659880403}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":659880403},{"type":"v1","addr":"172.21.15.172:6831","nonce":659880403}]},"public_addr":"172.21.15.172:6827/659880403","cluster_addr":"172.21.15.172:6829/659880403","heartbeat_back_addr":"172.21.15.172:6833/659880403","heartbeat_front_addr":"172.21.15.172:6831/659880403","state":["exists","up"]},{"osd":4,"uuid":"d2097369-4ab9-4e3a-8f60-ea410df7ec59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":1069750856},{"type":"v1","addr":"172.21.15.192:6801","nonce":1069750856}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":1069750856},{"type":"v1","addr":"172.21.15.192:6803","nonce":1069750856}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":1069750856},{"type":"v1","addr":"172.21.15.192:6807","nonce":1069750856}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":1069750856},{"type":"v1","addr":"172.21.15.192:6805","nonce":1069750856}]},"public_addr":"172.21.15.192:6801/1069750856","cluster_addr":"172.21.15.192:6803/1069750856","heartbeat_back_addr":"172.21.15.192:6807/1069750856","heartbeat_front_addr":"172.21.15.192:6805/1069750856","state":["exists","up"]},{"osd":5,"uuid":"3a3cb271-985d-412a-affd-6b6bceff3184","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":1165321246},{"type":"v1","addr":"172.21.15.192:6809","nonce":1165321246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":1165321246},{"type":"v1","addr":"172.21.15.192:6811","nonce":1165321246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":1165321246},{"type":"v1","addr":"172.21.15.192:6815","nonce":1165321246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":1165321246},{"type":"v1","addr":"172.21.15.192:6813","nonce":1165321246}]},"public_addr":"172.21.15.192:6809/1165321246","cluster_addr":"172.21.15.192:6811/1165321246","heartbeat_back_addr":"172.21.15.192:6815/1165321246","heartbeat_front_addr":"172.21.15.192:6813/1165321246","state":["exists","up"]},{"osd":6,"uuid":"a5883ed0-fa9b-4bc7-90a6-3f5e8700d8eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1409239444},{"type":"v1","addr":"172.21.15.192:6817","nonce":1409239444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1409239444},{"type":"v1","addr":"172.21.15.192:6819","nonce":1409239444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1409239444},{"type":"v1","addr":"172.21.15.192:6823","nonce":1409239444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1409239444},{"type":"v1","addr":"172.21.15.192:6821","nonce":1409239444}]},"public_addr":"172.21.15.192:6817/1409239444","cluster_addr":"172.21.15.192:6819/1409239444","heartbeat_back_addr":"172.21.15.192:6823/1409239444","heartbeat_front_addr":"172.21.15.192:6821/1409239444","state":["exists","up"]},{"osd":7,"uuid":"8092d059-facb-4b45-93cc-096da7c79f69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":4284606111},{"type":"v1","addr":"172.21.15.192:6825","nonce":4284606111}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":4284606111},{"type":"v1","addr":"172.21.15.192:6827","nonce":4284606111}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":4284606111},{"type":"v1","addr":"172.21.15.192:6831","nonce":4284606111}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":4284606111},{"type":"v1","addr":"172.21.15.192:6829","nonce":4284606111}]},"public_addr":"172.21.15.192:6825/4284606111","cluster_addr":"172.21.15.192:6827/4284606111","heartbeat_back_addr":"172.21.15.192:6831/4284606111","heartbeat_front_addr":"172.21.15.192:6829/4284606111","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T01:09:08.248824+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T01:09:31.253887+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T01:09:53.838044+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T01:10:17.046232+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T01:10:39.279156+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T01:11:02.686906+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T01:11:26.001012+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-11-28T01:11:49.088434+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.172:0/782488298":"2023-11-29T01:07:50.696325+0000","172.21.15.172:0/3903851192":"2023-11-29T01:07:50.696325+0000","172.21.15.172:0/1071659318":"2023-11-29T01:07:24.604351+0000","172.21.15.172:6801/2738031896":"2023-11-29T01:07:50.696325+0000","172.21.15.172:6801/618480980":"2023-11-29T01:07:24.604351+0000","172.21.15.172:6800/618480980":"2023-11-29T01:07:24.604351+0000","172.21.15.172:0/4125833062":"2023-11-29T01:07:24.604351+0000","172.21.15.172:0/3790567548":"2023-11-29T01:07:24.604351+0000","172.21.15.172:0/2095737196":"2023-11-29T01:07:50.696325+0000","172.21.15.172:6800/2738031896":"2023-11-29T01:07:50.696325+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-11-28T01:11:56.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.619+0000 7f1f38b58700 1 -- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1f1c06d380 msgr2=0x7f1f1c06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:56.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.619+0000 7f1f38b58700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1f1c06d380 0x7f1f1c06f840 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1f28009d70 tx=0x7f1f28009260 comp rx=0 tx=0).stop 2023-11-28T01:11:56.621 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.619+0000 7f1f38b58700 1 -- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f34100ef0 msgr2=0x7f1f34198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:56.622 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.619+0000 7f1f38b58700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f34100ef0 0x7f1f34198270 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1f24000c00 tx=0x7f1f24004730 comp rx=0 tx=0).stop 2023-11-28T01:11:56.622 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.620+0000 7f1f38b58700 1 -- 172.21.15.172:0/2803114270 shutdown_connections 2023-11-28T01:11:56.622 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.620+0000 7f1f38b58700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f1f1c06d380 0x7f1f1c06f840 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:56.622 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.620+0000 7f1f38b58700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f1f34106560 0x7f1f341a2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:56.622 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.620+0000 7f1f38b58700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f34100530 0x7f1f34197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:56.622 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.620+0000 7f1f38b58700 1 --2- 172.21.15.172:0/2803114270 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f1f34100ef0 0x7f1f34198270 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:56.622 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.620+0000 7f1f38b58700 1 -- 172.21.15.172:0/2803114270 >> 172.21.15.172:0/2803114270 conn(0x7f1f34076570 msgr2=0x7f1f34107270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:56.623 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.620+0000 7f1f38b58700 1 -- 172.21.15.172:0/2803114270 shutdown_connections 2023-11-28T01:11:56.623 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:56.620+0000 7f1f38b58700 1 -- 172.21.15.172:0/2803114270 wait complete. 2023-11-28T01:11:56.988 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:56 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2803114270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-28T01:11:57.186 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-11-28T01:09:57.022343+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-11-28T01:11:57.187 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph osd pool get .mgr pg_num 2023-11-28T01:11:57.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:56 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2803114270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-28T01:11:57.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:56 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2803114270' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-11-28T01:11:57.527 INFO:teuthology.orchestra.run.smithi172.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.a/config 2023-11-28T01:11:58.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:57 smithi172 ceph-mon[100627]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:11:58.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:57 smithi172 ceph-mon[118784]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:11:58.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:57 smithi192 ceph-mon[105371]: pgmap v171: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:11:58.851 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.847+0000 7fdc8183f700 1 -- 172.21.15.172:0/1226725789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c1066b0 msgr2=0x7fdc7c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:58.852 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.847+0000 7fdc8183f700 1 --2- 172.21.15.172:0/1226725789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c1066b0 0x7fdc7c106a90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdc700098f0 tx=0x7fdc70009c00 comp rx=0 tx=0).stop 2023-11-28T01:11:58.852 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.847+0000 7fdc8183f700 1 -- 172.21.15.172:0/1226725789 shutdown_connections 2023-11-28T01:11:58.852 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.847+0000 7fdc8183f700 1 --2- 172.21.15.172:0/1226725789 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc7c100910 0x7fdc7c10e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:58.853 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.847+0000 7fdc8183f700 1 --2- 172.21.15.172:0/1226725789 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c1066b0 0x7fdc7c106a90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:58.853 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.847+0000 7fdc8183f700 1 --2- 172.21.15.172:0/1226725789 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c0fff70 0x7fdc7c1003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:58.853 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.847+0000 7fdc8183f700 1 -- 172.21.15.172:0/1226725789 >> 172.21.15.172:0/1226725789 conn(0x7fdc7c0757f0 msgr2=0x7fdc7c075c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:58.853 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.847+0000 7fdc8183f700 1 -- 172.21.15.172:0/1226725789 shutdown_connections 2023-11-28T01:11:58.854 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.847+0000 7fdc8183f700 1 -- 172.21.15.172:0/1226725789 wait complete. 2023-11-28T01:11:58.854 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc8183f700 1 Processor -- start 2023-11-28T01:11:58.854 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc8183f700 1 -- start start 2023-11-28T01:11:58.854 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc8183f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc7c0fff70 0x7fdc7c196fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:58.855 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc8183f700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c100910 0x7fdc7c197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:58.855 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc8183f700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c1066b0 0x7fdc7c19b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:58.855 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc8183f700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c19bee0 con 0x7fdc7c1066b0 2023-11-28T01:11:58.855 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc8183f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c19c050 con 0x7fdc7c0fff70 2023-11-28T01:11:58.856 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc8183f700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c19c210 con 0x7fdc7c100910 2023-11-28T01:11:58.856 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc7a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c100910 0x7fdc7c197520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:58.856 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc7a7fc700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c100910 0x7fdc7c197520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.172:34444/0 (socket says 172.21.15.172:34444) 2023-11-28T01:11:58.857 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc7a7fc700 1 -- 172.21.15.172:0/1569043482 learned_addr learned my addr 172.21.15.172:0/1569043482 (peer_addr_for_me v2:172.21.15.172:0/0) 2023-11-28T01:11:58.857 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc7b7fe700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c1066b0 0x7fdc7c19b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:58.857 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.848+0000 7fdc7affd700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc7c0fff70 0x7fdc7c196fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:58.857 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc7a7fc700 1 -- 172.21.15.172:0/1569043482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc7c0fff70 msgr2=0x7fdc7c196fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:58.857 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc7a7fc700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc7c0fff70 0x7fdc7c196fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:58.857 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc7a7fc700 1 -- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c1066b0 msgr2=0x7fdc7c19b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:58.857 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc7a7fc700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c1066b0 0x7fdc7c19b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:58.858 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc7a7fc700 1 -- 172.21.15.172:0/1569043482 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c0095f0 con 0x7fdc7c100910 2023-11-28T01:11:58.858 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc7affd700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc7c0fff70 0x7fdc7c196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-11-28T01:11:58.858 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc7a7fc700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c100910 0x7fdc7c197520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00ec10 tx=0x7fdc6c00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:58.858 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc8083d700 1 -- 172.21.15.172:0/1569043482 <== mon.2 v2:172.21.15.172:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c00cd40 con 0x7fdc7c100910 2023-11-28T01:11:58.858 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc8183f700 1 -- 172.21.15.172:0/1569043482 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc70009520 con 0x7fdc7c100910 2023-11-28T01:11:58.858 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc8083d700 1 -- 172.21.15.172:0/1569043482 <== mon.2 v2:172.21.15.172:3301/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7fdc6c004470 con 0x7fdc7c100910 2023-11-28T01:11:58.858 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc8083d700 1 -- 172.21.15.172:0/1569043482 <== mon.2 v2:172.21.15.172:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc6c003e00 con 0x7fdc7c100910 2023-11-28T01:11:58.858 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.849+0000 7fdc8183f700 1 -- 172.21.15.172:0/1569043482 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c1a3b80 con 0x7fdc7c100910 2023-11-28T01:11:58.859 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.850+0000 7fdc8183f700 1 -- 172.21.15.172:0/1569043482 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c106230 con 0x7fdc7c100910 2023-11-28T01:11:58.859 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.850+0000 7fdc8083d700 1 -- 172.21.15.172:0/1569043482 <== mon.2 v2:172.21.15.172:3301/0 4 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7fdc6c003f60 con 0x7fdc7c100910 2023-11-28T01:11:58.859 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.851+0000 7fdc8083d700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fdc6406d380 0x7fdc6406f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:11:58.859 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.851+0000 7fdc8083d700 1 -- 172.21.15.172:0/1569043482 <== mon.2 v2:172.21.15.172:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5287+0+0 (secure 0 0 0) 0x7fdc6c014070 con 0x7fdc7c100910 2023-11-28T01:11:58.860 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.851+0000 7fdc7affd700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fdc6406d380 0x7fdc6406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:11:58.860 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.851+0000 7fdc7affd700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fdc6406d380 0x7fdc6406f840 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009810 tx=0x7fdc70019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:11:58.860 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:58.855+0000 7fdc8083d700 1 -- 172.21.15.172:0/1569043482 <== mon.2 v2:172.21.15.172:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7fdc6c056cf0 con 0x7fdc7c100910 2023-11-28T01:11:59.046 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.043+0000 7fdc8183f700 1 -- 172.21.15.172:0/1569043482 --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdc7c04f290 con 0x7fdc7c100910 2023-11-28T01:11:59.046 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.043+0000 7fdc8083d700 1 -- 172.21.15.172:0/1569043482 <== mon.2 v2:172.21.15.172:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdc6c05a310 con 0x7fdc7c100910 2023-11-28T01:11:59.046 INFO:teuthology.orchestra.run.smithi172.stdout:pg_num: 1 2023-11-28T01:11:59.048 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 -- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fdc6406d380 msgr2=0x7fdc6406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:59.049 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fdc6406d380 0x7fdc6406f840 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdc70009810 tx=0x7fdc70019040 comp rx=0 tx=0).stop 2023-11-28T01:11:59.049 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 -- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c100910 msgr2=0x7fdc7c197520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:11:59.049 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c100910 0x7fdc7c197520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c00ec10 tx=0x7fdc6c00ef20 comp rx=0 tx=0).stop 2023-11-28T01:11:59.049 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 -- 172.21.15.172:0/1569043482 shutdown_connections 2023-11-28T01:11:59.049 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fdc6406d380 0x7fdc6406f840 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:59.049 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fdc7c1066b0 0x7fdc7c19b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:59.050 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc7c0fff70 0x7fdc7c196fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:59.050 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 --2- 172.21.15.172:0/1569043482 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fdc7c100910 0x7fdc7c197520 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:11:59.050 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 -- 172.21.15.172:0/1569043482 >> 172.21.15.172:0/1569043482 conn(0x7fdc7c0757f0 msgr2=0x7fdc7c103370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:11:59.050 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 -- 172.21.15.172:0/1569043482 shutdown_connections 2023-11-28T01:11:59.050 INFO:teuthology.orchestra.run.smithi172.stderr:2023-11-28T01:11:59.046+0000 7fdc8183f700 1 -- 172.21.15.172:0/1569043482 wait complete. 2023-11-28T01:11:59.635 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi172 2023-11-28T01:11:59.635 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi172=foo.a' 2023-11-28T01:11:59.998 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:12:00.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:59 smithi172 ceph-mon[100627]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-28T01:12:00.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:11:59 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/1569043482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-28T01:12:00.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:59 smithi172 ceph-mon[118784]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-28T01:12:00.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:11:59 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/1569043482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-28T01:12:00.320 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:59 smithi192 ceph-mon[105371]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-11-28T01:12:00.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:11:59 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/1569043482' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-11-28T01:12:01.277 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:00 smithi192 ceph-mon[105371]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2023-11-28T01:12:01.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:00 smithi172 ceph-mon[100627]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2023-11-28T01:12:01.304 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:00 smithi172 ceph-mon[118784]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2023-11-28T01:12:01.490 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.485+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/935731326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bc4100510 msgr2=0x7f6bc4100930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.485+0000 7f6bcb0e3700 1 --2- 172.21.15.192:0/935731326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bc4100510 0x7f6bc4100930 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6bb80099c0 tx=0x7f6bb8009cd0 comp rx=0 tx=0).stop 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.485+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/935731326 shutdown_connections 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.485+0000 7f6bcb0e3700 1 --2- 172.21.15.192:0/935731326 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bc41020b0 0x7f6bc4108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.485+0000 7f6bcb0e3700 1 --2- 172.21.15.192:0/935731326 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bc4100510 0x7f6bc4100930 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.485+0000 7f6bcb0e3700 1 --2- 172.21.15.192:0/935731326 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bc4101710 0x7f6bc4101b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.485+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/935731326 >> 172.21.15.192:0/935731326 conn(0x7f6bc40fbad0 msgr2=0x7f6bc40fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.485+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/935731326 shutdown_connections 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.485+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/935731326 wait complete. 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bcb0e3700 1 Processor -- start 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bcb0e3700 1 -- start start 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bcb0e3700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bc4100510 0x7f6bc41959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bcb0e3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bc4101710 0x7f6bc4195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:01.491 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bcb0e3700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bc41020b0 0x7f6bc419cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:01.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bcb0e3700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc419d500 con 0x7f6bc4100510 2023-11-28T01:12:01.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bcb0e3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc419d640 con 0x7f6bc4101710 2023-11-28T01:12:01.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bcb0e3700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bc419d7d0 con 0x7f6bc41020b0 2023-11-28T01:12:01.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bc3fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bc4101710 0x7f6bc4195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:01.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bc3fff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bc4101710 0x7f6bc4195f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:49578/0 (socket says 172.21.15.192:49578) 2023-11-28T01:12:01.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bc3fff700 1 -- 172.21.15.192:0/1260727150 learned_addr learned my addr 172.21.15.192:0/1260727150 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:12:01.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bc8e7f700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bc4100510 0x7f6bc41959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:01.492 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bc9680700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bc41020b0 0x7f6bc419cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:01.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bc3fff700 1 -- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bc41020b0 msgr2=0x7f6bc419cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:01.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bc3fff700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bc41020b0 0x7f6bc419cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:01.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bc3fff700 1 -- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bc4100510 msgr2=0x7f6bc41959c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:01.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bc3fff700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bc4100510 0x7f6bc41959c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:01.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bc3fff700 1 -- 172.21.15.192:0/1260727150 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb80095f0 con 0x7f6bc4101710 2023-11-28T01:12:01.493 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.486+0000 7f6bc3fff700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bc4101710 0x7f6bc4195f00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000c920 tx=0x7f6bb000cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:12:01.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.487+0000 7f6bc1ffb700 1 -- 172.21.15.192:0/1260727150 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0004a30 con 0x7f6bc4101710 2023-11-28T01:12:01.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.487+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/1260727150 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bc419dab0 con 0x7f6bc4101710 2023-11-28T01:12:01.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.487+0000 7f6bc1ffb700 1 -- 172.21.15.192:0/1260727150 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f6bb0005020 con 0x7f6bc4101710 2023-11-28T01:12:01.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.487+0000 7f6bc1ffb700 1 -- 172.21.15.192:0/1260727150 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb00079d0 con 0x7f6bc4101710 2023-11-28T01:12:01.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.487+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/1260727150 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bc419dfb0 con 0x7f6bc4101710 2023-11-28T01:12:01.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.488+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/1260727150 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bc4107a30 con 0x7f6bc4101710 2023-11-28T01:12:01.494 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.488+0000 7f6bc1ffb700 1 -- 172.21.15.192:0/1260727150 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7f6bb0004b90 con 0x7f6bc4101710 2023-11-28T01:12:01.495 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.489+0000 7f6bc1ffb700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6bb406d380 0x7f6bb406f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:01.495 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.489+0000 7f6bc1ffb700 1 -- 172.21.15.192:0/1260727150 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f6bb008c270 con 0x7f6bc4101710 2023-11-28T01:12:01.495 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.489+0000 7f6bc8e7f700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6bb406d380 0x7f6bb406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:01.496 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.489+0000 7f6bc8e7f700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6bb406d380 0x7f6bb406f840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800c010 tx=0x7f6bb8005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:12:01.496 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.493+0000 7f6bc1ffb700 1 -- 172.21.15.192:0/1260727150 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f6bb0056240 con 0x7f6bc4101710 2023-11-28T01:12:01.697 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.695+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/1260727150 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi172=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f6bc410b430 con 0x7f6bb406d380 2023-11-28T01:12:01.704 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.703+0000 7f6bc1ffb700 1 -- 172.21.15.192:0/1260727150 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7f6bc410b430 con 0x7f6bb406d380 2023-11-28T01:12:01.705 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled rgw.foo.a update... 2023-11-28T01:12:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6bb406d380 msgr2=0x7f6bb406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6bb406d380 0x7f6bb406f840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6bb800c010 tx=0x7f6bb8005c60 comp rx=0 tx=0).stop 2023-11-28T01:12:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/1260727150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bc4101710 msgr2=0x7f6bc4195f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bc4101710 0x7f6bc4195f00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000c920 tx=0x7f6bb000cc30 comp rx=0 tx=0).stop 2023-11-28T01:12:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/1260727150 shutdown_connections 2023-11-28T01:12:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f6bb406d380 0x7f6bb406f840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f6bc41020b0 0x7f6bc419cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:01.707 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f6bc4100510 0x7f6bc41959c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:01.708 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 --2- 172.21.15.192:0/1260727150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bc4101710 0x7f6bc4195f00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:01.708 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/1260727150 >> 172.21.15.192:0/1260727150 conn(0x7f6bc40fbad0 msgr2=0x7f6bc4104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:12:01.708 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/1260727150 shutdown_connections 2023-11-28T01:12:01.708 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:01.705+0000 7f6bcb0e3700 1 -- 172.21.15.192:0/1260727150 wait complete. 2023-11-28T01:12:02.391 DEBUG:teuthology.orchestra.run.smithi172:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@rgw.foo.a.service 2023-11-28T01:12:02.394 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi192 2023-11-28T01:12:02.394 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2023-11-28T01:12:02.422 INFO:journalctl@ceph.rgw.foo.a.smithi172.stdout:-- Logs begin at Tue 2023-11-28 00:47:41 UTC. -- 2023-11-28T01:12:02.730 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:12:02.960 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi172=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:12:02.960 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: Saving service rgw.foo.a spec with placement smithi172=foo.a;count:1 2023-11-28T01:12:02.960 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:02.960 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:12:02.960 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:02.960 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:12:02.960 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:02.960 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-11-28T01:12:02.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-11-28T01:12:02.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:02.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:02.961 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:02 smithi192 ceph-mon[105371]: Deploying daemon rgw.foo.a on smithi172 2023-11-28T01:12:03.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi172=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:12:03.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: Saving service rgw.foo.a spec with placement smithi172=foo.a;count:1 2023-11-28T01:12:03.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:03.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:12:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:12:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-11-28T01:12:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-11-28T01:12:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:03.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[100627]: Deploying daemon rgw.foo.a on smithi172 2023-11-28T01:12:03.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi172=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:12:03.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: Saving service rgw.foo.a spec with placement smithi172=foo.a;count:1 2023-11-28T01:12:03.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:03.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:12:03.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:03.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:12:03.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:03.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2023-11-28T01:12:03.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2023-11-28T01:12:03.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:03.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:03.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:02 smithi172 ceph-mon[118784]: Deploying daemon rgw.foo.a on smithi172 2023-11-28T01:12:03.754 INFO:journalctl@ceph.rgw.foo.a.smithi172.stdout:Nov 28 01:12:03 smithi172 systemd[1]: Starting Ceph rgw.foo.a for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:12:04.053 INFO:journalctl@ceph.rgw.foo.a.smithi172.stdout:Nov 28 01:12:03 smithi172 bash[161553]: 1cae863c866cb6c3829bbd0c7de5c1b07f86691bde28ff5130ad5a21b0395efc 2023-11-28T01:12:04.053 INFO:journalctl@ceph.rgw.foo.a.smithi172.stdout:Nov 28 01:12:03 smithi172 systemd[1]: Started Ceph rgw.foo.a for 31861c10-8d8a-11ee-95a2-87774f69a715. 2023-11-28T01:12:04.216 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.215+0000 7f07d2907700 1 -- 172.21.15.192:0/3231282423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07cc102820 msgr2=0x7f07cc102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:04.216 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.215+0000 7f07d2907700 1 --2- 172.21.15.192:0/3231282423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07cc102820 0x7f07cc102c80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f07bc0099c0 tx=0x7f07bc009cd0 comp rx=0 tx=0).stop 2023-11-28T01:12:04.216 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.215+0000 7f07d2907700 1 -- 172.21.15.192:0/3231282423 shutdown_connections 2023-11-28T01:12:04.217 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.215+0000 7f07d2907700 1 --2- 172.21.15.192:0/3231282423 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07cc1031c0 0x7f07cc109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:04.217 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.215+0000 7f07d2907700 1 --2- 172.21.15.192:0/3231282423 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07cc101620 0x7f07cc101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:04.217 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.215+0000 7f07d2907700 1 --2- 172.21.15.192:0/3231282423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07cc102820 0x7f07cc102c80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:04.217 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.215+0000 7f07d2907700 1 -- 172.21.15.192:0/3231282423 >> 172.21.15.192:0/3231282423 conn(0x7f07cc0fcbc0 msgr2=0x7f07cc0ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:12:04.217 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.215+0000 7f07d2907700 1 -- 172.21.15.192:0/3231282423 shutdown_connections 2023-11-28T01:12:04.217 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.215+0000 7f07d2907700 1 -- 172.21.15.192:0/3231282423 wait complete. 2023-11-28T01:12:04.218 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.216+0000 7f07d2907700 1 Processor -- start 2023-11-28T01:12:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.216+0000 7f07d2907700 1 -- start start 2023-11-28T01:12:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.216+0000 7f07d2907700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07cc101620 0x7f07cc196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.216+0000 7f07d2907700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07cc1031c0 0x7f07cc1972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07d2907700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07cc19c270 0x7f07cc19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07d2907700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc19ec30 con 0x7f07cc19c270 2023-11-28T01:12:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07d2907700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc19eda0 con 0x7f07cc1031c0 2023-11-28T01:12:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07d2907700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc19ef60 con 0x7f07cc101620 2023-11-28T01:12:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07cb7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07cc1031c0 0x7f07cc1972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07cbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07cc101620 0x7f07cc196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07cb7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07cc1031c0 0x7f07cc1972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:49592/0 (socket says 172.21.15.192:49592) 2023-11-28T01:12:04.219 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07cbfff700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07cc101620 0x7f07cc196da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.172:3301/0 says I am v2:172.21.15.192:59516/0 (socket says 172.21.15.192:59516) 2023-11-28T01:12:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07cb7fe700 1 -- 172.21.15.192:0/2291894804 learned_addr learned my addr 172.21.15.192:0/2291894804 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:12:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07d0ea4700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07cc19c270 0x7f07cc19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07cb7fe700 1 -- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07cc101620 msgr2=0x7f07cc196da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07cb7fe700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07cc101620 0x7f07cc196da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07cb7fe700 1 -- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07cc19c270 msgr2=0x7f07cc19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07cb7fe700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07cc19c270 0x7f07cc19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07cb7fe700 1 -- 172.21.15.192:0/2291894804 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc0095f0 con 0x7f07cc1031c0 2023-11-28T01:12:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07cb7fe700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07cc1031c0 0x7f07cc1972e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f07bc0098e0 tx=0x7f07bc0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:12:04.220 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07c97fa700 1 -- 172.21.15.192:0/2291894804 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc01d070 con 0x7f07cc1031c0 2023-11-28T01:12:04.221 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07d2907700 1 -- 172.21.15.192:0/2291894804 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07cc073800 con 0x7f07cc1031c0 2023-11-28T01:12:04.221 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07c97fa700 1 -- 172.21.15.192:0/2291894804 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7f07bc003ce0 con 0x7f07cc1031c0 2023-11-28T01:12:04.221 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07c97fa700 1 -- 172.21.15.192:0/2291894804 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc00f770 con 0x7f07cc1031c0 2023-11-28T01:12:04.221 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.217+0000 7f07d2907700 1 -- 172.21.15.192:0/2291894804 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07cc073ca0 con 0x7f07cc1031c0 2023-11-28T01:12:04.222 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.218+0000 7f07d2907700 1 -- 172.21.15.192:0/2291894804 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07cc108c60 con 0x7f07cc1031c0 2023-11-28T01:12:04.222 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.219+0000 7f07c97fa700 1 -- 172.21.15.192:0/2291894804 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7f07bc003890 con 0x7f07cc1031c0 2023-11-28T01:12:04.222 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.219+0000 7f07c97fa700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f07b406d380 0x7f07b406f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:04.223 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.219+0000 7f07c97fa700 1 -- 172.21.15.192:0/2291894804 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5287+0+0 (secure 0 0 0) 0x7f07bc08d730 con 0x7f07cc1031c0 2023-11-28T01:12:04.223 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.219+0000 7f07cbfff700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f07b406d380 0x7f07b406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:04.223 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.220+0000 7f07cbfff700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f07b406d380 0x7f07b406f840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f07cc1981e0 tx=0x7f07c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:12:04.224 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.224+0000 7f07c97fa700 1 -- 172.21.15.192:0/2291894804 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7f07bc057700 con 0x7f07cc1031c0 2023-11-28T01:12:04.321 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:04 smithi192 ceph-mon[105371]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2023-11-28T01:12:04.416 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.416+0000 7f07d2907700 1 -- 172.21.15.192:0/2291894804 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7f07cc10c440 con 0x7f07cc1031c0 2023-11-28T01:12:04.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:04 smithi172 ceph-mon[100627]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2023-11-28T01:12:04.554 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:04 smithi172 ceph-mon[118784]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s, 0 objects/s recovering 2023-11-28T01:12:04.710 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.709+0000 7f07c97fa700 1 -- 172.21.15.192:0/2291894804 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v59) v1 ==== 160+0+0 (secure 0 0 0) 0x7f07bc003b40 con 0x7f07cc1031c0 2023-11-28T01:12:04.714 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 -- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f07b406d380 msgr2=0x7f07b406f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:04.714 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f07b406d380 0x7f07b406f840 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f07cc1981e0 tx=0x7f07c0009260 comp rx=0 tx=0).stop 2023-11-28T01:12:04.714 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 -- 172.21.15.192:0/2291894804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07cc1031c0 msgr2=0x7f07cc1972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:04.714 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07cc1031c0 0x7f07cc1972e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f07bc0098e0 tx=0x7f07bc0045e0 comp rx=0 tx=0).stop 2023-11-28T01:12:04.714 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 -- 172.21.15.192:0/2291894804 shutdown_connections 2023-11-28T01:12:04.714 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7f07b406d380 0x7f07b406f840 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:04.714 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7f07cc101620 0x7f07cc196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:04.714 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7f07cc19c270 0x7f07cc19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:04.714 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 --2- 172.21.15.192:0/2291894804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f07cc1031c0 0x7f07cc1972e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:04.714 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 -- 172.21.15.192:0/2291894804 >> 172.21.15.192:0/2291894804 conn(0x7f07cc0fcbc0 msgr2=0x7f07cc105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:12:04.714 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 -- 172.21.15.192:0/2291894804 shutdown_connections 2023-11-28T01:12:04.715 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:04.713+0000 7f07d2907700 1 -- 172.21.15.192:0/2291894804 wait complete. 2023-11-28T01:12:04.718 INFO:teuthology.orchestra.run.smithi192.stderr:pool 'datapool' created 2023-11-28T01:12:05.429 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: Saving service rgw.foo.a spec with placement smithi172=foo.a;count:1 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/2291894804' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: osdmap e59: 8 total, 8 up, 8 in 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/537297196' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-11-28T01:12:05.430 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:05 smithi192 ceph-mon[105371]: pgmap v176: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-28T01:12:05.431 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- rbd pool init datapool 2023-11-28T01:12:05.553 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: Saving service rgw.foo.a spec with placement smithi172=foo.a;count:1 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/2291894804' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: osdmap e59: 8 total, 8 up, 8 in 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/537297196' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-11-28T01:12:05.554 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[100627]: pgmap v176: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-28T01:12:05.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: Saving service rgw.foo.a spec with placement smithi172=foo.a;count:1 2023-11-28T01:12:05.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:05.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-28T01:12:05.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/2291894804' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-11-28T01:12:05.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2023-11-28T01:12:05.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2023-11-28T01:12:05.555 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: osdmap e59: 8 total, 8 up, 8 in 2023-11-28T01:12:05.556 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/537297196' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2023-11-28T01:12:05.556 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:05 smithi172 ceph-mon[118784]: pgmap v176: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 59 KiB/s, 0 objects/s recovering 2023-11-28T01:12:05.783 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:12:06.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:06 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:06.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:06 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:06.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:06 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:06.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:06 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:12:06.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:06 smithi192 ceph-mon[105371]: Checking dashboard <-> RGW credentials 2023-11-28T01:12:06.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:06 smithi192 ceph-mon[105371]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-11-28T01:12:06.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:06 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/537297196' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-11-28T01:12:06.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:06 smithi192 ceph-mon[105371]: osdmap e60: 8 total, 8 up, 8 in 2023-11-28T01:12:06.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:06 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:07.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:07.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:07.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:07.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:12:07.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[118784]: Checking dashboard <-> RGW credentials 2023-11-28T01:12:07.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[118784]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-11-28T01:12:07.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/537297196' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-11-28T01:12:07.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[118784]: osdmap e60: 8 total, 8 up, 8 in 2023-11-28T01:12:07.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:07.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:07.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:07.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:07.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-11-28T01:12:07.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[100627]: Checking dashboard <-> RGW credentials 2023-11-28T01:12:07.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[100627]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-11-28T01:12:07.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/537297196' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2023-11-28T01:12:07.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[100627]: osdmap e60: 8 total, 8 up, 8 in 2023-11-28T01:12:07.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:06 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:08.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[118784]: osdmap e61: 8 total, 8 up, 8 in 2023-11-28T01:12:08.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-11-28T01:12:08.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/338909343' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-11-28T01:12:08.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[118784]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-11-28T01:12:08.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[118784]: pgmap v179: 68 pgs: 64 unknown, 4 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:12:08.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[118784]: from='client.? 172.21.15.192:0/2125368698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-11-28T01:12:08.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[118784]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-11-28T01:12:08.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[100627]: osdmap e61: 8 total, 8 up, 8 in 2023-11-28T01:12:08.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-11-28T01:12:08.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/338909343' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-11-28T01:12:08.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[100627]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-11-28T01:12:08.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[100627]: pgmap v179: 68 pgs: 64 unknown, 4 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:12:08.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[100627]: from='client.? 172.21.15.192:0/2125368698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-11-28T01:12:08.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:07 smithi172 ceph-mon[100627]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-11-28T01:12:08.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:07 smithi192 ceph-mon[105371]: osdmap e61: 8 total, 8 up, 8 in 2023-11-28T01:12:08.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:07 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-11-28T01:12:08.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:07 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/338909343' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-11-28T01:12:08.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:07 smithi192 ceph-mon[105371]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2023-11-28T01:12:08.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:07 smithi192 ceph-mon[105371]: pgmap v179: 68 pgs: 64 unknown, 4 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-11-28T01:12:08.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:07 smithi192 ceph-mon[105371]: from='client.? 172.21.15.192:0/2125368698' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-11-28T01:12:08.072 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:07 smithi192 ceph-mon[105371]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2023-11-28T01:12:09.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:08 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2023-11-28T01:12:09.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:08 smithi172 ceph-mon[100627]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2023-11-28T01:12:09.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:08 smithi172 ceph-mon[100627]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2023-11-28T01:12:09.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:08 smithi172 ceph-mon[100627]: osdmap e62: 8 total, 8 up, 8 in 2023-11-28T01:12:09.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:08 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2023-11-28T01:12:09.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:08 smithi172 ceph-mon[118784]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2023-11-28T01:12:09.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:08 smithi172 ceph-mon[118784]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2023-11-28T01:12:09.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:08 smithi172 ceph-mon[118784]: osdmap e62: 8 total, 8 up, 8 in 2023-11-28T01:12:09.070 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:08 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2023-11-28T01:12:09.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:08 smithi192 ceph-mon[105371]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2023-11-28T01:12:09.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:08 smithi192 ceph-mon[105371]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2023-11-28T01:12:09.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:08 smithi192 ceph-mon[105371]: osdmap e62: 8 total, 8 up, 8 in 2023-11-28T01:12:10.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:09 smithi172 ceph-mon[100627]: osdmap e63: 8 total, 8 up, 8 in 2023-11-28T01:12:10.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:09 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-11-28T01:12:10.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:09 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/338909343' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-11-28T01:12:10.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:09 smithi172 ceph-mon[100627]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-11-28T01:12:10.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:09 smithi172 ceph-mon[100627]: pgmap v182: 100 pgs: 10 creating+peering, 15 active+clean, 75 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2023-11-28T01:12:10.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:09 smithi172 ceph-mon[118784]: osdmap e63: 8 total, 8 up, 8 in 2023-11-28T01:12:10.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:09 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-11-28T01:12:10.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:09 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/338909343' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-11-28T01:12:10.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:09 smithi172 ceph-mon[118784]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-11-28T01:12:10.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:09 smithi172 ceph-mon[118784]: pgmap v182: 100 pgs: 10 creating+peering, 15 active+clean, 75 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2023-11-28T01:12:10.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:09 smithi192 ceph-mon[105371]: osdmap e63: 8 total, 8 up, 8 in 2023-11-28T01:12:10.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:09 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-11-28T01:12:10.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:09 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/338909343' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-11-28T01:12:10.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:09 smithi192 ceph-mon[105371]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2023-11-28T01:12:10.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:09 smithi192 ceph-mon[105371]: pgmap v182: 100 pgs: 10 creating+peering, 15 active+clean, 75 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2023-11-28T01:12:10.501 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.192 --placement '1;smithi192=iscsi.a' 2023-11-28T01:12:10.843 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:12:11.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:10 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-11-28T01:12:11.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:10 smithi172 ceph-mon[100627]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-11-28T01:12:11.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:10 smithi172 ceph-mon[100627]: osdmap e64: 8 total, 8 up, 8 in 2023-11-28T01:12:11.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:10 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-11-28T01:12:11.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:10 smithi172 ceph-mon[118784]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-11-28T01:12:11.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:10 smithi172 ceph-mon[118784]: osdmap e64: 8 total, 8 up, 8 in 2023-11-28T01:12:11.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:10 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2857567882' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-11-28T01:12:11.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:10 smithi192 ceph-mon[105371]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2023-11-28T01:12:11.057 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:10 smithi192 ceph-mon[105371]: osdmap e64: 8 total, 8 up, 8 in 2023-11-28T01:12:12.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[100627]: pgmap v184: 100 pgs: 26 creating+peering, 53 active+clean, 21 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2023-11-28T01:12:12.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[100627]: osdmap e65: 8 total, 8 up, 8 in 2023-11-28T01:12:12.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-11-28T01:12:12.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2694181212' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-11-28T01:12:12.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[100627]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-11-28T01:12:12.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[100627]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-11-28T01:12:12.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[100627]: Cluster is now healthy 2023-11-28T01:12:12.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[118784]: pgmap v184: 100 pgs: 26 creating+peering, 53 active+clean, 21 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2023-11-28T01:12:12.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[118784]: osdmap e65: 8 total, 8 up, 8 in 2023-11-28T01:12:12.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-11-28T01:12:12.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2694181212' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-11-28T01:12:12.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[118784]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-11-28T01:12:12.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[118784]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-11-28T01:12:12.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:11 smithi172 ceph-mon[118784]: Cluster is now healthy 2023-11-28T01:12:12.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:11 smithi192 ceph-mon[105371]: pgmap v184: 100 pgs: 26 creating+peering, 53 active+clean, 21 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2023-11-28T01:12:12.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:11 smithi192 ceph-mon[105371]: osdmap e65: 8 total, 8 up, 8 in 2023-11-28T01:12:12.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:11 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-11-28T01:12:12.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:11 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2694181212' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-11-28T01:12:12.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:11 smithi192 ceph-mon[105371]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2023-11-28T01:12:12.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:11 smithi192 ceph-mon[105371]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-11-28T01:12:12.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:11 smithi192 ceph-mon[105371]: Cluster is now healthy 2023-11-28T01:12:12.317 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.316+0000 7fe43bfad700 1 -- 172.21.15.192:0/2454872891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe434102090 msgr2=0x7fe434108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:12.317 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.316+0000 7fe43bfad700 1 --2- 172.21.15.192:0/2454872891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe434102090 0x7fe434108940 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe42c00b0d0 tx=0x7fe42c00b3e0 comp rx=0 tx=0).stop 2023-11-28T01:12:12.318 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.316+0000 7fe43bfad700 1 -- 172.21.15.192:0/2454872891 shutdown_connections 2023-11-28T01:12:12.318 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.316+0000 7fe43bfad700 1 --2- 172.21.15.192:0/2454872891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe434102090 0x7fe434108940 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:12.318 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.316+0000 7fe43bfad700 1 --2- 172.21.15.192:0/2454872891 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe4341004f0 0x7fe434100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:12.318 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.316+0000 7fe43bfad700 1 --2- 172.21.15.192:0/2454872891 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4341016f0 0x7fe434101b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:12.318 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.316+0000 7fe43bfad700 1 -- 172.21.15.192:0/2454872891 >> 172.21.15.192:0/2454872891 conn(0x7fe4340fba90 msgr2=0x7fe4340fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:12:12.318 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.316+0000 7fe43bfad700 1 -- 172.21.15.192:0/2454872891 shutdown_connections 2023-11-28T01:12:12.318 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.316+0000 7fe43bfad700 1 -- 172.21.15.192:0/2454872891 wait complete. 2023-11-28T01:12:12.320 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.317+0000 7fe43bfad700 1 Processor -- start 2023-11-28T01:12:12.320 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.317+0000 7fe43bfad700 1 -- start start 2023-11-28T01:12:12.320 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.317+0000 7fe43bfad700 1 --2- >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4341004f0 0x7fe43406a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:12.320 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.317+0000 7fe43bfad700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4341016f0 0x7fe43419b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:12.320 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.317+0000 7fe43bfad700 1 --2- >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe434102090 0x7fe43419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:12.320 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.317+0000 7fe43bfad700 1 -- --> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43419e610 con 0x7fe434102090 2023-11-28T01:12:12.320 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.317+0000 7fe43bfad700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43419e780 con 0x7fe4341016f0 2023-11-28T01:12:12.320 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.317+0000 7fe43bfad700 1 -- --> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43419e940 con 0x7fe4341004f0 2023-11-28T01:12:12.320 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.317+0000 7fe439548700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4341016f0 0x7fe43419b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:12.320 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe439548700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4341016f0 0x7fe43419b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:49616/0 (socket says 172.21.15.192:49616) 2023-11-28T01:12:12.320 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe439548700 1 -- 172.21.15.192:0/1041726537 learned_addr learned my addr 172.21.15.192:0/1041726537 (peer_addr_for_me v2:172.21.15.192:0/0) 2023-11-28T01:12:12.321 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe439d49700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4341004f0 0x7fe43406a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:12.321 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe43a54a700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe434102090 0x7fe43419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:12.321 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe439548700 1 -- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4341004f0 msgr2=0x7fe43406a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:12.321 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe439548700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4341004f0 0x7fe43406a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:12.321 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe439548700 1 -- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe434102090 msgr2=0x7fe43419de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:12.321 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe439548700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe434102090 0x7fe43419de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:12.321 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe439548700 1 -- 172.21.15.192:0/1041726537 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c009d00 con 0x7fe4341016f0 2023-11-28T01:12:12.321 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe439548700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4341016f0 0x7fe43419b880 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe43000d950 tx=0x7fe43000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:12:12.321 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe42affd700 1 -- 172.21.15.192:0/1041726537 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43000f890 con 0x7fe4341016f0 2023-11-28T01:12:12.321 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe42affd700 1 -- 172.21.15.192:0/1041726537 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(23 keys) v1 ==== 1007+0+0 (secure 0 0 0) 0x7fe43000fe40 con 0x7fe4341016f0 2023-11-28T01:12:12.322 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe43bfad700 1 -- 172.21.15.192:0/1041726537 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe43419ec20 con 0x7fe4341016f0 2023-11-28T01:12:12.322 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe42affd700 1 -- 172.21.15.192:0/1041726537 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43000e3b0 con 0x7fe4341016f0 2023-11-28T01:12:12.322 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.318+0000 7fe43bfad700 1 -- 172.21.15.192:0/1041726537 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43419f170 con 0x7fe4341016f0 2023-11-28T01:12:12.322 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.319+0000 7fe43bfad700 1 -- 172.21.15.192:0/1041726537 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4341079d0 con 0x7fe4341016f0 2023-11-28T01:12:12.323 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.320+0000 7fe42affd700 1 -- 172.21.15.192:0/1041726537 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 90989+0+0 (secure 0 0 0) 0x7fe43000f9f0 con 0x7fe4341016f0 2023-11-28T01:12:12.323 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.320+0000 7fe42affd700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fe42006d380 0x7fe42006f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-11-28T01:12:12.323 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.320+0000 7fe42affd700 1 -- 172.21.15.192:0/1041726537 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7132+0+0 (secure 0 0 0) 0x7fe43008cb10 con 0x7fe4341016f0 2023-11-28T01:12:12.323 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.320+0000 7fe439d49700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fe42006d380 0x7fe42006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-11-28T01:12:12.323 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.320+0000 7fe439d49700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fe42006d380 0x7fe42006f840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe434101550 tx=0x7fe42400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-28T01:12:12.325 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.324+0000 7fe42affd700 1 -- 172.21.15.192:0/1041726537 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180955 (secure 0 0 0) 0x7fe430056430 con 0x7fe4341016f0 2023-11-28T01:12:12.524 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.522+0000 7fe43bfad700 1 -- 172.21.15.192:0/1041726537 --> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.192", "placement": "1;smithi192=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7fe434061a10 con 0x7fe42006d380 2023-11-28T01:12:12.532 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.531+0000 7fe42affd700 1 -- 172.21.15.192:0/1041726537 <== mgr.14152 v2:172.21.15.172:6800/3346003710 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7fe434061a10 con 0x7fe42006d380 2023-11-28T01:12:12.532 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled iscsi.datapool update... 2023-11-28T01:12:12.534 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.533+0000 7fe43bfad700 1 -- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fe42006d380 msgr2=0x7fe42006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:12.534 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.533+0000 7fe43bfad700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fe42006d380 0x7fe42006f840 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe434101550 tx=0x7fe42400b410 comp rx=0 tx=0).stop 2023-11-28T01:12:12.534 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.533+0000 7fe43bfad700 1 -- 172.21.15.192:0/1041726537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4341016f0 msgr2=0x7fe43419b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-28T01:12:12.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.533+0000 7fe43bfad700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4341016f0 0x7fe43419b880 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe43000d950 tx=0x7fe43000dc60 comp rx=0 tx=0).stop 2023-11-28T01:12:12.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.533+0000 7fe43bfad700 1 -- 172.21.15.192:0/1041726537 shutdown_connections 2023-11-28T01:12:12.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.534+0000 7fe43bfad700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:6800/3346003710,v1:172.21.15.172:6801/3346003710] conn(0x7fe42006d380 0x7fe42006f840 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:12.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.534+0000 7fe43bfad700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:3301/0,v1:172.21.15.172:6790/0] conn(0x7fe4341004f0 0x7fe43406a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:12.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.534+0000 7fe43bfad700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] conn(0x7fe434102090 0x7fe43419de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:12.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.534+0000 7fe43bfad700 1 --2- 172.21.15.192:0/1041726537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe4341016f0 0x7fe43419b880 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-11-28T01:12:12.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.534+0000 7fe43bfad700 1 -- 172.21.15.192:0/1041726537 >> 172.21.15.192:0/1041726537 conn(0x7fe4340fba90 msgr2=0x7fe434104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-28T01:12:12.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.534+0000 7fe43bfad700 1 -- 172.21.15.192:0/1041726537 shutdown_connections 2023-11-28T01:12:12.535 INFO:teuthology.orchestra.run.smithi192.stderr:2023-11-28T01:12:12.534+0000 7fe43bfad700 1 -- 172.21.15.192:0/1041726537 wait complete. 2023-11-28T01:12:12.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:12 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-11-28T01:12:12.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:12 smithi192 ceph-mon[105371]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-11-28T01:12:12.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:12 smithi192 ceph-mon[105371]: osdmap e66: 8 total, 8 up, 8 in 2023-11-28T01:12:12.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:12 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2694181212' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-11-28T01:12:12.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:12 smithi192 ceph-mon[105371]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-11-28T01:12:12.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:12 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-11-28T01:12:12.822 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:12 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:13.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-11-28T01:12:13.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[100627]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-11-28T01:12:13.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[100627]: osdmap e66: 8 total, 8 up, 8 in 2023-11-28T01:12:13.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2694181212' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-11-28T01:12:13.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[100627]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-11-28T01:12:13.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-11-28T01:12:13.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:13.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-11-28T01:12:13.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[118784]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2023-11-28T01:12:13.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[118784]: osdmap e66: 8 total, 8 up, 8 in 2023-11-28T01:12:13.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2694181212' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-11-28T01:12:13.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[118784]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-11-28T01:12:13.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2023-11-28T01:12:13.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:12 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:13.337 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2023-11-28T01:12:13.338 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2023-11-28T01:12:13.338 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2023-11-28T01:12:13.368 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2023-11-28T01:12:13.368 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2023-11-28T01:12:13.401 DEBUG:teuthology.orchestra.run.smithi192:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@iscsi.iscsi.a.service 2023-11-28T01:12:13.443 INFO:tasks.cephadm:Adding prometheus.a on smithi192 2023-11-28T01:12:13.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch apply prometheus '1;smithi192=a' 2023-11-28T01:12:13.472 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:-- Logs begin at Tue 2023-11-28 00:47:11 UTC. -- 2023-11-28T01:12:14.024 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.192", "placement": "1;smithi192=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:12:14.024 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: Saving service iscsi.datapool spec with placement smithi192=iscsi.a;count:1 2023-11-28T01:12:14.024 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: pgmap v187: 132 pgs: 21 creating+peering, 97 active+clean, 14 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1.2 KiB/s wr, 2 op/s 2023-11-28T01:12:14.024 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-11-28T01:12:14.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-11-28T01:12:14.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: osdmap e67: 8 total, 8 up, 8 in 2023-11-28T01:12:14.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:14.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:14.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:14.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-11-28T01:12:14.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-11-28T01:12:14.025 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:13 smithi192 ceph-mon[105371]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:14.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.192", "placement": "1;smithi192=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:12:14.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: Saving service iscsi.datapool spec with placement smithi192=iscsi.a;count:1 2023-11-28T01:12:14.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: pgmap v187: 132 pgs: 21 creating+peering, 97 active+clean, 14 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1.2 KiB/s wr, 2 op/s 2023-11-28T01:12:14.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-11-28T01:12:14.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-11-28T01:12:14.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: osdmap e67: 8 total, 8 up, 8 in 2023-11-28T01:12:14.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:14.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:14.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:14.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-11-28T01:12:14.054 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-11-28T01:12:14.055 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[100627]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:14.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.192", "placement": "1;smithi192=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2023-11-28T01:12:14.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: Saving service iscsi.datapool spec with placement smithi192=iscsi.a;count:1 2023-11-28T01:12:14.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: pgmap v187: 132 pgs: 21 creating+peering, 97 active+clean, 14 unknown; 578 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 1.2 KiB/s wr, 2 op/s 2023-11-28T01:12:14.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-11-28T01:12:14.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: from='client.? 172.21.15.172:0/2297679819' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2023-11-28T01:12:14.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: osdmap e67: 8 total, 8 up, 8 in 2023-11-28T01:12:14.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:14.055 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:14.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' 2023-11-28T01:12:14.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2023-11-28T01:12:14.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2023-11-28T01:12:14.056 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:13 smithi172 ceph-mon[118784]: from='mgr.14152 172.21.15.172:0/2948316187' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-28T01:12:14.574 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/mon.b/config 2023-11-28T01:12:14.882 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:14 smithi192 ceph-mon[105371]: Deploying daemon iscsi.iscsi.a on smithi192 2023-11-28T01:12:15.053 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:14 smithi172 ceph-mon[100627]: Deploying daemon iscsi.iscsi.a on smithi192 2023-11-28T01:12:15.054 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:14 smithi172 ceph-mon[118784]: Deploying daemon iscsi.iscsi.a on smithi192 2023-11-28T01:12:15.389 INFO:teuthology.orchestra.run.smithi192.stderr:Non-zero exit code 127 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 -e NODE_NAME=smithi192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 -c %u %g /var/lib/ceph 2023-11-28T01:12:15.389 INFO:teuthology.orchestra.run.smithi192.stderr:stat: stderr Error: OCI runtime error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: Unit libpod-642d077e1303deb894428cbd9f2a3038679b8a6e3970f392334e9ab65f89fe66.scope not found. 2023-11-28T01:12:15.390 INFO:teuthology.orchestra.run.smithi192.stderr:ERROR: Failed to extract uid/gid for path /var/lib/ceph: Failed command: /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 -e NODE_NAME=smithi192 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 -c %u %g /var/lib/ceph: Error: OCI runtime error: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: Unit libpod-642d077e1303deb894428cbd9f2a3038679b8a6e3970f392334e9ab65f89fe66.scope not found. 2023-11-28T01:12:15.390 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-11-28T01:12:15.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T01:12:15.424 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa/tasks/cephadm.py", line 1088, in ceph_monitoring _shell(ctx, cluster_name, remote, [ File "/home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 1: "sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch apply prometheus '1;smithi192=a'" 2023-11-28T01:12:15.426 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-28T01:12:15.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-28T01:12:15.494 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-28T01:12:15.494 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-28T01:12:15.519 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-28T01:12:15.552 INFO:tasks.cephadm:Stopping all daemons... 2023-11-28T01:12:15.553 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-11-28T01:12:15.553 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.a 2023-11-28T01:12:15.773 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:15 smithi172 systemd[1]: Stopping Ceph mon.a for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:12:15.773 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:15 smithi172 ceph-mon[100627]: pgmap v189: 132 pgs: 5 creating+peering, 127 active+clean; 580 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 4.4 KiB/s wr, 181 op/s 2023-11-28T01:12:16.038 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:15 smithi172 conmon[100603]: 2023-11-28T01:12:15.833+0000 7f1a08a77700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:12:16.038 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:15 smithi172 conmon[100603]: 2023-11-28T01:12:15.833+0000 7f1a08a77700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-11-28T01:12:16.039 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:15 smithi172 ceph-mon[118784]: pgmap v189: 132 pgs: 5 creating+peering, 127 active+clean; 580 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 4.4 KiB/s wr, 181 op/s 2023-11-28T01:12:16.071 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:15 smithi192 ceph-mon[105371]: pgmap v189: 132 pgs: 5 creating+peering, 127 active+clean; 580 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s rd, 4.4 KiB/s wr, 181 op/s 2023-11-28T01:12:16.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:12:15 smithi192 bash[148600]: a1eccf8808a6363f5c871de864dcf090a869b129dac8636bdd0babcf410e98b3 2023-11-28T01:12:16.303 INFO:journalctl@ceph.mon.a.smithi172.stdout:Nov 28 01:12:16 smithi172 bash[163322]: e5cbc746eb8f5a1ee9362b369e24779a2c030505c07b7ec96f9e153afcbba06a 2023-11-28T01:12:16.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:12:16 smithi192 bash[148849]: fe664fb264201c076745a7cbbee1544385bc2bcf8aaf50bc7de10914e0009cb2 2023-11-28T01:12:16.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:12:16 smithi192 systemd[1]: Started Ceph iscsi.iscsi.a for 31861c10-8d8a-11ee-95a2-87774f69a715. 2023-11-28T01:12:16.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:12:16 smithi192 conmon[148891]: debug Started the configuration object watcher 2023-11-28T01:12:16.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:12:16 smithi192 conmon[148891]: debug Checking for config object changes every 1s 2023-11-28T01:12:16.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:12:16 smithi192 conmon[148891]: debug Processing osd blocklist entries for this node 2023-11-28T01:12:16.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.a.service' 2023-11-28T01:12:17.049 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:12:17.049 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-11-28T01:12:17.049 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2023-11-28T01:12:17.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.c 2023-11-28T01:12:17.303 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:17 smithi172 systemd[1]: Stopping Ceph mon.c for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:12:17.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:12:16 smithi192 conmon[148891]: debug Reading the configuration object to update local LIO configuration 2023-11-28T01:12:17.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:12:16 smithi192 conmon[148891]: debug Configuration does not have an entry for this host(smithi192.front.sepia.ceph.com) - nothing to define to LIO 2023-11-28T01:12:17.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:12:16 smithi192 conmon[148891]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2023-11-28T01:12:17.604 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:17 smithi172 conmon[118760]: 2023-11-28T01:12:17.308+0000 7f1be7616700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:12:17.604 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:17 smithi172 conmon[118760]: 2023-11-28T01:12:17.308+0000 7f1be7616700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2023-11-28T01:12:18.053 INFO:journalctl@ceph.mon.c.smithi172.stdout:Nov 28 01:12:17 smithi172 bash[163625]: a71d29d005226851d7d1217691dab8dbf5848cbb937e63365d8f7dc471672c4d 2023-11-28T01:12:18.230 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.c.service' 2023-11-28T01:12:18.266 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:12:18.266 INFO:tasks.cephadm.mon.b:Stopped mon.c 2023-11-28T01:12:18.266 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2023-11-28T01:12:18.266 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.b 2023-11-28T01:12:18.549 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:18 smithi192 systemd[1]: Stopping Ceph mon.b for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:12:18.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:18 smithi192 conmon[105347]: 2023-11-28T01:12:18.547+0000 7f9177f59700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:12:18.821 INFO:journalctl@ceph.mon.b.smithi192.stdout:Nov 28 01:12:18 smithi192 conmon[105347]: 2023-11-28T01:12:18.547+0000 7f9177f59700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2023-11-28T01:12:19.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mon.b.service' 2023-11-28T01:12:19.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:12:19.582 INFO:tasks.cephadm.mon.b:Stopped mon.b 2023-11-28T01:12:19.582 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-11-28T01:12:19.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mgr.y 2023-11-28T01:12:19.876 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:12:19 smithi172 systemd[1]: Stopping Ceph mgr.y for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:12:20.303 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:12:19 smithi172 conmon[101049]: 2023-11-28T01:12:19.875+0000 7fb9ca259700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:12:20.303 INFO:journalctl@ceph.mgr.y.smithi172.stdout:Nov 28 01:12:19 smithi172 conmon[101049]: 2023-11-28T01:12:19.875+0000 7fb9ca259700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-11-28T01:12:20.572 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mgr.y.service' 2023-11-28T01:12:20.609 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:12:20.609 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-11-28T01:12:20.609 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-11-28T01:12:20.610 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mgr.x 2023-11-28T01:12:21.032 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Nov 28 01:12:20 smithi192 systemd[1]: Stopping Ceph mgr.x for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:12:21.674 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@mgr.x.service' 2023-11-28T01:12:21.714 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:12:21.715 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-11-28T01:12:21.715 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-11-28T01:12:21.715 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.0 2023-11-28T01:12:22.053 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:12:21 smithi172 systemd[1]: Stopping Ceph osd.0 for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:12:22.053 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:12:21 smithi172 conmon[125062]: 2023-11-28T01:12:21.984+0000 7f416d26c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:12:22.053 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:12:21 smithi172 conmon[125062]: 2023-11-28T01:12:21.984+0000 7f416d26c700 -1 osd.0 67 *** Got signal Terminated *** 2023-11-28T01:12:22.053 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:12:21 smithi172 conmon[125062]: 2023-11-28T01:12:21.984+0000 7f416d26c700 -1 osd.0 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T01:12:28.053 INFO:journalctl@ceph.osd.0.smithi172.stdout:Nov 28 01:12:27 smithi172 bash[164221]: a5c9888488333ed6fa15dad83269da6112acdc24d0b08988760d39df07181449 2023-11-28T01:12:29.376 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.0.service' 2023-11-28T01:12:29.417 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:12:29.418 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-11-28T01:12:29.418 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-11-28T01:12:29.418 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.1 2023-11-28T01:12:29.708 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:12:29 smithi172 systemd[1]: Stopping Ceph osd.1 for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:12:30.053 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:12:29 smithi172 conmon[134833]: 2023-11-28T01:12:29.706+0000 7fca48aaa700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:12:30.054 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:12:29 smithi172 conmon[134833]: 2023-11-28T01:12:29.706+0000 7fca48aaa700 -1 osd.1 67 *** Got signal Terminated *** 2023-11-28T01:12:30.054 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:12:29 smithi172 conmon[134833]: 2023-11-28T01:12:29.706+0000 7fca48aaa700 -1 osd.1 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T01:12:36.053 INFO:journalctl@ceph.osd.1.smithi172.stdout:Nov 28 01:12:35 smithi172 bash[164793]: bf2e1f951d36a24f67192215a7e80db60d6b7619df94be98f993e3010f8f5925 2023-11-28T01:12:37.065 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.1.service' 2023-11-28T01:12:37.106 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:12:37.107 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-11-28T01:12:37.107 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-11-28T01:12:37.107 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.2 2023-11-28T01:12:37.387 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:12:37 smithi172 systemd[1]: Stopping Ceph osd.2 for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:12:37.803 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:12:37 smithi172 conmon[143893]: 2023-11-28T01:12:37.385+0000 7f214516b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:12:37.803 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:12:37 smithi172 conmon[143893]: 2023-11-28T01:12:37.385+0000 7f214516b700 -1 osd.2 67 *** Got signal Terminated *** 2023-11-28T01:12:37.803 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:12:37 smithi172 conmon[143893]: 2023-11-28T01:12:37.385+0000 7f214516b700 -1 osd.2 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T01:12:43.803 INFO:journalctl@ceph.osd.2.smithi172.stdout:Nov 28 01:12:43 smithi172 bash[165371]: 258539a6cda8602485b6e5b6e17bbff34018945c57e09059f848c87d6248e91c 2023-11-28T01:12:44.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.2.service' 2023-11-28T01:12:44.827 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:12:44.828 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-11-28T01:12:44.828 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-11-28T01:12:44.828 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.3 2023-11-28T01:12:45.053 INFO:journalctl@ceph.osd.3.smithi172.stdout:Nov 28 01:12:44 smithi172 systemd[1]: Stopping Ceph osd.3 for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:12:45.553 INFO:journalctl@ceph.osd.3.smithi172.stdout:Nov 28 01:12:45 smithi172 conmon[153551]: 2023-11-28T01:12:45.110+0000 7ff6c41e1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:12:45.553 INFO:journalctl@ceph.osd.3.smithi172.stdout:Nov 28 01:12:45 smithi172 conmon[153551]: 2023-11-28T01:12:45.110+0000 7ff6c41e1700 -1 osd.3 67 *** Got signal Terminated *** 2023-11-28T01:12:45.553 INFO:journalctl@ceph.osd.3.smithi172.stdout:Nov 28 01:12:45 smithi172 conmon[153551]: 2023-11-28T01:12:45.110+0000 7ff6c41e1700 -1 osd.3 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T01:12:48.804 INFO:journalctl@ceph.osd.3.smithi172.stdout:Nov 28 01:12:48 smithi172 conmon[153551]: 2023-11-28T01:12:48.451+0000 7ff6bdfd7700 -1 osd.3 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:24.576370+0000 front 2023-11-28T01:12:24.576561+0000 (oldest deadline 2023-11-28T01:12:48.076626+0000) 2023-11-28T01:12:49.571 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:49 smithi192 conmon[113405]: 2023-11-28T01:12:49.209+0000 7f0b5475c700 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.186907+0000 front 2023-11-28T01:12:25.187044+0000 (oldest deadline 2023-11-28T01:12:48.686957+0000) 2023-11-28T01:12:49.803 INFO:journalctl@ceph.osd.3.smithi172.stdout:Nov 28 01:12:49 smithi172 conmon[153551]: 2023-11-28T01:12:49.453+0000 7ff6bdfd7700 -1 osd.3 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:24.576370+0000 front 2023-11-28T01:12:24.576561+0000 (oldest deadline 2023-11-28T01:12:48.076626+0000) 2023-11-28T01:12:50.071 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:49 smithi192 conmon[122308]: 2023-11-28T01:12:49.813+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:12:50.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:12:49 smithi192 conmon[131202]: 2023-11-28T01:12:49.924+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:12:50.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:49 smithi192 conmon[140126]: 2023-11-28T01:12:49.769+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:12:50.571 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:50 smithi192 conmon[113405]: 2023-11-28T01:12:50.237+0000 7f0b5475c700 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.186907+0000 front 2023-11-28T01:12:25.187044+0000 (oldest deadline 2023-11-28T01:12:48.686957+0000) 2023-11-28T01:12:51.053 INFO:journalctl@ceph.osd.3.smithi172.stdout:Nov 28 01:12:50 smithi172 bash[165946]: fd258dd1b4c771a3ebc9d5c1623006bd2be681d5af9a10017bf75799dae67624 2023-11-28T01:12:51.070 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:50 smithi192 conmon[122308]: 2023-11-28T01:12:50.817+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:12:51.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:12:50 smithi192 conmon[131202]: 2023-11-28T01:12:50.881+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:12:51.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:50 smithi192 conmon[140126]: 2023-11-28T01:12:50.791+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:12:51.570 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:51 smithi192 conmon[113405]: 2023-11-28T01:12:51.251+0000 7f0b5475c700 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.186907+0000 front 2023-11-28T01:12:25.187044+0000 (oldest deadline 2023-11-28T01:12:48.686957+0000) 2023-11-28T01:12:52.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:51 smithi192 conmon[140126]: 2023-11-28T01:12:51.781+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:12:52.071 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:51 smithi192 conmon[122308]: 2023-11-28T01:12:51.866+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:12:52.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:12:51 smithi192 conmon[131202]: 2023-11-28T01:12:51.833+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:12:52.411 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.3.service' 2023-11-28T01:12:52.450 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:12:52.450 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-11-28T01:12:52.450 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-11-28T01:12:52.451 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.4 2023-11-28T01:12:52.571 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:52 smithi192 conmon[113405]: 2023-11-28T01:12:52.251+0000 7f0b5475c700 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.186907+0000 front 2023-11-28T01:12:25.187044+0000 (oldest deadline 2023-11-28T01:12:48.686957+0000) 2023-11-28T01:12:52.571 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:52 smithi192 systemd[1]: Stopping Ceph osd.4 for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:12:53.071 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:52 smithi192 conmon[122308]: 2023-11-28T01:12:52.828+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:12:53.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:12:52 smithi192 conmon[131202]: 2023-11-28T01:12:52.808+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:12:53.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:52 smithi192 conmon[140126]: 2023-11-28T01:12:52.804+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:12:53.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:52 smithi192 conmon[113405]: 2023-11-28T01:12:52.716+0000 7f0b58161700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:12:53.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:52 smithi192 conmon[113405]: 2023-11-28T01:12:52.716+0000 7f0b58161700 -1 osd.4 67 *** Got signal Terminated *** 2023-11-28T01:12:53.072 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:52 smithi192 conmon[113405]: 2023-11-28T01:12:52.716+0000 7f0b58161700 -1 osd.4 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T01:12:53.571 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:53 smithi192 conmon[113405]: 2023-11-28T01:12:53.261+0000 7f0b5475c700 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.186907+0000 front 2023-11-28T01:12:25.187044+0000 (oldest deadline 2023-11-28T01:12:48.686957+0000) 2023-11-28T01:12:54.070 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:53 smithi192 conmon[140126]: 2023-11-28T01:12:53.799+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:12:54.071 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:53 smithi192 conmon[122308]: 2023-11-28T01:12:53.795+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:12:54.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:12:53 smithi192 conmon[131202]: 2023-11-28T01:12:53.821+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:12:54.570 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:54 smithi192 conmon[113405]: 2023-11-28T01:12:54.221+0000 7f0b5475c700 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.186907+0000 front 2023-11-28T01:12:25.187044+0000 (oldest deadline 2023-11-28T01:12:48.686957+0000) 2023-11-28T01:12:55.070 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:54 smithi192 conmon[122308]: 2023-11-28T01:12:54.814+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:12:55.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:54 smithi192 conmon[140126]: 2023-11-28T01:12:54.755+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:12:55.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:12:54 smithi192 conmon[131202]: 2023-11-28T01:12:54.784+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:12:55.570 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:55 smithi192 conmon[113405]: 2023-11-28T01:12:55.210+0000 7f0b5475c700 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.186907+0000 front 2023-11-28T01:12:25.187044+0000 (oldest deadline 2023-11-28T01:12:48.686957+0000) 2023-11-28T01:12:56.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:12:55 smithi192 conmon[131202]: 2023-11-28T01:12:55.746+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:12:56.071 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:55 smithi192 conmon[122308]: 2023-11-28T01:12:55.850+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:12:56.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:55 smithi192 conmon[140126]: 2023-11-28T01:12:55.798+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:12:56.570 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:56 smithi192 conmon[113405]: 2023-11-28T01:12:56.171+0000 7f0b5475c700 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.186907+0000 front 2023-11-28T01:12:25.187044+0000 (oldest deadline 2023-11-28T01:12:48.686957+0000) 2023-11-28T01:12:57.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:12:56 smithi192 conmon[131202]: 2023-11-28T01:12:56.745+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:12:57.071 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:56 smithi192 conmon[122308]: 2023-11-28T01:12:56.863+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:12:57.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:56 smithi192 conmon[140126]: 2023-11-28T01:12:56.802+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:12:57.571 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:57 smithi192 conmon[113405]: 2023-11-28T01:12:57.138+0000 7f0b5475c700 -1 osd.4 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.186907+0000 front 2023-11-28T01:12:25.187044+0000 (oldest deadline 2023-11-28T01:12:48.686957+0000) 2023-11-28T01:12:58.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:12:57 smithi192 conmon[131202]: 2023-11-28T01:12:57.753+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:12:58.071 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:57 smithi192 conmon[122308]: 2023-11-28T01:12:57.883+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:12:58.071 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:57 smithi192 conmon[122308]: 2023-11-28T01:12:57.883+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:32.005324+0000 front 2023-11-28T01:12:32.005336+0000 (oldest deadline 2023-11-28T01:12:57.305290+0000) 2023-11-28T01:12:58.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:57 smithi192 conmon[140126]: 2023-11-28T01:12:57.777+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:12:59.071 INFO:journalctl@ceph.osd.4.smithi192.stdout:Nov 28 01:12:58 smithi192 bash[149704]: 59f8a7ac9f48e4948e753e72a22f35c1199b69c7b29d7b8dc0afb26fa8f37002 2023-11-28T01:12:59.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:58 smithi192 conmon[122308]: 2023-11-28T01:12:58.884+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:12:59.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:58 smithi192 conmon[122308]: 2023-11-28T01:12:58.884+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:32.005324+0000 front 2023-11-28T01:12:32.005336+0000 (oldest deadline 2023-11-28T01:12:57.305290+0000) 2023-11-28T01:12:59.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:12:58 smithi192 conmon[131202]: 2023-11-28T01:12:58.715+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:12:59.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:58 smithi192 conmon[140126]: 2023-11-28T01:12:58.744+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:00.071 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:59 smithi192 conmon[122308]: 2023-11-28T01:12:59.916+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:13:00.071 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:12:59 smithi192 conmon[122308]: 2023-11-28T01:12:59.916+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:32.005324+0000 front 2023-11-28T01:12:32.005336+0000 (oldest deadline 2023-11-28T01:12:57.305290+0000) 2023-11-28T01:13:00.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:12:59 smithi192 conmon[131202]: 2023-11-28T01:12:59.683+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:00.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:59 smithi192 conmon[140126]: 2023-11-28T01:12:59.785+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:00.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:12:59 smithi192 conmon[140126]: 2023-11-28T01:12:59.785+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:00.275 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.4.service' 2023-11-28T01:13:00.315 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:13:00.316 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-11-28T01:13:00.316 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-11-28T01:13:00.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.5 2023-11-28T01:13:00.571 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:00 smithi192 systemd[1]: Stopping Ceph osd.5 for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:13:00.934 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:00 smithi192 conmon[140126]: 2023-11-28T01:13:00.769+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:00.934 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:00 smithi192 conmon[140126]: 2023-11-28T01:13:00.769+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:00.934 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:00 smithi192 conmon[122308]: 2023-11-28T01:13:00.608+0000 7fd96239a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:13:00.934 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:00 smithi192 conmon[122308]: 2023-11-28T01:13:00.608+0000 7fd96239a700 -1 osd.5 67 *** Got signal Terminated *** 2023-11-28T01:13:00.935 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:00 smithi192 conmon[122308]: 2023-11-28T01:13:00.608+0000 7fd96239a700 -1 osd.5 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T01:13:00.935 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:00 smithi192 conmon[131202]: 2023-11-28T01:13:00.704+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:00.935 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:00 smithi192 conmon[131202]: 2023-11-28T01:13:00.704+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:01.321 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:00 smithi192 conmon[122308]: 2023-11-28T01:13:00.933+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:13:01.321 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:00 smithi192 conmon[122308]: 2023-11-28T01:13:00.933+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:32.005324+0000 front 2023-11-28T01:12:32.005336+0000 (oldest deadline 2023-11-28T01:12:57.305290+0000) 2023-11-28T01:13:01.973 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:01 smithi192 conmon[131202]: 2023-11-28T01:13:01.668+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:01.973 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:01 smithi192 conmon[131202]: 2023-11-28T01:13:01.668+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:01.973 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:01 smithi192 conmon[140126]: 2023-11-28T01:13:01.768+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:01.973 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:01 smithi192 conmon[140126]: 2023-11-28T01:13:01.768+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:02.321 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:01 smithi192 conmon[122308]: 2023-11-28T01:13:01.972+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:13:02.321 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:01 smithi192 conmon[122308]: 2023-11-28T01:13:01.972+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:32.005324+0000 front 2023-11-28T01:12:32.005336+0000 (oldest deadline 2023-11-28T01:12:57.305290+0000) 2023-11-28T01:13:03.002 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:02 smithi192 conmon[131202]: 2023-11-28T01:13:02.635+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:03.002 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:02 smithi192 conmon[131202]: 2023-11-28T01:13:02.635+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:03.003 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:02 smithi192 conmon[140126]: 2023-11-28T01:13:02.788+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:03.003 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:02 smithi192 conmon[140126]: 2023-11-28T01:13:02.788+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:03.320 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:03 smithi192 conmon[122308]: 2023-11-28T01:13:03.001+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:13:03.321 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:03 smithi192 conmon[122308]: 2023-11-28T01:13:03.001+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:32.005324+0000 front 2023-11-28T01:12:32.005336+0000 (oldest deadline 2023-11-28T01:12:57.305290+0000) 2023-11-28T01:13:04.023 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:03 smithi192 conmon[131202]: 2023-11-28T01:13:03.668+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:04.023 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:03 smithi192 conmon[131202]: 2023-11-28T01:13:03.668+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:04.024 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:03 smithi192 conmon[140126]: 2023-11-28T01:13:03.758+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:04.024 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:03 smithi192 conmon[140126]: 2023-11-28T01:13:03.758+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:04.321 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[122308]: 2023-11-28T01:13:04.021+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:13:04.321 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[122308]: 2023-11-28T01:13:04.021+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:32.005324+0000 front 2023-11-28T01:12:32.005336+0000 (oldest deadline 2023-11-28T01:12:57.305290+0000) 2023-11-28T01:13:04.321 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[122308]: 2023-11-28T01:13:04.021+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:37.305843+0000 front 2023-11-28T01:12:37.305744+0000 (oldest deadline 2023-11-28T01:13:03.706028+0000) 2023-11-28T01:13:04.993 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[131202]: 2023-11-28T01:13:04.688+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:04.993 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[131202]: 2023-11-28T01:13:04.688+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:04.993 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[131202]: 2023-11-28T01:13:04.688+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:40.199434+0000 front 2023-11-28T01:12:40.199388+0000 (oldest deadline 2023-11-28T01:13:03.699347+0000) 2023-11-28T01:13:04.994 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[140126]: 2023-11-28T01:13:04.718+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:04.994 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[140126]: 2023-11-28T01:13:04.718+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:04.994 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[140126]: 2023-11-28T01:13:04.718+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:05.320 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[122308]: 2023-11-28T01:13:04.991+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:25.004249+0000 front 2023-11-28T01:12:25.004212+0000 (oldest deadline 2023-11-28T01:12:49.704122+0000) 2023-11-28T01:13:05.321 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[122308]: 2023-11-28T01:13:04.991+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:32.005324+0000 front 2023-11-28T01:12:32.005336+0000 (oldest deadline 2023-11-28T01:12:57.305290+0000) 2023-11-28T01:13:05.321 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:04 smithi192 conmon[122308]: 2023-11-28T01:13:04.991+0000 7fd95d993700 -1 osd.5 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:37.305843+0000 front 2023-11-28T01:12:37.305744+0000 (oldest deadline 2023-11-28T01:13:03.706028+0000) 2023-11-28T01:13:06.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:05 smithi192 conmon[131202]: 2023-11-28T01:13:05.669+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:06.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:05 smithi192 conmon[131202]: 2023-11-28T01:13:05.669+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:06.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:05 smithi192 conmon[131202]: 2023-11-28T01:13:05.669+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:40.199434+0000 front 2023-11-28T01:12:40.199388+0000 (oldest deadline 2023-11-28T01:13:03.699347+0000) 2023-11-28T01:13:06.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:05 smithi192 conmon[140126]: 2023-11-28T01:13:05.761+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:06.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:05 smithi192 conmon[140126]: 2023-11-28T01:13:05.761+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:06.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:05 smithi192 conmon[140126]: 2023-11-28T01:13:05.761+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:07.071 INFO:journalctl@ceph.osd.5.smithi192.stdout:Nov 28 01:13:06 smithi192 bash[150283]: 15e1e02df0701f0a8ba4da2be3a642fcea6c138025db96e37c2095661fc351b0 2023-11-28T01:13:07.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:06 smithi192 conmon[131202]: 2023-11-28T01:13:06.638+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:07.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:06 smithi192 conmon[131202]: 2023-11-28T01:13:06.638+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:07.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:06 smithi192 conmon[131202]: 2023-11-28T01:13:06.638+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:40.199434+0000 front 2023-11-28T01:12:40.199388+0000 (oldest deadline 2023-11-28T01:13:03.699347+0000) 2023-11-28T01:13:07.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:06 smithi192 conmon[140126]: 2023-11-28T01:13:06.776+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:07.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:06 smithi192 conmon[140126]: 2023-11-28T01:13:06.776+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:07.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:06 smithi192 conmon[140126]: 2023-11-28T01:13:06.776+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:08.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:07 smithi192 conmon[140126]: 2023-11-28T01:13:07.754+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:08.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:07 smithi192 conmon[140126]: 2023-11-28T01:13:07.754+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:08.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:07 smithi192 conmon[140126]: 2023-11-28T01:13:07.754+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:08.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:07 smithi192 conmon[131202]: 2023-11-28T01:13:07.589+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:08.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:07 smithi192 conmon[131202]: 2023-11-28T01:13:07.589+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:08.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:07 smithi192 conmon[131202]: 2023-11-28T01:13:07.589+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:40.199434+0000 front 2023-11-28T01:12:40.199388+0000 (oldest deadline 2023-11-28T01:13:03.699347+0000) 2023-11-28T01:13:08.208 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.5.service' 2023-11-28T01:13:08.249 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:13:08.249 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-11-28T01:13:08.250 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-11-28T01:13:08.250 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.6 2023-11-28T01:13:08.549 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:08 smithi192 systemd[1]: Stopping Ceph osd.6 for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:13:08.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:08 smithi192 conmon[140126]: 2023-11-28T01:13:08.746+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:08.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:08 smithi192 conmon[140126]: 2023-11-28T01:13:08.746+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:08.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:08 smithi192 conmon[140126]: 2023-11-28T01:13:08.746+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:08.821 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:08 smithi192 conmon[131202]: 2023-11-28T01:13:08.547+0000 7f4a58429700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:13:08.821 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:08 smithi192 conmon[131202]: 2023-11-28T01:13:08.547+0000 7f4a58429700 -1 osd.6 67 *** Got signal Terminated *** 2023-11-28T01:13:08.821 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:08 smithi192 conmon[131202]: 2023-11-28T01:13:08.547+0000 7f4a58429700 -1 osd.6 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T01:13:08.821 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:08 smithi192 conmon[131202]: 2023-11-28T01:13:08.581+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:08.821 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:08 smithi192 conmon[131202]: 2023-11-28T01:13:08.581+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:08.821 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:08 smithi192 conmon[131202]: 2023-11-28T01:13:08.581+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:40.199434+0000 front 2023-11-28T01:12:40.199388+0000 (oldest deadline 2023-11-28T01:13:03.699347+0000) 2023-11-28T01:13:10.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:09 smithi192 conmon[131202]: 2023-11-28T01:13:09.624+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:10.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:09 smithi192 conmon[131202]: 2023-11-28T01:13:09.624+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:10.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:09 smithi192 conmon[131202]: 2023-11-28T01:13:09.624+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:40.199434+0000 front 2023-11-28T01:12:40.199388+0000 (oldest deadline 2023-11-28T01:13:03.699347+0000) 2023-11-28T01:13:10.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:09 smithi192 conmon[140126]: 2023-11-28T01:13:09.712+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:10.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:09 smithi192 conmon[140126]: 2023-11-28T01:13:09.712+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:10.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:09 smithi192 conmon[140126]: 2023-11-28T01:13:09.712+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:11.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:10 smithi192 conmon[140126]: 2023-11-28T01:13:10.675+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:11.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:10 smithi192 conmon[140126]: 2023-11-28T01:13:10.675+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:11.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:10 smithi192 conmon[140126]: 2023-11-28T01:13:10.675+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:11.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:10 smithi192 conmon[131202]: 2023-11-28T01:13:10.654+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:11.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:10 smithi192 conmon[131202]: 2023-11-28T01:13:10.654+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:11.072 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:10 smithi192 conmon[131202]: 2023-11-28T01:13:10.654+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:40.199434+0000 front 2023-11-28T01:12:40.199388+0000 (oldest deadline 2023-11-28T01:13:03.699347+0000) 2023-11-28T01:13:12.070 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:11 smithi192 conmon[140126]: 2023-11-28T01:13:11.628+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:12.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:11 smithi192 conmon[140126]: 2023-11-28T01:13:11.628+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:12.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:11 smithi192 conmon[140126]: 2023-11-28T01:13:11.628+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:12.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:11 smithi192 conmon[131202]: 2023-11-28T01:13:11.664+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:12.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:11 smithi192 conmon[131202]: 2023-11-28T01:13:11.664+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:12.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:11 smithi192 conmon[131202]: 2023-11-28T01:13:11.664+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:40.199434+0000 front 2023-11-28T01:12:40.199388+0000 (oldest deadline 2023-11-28T01:13:03.699347+0000) 2023-11-28T01:13:12.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:11 smithi192 conmon[131202]: 2023-11-28T01:13:11.664+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2023-11-28T01:12:46.000364+0000 front 2023-11-28T01:12:46.000342+0000 (oldest deadline 2023-11-28T01:13:11.300174+0000) 2023-11-28T01:13:13.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:12 smithi192 conmon[131202]: 2023-11-28T01:13:12.709+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:23.097953+0000 front 2023-11-28T01:12:23.098029+0000 (oldest deadline 2023-11-28T01:12:48.997795+0000) 2023-11-28T01:13:13.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:12 smithi192 conmon[131202]: 2023-11-28T01:13:12.709+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.298860+0000 front 2023-11-28T01:12:34.298828+0000 (oldest deadline 2023-11-28T01:13:00.198691+0000) 2023-11-28T01:13:13.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:12 smithi192 conmon[131202]: 2023-11-28T01:13:12.709+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:40.199434+0000 front 2023-11-28T01:12:40.199388+0000 (oldest deadline 2023-11-28T01:13:03.699347+0000) 2023-11-28T01:13:13.071 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:12 smithi192 conmon[131202]: 2023-11-28T01:13:12.709+0000 7f4a54223700 -1 osd.6 67 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2023-11-28T01:12:46.000364+0000 front 2023-11-28T01:12:46.000342+0000 (oldest deadline 2023-11-28T01:13:11.300174+0000) 2023-11-28T01:13:13.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:12 smithi192 conmon[140126]: 2023-11-28T01:13:12.582+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:13.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:12 smithi192 conmon[140126]: 2023-11-28T01:13:12.582+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:13.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:12 smithi192 conmon[140126]: 2023-11-28T01:13:12.582+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:13.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:12 smithi192 conmon[140126]: 2023-11-28T01:13:12.582+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2023-11-28T01:12:49.077616+0000 front 2023-11-28T01:12:49.077636+0000 (oldest deadline 2023-11-28T01:13:12.577627+0000) 2023-11-28T01:13:14.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:13 smithi192 conmon[140126]: 2023-11-28T01:13:13.612+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:14.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:13 smithi192 conmon[140126]: 2023-11-28T01:13:13.612+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:14.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:13 smithi192 conmon[140126]: 2023-11-28T01:13:13.612+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:14.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:13 smithi192 conmon[140126]: 2023-11-28T01:13:13.612+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2023-11-28T01:12:49.077616+0000 front 2023-11-28T01:12:49.077636+0000 (oldest deadline 2023-11-28T01:13:12.577627+0000) 2023-11-28T01:13:14.820 INFO:journalctl@ceph.osd.6.smithi192.stdout:Nov 28 01:13:14 smithi192 bash[150857]: 0920914c02ce7676c7b1f2322193fd0a0a15803f1a5219f791b9188a2a6ea346 2023-11-28T01:13:14.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:14 smithi192 conmon[140126]: 2023-11-28T01:13:14.601+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:14.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:14 smithi192 conmon[140126]: 2023-11-28T01:13:14.601+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:14.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:14 smithi192 conmon[140126]: 2023-11-28T01:13:14.601+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:14.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:14 smithi192 conmon[140126]: 2023-11-28T01:13:14.601+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2023-11-28T01:12:49.077616+0000 front 2023-11-28T01:12:49.077636+0000 (oldest deadline 2023-11-28T01:13:12.577627+0000) 2023-11-28T01:13:15.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:15 smithi192 conmon[140126]: 2023-11-28T01:13:15.623+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:15.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:15 smithi192 conmon[140126]: 2023-11-28T01:13:15.623+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:15.822 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:15 smithi192 conmon[140126]: 2023-11-28T01:13:15.623+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:15.822 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:15 smithi192 conmon[140126]: 2023-11-28T01:13:15.623+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2023-11-28T01:12:49.077616+0000 front 2023-11-28T01:12:49.077636+0000 (oldest deadline 2023-11-28T01:13:12.577627+0000) 2023-11-28T01:13:16.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.6.service' 2023-11-28T01:13:16.153 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:13:16.153 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-11-28T01:13:16.153 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-11-28T01:13:16.153 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.7 2023-11-28T01:13:16.448 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:16 smithi192 systemd[1]: Stopping Ceph osd.7 for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:13:16.820 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:16 smithi192 conmon[140126]: 2023-11-28T01:13:16.446+0000 7fe22fdb0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:13:16.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:16 smithi192 conmon[140126]: 2023-11-28T01:13:16.446+0000 7fe22fdb0700 -1 osd.7 67 *** Got signal Terminated *** 2023-11-28T01:13:16.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:16 smithi192 conmon[140126]: 2023-11-28T01:13:16.446+0000 7fe22fdb0700 -1 osd.7 67 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-11-28T01:13:16.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:16 smithi192 conmon[140126]: 2023-11-28T01:13:16.631+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:16.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:16 smithi192 conmon[140126]: 2023-11-28T01:13:16.631+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:16.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:16 smithi192 conmon[140126]: 2023-11-28T01:13:16.631+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:16.821 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:16 smithi192 conmon[140126]: 2023-11-28T01:13:16.631+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2023-11-28T01:12:49.077616+0000 front 2023-11-28T01:12:49.077636+0000 (oldest deadline 2023-11-28T01:13:12.577627+0000) 2023-11-28T01:13:18.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:17 smithi192 conmon[140126]: 2023-11-28T01:13:17.583+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:18.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:17 smithi192 conmon[140126]: 2023-11-28T01:13:17.583+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:18.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:17 smithi192 conmon[140126]: 2023-11-28T01:13:17.583+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:18.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:17 smithi192 conmon[140126]: 2023-11-28T01:13:17.583+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2023-11-28T01:12:49.077616+0000 front 2023-11-28T01:12:49.077636+0000 (oldest deadline 2023-11-28T01:13:12.577627+0000) 2023-11-28T01:13:19.070 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:18 smithi192 conmon[140126]: 2023-11-28T01:13:18.623+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:19.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:18 smithi192 conmon[140126]: 2023-11-28T01:13:18.623+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:19.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:18 smithi192 conmon[140126]: 2023-11-28T01:13:18.623+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:19.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:18 smithi192 conmon[140126]: 2023-11-28T01:13:18.623+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2023-11-28T01:12:49.077616+0000 front 2023-11-28T01:12:49.077636+0000 (oldest deadline 2023-11-28T01:13:12.577627+0000) 2023-11-28T01:13:20.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:19 smithi192 conmon[140126]: 2023-11-28T01:13:19.639+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:20.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:19 smithi192 conmon[140126]: 2023-11-28T01:13:19.639+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:20.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:19 smithi192 conmon[140126]: 2023-11-28T01:13:19.639+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:20.072 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:19 smithi192 conmon[140126]: 2023-11-28T01:13:19.639+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2023-11-28T01:12:49.077616+0000 front 2023-11-28T01:12:49.077636+0000 (oldest deadline 2023-11-28T01:13:12.577627+0000) 2023-11-28T01:13:21.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:20 smithi192 conmon[140126]: 2023-11-28T01:13:20.638+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6806 osd.0 since back 2023-11-28T01:12:26.473597+0000 front 2023-11-28T01:12:26.473717+0000 (oldest deadline 2023-11-28T01:12:49.373756+0000) 2023-11-28T01:13:21.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:20 smithi192 conmon[140126]: 2023-11-28T01:13:20.638+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6814 osd.1 since back 2023-11-28T01:12:34.674895+0000 front 2023-11-28T01:12:34.674841+0000 (oldest deadline 2023-11-28T01:12:58.774746+0000) 2023-11-28T01:13:21.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:20 smithi192 conmon[140126]: 2023-11-28T01:13:20.638+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6822 osd.2 since back 2023-11-28T01:12:42.076466+0000 front 2023-11-28T01:12:42.076385+0000 (oldest deadline 2023-11-28T01:13:03.776474+0000) 2023-11-28T01:13:21.071 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:20 smithi192 conmon[140126]: 2023-11-28T01:13:20.638+0000 7fe22aba8700 -1 osd.7 67 heartbeat_check: no reply from 172.21.15.172:6830 osd.3 since back 2023-11-28T01:12:49.077616+0000 front 2023-11-28T01:12:49.077636+0000 (oldest deadline 2023-11-28T01:13:12.577627+0000) 2023-11-28T01:13:22.571 INFO:journalctl@ceph.osd.7.smithi192.stdout:Nov 28 01:13:22 smithi192 bash[151429]: 1a2468c6ca82bfe067eac6f48cfd06b8af285522a87c3ad8a0d1fad638ed8b8b 2023-11-28T01:13:23.843 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@osd.7.service' 2023-11-28T01:13:23.884 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:13:23.885 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-11-28T01:13:23.885 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2023-11-28T01:13:23.885 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-31861c10-8d8a-11ee-95a2-87774f69a715@rgw.foo.a 2023-11-28T01:13:24.173 INFO:journalctl@ceph.rgw.foo.a.smithi172.stdout:Nov 28 01:13:23 smithi172 systemd[1]: Stopping Ceph rgw.foo.a for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:13:24.553 INFO:journalctl@ceph.rgw.foo.a.smithi172.stdout:Nov 28 01:13:24 smithi172 conmon[161594]: 2023-11-28T01:13:24.171+0000 7f9eafd5e700 -1 received signal: Terminated from /dev/init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-11-28T01:13:24.553 INFO:journalctl@ceph.rgw.foo.a.smithi172.stdout:Nov 28 01:13:24 smithi172 conmon[161594]: 2023-11-28T01:13:24.171+0000 7f9ebe1f87c0 -1 shutting down 2023-11-28T01:13:34.700 INFO:journalctl@ceph.rgw.foo.a.smithi172.stdout:Nov 28 01:13:34 smithi172 bash[166529]: 1cae863c866cb6c3829bbd0c7de5c1b07f86691bde28ff5130ad5a21b0395efc 2023-11-28T01:13:34.701 INFO:journalctl@ceph.rgw.foo.a.smithi172.stdout:Nov 28 01:13:34 smithi172 systemd[1]: ceph-31861c10-8d8a-11ee-95a2-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2023-11-28T01:13:34.707 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-31861c10-8d8a-11ee-95a2-87774f69a715@rgw.foo.a.service' 2023-11-28T01:13:34.742 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-28T01:13:34.742 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2023-11-28T01:13:34.742 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm rm-cluster --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 --force --keep-logs 2023-11-28T01:13:35.015 INFO:teuthology.orchestra.run.smithi172.stdout:Deleting cluster with fsid: 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:13:36.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm rm-cluster --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 --force --keep-logs 2023-11-28T01:13:36.478 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:13:37.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:13:37 smithi192 systemd[1]: Stopping Ceph iscsi.iscsi.a for 31861c10-8d8a-11ee-95a2-87774f69a715... 2023-11-28T01:13:37.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:13:37 smithi192 conmon[148891]: debug Shutdown received 2023-11-28T01:13:48.070 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:13:47 smithi192 bash[152291]: fe664fb264201c076745a7cbbee1544385bc2bcf8aaf50bc7de10914e0009cb2 2023-11-28T01:13:48.071 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:13:47 smithi192 systemd[1]: ceph-31861c10-8d8a-11ee-95a2-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2023-11-28T01:13:48.696 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:13:48 smithi192 bash[152467]: a1eccf8808a6363f5c871de864dcf090a869b129dac8636bdd0babcf410e98b3 2023-11-28T01:13:48.697 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:13:48 smithi192 systemd[1]: ceph-31861c10-8d8a-11ee-95a2-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2023-11-28T01:13:48.697 INFO:journalctl@ceph.iscsi.iscsi.a.smithi192.stdout:Nov 28 01:13:48 smithi192 systemd[1]: Stopped Ceph iscsi.iscsi.a for 31861c10-8d8a-11ee-95a2-87774f69a715. 2023-11-28T01:13:48.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-28T01:13:48.879 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-11-28T01:13:48.909 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-28T01:13:48.911 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380/remote/smithi172/crash 2023-11-28T01:13:48.912 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/crash -- . 2023-11-28T01:13:48.957 INFO:teuthology.orchestra.run.smithi172.stderr:tar: /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/crash: Cannot open: No such file or directory 2023-11-28T01:13:48.957 INFO:teuthology.orchestra.run.smithi172.stderr:tar: Error is not recoverable: exiting now 2023-11-28T01:13:48.959 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380/remote/smithi192/crash 2023-11-28T01:13:48.960 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/crash -- . 2023-11-28T01:13:48.998 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/crash: Cannot open: No such file or directory 2023-11-28T01:13:48.998 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2023-11-28T01:13:48.999 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-28T01:13:48.999 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2023-11-28T01:13:49.030 INFO:teuthology.orchestra.run.smithi172.stderr:grep: /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph.log: No such file or directory 2023-11-28T01:13:49.032 INFO:tasks.cephadm:Compressing logs... 2023-11-28T01:13:49.033 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-28T01:13:49.075 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-28T01:13:49.101 INFO:teuthology.orchestra.run.smithi172.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-28T01:13:49.101 INFO:teuthology.orchestra.run.smithi172.stderr:: No such file or directory 2023-11-28T01:13:49.101 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.2.log 2023-11-28T01:13:49.101 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.0.log 2023-11-28T01:13:49.104 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-client.rgw.foo.a.log 2023-11-28T01:13:49.104 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.3.log 2023-11-28T01:13:49.104 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-client.rgw.foo.a.log: -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.1.log 2023-11-28T01:13:49.104 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mon.c.log 2023-11-28T01:13:49.105 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log : No such file or directory 2023-11-28T01:13:49.105 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-11-28T01:13:49.105 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.5.log 2023-11-28T01:13:49.105 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.6.log 2023-11-28T01:13:49.105 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.4.log 2023-11-28T01:13:49.105 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mgr.x.log 2023-11-28T01:13:49.106 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/tcmu-runner.log 2023-11-28T01:13:49.106 INFO:teuthology.orchestra.run.smithi192.stderr: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-28T01:13:49.106 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-volume.log 2023-11-28T01:13:49.106 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.7.log 2023-11-28T01:13:49.106 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/tcmu-runner.log: 67.6% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/tcmu-runner.log.gz 2023-11-28T01:13:49.106 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mon.b.log 2023-11-28T01:13:49.106 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.7.log: /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mon.b.log: 90.8% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mgr.x.log.gz 2023-11-28T01:13:49.107 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.1.log: /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-volume.log 2023-11-28T01:13:49.115 INFO:teuthology.orchestra.run.smithi192.stderr: 91.2% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-volume.log.gz 2023-11-28T01:13:49.117 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-volume.log: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-28T01:13:49.118 INFO:teuthology.orchestra.run.smithi172.stderr: 93.5% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-client.rgw.foo.a.log.gz 2023-11-28T01:13:49.118 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mon.a.log 2023-11-28T01:13:49.118 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mgr.y.log 2023-11-28T01:13:49.121 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mon.a.log: /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mgr.y.log: 91.3% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-volume.log.gz 2023-11-28T01:13:49.193 INFO:teuthology.orchestra.run.smithi172.stderr: 92.4% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mon.c.log.gz 2023-11-28T01:13:49.200 INFO:teuthology.orchestra.run.smithi192.stderr: 93.6% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.7.log.gz 2023-11-28T01:13:49.200 INFO:teuthology.orchestra.run.smithi192.stderr: 93.3% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.5.log.gz 2023-11-28T01:13:49.203 INFO:teuthology.orchestra.run.smithi192.stderr: 92.1% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mon.b.log.gz 2023-11-28T01:13:49.208 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mgr.y.log.gz 2023-11-28T01:13:49.215 INFO:teuthology.orchestra.run.smithi172.stderr: 93.3% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.2.log.gz 2023-11-28T01:13:49.236 INFO:teuthology.orchestra.run.smithi192.stderr: 93.3% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.6.log.gz 2023-11-28T01:13:49.237 INFO:teuthology.orchestra.run.smithi192.stderr: 93.6% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.4.log.gz 2023-11-28T01:13:49.238 INFO:teuthology.orchestra.run.smithi192.stderr: 2023-11-28T01:13:49.238 INFO:teuthology.orchestra.run.smithi192.stderr:real 0m0.147s 2023-11-28T01:13:49.238 INFO:teuthology.orchestra.run.smithi192.stderr:user 0m0.562s 2023-11-28T01:13:49.238 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m0.041s 2023-11-28T01:13:49.249 INFO:teuthology.orchestra.run.smithi172.stderr: 93.5% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.3.log.gz 2023-11-28T01:13:49.272 INFO:teuthology.orchestra.run.smithi172.stderr: 93.5% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.0.log.gz 2023-11-28T01:13:49.315 INFO:teuthology.orchestra.run.smithi172.stderr: 93.6% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-osd.1.log.gz 2023-11-28T01:13:49.343 INFO:teuthology.orchestra.run.smithi172.stderr: 91.3% -- replaced with /var/log/ceph/31861c10-8d8a-11ee-95a2-87774f69a715/ceph-mon.a.log.gz 2023-11-28T01:13:49.345 INFO:teuthology.orchestra.run.smithi172.stderr: 2023-11-28T01:13:49.345 INFO:teuthology.orchestra.run.smithi172.stderr:real 0m0.255s 2023-11-28T01:13:49.345 INFO:teuthology.orchestra.run.smithi172.stderr:user 0m1.056s 2023-11-28T01:13:49.345 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m0.053s 2023-11-28T01:13:49.345 INFO:tasks.cephadm:Archiving logs... 2023-11-28T01:13:49.345 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380/remote/smithi172/log 2023-11-28T01:13:49.347 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T01:13:49.554 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380/remote/smithi192/log 2023-11-28T01:13:49.555 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-28T01:13:49.657 INFO:tasks.cephadm:Removing cluster... 2023-11-28T01:13:49.657 DEBUG:teuthology.orchestra.run.smithi172:> sudo cephadm rm-cluster --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 --force 2023-11-28T01:13:49.897 INFO:teuthology.orchestra.run.smithi172.stdout:Deleting cluster with fsid: 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:13:50.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo cephadm rm-cluster --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 --force 2023-11-28T01:13:50.437 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: 31861c10-8d8a-11ee-95a2-87774f69a715 2023-11-28T01:13:50.735 INFO:tasks.cephadm:Teardown complete 2023-11-28T01:13:50.735 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa/tasks/cephadm.py", line 1720, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa/tasks/cephadm.py", line 1635, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa/tasks/cephadm.py", line 1088, in ceph_monitoring _shell(ctx, cluster_name, remote, [ File "/home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 1: "sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch apply prometheus '1;smithi192=a'" 2023-11-28T01:13:50.809 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d8226c094be849e49eee4a94c5cbd7cd Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa/tasks/cephadm.py", line 1720, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa/tasks/cephadm.py", line 1635, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa/tasks/cephadm.py", line 1088, in ceph_monitoring _shell(ctx, cluster_name, remote, [ File "/home/teuthworker/src/github.com_ceph_ceph-c_369173db14b6995b2bd07c60ec5f63d01cf21631/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_2442239d2653456406c25ae0c71b689c8f2657b6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 1: "sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch apply prometheus '1;smithi192=a'" 2023-11-28T01:13:50.813 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-28T01:13:50.822 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-28T01:13:50.835 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-28T01:13:50.835 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T01:13:50.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-28T01:13:50.931 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-28T01:13:50.940 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-28T01:13:50.941 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T01:13:50.944 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-28T01:13:50.959 INFO:teuthology.orchestra.run.smithi172.stderr:bash: ntpq: command not found 2023-11-28T01:13:50.960 INFO:teuthology.orchestra.run.smithi192.stderr:bash: ntpq: command not found 2023-11-28T01:13:50.964 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T01:13:50.964 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2023-11-28T01:13:50.965 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +906us[ +906us] +/- 75ms 2023-11-28T01:13:50.965 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -747us[ -747us] +/- 70ms 2023-11-28T01:13:50.965 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +58us[ +58us] +/- 87ms 2023-11-28T01:13:50.965 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T01:13:50.967 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-28T01:13:50.967 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2023-11-28T01:13:50.967 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 46 +878us[ +885us] +/- 75ms 2023-11-28T01:13:50.967 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -783us[ -776us] +/- 70ms 2023-11-28T01:13:50.968 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +53us[ +60us] +/- 87ms 2023-11-28T01:13:50.968 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-28T01:13:50.968 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-28T01:13:50.977 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-28T01:13:50.977 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-28T01:13:50.986 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-28T01:13:51.106 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-28T01:13:51.199 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T01:13:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T01:13:51.232 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-28T01:13:51.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T01:13:51.264 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-28T01:13:51.273 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-28T01:13:51.282 INFO:teuthology.task.internal:Duration was 1503.190871 seconds 2023-11-28T01:13:51.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-28T01:13:51.291 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-28T01:13:51.292 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T01:13:51.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-28T01:13:51.339 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T01:13:51.351 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-28T01:13:51.566 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-28T01:13:51.566 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2023-11-28T01:13:51.566 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T01:13:51.588 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2023-11-28T01:13:51.588 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-28T01:13:51.610 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-28T01:13:51.611 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T01:13:51.630 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-28T01:13:51.695 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-28T01:13:51.696 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T01:13:51.698 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-28T01:13:51.822 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-28T01:13:51.832 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-28T01:13:51.832 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T01:13:51.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-28T01:13:51.888 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-28T01:13:51.902 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T01:13:51.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-28T01:13:51.928 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2023-11-28T01:13:51.954 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2023-11-28T01:13:51.976 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T01:13:52.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T01:13:52.003 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-28T01:13:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-28T01:13:52.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-28T01:13:52.042 INFO:teuthology.task.internal:Transferring archived files... 2023-11-28T01:13:52.043 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380/remote/smithi172 2023-11-28T01:13:52.043 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-28T01:13:52.084 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380/remote/smithi192 2023-11-28T01:13:52.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-28T01:13:52.124 INFO:teuthology.task.internal:Removing archive directory... 2023-11-28T01:13:52.124 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-28T01:13:52.127 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-28T01:13:52.175 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-28T01:13:52.186 INFO:teuthology.task.internal:Not uploading archives. 2023-11-28T01:13:52.186 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-28T01:13:52.198 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-28T01:13:52.198 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T01:13:52.200 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-28T01:13:52.215 INFO:teuthology.orchestra.run.smithi172.stdout: 262320 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 01:13 /home/ubuntu/cephtest 2023-11-28T01:13:52.232 INFO:teuthology.orchestra.run.smithi192.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 28 01:13 /home/ubuntu/cephtest 2023-11-28T01:13:52.233 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-28T01:13:52.243 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-28T01:13:52.265 INFO:teuthology.nuke:Checking targets against current locks 2023-11-28T01:13:52.293 DEBUG:teuthology.nuke:shortname: smithi172 2023-11-28T01:13:52.293 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T01:13:52.322 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 00:42:52.078435', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-28T01:13:52.323 DEBUG:teuthology.nuke:shortname: smithi192 2023-11-28T01:13:52.324 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-28T01:13:52.354 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-27_17:19:32-orch-wip-yuri10-testing-2023-11-22-1112-distro-default-smithi/7468380', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-28 00:42:52.077023', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-11-28T01:13:52.380 INFO:teuthology.orchestra.console.smithi172:Power off 2023-11-28T01:13:52.380 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T01:13:52.406 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2023-11-28T01:13:52.407 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T01:13:52.422 INFO:teuthology.orchestra.console.smithi192:Power off 2023-11-28T01:13:52.423 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-28T01:13:52.436 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-11-28T01:13:52.451 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2023-11-28T01:13:52.451 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T01:13:52.474 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-11-28T01:13:56.438 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T01:13:56.475 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T01:13:56.554 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2023-11-28T01:13:56.585 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2023-11-28T01:14:00.558 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T01:14:00.586 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-28T01:14:00.668 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2023-11-28T01:14:00.696 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2023-11-28T01:14:00.770 INFO:teuthology.orchestra.console.smithi172:Power off completed 2023-11-28T01:14:00.798 INFO:teuthology.orchestra.console.smithi192:Power off completed 2023-11-28T01:14:00.904 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/rhel_8.6_container_tools_3.0 fixed-2 mode/packaged mon_election/classic msgr/async start tasks/rados_api_tests} duration: 1503.1908712387085 failure_reason: 'Command failed on smithi192 with status 1: "sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:369173db14b6995b2bd07c60ec5f63d01cf21631 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 31861c10-8d8a-11ee-95a2-87774f69a715 -- ceph orch apply prometheus ''1;smithi192=a''"' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d8226c094be849e49eee4a94c5cbd7cd status: fail success: false 2023-11-28T01:14:00.904 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-28T01:14:00.977 INFO:teuthology.run:FAIL